(thread crossed over to pgsql-performance, where it belongs, from
pgsql-advocacy)

Greg,

> I think TPC-E will make both of these major improvements much more important.
> I suspect it would be hard to get 8.2 to even pass TPC-E due to the checkpoint
> dropouts.
>

You'd be surprised, then. We're still horribly, horribly lock-bound on
TPC-E; on anything over 4 cores lock resolution chokes us to death. See
Jignesh's and Paul's various posts about attempts to fix this.

Without resolving the locking issues, HOT and checkpoint doesn't have
much effect on TPCE.

--Josh

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

"Josh Berkus" <josh@agliodbs.com> writes:

>> I think TPC-E will make both of these major improvements much more important.
>> I suspect it would be hard to get 8.2 to even pass TPC-E due to the checkpoint
>> dropouts.
>
> You'd be surprised, then. We're still horribly, horribly lock-bound on TPC-E;
> on anything over 4 cores lock resolution chokes us to death. See Jignesh's and
> Paul's various posts about attempts to fix this.

Most of those posts have been about scalability issues with extremely large
numbers of sessions. Those are interesting too and they may be limiting our
results in benchmarks which depend on such a configuration (which I don't
think includes TPC-E, but the benchmark Jignesh has been writing about is some
Java application benchmark which may be such a beast) but they don't directly
relate to whether we're "passing" TPC-E.

What I was referring to by "passing" TPC-E was the criteria for a conformant
benchmark run. TPC-C has iirc, only two relevant criteria: "95th percentile
response time < 5s" and "average response time < 95th percentile response
time". You can pass those even if 1 transaction in 20 takes 10-20s which is
more than enough to cover checkpoints and other random sources of inconsistent
performance.

TPC-E has more stringent requirements which explicitly require very consistent
response times and I doubt 8.2 would have been able to pass them. So the
performance limiting factors whether they be i/o, cpu, lock contention, or
whatever don't even come into play. We wouldn't have any conformant results
whatsoever, not even low values limited by contention. 8.3 however should be
in a better position to pass.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's 24x7 Postgres support!

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Gregory Stark wrote:
> TPC-E has more stringent requirements which explicitly require very consistent
> response times and I doubt 8.2 would have been able to pass them.

Sure it would. Just not for a very large scale factor ;-).

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Greg,

> What I was referring to by "passing" TPC-E was the criteria for a conformant
> benchmark run. TPC-C has iirc, only two relevant criteria: "95th percentile
> response time < 5s" and "average response time < 95th percentile response
> time". You can pass those even if 1 transaction in 20 takes 10-20s which is
> more than enough to cover checkpoints and other random sources of inconsistent
> performance.

We can do this now. I'm unhappy because we're at about 1/4 of Oracle
performance, but we certainly pass -- even with 8.2.

--Josh

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

"Josh Berkus" <josh@agliodbs.com> writes:

> Greg,
>
>> What I was referring to by "passing" TPC-E was the criteria for a conformant
>> benchmark run. TPC-C has iirc, only two relevant criteria: "95th percentile
>> response time < 5s" and "average response time < 95th percentile response
>> time". You can pass those even if 1 transaction in 20 takes 10-20s which is
>> more than enough to cover checkpoints and other random sources of inconsistent
>> performance.
>
> We can do this now. I'm unhappy because we're at about 1/4 of Oracle
> performance, but we certainly pass -- even with 8.2.

We certainly can pass TPC-C. I'm curious what you mean by 1/4 though? On
similar hardware? Or the maximum we can scale to is 1/4 as large as Oracle?
Can you point me to the actual benchmark runs you're referring to?

But I just made an off-hand comment that I doubt 8.2 could pass TPC-E which
has much more stringent requirements. It has requirements like:

the throughput computed over any period of one hour, sliding over the Steady
State by increments of ten minutes, varies from the Reported Throughput by no
more than 2%


--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's Slony Replication support!

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

On Mon, 28 Apr 2008 14:40:25 -0400
Gregory Stark <stark@enterprisedb.com> wrote:


> We certainly can pass TPC-C. I'm curious what you mean by 1/4 though?
> On similar hardware? Or the maximum we can scale to is 1/4 as large
> as Oracle? Can you point me to the actual benchmark runs you're
> referring to?

I would be curious as well considering there has been zero evidence
provided to make such a statement. I am not saying it isn't true, it
wouldn't be surprising to me if Oracle outperformed PostgreSQL in TPC-C
but I would sure like to see in general how wel we do (or don't).


Sincerely,

Joshua D. Drake

--
The PostgreSQL Company since 1997: http://www.commandprompt.com/
PostgreSQL Community Conference: http://www.postgresqlconference.org/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL SPI Liaison | SPI Director | PostgreSQL political pundit


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFIFjm/ATb/zqfZUUQRAm7hAJoDtOjStcWNrPHk+t21VEjfQEgZKACfZr1U
2izBCGWjJxkkipTK/nUgTd0=
=jqo8
-----END PGP SIGNATURE-----

Joshua D. Drake wrote:
> On Mon, 28 Apr 2008 14:40:25 -0400
> Gregory Stark <stark@enterprisedb.com> wrote:
>
>
>
>> We certainly can pass TPC-C. I'm curious what you mean by 1/4 though?
>> On similar hardware? Or the maximum we can scale to is 1/4 as large
>> as Oracle? Can you point me to the actual benchmark runs you're
>> referring to?
>>
>
> I would be curious as well considering there has been zero evidence
> provided to make such a statement. I am not saying it isn't true, it
> wouldn't be surprising to me if Oracle outperformed PostgreSQL in TPC-C
> but I would sure like to see in general how wel we do (or don't).
>
>
> Sincerely,
>
> Joshua D. Drake
>

I am sorry but I am far from catching my emails:

Best thing is to work with TPC-E benchmarks involving the community.
(TPC-C requirements is way too high on storage and everybody seems to be
getting on the TPC-E bandwagon slowly.)

Where can I get the latest DBT5 (TPC-E) kit ? Using the kit should allow
me to recreate setups which can then be made available for various
PostgreSQL Performance engineers to look at it.



Regards,
Jignesh





--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance