Hi,
I've two questions for which I not really found answers in the web.

Intro:
I've a Website with some traffic.
2 Million queries a day, during daylight.
Postgres is running on a dedicated server P4 DualCore, 4 Gig Ram.
Mainly updates on 1 tuple. And more or less complex SELECT statements.
I noticed that the overall performance of postgres is decreasing
when one or more long
readers are present. Where a long reader here is already a Select
count(*) from table.

As postgres gets slower an slower, and users still hammering on the
reload button to get their
page loaded. Postgres begins to reach max connections, and web site
is stuck.
It's not because of a bad schema or bad select statements. As I said,
a select count(*) on big table is already
triggering this behaviour.

Why do long readers influence the rest of the transactions in such a
heavy way?
Any configuration changes which can help here?
Is it a disc-IO bottleneck thing?

Second question. What is the right choice for the shared_buffers size?
On a dedicated postgres server with 4 Giga RAM. Is there any rule of
thumb?
Actually I set it to +-256M.


thanks for any suggestions.

Patric


My Setup:

Debian Etch
PSQL: 8.1.4

WAL files are located on another disc than the dbase itself.

max_connections = 190
shared_buffers = 30000
temp_buffers = 3000
work_mem = 4096
maintenance_work_mem = 16384
fsync = on
wal_buffers = 16
effective_cache_size = 5000


---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

We have a few tables that we need to pull relatively accurate aggregate
counts from, and we found the performance of SELECT COUNT(*) to be
unacceptable. We solved this by creating triggers on insert and delete to
update counts in a secondary table which we join to when we need the count
information.

This may or may not work in your scenario, but it was a reasonable trade off
for us.

Bryan

On 7/11/07, Patric de Waha <lists@p-dw.com> wrote:
>
> Hi,
> I've two questions for which I not really found answers in the web.
>
> Intro:
> I've a Website with some traffic.
> 2 Million queries a day, during daylight.
> Postgres is running on a dedicated server P4 DualCore, 4 Gig Ram.
> Mainly updates on 1 tuple. And more or less complex SELECT statements.
> I noticed that the overall performance of postgres is decreasing
> when one or more long
> readers are present. Where a long reader here is already a Select
> count(*) from table.
>
> As postgres gets slower an slower, and users still hammering on the
> reload button to get their
> page loaded. Postgres begins to reach max connections, and web site
> is stuck.
> It's not because of a bad schema or bad select statements. As I said,
> a select count(*) on big table is already
> triggering this behaviour.
>
> Why do long readers influence the rest of the transactions in such a
> heavy way?
> Any configuration changes which can help here?
> Is it a disc-IO bottleneck thing?
>
> Second question. What is the right choice for the shared_buffers size?
> On a dedicated postgres server with 4 Giga RAM. Is there any rule of
> thumb?
> Actually I set it to +-256M.
>
>
> thanks for any suggestions.
>
> Patric
>
>
> My Setup:
>
> Debian Etch
> PSQL: 8.1.4
>
> WAL files are located on another disc than the dbase itself.
>
> max_connections = 190
> shared_buffers = 30000
> temp_buffers = 3000
> work_mem = 4096
> maintenance_work_mem = 16384
> fsync = on
> wal_buffers = 16
> effective_cache_size = 5000
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 7: You can help support the PostgreSQL project by donating at
>
> http://www.postgresql.org/about/donate
>

Patric de Waha <lists@p-dw.com> writes:
> Postgres is running on a dedicated server P4 DualCore, 4 Gig Ram.

When you don't even mention your disk hardware, that's a bad sign.
In a database server the disk is usually more important than the CPU.

> Why do long readers influence the rest of the transactions in such a
> heavy way?
> Any configuration changes which can help here?
> Is it a disc-IO bottleneck thing?

Very possibly. Have you spent any time watching "vmstat 1" output
to get a sense of whether your I/O is saturated?

> WAL files are located on another disc than the dbase itself.

That's good, but it only relates to update performance not SELECT
performance.

> effective_cache_size = 5000

That's way too small for a 4G machine. You could probably stand to
boost maintenance_work_mem too. However, neither of these have any
immediate relationship to your problem.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

On Wed, Jul 11, 2007 at 05:35:33PM +0200, Patric de Waha wrote:
> Mainly updates on 1 tuple.

Are you vacuuming that table enough?

> And more or less complex SELECT statements.
> I noticed that the overall performance of postgres is decreasing
> when one or more long
> readers are present. Where a long reader here is already a Select
> count(*) from table.

SELECT count(*) is expensive in Postgres. Do you really need it?
Unqualified count() in PostgreSQL is just a bad thing to do, so if
you can work around it (by doing limited subselects, for instance,
where you never scan more than 50 rows, or by keeping counts using
triggers, or various other tricks), it's a good idea.

> Why do long readers influence the rest of the transactions in such a
> heavy way?

It could be because of all those updated tuples not getting vacuumed
(which results in a bad plan). Or it could be that your connection
pool is exhausted: note that when someone hits "reload", that doesn't
mean your old query goes away. It is still crunching through
whatever work it was doing.

> Second question. What is the right choice for the shared_buffers size?
> On a dedicated postgres server with 4 Giga RAM. Is there any rule of
> thumb?
> Actually I set it to +-256M.

There has been Much Discussion of this lately on this list. I
suggest you have a look through the recent archives on that topic.

A

--
Andrew Sullivan | ajs@crankycanuck.ca
"The year's penultimate month" is not in truth a good way of saying
November.
--H.W. Fowler

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

Ok thanks.

iostat confirmed it's an IO bottleneck.
Will add some discs to the RAID unit.

Used 4 Raptor discs in Raid 10 until now.


best regards,
patric


Tom Lane wrote:
> Patric de Waha <lists@p-dw.com> writes:
>
>> Postgres is running on a dedicated server P4 DualCore, 4 Gig Ram.
>>
>
> When you don't even mention your disk hardware, that's a bad sign.
> In a database server the disk is usually more important than the CPU.
>
>
>> Why do long readers influence the rest of the transactions in such a
>> heavy way?
>> Any configuration changes which can help here?
>> Is it a disc-IO bottleneck thing?
>>
>
> Very possibly. Have you spent any time watching "vmstat 1" output
> to get a sense of whether your I/O is saturated?
>
>
>> WAL files are located on another disc than the dbase itself.
>>
>
> That's good, but it only relates to update performance not SELECT
> performance.
>
>
>> effective_cache_size = 5000
>>
>
> That's way too small for a 4G machine. You could probably stand to
> boost maintenance_work_mem too. However, neither of these have any
> immediate relationship to your problem.
>
> regards, tom lane
>


---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match