I agree, I think these say you are getting 240MB/s sequential reads and 1000 seeks per second.

That's pretty much the best you'd expect.

- Luke

Sent from my GoodLink synchronized handheld (www.good.com)


-----Original Message-----
From: Alex Turner [mailto:armtuk@gmail.com]
Sent: Tuesday, August 08, 2006 02:40 AM Eastern Standard Time
To: steve.poe@gmail.com
Cc: Luke Lonergan; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Postgresql Performance on an HP DL385 and

These number are pretty darn good for a four disk RAID 10, pretty close to
perfect infact. Nice advert for the 642 - I guess we have a Hardware RAID
controller than will read indpendently from mirrors.

Alex

On 8/8/06, Steve Poe <steve.poe@gmail.com> wrote:
>
> Luke,
>
> Here are the results of two runs of 16GB file tests on XFS.
>
> scsi disc array
> xfs ,16G,81024,99,153016,24,73422,10,82092,97,243210,1 7,1043.1
> ,0,16,3172,7,+++++,+++,2957,9,3197,10,+++++,+++,24 84,8
> scsi disc array
> xfs ,16G,83320,99,155641,25,73662,10,81756,96,243352,1 8,1029.1
> ,0,16,3119,10,+++++,+++,2789,7,3263,11,+++++,+++,2 014,6
>
> Thanks.
>
> Steve
>
>
>
> > Can you run bonnie++ version 1.03a on the machine and report the results
> > here?
> >
> > It could be OK if you have the latest Linux driver for cciss, someone
> has
> > reported good results to this list with the latest, bleeding edge
> version of
> > Linux (2.6.17).
> >
> > - Luke
> >
>
>
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 9: In versions below 8.0, the planner will ignore your desire to
> > choose an index scan if your joining column's datatypes do not
> > match
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq
>


---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

Luke,

Thanks for the feedback. I use the same database test that I've run a Sun
dual Opteron with 4Gb RAM and (2) four disk arrays in RAID10. The sun box
with one disc on an LSI MegaRAID 2-channel adapter outperforms this HP box.
I though I was doing something wrong or there is something wrong with the
box.

Steve

On 8/8/06, Luke Lonergan <LLonergan@greenplum.com> wrote:
>
> I agree, I think these say you are getting 240MB/s sequential reads and
> 1000 seeks per second.
>
> That's pretty much the best you'd expect.
>
> - Luke
>
> Sent from my GoodLink synchronized handheld (www.good.com)
>
>
> -----Original Message-----
> From: Alex Turner [mailto:armtuk@gmail.com]
> Sent: Tuesday, August 08, 2006 02:40 AM Eastern Standard Time
> To: steve.poe@gmail.com
> Cc: Luke Lonergan; pgsql-performance@postgresql.org
> Subject: Re: [PERFORM] Postgresql Performance on an HP DL385 and
>
> These number are pretty darn good for a four disk RAID 10, pretty close to
> perfect infact. Nice advert for the 642 - I guess we have a Hardware RAID
> controller than will read indpendently from mirrors.
>
> Alex
>
> On 8/8/06, Steve Poe <steve.poe@gmail.com> wrote:
> >
> > Luke,
> >
> > Here are the results of two runs of 16GB file tests on XFS.
> >
> > scsi disc array
> > xfs ,16G,81024,99,153016,24,73422,10,82092,97,243210,1 7,1043.1
> > ,0,16,3172,7,+++++,+++,2957,9,3197,10,+++++,+++,24 84,8
> > scsi disc array
> > xfs ,16G,83320,99,155641,25,73662,10,81756,96,243352,1 8,1029.1
> > ,0,16,3119,10,+++++,+++,2789,7,3263,11,+++++,+++,2 014,6
> >
> > Thanks.
> >
> > Steve
> >
> >
> >
> > > Can you run bonnie++ version 1.03a on the machine and report the
> results
> > > here?
> > >
> > > It could be OK if you have the latest Linux driver for cciss, someone
> > has
> > > reported good results to this list with the latest, bleeding edge
> > version of
> > > Linux (2.6.17).
> > >
> > > - Luke
> > >
> >
> >
> > >
> > >
> > > ---------------------------(end of
> broadcast)---------------------------
> > > TIP 9: In versions below 8.0, the planner will ignore your desire to
> > > choose an index scan if your joining column's datatypes do not
> > > match
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 3: Have you checked our extensive FAQ?
> >
> > http://www.postgresql.org/docs/faq
> >
>
>

Steve,

On 8/8/06 8:01 AM, "Steve Poe" <steve.poe@gmail.com> wrote:

> Thanks for the feedback. I use the same database test that I've run a Sun
> dual Opteron with 4Gb RAM and (2) four disk arrays in RAID10. The sun box with
> one disc on an LSI MegaRAID 2-channel adapter outperforms this HP box. I
> though I was doing something wrong or there is something wrong with the box.

Given the circumstances (benchmarked I/O is great, comparable perf on
another box with single disk is better), seems that one of:
1) something wrong with the CPU/memory on the box
2) something with the OS version / kernel
3) something with the postgres configuration

Can you post the database benchmark results?

- Luke



---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

Luke,

Here's some background:
I use Pg 7.4.13 (I've tested as far back as 7.4.8). I use an 8GB data with a
program called odbc-bench. I run an 18 minute test. With each run, HP box
excluded, I unmount the discs involved, reformat, un-tar the backup of
PGDATA and pg_xlog back on the discs, start-up Postgresql, then run the
odbc-bench.

On the Sun box, I've benchmarked an average of 3 to 4 runs with each disc
(up to 8) in succession in RAID0, RAID5, and RAID10 where applicable. I've
done with with pg_xlog on the same discs as PGDATA and separately, so I've
felt like I had a good understanding of how the performance works. I've
notice performance seems to level off at around 6 discs with another 10-15%
with two more discs.

When I run odbc-bench, I also run vmstat in the background (through a python
script) which averages/summarzies the high/low/average of each category for
each minute then a final summary after the run.

On the Sun box, with 4 discs (RAID10) to one channel on the LSI RAID card, I
see an average TPS around 70. If I ran this off of one disc, I see an
average TPS of 32.

on the HP box, with 6-discs in RAID10 and 1 spare. I see a TPS of 34. I
don't have my vmstat reports with me, but I recall the CPU utilitization on
the HP was about 50% higher. I need to check on this.

Steve







On 8/8/06, Luke Lonergan <llonergan@greenplum.com> wrote:
>
> Steve,
>
> On 8/8/06 8:01 AM, "Steve Poe" <steve.poe@gmail.com> wrote:
>
> > Thanks for the feedback. I use the same database test that I've run a
> Sun
> > dual Opteron with 4Gb RAM and (2) four disk arrays in RAID10. The sun
> box with
> > one disc on an LSI MegaRAID 2-channel adapter outperforms this HP box. I
> > though I was doing something wrong or there is something wrong with the
> box.
>
> Given the circumstances (benchmarked I/O is great, comparable perf on
> another box with single disk is better), seems that one of:
> 1) something wrong with the CPU/memory on the box
> 2) something with the OS version / kernel
> 3) something with the postgres configuration
>
> Can you post the database benchmark results?
>
> - Luke
>
>
>

Steve,

On 8/8/06 9:57 AM, "Steve Poe" <steve.poe@gmail.com> wrote:

> On the Sun box, with 4 discs (RAID10) to one channel on the LSI RAID card, I
> see an average TPS around 70. If I ran this off of one disc, I see an average
> TPS of 32.
>
> on the HP box, with 6-discs in RAID10 and 1 spare. I see a TPS of 34. I don't
> have my vmstat reports with me, but I recall the CPU utilitization on the HP
> was about 50% higher. I need to check on this.

Sounds like there are a few moving parts here, one of which is the ODBC
driver.

First - using 7.4.x postgres is a big variable - not much experience on this
list with 7.4.x anymore.

What OS versions are on the two machines?

What is the network configuration of each - is a caching DNS server
available to each? What are the contents of /etc/resolv.conf?

Have you run "top" on the machines while the benchmark is running? What is
the top running process, what is it doing (RSS, swap, I/O wait, etc)?

Are any of the disks not healthy? Do you see any I/O errors in dmesg?

Note that tarring up the database directory and untarring it actually
changes the block layout of the files on the disk from what the database
might have done when it was created. When you create a tar archive of the
files in the DB directory, their contents will be packed in file name order
in the tar archive and unpacked that way as well. By comparison, the
ordering when the database lays them on disk might have been quite
different. This doesn't impact the problem you describe as you are
unpacking the tar file on both machines to start the process (right?).

- Luke



---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

>
> >Sounds like there are a few moving parts here, one of which is the ODBC
> >driver.


Yes, I need to use it since my clients use it for their veterinary
application.


>First - using 7.4.x postgres is a big variable - not much experience on
> this
> >list with 7.4.x anymore.


Like the previous, we have to use it since the manufacturer/vendor uses a
4GL language which only supports Postgresql 7.4.x

>What OS versions are on the two machines?


Centos 4.3 x84_64 on both boxes.


>What is the network configuration of each - is a caching DNS server
> >available to each? What are the contents of /etc/resolv.conf?


The database is configured for the local/loopback on 127.0.0.1. This is my
local network. No DNS.


>Have you run "top" on the machines while the benchmark is running? What is
> >the top running process, what is it doing (RSS, swap, I/O wait, etc)?


I am not running top, but here's an average per second for the 20-25min
run from vmstat presented in a high/peak, low and median

Sun box with 4-disc array (4GB RAM. 4 167GB 10K SCSI RAID10 LSI MegaRAID
128MB). This is after 8 runs.

dbserver-dual-opteron-centos,08/08/06,Tuesday,20,swapd,128,128,128
dbserver-dual-opteron-centos,08/08/06,Tuesday,20,free,21596,21050,21327
dbserver-dual-opteron-centos,08/08/06,Tuesday,20,buffers,1171,174,595
dbserver-dual-opteron-centos,08/08/06,Tuesday,20,cache,3514368,3467427,3495081
dbserver-dual-opteron-centos,08/08/06,Tuesday,20,bi,97276,1720,31745
dbserver-dual-opteron-centos,08/08/06,Tuesday,20,bo,9209,832,4674
dbserver-dual-opteron-centos,08/08/06,Tuesday,20,in,25906,23204,24115
dbserver-dual-opteron-centos,08/08/06,Tuesday,20,cs,49849,46035,47617
dbserver-dual-opteron-centos,08/08/06,Tuesday,20,us,12,2,5
dbserver-dual-opteron-centos,08/08/06,Tuesday,20,sy,59,50,53
dbserver-dual-opteron-centos,08/08/06,Tuesday,20,wa,1,0,0
dbserver-dual-opteron-centos,08/08/06,Tuesday,20,id,45,26,38

Average TPS is 75

HP box with 8GB RAM. six disc array RAID10 on SmartArray 642 with 192MB RAM.
After 8 runs, I see:

intown-vetstar-amd64,08/09/06,Tuesday,23,r,0,0,0
intown-vetstar-amd64,08/09/06,Tuesday,23,b,2,0,0
intown-vetstar-amd64,08/09/06,Tuesday,23,swapd,0,0,0
intown-vetstar-amd64,08/09/06,Tuesday,23,free,33760,16501,17931
intown-vetstar-amd64,08/09/06,Tuesday,23,buffers,1578,673,1179
intown-vetstar-amd64,08/09/06,Tuesday,23,cache,7881745,7867700,7876327
intown-vetstar-amd64,08/09/06,Tuesday,23,bi,66536,0,4480
intown-vetstar-amd64,08/09/06,Tuesday,23,bo,5991,2,2806
intown-vetstar-amd64,08/09/06,Tuesday,23,in,1624,260,573
intown-vetstar-amd64,08/09/06,Tuesday,23,cs,2342,17,1464
intown-vetstar-amd64,08/09/06,Tuesday,23,us,31,0,3
intown-vetstar-amd64,08/09/06,Tuesday,23,sy,16,0,1
intown-vetstar-amd64,08/09/06,Tuesday,23,wa,99,6,50
intown-vetstar-amd64,08/09/06,Tuesday,23,id,78,0,42

Average TPS is 31.


>Are any of the disks not healthy? Do you see any I/O errors in dmesg?


I don't know. I do the following message:
"PCI: MSI quirk detected. PCI_BUS_FLAGS_NO_MSI set for subordinate bus"

Otherwise, no disc error messages.

Note that tarring up the database directory and untarring it actually
> changes the block layout of the files on the disk from what the database
> might have done when it was created. When you create a tar archive of the
> files in the DB directory, their contents will be packed in file name
> order
> in the tar archive and unpacked that way as well. By comparison, the
> ordering when the database lays them on disk might have been quite
> different. This doesn't impact the problem you describe as you are
> unpacking the tar file on both machines to start the process (right?).


Yes, I am running this on both machines with the same RPMs of Postgresql and
same conf files.

Also, just for this testing, I am not unmounting, formatting, untaring. I am
doing it once than running the series of tests (usually 10 runs).

Thanks again for your time. If you're in the SF area, I'll owe you lunch and
> beer.



Steve

> Are any of the disks not healthy? Do you see any I/O errors in dmesg?

Luke,

In my vmstat report, I it is an average per minute not per-second. Also,
I found that in the first minute of the very first run, the HP's "bi"
value hits a high of 221184 then it tanks after that.

Steve


---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq