Hello,



I've recently been tasked with scalability/performance testing of a Dell
PowerEdge 2950. This is the one with the new Intel Woodcrest Xeons.
Since I haven't seen any info on this box posted to the list, I figured
people might be interested in the results, and maybe in return share a
few tips on performance tweaks.



After doing some reading on the performance list, I realize that there's
a preference for Opteron; however, the goal of these experiments is to
see what I can get the 2950 to do. I will also be comparing performance
vs. a 1850 at some point, if there's any interest I can post those
numbers too.



Here's the hardware:

2xDual Core 3.0 Ghz CPU (Xeon 5160- 1333Mhz FSB, 4 MB shared cache per
socket)

8 GB RAM (DDR2, fully buffered, Dual Ranked, 667 Mhz)

6x300 10k RPM SAS drives

Perc 5i w/256 MB battery backed cache



The target application:

Mostly OLAP (large bulk loads, then lots of reporting, possibly moving
to real-time loads in the future). All of this will be run on FreeBSD
6.1 amd64. (If I have some extra time, I might be able to run a few
tests on linux just for comparison's sake)



Test strategy:

Make sure the RAID is giving reasonable performance:

bonnie++ -d /u/bonnie -s 1000:8k

time bash -c "(dd if=/dev/zero of=bigfile count=125000 bs=8k && sync)"



Now, I realize that the above are overly simple, and not indicative of
overall performance, however here's what I'm seeing:

Single 10K 300 GB drive - ~75 Mb/s on both tests, more or less

RAID 10, 6 disks (3 sets of mirrored pairs) - ~117 Mb/s



The RAID 10 numbers look way off to me, so my next step is to go test
some different RAID configs. I'm going to look at a mirrored pair, and a
striped pair first, just to make sure the setup is sane. Then, RAID 5 x
6 disks, and mirrored pair + raid 10 with 4. Possibly software raid,
however I'm not very familiar with this on FreeBSD.



Once I get the RAID giving me reasonable results (I would think that a
raid 10 with 6 10k drives should be able to push >200 MB/s sustained
IO...no?) I will move on to other more DB specific tests.



A few questions:

1) Does anyone have other suggestions for testing raw IO for the RAID?



2) What is reasonable IO (bonnie++, dd) for 4 or 6 disks- RAID 10?



3) For DB tests, I would like to compare performance on the different
RAID configs and vs. the 1850. Maybe to assist also in some basic
postgresql.conf and OS tuning (but that will be saved mostly for when I
start application level testing). I realize that benchmarks don't
necessarily map to application performance, but it helps me establish a
baseline for the hardware. I'm currently running pgbench, but would like
something with a few more features (but hopefully without too much setup
time). I've heard mention of the OSDL's DBT tests, and I'm specifically
interested in DBT-2 and DBT-3. Any suggestions here?



Here's some initial numbers from pgbench (-s 50 -c 10 -t 100). Please
keep in mind that these are default installs of FreeBSD 6.1 and Postgres
8.1.4- NO tuning yet.

1850: run1: 121 tps, run2: 132 tps, run3: 229 tps

2950: run1: 178 tps, run2: 201 tps, run3:259 tps



Obviously neither PG nor FreeBSD are taking advantage of all the
hardware available in either case.



I will post the additional RAID numbers shortly...



Thanks,



Bucky

On Aug 9, 2006, at 11:56 AM, Bucky Jordan wrote:

> Here’s the hardware:
>
> 2xDual Core 3.0 Ghz CPU (Xeon 5160- 1333Mhz FSB, 4 MB shared cache
> per socket)
>
> 8 GB RAM (DDR2, fully buffered, Dual Ranked, 667 Mhz)
>
> 6x300 10k RPM SAS drives
>
> Perc 5i w/256 MB battery backed cache

Is the PERC 5/i dual channel? If so, are 1/2 the drives on one
channel and the other half on the other channel? I find this helps
RAID10 performance when the mirrored pairs are on separate channels.

Your transfer rate seems pretty good for Dell hardware, but I'm not
experienced with SAS drives to know if those numbers are good in an
absolute sense.

Also, which driver picked up the SAS controller? amr(4) or aac(4) or
some other? That makes a big difference too. I think the amr driver
is "better" than the aac driver.

....
Is the PERC 5/i dual channel?* If so, are 1/2 the drives on one channel and the other half on the other channel?* I find this helps RAID10 performance when the mirrored pairs are on separate channels.
....

With the SAS controller (PERC 5/i), every drive gets it's own 3 GB/s port.

....
Your transfer rate seems pretty good for Dell hardware, but I'm not experienced with SAS drives to know if those numbers are good in an absolute sense.

Also, which driver picked up the SAS controller?* amr(4) or aac(4) or some other?* That makes a big difference too.* I think the amr driver is "better" than the aac driver.
...

The internals of the current SAS drives are similar to the U320's they replaced in terms of read/write/seek performance, however the benefit is the SAS bus, which helps eliminate some of the U320 limitations (e.g. with Perc4, you only get 160 MB/s per channel as you mentioned). It's using the mfi driver...

Here's some simplistic performance numbers:
time bash -c "(dd if=/dev/zero of=bigfile count=125000 bs=8k && sync)"

Raid0 x 2 (2 spindles) ~138 MB/s on BSD
Raid5 x 4 ~160 MB/s BSD, ~274 MB/s Knoppix (ext2)
Raid5 x 6 ~255 MB/s BSD, 265 MB/s Knoppix (ext3)
Raid10 x 4 ~25 MB/s BSD
Raid50 x 6 ~144 MB/s BSD, 271 MB/s Knoppix

* BSD is 6.1-RELEASE amd64 with UFS + Soft updates, Knoppix is 5.1 (ext2 didn't like the > 1TB partition for the 6 disk RAID 5, hence ext3)

Seems to me the PERC5 has issues with layered raid (10, 50) as others have suggested on this list is a common problem with lower end raid cards. For now, I'm going with the RAID 5 option, however if I have time, I would like to test having the hardware do raid 0 and doing raid 1 in the os, or vice versa, as proposed in other posts.

Also, I ran a pgbench -s 50 -c 10 -t 1000 on a completely default BSD 6.1 and PG 8.1.4 install with RAID5 x 6 disks, and got 442 tps on a fresh run (the numbers climb very rapidly due to caching after running simultaneous tests without reinitializing the test db. I'm guessing this is due to OS caching since the default postgresql.conf is pretty limited in terms of resource use). I probably need to up the scaling factor significantly so the whole data set doesn't get cached in RAM if I want realistic results from simultaneous tests, but it seems quicker to just reinit each time at this point.

On to some kernel tweaks and some adjustments to postgresql.conf...

- Bucky


---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster