My company is purchasing a Sunfire x4500 to run our most I/O-bound databases, and I'd like to get some advice on configuration and tuning. We're currently looking at:
- Solaris 10 + zfs + RAID Z
- CentOS 4 + xfs + RAID 10
- CentOS 4 + ext3 + RAID 10
but we're open to other suggestions.

From previous message threads, it looks like some of you have achieved stellar performance under both Solaris 10 U2/U3 with zfs and CentOS 4.4 with xfs. Would those of you who posted such results please describe how you tuned the OS/fs to yield those figures (e.g. patches, special drivers, read-ahead, checksumming, write-through cache settings, etc.)?

Most of our servers currently run CentOS/RedHat, and we have little experience with Solaris, but we're not opposed to Solaris if there's a compelling reason to switch. For example, it sounds like zfs snapshots may have a lighter performance penalty than LVM snapshots. We've heard that just using LVM (even without active snapshots) imposes a maximum sequential I/O rate of around 600 MB/s (although we haven't yet reached this limit experimentally).

By the way, we've also heard that Solaris is "more stable" under heavy I/O load than Linux. Have any of you experienced this? It's hard to put much stock in such a blanket statement, but naturally we don't want to introduce instabilities.

Thanks in advance for your thoughts!

For reference:

Our database cluster will be 3-6 TB in size. The Postgres installation will be 8.1 (at least initially), compiled to use 32 KB blocks (rather than 8 KB). The workload will be predominantly OLAP. The Sunfire X4500 has 2 dual-core Opterons, 16 GB RAM, 48 SATA disks (500 GB/disk * 48 = 24 TB raw -> 12 TB usable under RAID 10).

So far, we've seen the X4500 deliver impressive but suboptimal results using the out-of-the-box installation of Solaris + zfs. The Linux testing is in the early stages (no xfs, yet), but so far it yeilds comparatively modest write rates and very poor read and rewrite rates.

===============================
Results under Solaris with zfs:
===============================

Four concurrent writers:
% time dd if=/dev/zero of=/zpool1/test/50GB-zero1 bs=1024k count=51200 ; time sync
% time dd if=/dev/zero of=/zpool1/test/50GB-zero2 bs=1024k count=51200 ; time sync
% time dd if=/dev/zero of=/zpool1/test/50GB-zero3 bs=1024k count=51200 ; time sync
% time dd if=/dev/zero of=/zpool1/test/50GB-zero4 bs=1024k count=51200 ; time sync

Seq Write (bs = 1 MB): 128 + 122 + 131 + 124 = 505 MB/s

Four concurrent readers:
% time dd if=/zpool1/test/50GB-zero1 of=/dev/null bs=1024k
% time dd if=/zpool1/test/50GB-zero2 of=/dev/null bs=1024k
% time dd if=/zpool1/test/50GB-zero3 of=/dev/null bs=1024k
% time dd if=/zpool1/test/50GB-zero4 of=/dev/null bs=1024k

Seq Read (bs = 1 MB): 181 + 177 + 180 + 178 = 716 MB/s


One bonnie++ process:
% bonnie++ -r 16384 -s 32g:32k -f -n0 -d /zpool1/test/bonnie_scratch

Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
thumper1 32G:32k 604173 98 268893 43 543389 59 519.2 3
thumper1,32G:32k,,,604173,98,268893,43,,,543389,59 ,519.2,3,,,,,,,,,,,,,


4 concurrent synchronized bonnie++ processes:
% bonnie++ -p4
% bonnie++ -r 16384 -s 32g:32k -y -f -n0 -d /zpool1/test/bonnie_scratch
% bonnie++ -r 16384 -s 32g:32k -y -f -n0 -d /zpool1/test/bonnie_scratch
% bonnie++ -r 16384 -s 32g:32k -y -f -n0 -d /zpool1/test/bonnie_scratch
% bonnie++ -r 16384 -s 32g:32k -y -f -n0 -d /zpool1/test/bonnie_scratch
% bonnie++ -p-1

Combined results of 4 sessions:
Seq Output: 124 + 124 + 124 + 140 = 512 MB/s
Rewrite: 93 + 94 + 93 + 96 = 376 MB/s
Seq Input: 192 + 194 + 193 + 197 = 776 MB/s
Random Seek: 327 + 327 + 335 + 332 = 1321 seeks/s


=========================================
Results under CentOS 4 with ext3 and LVM:
=========================================

% bonnie++ -s 32g:32k -f -n0 -d /large_lvm_stripe/test/bonnie_scratch
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
thumper1.rt 32G:32k 346595 94 59448 11 132471 12 479.4 2
thumper1.rtkinternal,32G:32k,,,346595,94,59448,11, ,,132471,12,479.4,2,,,,,,,,,,,,,


============================
Summary of bonnie++ results:
============================

sequential sequential sequential scattered
Test case write MB/s rewrite MB/s read MB/s seeks/s
------------------------- ---------- ------------ ---------- ---------
Sol10+zfs, 1 process 604 269 543 519
Sol10+zfs, 4 processes 512 376 776 1321
Cent4+ext3+LVM, 1 process 347 59 132 479



---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

On Friday 23 March 2007 03:20, Matt Smiley wrote:
> My company is purchasing a Sunfire x4500 to run our most I/O-bound
> databases, and I'd like to get some advice on configuration and tuning.
> We're currently looking at: - Solaris 10 + zfs + RAID Z
> - CentOS 4 + xfs + RAID 10
> - CentOS 4 + ext3 + RAID 10
> but we're open to other suggestions.
>

Matt,

for Solaris + ZFS you may find answers to all your questions here:

http://blogs.sun.com/roch/category/ZFS
http://blogs.sun.com/realneel/entry/zfs_and_databases

Think to measure log (WAL) activity and use separated pool for logs if needed.
Also, RAID-Z is more security-oriented rather performance, RAID-10 should be
a better choice...

Rgds,
-Dimitri

---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate