All tests are with bonnie++ 1.03a

Main components of system:
16 WD Raptor 150GB 10000 RPM drives all in a RAID 10
ARECA 1280 PCI-Express RAID adapter with 1GB BB Cache (Thanks for the
recommendation, Ron!)
32 GB RAM
Dual Intel 5160 Xeon Woodcrest 3.0 GHz processors
OS: SUSE Linux 10.1

All runs are with the write cache disabled on the hard disks, except
for one additional test for xfs where it was enabled. I tested with
ordered and writeback journaling modes for ext3 to see if writeback
journaling would help over the default of ordered. The 1GB of battery
backed cache on the RAID card was enabled for all tests as well.
Tests are in order of increasing random seek performance. In my tests
on this hardware, xfs is the decisive winner, beating all of the
other file systems in performance on every single metric. 658 random
seeks per second, 433 MB/sec sequential read, and 350 MB/sec
sequential write seems decent enough, but not as high as numbers
other people have suggested are attainable with a 16 disk RAID 10.
350 MB/sec sequential write with disk caches enabled versus 280 MB/
sec sequential write with disk caches disabled sure makes enabling
the disk write cache tempting. Anyone run their RAIDs with disk
caches enabled, or is this akin to having fsync off?

ext3 (writeback data journaling mode):
/usr/local/sbin/bonnie++ -d bonnie -s 64368:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
hulk4 64368M 78625 91 279921 51 112346 13 89463 96 417695
22 545.7 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --
Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 5903 99 +++++ +++ +++++ +++ 6112 99 +++++ ++
+ 18620 100
hulk4,64368M,
78625,91,279921,51,112346,13,89463,96,417695,22,54 5.7,0,16,5903,99,+++
++,+++,+++++,+++,6112,99,+++++,+++,18620,100

ext3 (ordered data journaling mode):
/usr/local/sbin/bonnie++ -d bonnie -s 64368:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
hulk4 64368M 74902 89 250274 52 123637 16 88992 96 417222
23 548.3 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --
Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 5941 97 +++++ +++ +++++ +++ 6270 99 +++++ ++
+ 18670 99
hulk4,64368M,
74902,89,250274,52,123637,16,88992,96,417222,23,54 8.3,0,16,5941,97,+++
++,+++,+++++,+++,6270,99,+++++,+++,18670,99


reiserfs:
/usr/local/sbin/bonnie++ -d bonnie -s 64368:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
hulk4 64368M 81004 99 269191 50 128322 16 87865 96 407035
28 550.3 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --
Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ ++
+ +++++ +++
hulk4,64368M,
81004,99,269191,50,128322,16,87865,96,407035,28,55 0.3,0,16,+++++,+++,+
++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

jfs:
/usr/local/sbin/bonnie++ -d bonnie/ -s 64368:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
hulk4 64368M 73246 80 268886 28 110465 9 89516 96 413897
21 639.5 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --
Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 3756 5 +++++ +++ +++++ +++ 23763 90 +++++ ++
+ 22371 70
hulk4,64368M,
73246,80,268886,28,110465,9,89516,96,413897,21,639 .5,0,16,3756,5,++++
+,+++,+++++,+++,23763,90,+++++,+++,22371,70

xfs (with write cache disabled on disks):
/usr/local/sbin/bonnie++ -d bonnie/ -s 64368:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
hulk4 64368M 90621 99 283916 35 105871 11 88569 97 433890
23 644.5 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --
Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 28435 95 +++++ +++ 28895 82 28523 91 +++++ ++
+ 24369 86
hulk4,64368M,
90621,99,283916,35,105871,11,88569,97,433890,23,64 4.5,0,16,28435,95,++
+++,+++,28895,82,28523,91,+++++,+++,24369,86

xfs (with write cache enabled on disks):
/usr/local/sbin/bonnie++ -d bonnie -s 64368:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
hulk4 64368M 90861 99 348401 43 131887 14 89412 97 432964
23 658.7 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --
Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 28871 90 +++++ +++ 28923 91 30879 93 +++++ ++
+ 28012 94
hulk4,64368M,
90861,99,348401,43,131887,14,89412,97,432964,23,65 8.7,0,16,28871,90,++
+++,+++,28923,91,30879,93,+++++,+++,28012,94




---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

Hi,

Alexander Staubo wrote:
> Care to post these numbers *without* word wrapping? Thanks.

How is one supposed to do that? Care giving an example?

Markus


---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

> Anyone run their RAIDs with disk caches enabled, or is this akin to
> having fsync off?

Disk write caches are basically always akin to having fsync off. The
only time a write-cache is (more or less) safe to enable is when it is
backed by a battery or in some other way made non-volatile.

So a RAID controller with a battery-backed write cache can enable its
own write cache, but can't safely enable the write-caches on the disk
drives it manages.

-- Mark Lewis

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

* Brian Wipf:

> Anyone run their RAIDs with disk caches enabled, or is this akin to
> having fsync off?

If your cache is backed by a battery, enabling write cache shouldn't
be a problem. You can check if the whole thing is working well by
running this test script: <http://brad.livejournal.com/2116715.html>

Enabling write cache leads to various degrees of data corruption in
case of a power outage (possibly including file system corruption
requiring manual recover).

--
Florian Weimer <fweimer@bfk.de>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

On Wed, Dec 06, 2006 at 05:31:01PM +0100, Markus Schiltknecht wrote:
>> Care to post these numbers *without* word wrapping? Thanks.
> How is one supposed to do that? Care giving an example?

This is a rather long sentence without any kind of word wrapping except what would be imposed on your own side -- how to set that up properly depends on the sending e-mail client, but in mine it's just a matter of turning off the word wrapping in your editor :-)

/* Steinar */
--
Homepage: http://www.sesse.net/

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

Hi,

Steinar H. Gunderson wrote:
> This is a rather long sentence without any kind of word wrapping except what would be imposed on your own side -- how to set that up properly depends on the sending e-mail client, but in mine it's just a matter of turning off the word wrapping in your editor :-)

Duh!

Cool, thank you for the example :-) I thought the MTA or at least the the mailing list would wrap mails at some limit. I've now set word-wrap to 9999 characters (it seems not possible to turn it off completely in thunderbird). But when writing, I'm now getting one long line.

What's common practice? What's it on the pgsql mailing lists?

Regards

Markus


---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

Markus Schiltknecht a écrit :
> What's common practice? What's it on the pgsql mailing lists?

The netiquette usually advise mailers to wrap after 72 characters
on mailing lists.
This does not apply for format=flowed I guess (that's the format
used in Steinar's message).

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

On Wed, Dec 06, 2006 at 06:59:12PM +0100, Arnaud Lesauvage wrote:
>Markus Schiltknecht a écrit :
>>What's common practice? What's it on the pgsql mailing lists?
>
>The netiquette usually advise mailers to wrap after 72 characters
>on mailing lists.
>This does not apply for format=flowed I guess (that's the format
>used in Steinar's message).

It would apply to either; format=flowed can be wrapped at the receiver's
end, but still be formatted to a particular column for readers that
don't understand format=flowed. (Which is likely to be many, since
that's a standard that never really took off.) No wrap netiquette
applies to formatted text blocks which are unreadable if wrapped (such
as bonnie or EXPLAIN output).

Mike Stone

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

On Dec 6, 2006, at 16:40 , Brian Wipf wrote:

> All tests are with bonnie++ 1.03a
[snip]

Care to post these numbers *without* word wrapping? Thanks.

Alexander.

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

Brian Wipf wrote:

> All tests are with bonnie++ 1.03a

Thanks for posting these tests. Now I have actual numbers to beat our
storage server provider about the head and shoulders with. Also, I
found them interesting in and of themselves.

These numbers are close enough to bus-saturation rates that I'd strongly
advise new people setting up systems to go this route over spending
money on some fancy storage area network solution- unless you need more
HD space than fits nicely in one of these raids. If reliability is a
concern, buy 2 servers and implement Sloni for failover.

Brian


---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend