On Mon, 2006-12-04 at 10:25, Scott Marlowe wrote:
>
> OTOH, with the choice at my last place of employment being LSI or
> Adaptec, LSI was a much better choice.
>
> I'd ask which LSI megaraid you've tested, and what driver was used.
> Does RHEL4 have the megaraid 2 driver?

Just wanted to add that what we used our database for at my last company
was for lots of mostly small writes / reads. I.e. sequential throughput
didn't really matter, but random write speed did. for that application,
the LSI Megaraid with battery backed cache was great.

Last point, bonnie++ is a good benchmarking tool, but until you test
your app / postgresql on top of the hardware, you can't really say how
well it will perform.

A controller that looks fast under a single bonnie++ thread might
perform poorly when there are 100+ pending writes, and vice versa, a
controller that looks mediocre under bonnie++ might shine when there's
heavy parallel write load to handle.

---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

The RAID 10 was in there merely for filling in, not really as a compare,
indeed it would be ludicrous to compare a RAID 1 to a 6 drive RAID 10!!

How do I find out if it has version 2 of the driver?

This discussion I think is important, as I think it would be useful for this
list to have a list of RAID cards that _do_ work well under Linux/BSD for
people as recommended hardware for Postgresql. So far, all I can recommend
is what I've found to be good, which is 3ware 9500 series cards with 10k
SATA drives. Throughput was great until you reached higher levels of RAID
10 (the bonnie++ mark I posted showed write speed is a bit slow). But that
doesn't solve the problem for SCSI. What cards in the SCSI arena solve the
problem optimally? Why should we settle for sub-optimal performance in SCSI
when there are a number of almost optimally performing cards in the SATA
world (Areca, 3Ware/AMCC, LSI).

Thanks,

Alex

On 12/4/06, Scott Marlowe <smarlowe@g2switchworks.com> wrote:
>
> On Mon, 2006-12-04 at 01:17, Alex Turner wrote:
> > People recommend LSI MegaRAID controllers on here regularly, but I
> > have found that they do not work that well. I have bonnie++ numbers
> > that show the controller is not performing anywhere near the disk's
> > saturation level in a simple RAID 1 on RedHat Linux EL4 on two
> > seperate machines provided by two different hosting companies. In one
> > case I asked them to replace the card, and the numbers got a bit
> > better, but still not optimal.
> >
> > LSI MegaRAID has proved to be a bit of a disapointment. I have seen
> > better numbers from the HP SmartArray 6i, and from 3ware cards with
> > 7200RPM SATA drives.
> >
> > for the output: http://www.infoconinc.com/test/bonnie++.html (the
> > first line is a six drive RAID 10 on a 3ware 9500S, the next three are
> > all RAID 1s on LSI MegaRAID controllers, verified by lspci).
>
> Wait, you're comparing a MegaRAID running a RAID 1 against another
> controller running a 6 disk RAID10? That's hardly fair.
>
> My experience with the LSI was that with the 1.18 series drivers, they
> were slow but stable.
>
> With the version 2.x drivers, I found that the performance was very good
> with RAID-5 and fair with RAID-1 and that layered RAID was not any
> better than unlayered (i.e. layering RAID0 over RAID1 resulted in basic
> RAID-1 performance).
>
> OTOH, with the choice at my last place of employment being LSI or
> Adaptec, LSI was a much better choice.
>
> I'd ask which LSI megaraid you've tested, and what driver was used.
> Does RHEL4 have the megaraid 2 driver?
>

On Mon, Dec 04, 2006 at 12:37:29PM -0500, Alex Turner wrote:
>This discussion I think is important, as I think it would be useful for this
>list to have a list of RAID cards that _do_ work well under Linux/BSD for
>people as recommended hardware for Postgresql. So far, all I can recommend
>is what I've found to be good, which is 3ware 9500 series cards with 10k
>SATA drives. Throughput was great until you reached higher levels of RAID
>10 (the bonnie++ mark I posted showed write speed is a bit slow). But that
>doesn't solve the problem for SCSI. What cards in the SCSI arena solve the
>problem optimally? Why should we settle for sub-optimal performance in SCSI
>when there are a number of almost optimally performing cards in the SATA
>world (Areca, 3Ware/AMCC, LSI).

Well, one factor is to be more precise about what you're looking for; a
HBA != RAID controller, and you may be comparing apples and oranges. (If
you have an external array with an onboard controller you probably want
a simple HBA rather than a RAID controller.)

Mike Stone

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

http://en.wikipedia.org/wiki/RAID_controller

Alex

On 12/4/06, Michael Stone <mstone+postgres@mathom.us> wrote:
>
> On Mon, Dec 04, 2006 at 12:37:29PM -0500, Alex Turner wrote:
> >This discussion I think is important, as I think it would be useful for
> this
> >list to have a list of RAID cards that _do_ work well under Linux/BSD for
> >people as recommended hardware for Postgresql. So far, all I can
> recommend
> >is what I've found to be good, which is 3ware 9500 series cards with 10k
> >SATA drives. Throughput was great until you reached higher levels of
> RAID
> >10 (the bonnie++ mark I posted showed write speed is a bit slow). But
> that
> >doesn't solve the problem for SCSI. What cards in the SCSI arena solve
> the
> >problem optimally? Why should we settle for sub-optimal performance in
> SCSI
> >when there are a number of almost optimally performing cards in the SATA
> >world (Areca, 3Ware/AMCC, LSI).
>
> Well, one factor is to be more precise about what you're looking for; a
> HBA != RAID controller, and you may be comparing apples and oranges. (If
> you have an external array with an onboard controller you probably want
> a simple HBA rather than a RAID controller.)
>
> Mike Stone
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend
>

On Mon, Dec 04, 2006 at 12:52:46PM -0500, Alex Turner wrote:
>http://en.wikipedia.org/wiki/RAID_controller

What is the wikipedia quote supposed to prove? Pray tell, if you
consider RAID==HBA, what would you call a SCSI (e.g.) controller that
has no RAID functionality? If you'd call it an HBA, then there is a
useful distinction to be made, no?

Mike Stone

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

On Mon, 2006-12-04 at 11:43, Michael Stone wrote:
> On Mon, Dec 04, 2006 at 12:37:29PM -0500, Alex Turner wrote:
> >This discussion I think is important, as I think it would be useful for this
> >list to have a list of RAID cards that _do_ work well under Linux/BSD for
> >people as recommended hardware for Postgresql. So far, all I can recommend
> >is what I've found to be good, which is 3ware 9500 series cards with 10k
> >SATA drives. Throughput was great until you reached higher levels of RAID
> >10 (the bonnie++ mark I posted showed write speed is a bit slow). But that
> >doesn't solve the problem for SCSI. What cards in the SCSI arena solve the
> >problem optimally? Why should we settle for sub-optimal performance in SCSI
> >when there are a number of almost optimally performing cards in the SATA
> >world (Areca, 3Ware/AMCC, LSI).
>
> Well, one factor is to be more precise about what you're looking for; a
> HBA != RAID controller, and you may be comparing apples and oranges. (If
> you have an external array with an onboard controller you probably want
> a simple HBA rather than a RAID controller.)

I think he's been pretty clear. He's just talking about SCSI based RAID
controllers is all.

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

On Mon, 2006-12-04 at 11:37, Alex Turner wrote:
> The RAID 10 was in there merely for filling in, not really as a
> compare, indeed it would be ludicrous to compare a RAID 1 to a 6 drive
> RAID 10!!
>
> How do I find out if it has version 2 of the driver?

Go to the directory it lives in (on my Fedora Core 2 box, it's in
something like: /lib/modules/2.6.10-1.9_FC2/kernel/drivers/scsi )
and run modinfo on the driver:

modinfo megaraid.ko
author: LSI Logic Corporation
description: LSI Logic MegaRAID driver
license: GPL
version: 2.00.3

SNIPPED extra stuff

> This discussion I think is important, as I think it would be useful
> for this list to have a list of RAID cards that _do_ work well under
> Linux/BSD for people as recommended hardware for Postgresql. So far,
> all I can recommend is what I've found to be good, which is 3ware 9500
> series cards with 10k SATA drives. Throughput was great until you
> reached higher levels of RAID 10 (the bonnie++ mark I posted showed
> write speed is a bit slow). But that doesn't solve the problem for
> SCSI. What cards in the SCSI arena solve the problem optimally? Why
> should we settle for sub-optimal performance in SCSI when there are a
> number of almost optimally performing cards in the SATA world (Areca,
> 3Ware/AMCC, LSI).

Well, I think the LSI works VERY well under linux. And I've always made
it quite clear in my posts that while I find it an acceptable performer,
my main recommendation is based on it's stability, not speed, and that
the Areca and 3Ware cards are generally regarded as faster. And all
three beat the adaptecs which are observed as being rather unstable.

Does this LSI have battery backed cache? Are you testing it under heavy
parallel load versus single threaded to get an idea how it scales with
multiple processes hitting it at once?

Don't get me wrong, I'm a big fan of running tools like bonnie to get a
basic idea of how good the hardware is, but benchmarks that simulate
real production loads are the only ones worth putting your trust in.

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

On Mon, 4 Dec 2006, Alex Turner wrote:

> People recommend LSI MegaRAID controllers on here regularly, but I have
> found that they do not work that well. I have bonnie++ numbers that
> show the controller is not performing anywhere near the disk's
> saturation level in a simple RAID 1 on RedHat Linux EL4 on two seperate
> machines provided by two different hosting companies.
> http://www.infoconinc.com/test/bonnie++.html

I don't know what's going on with your www-september-06 machine, but the
other two are giving 32-40MB/s writes and 53-68MB/s reads. For a RAID-1
volume, these aren't awful numbers, but I agree they're not great.

My results are no better. For your comparison, here's a snippet of
bonnie++ results from one of my servers: RHEL 4, P4 3GHz, MegaRAID
firmware 1L37, write-thru cache setup, RAID 1; I think the drives are 10K
RPM Seagate Cheetahs. This is from the end of the drive where performance
is the worst (I partitioned the important stuff at the beginning where
it's fastest and don't have enough free space to run bonnie there):

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
20708 50 21473 9 9603 3 34419 72 55799 7 467.1 1

21Mb/s writes, 56MB/s reads. Not too different from yours (especially if
your results were from the beginning of the disk), and certainly nothing
special. I might be able to tune the write performance higher if I cared;
the battery backed cache sits unused and everything is tuned for paranoia
rather than performance. On this machine it doesn't matter.

The thing is, even though it's rarely the top performing card even when
setup perfectly, the LSI SCSI Megaraid just works. The driver is stable,
caching behavior is well defined, it's a pleasure to administer. I'm
never concerned that it's lying to me or doing anything to put data at
risk. The command-line tools for Linux work perfectly, let me look at or
control whatever I want, and it was straighforward for me to make my own
customized monitoring script using them.

> LSI MegaRAID has proved to be a bit of a disapointment. I have seen
> better numbers from the HP SmartArray 6i, and from 3ware cards with
> 7200RPM SATA drives.

Whereas although I use 7200RPM SATA drives, I always try to keep an eye on
them because I never really trust them. The performance list archives
here also have plenty of comments about people having issues with the
SmartArray controllers; search the archives for "cciss" and you'll see
what I'm talking about.

The Megaraid controller is very boring. That's why I like it. As a Linux
distribution, RedHat has similar characteristics. If I were going for a
performance setup, I'd dump that, too, for something sexier with a newish
kernel. It all depends on which side of the performance/stability
tradeoff you're aiming at.

On Mon, 4 Dec 2006, Scott Marlowe wrote:
> Does RHEL4 have the megaraid 2 driver?

This is from the moderately current RHEL4 installation I had results from
above. Redhat has probably done a kernel rev since I last updated back in
September, haven't needed or wanted to reboot since then:

megaraid cmm: 2.20.2.6 (Release Date: Mon Mar 7 00:01:03 EST 2005)
megaraid: 2.20.4.6-rh2 (Release Date: Wed Jun 28 12:27:22 EST 2006)

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

I agree, that MegaRAID is very stable, and it's very appealing from that
perspective. And two years ago I would have never even mentioned cciss
based cards on this list, because they sucked wind big time, but I believe
some people have started seeing better number from the 6i. 20MB/sec write,
when the number should be closer to 60.... thats off by a factor of 3. For
my data wharehouse application, thats a big difference, and if I can get a
better number from 7200RPM drives and a good SATA controller, I'm gonna do
that because my data isn't OLTP, and I don't care if the whole system shits
itself and I have to restore from backup one day.

My other and most important point is that I can't find any solid
recommendations for a SCSI card that can perform optimally in Linux or
*BSD. Off by a factor of 3x is pretty sad IMHO. (and yes, we know the
Adaptec cards suck worse, that doesn't bring us to a _good_ card).

Alex.

On 12/4/06, Greg Smith <gsmith@gregsmith.com> wrote:
>
> On Mon, 4 Dec 2006, Alex Turner wrote:
>
> > People recommend LSI MegaRAID controllers on here regularly, but I have
> > found that they do not work that well. I have bonnie++ numbers that
> > show the controller is not performing anywhere near the disk's
> > saturation level in a simple RAID 1 on RedHat Linux EL4 on two seperate
> > machines provided by two different hosting companies.
> > http://www.infoconinc.com/test/bonnie++.html
>
> I don't know what's going on with your www-september-06 machine, but the
> other two are giving 32-40MB/s writes and 53-68MB/s reads. For a RAID-1
> volume, these aren't awful numbers, but I agree they're not great.
>
> My results are no better. For your comparison, here's a snippet of
> bonnie++ results from one of my servers: RHEL 4, P4 3GHz, MegaRAID
> firmware 1L37, write-thru cache setup, RAID 1; I think the drives are 10K
> RPM Seagate Cheetahs. This is from the end of the drive where performance
> is the worst (I partitioned the important stuff at the beginning where
> it's fastest and don't have enough free space to run bonnie there):
>
> ------Sequential Output------ --Sequential Input- --Random-
> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> 20708 50 21473 9 9603 3 34419 72 55799 7 467.1 1
>
> 21Mb/s writes, 56MB/s reads. Not too different from yours (especially if
> your results were from the beginning of the disk), and certainly nothing
> special. I might be able to tune the write performance higher if I cared;
> the battery backed cache sits unused and everything is tuned for paranoia
> rather than performance. On this machine it doesn't matter.
>
> The thing is, even though it's rarely the top performing card even when
> setup perfectly, the LSI SCSI Megaraid just works. The driver is stable,
> caching behavior is well defined, it's a pleasure to administer. I'm
> never concerned that it's lying to me or doing anything to put data at
> risk. The command-line tools for Linux work perfectly, let me look at or
> control whatever I want, and it was straighforward for me to make my own
> customized monitoring script using them.
>
> > LSI MegaRAID has proved to be a bit of a disapointment. I have seen
> > better numbers from the HP SmartArray 6i, and from 3ware cards with
> > 7200RPM SATA drives.
>
> Whereas although I use 7200RPM SATA drives, I always try to keep an eye on
> them because I never really trust them. The performance list archives
> here also have plenty of comments about people having issues with the
> SmartArray controllers; search the archives for "cciss" and you'll see
> what I'm talking about.
>
> The Megaraid controller is very boring. That's why I like it. As a Linux
> distribution, RedHat has similar characteristics. If I were going for a
> performance setup, I'd dump that, too, for something sexier with a newish
> kernel. It all depends on which side of the performance/stability
> tradeoff you're aiming at.
>
> On Mon, 4 Dec 2006, Scott Marlowe wrote:
> > Does RHEL4 have the megaraid 2 driver?
>
> This is from the moderately current RHEL4 installation I had results from
> above. Redhat has probably done a kernel rev since I last updated back in
> September, haven't needed or wanted to reboot since then:
>
> megaraid cmm: 2.20.2.6 (Release Date: Mon Mar 7 00:01:03 EST 2005)
> megaraid: 2.20.4.6-rh2 (Release Date: Wed Jun 28 12:27:22 EST 2006)
>
> --
> * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>

On Tue, Dec 05, 2006 at 01:21:38AM -0500, Alex Turner wrote:
>My other and most important point is that I can't find any solid
>recommendations for a SCSI card that can perform optimally in Linux or
>*BSD. Off by a factor of 3x is pretty sad IMHO. (and yes, we know the
>Adaptec cards suck worse, that doesn't bring us to a _good_ card).

This gets back to my point about terminology. As a SCSI HBA the Adaptec
is decent: I can sustain about 300MB/s off a single channel of the
39320A using an external RAID controller. As a RAID controller I can't
even imagine using the Adaptec; I'm fairly certain they put that
"functionality" on there just so they could charge more for the card. It
may be that there's not much market for on-board SCSI RAID controllers;
between SATA on the low end and SAS & FC on the high end, there isn't a
whole lotta space left for SCSI. I definitely don't think much
R&D is going into SCSI controllers any more, compared to other solutions
like SATA or SAS RAID (the 39320 hasn't change in at least 3 years,
IIRC). Anyway, since the Adaptec part is a decent SCSI controller and a
lousy RAID controller, have you tried just using software RAID?

Mike Stone

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings