I am wanting some ideas about improving the performance of ORDER BY in
our use. I have a DB on the order of 500,000 rows and 50 columns.
The results are always sorted with ORDER BY. Sometimes, the users end up
with a search that matches most of the rows. In that case, I have a
LIMIT 5000 to keep the returned results under control. However, the
sorting seems to take 10-60 sec. If I do the same search without the
ORDER BY, it takes about a second.

I am currently on version 8.0.1 on Windows XP using a Dell Optiplex 280
with 1Gb of ram. I have set sort_mem=100000 set.

Any ideas?

Thanks,
Glenn


---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

Glenn,

On 12/5/06 9:12 AM, "Glenn Sullivan" <glenn.sullivan@varianinc.com> wrote:

> I am wanting some ideas about improving the performance of ORDER BY in
> our use. I have a DB on the order of 500,000 rows and 50 columns.
> The results are always sorted with ORDER BY. Sometimes, the users end up
> with a search that matches most of the rows. In that case, I have a
> LIMIT 5000 to keep the returned results under control. However, the
> sorting seems to take 10-60 sec. If I do the same search without the
> ORDER BY, it takes about a second.
>
> I am currently on version 8.0.1 on Windows XP using a Dell Optiplex 280
> with 1Gb of ram. I have set sort_mem=100000 set.
>
> Any ideas?

Upgrade to 8.1 or 8.2, there were very large performance improvements to the
sort code made for 8.1+. Also, when you've upgraded, you can experiment
with increasing work_mem to get better performance. At some value of
work_mem (probably > 32MB) you will reach a plateau of performance, probably
4-5 times faster than what you see with 8.0.

- Luke



---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

Glenn Sullivan <glenn.sullivan@varianinc.com> writes:
> I am wanting some ideas about improving the performance of ORDER BY in
> our use. I have a DB on the order of 500,000 rows and 50 columns.
> The results are always sorted with ORDER BY. Sometimes, the users end up
> with a search that matches most of the rows. In that case, I have a
> LIMIT 5000 to keep the returned results under control. However, the
> sorting seems to take 10-60 sec. If I do the same search without the
> ORDER BY, it takes about a second.

Does the ORDER BY match an index? If so, is it using the index?
(See EXPLAIN.)

> I am currently on version 8.0.1 on Windows XP using a Dell Optiplex 280
> with 1Gb of ram. I have set sort_mem=100000 set.

In 8.0 that might be counterproductively high --- we have seen cases
where more sort_mem = slower with the older sorting code. I concur
with Luke's advice that you should update to 8.2 (not 8.1) to get the
improved sorting code.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

On Tue, Dec 05, 2006 at 01:02:06PM -0500, Tom Lane wrote:
> In 8.0 that might be counterproductively high --- we have seen cases
> where more sort_mem = slower with the older sorting code. I concur
> with Luke's advice that you should update to 8.2 (not 8.1) to get the
> improved sorting code.

By the way, is the new sorting code any better for platforms that already
have a decent qsort() (like Linux)?

/* Steinar */
--
Homepage: http://www.sesse.net/

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

am Tue, dem 05.12.2006, um 13:02:06 -0500 mailte Tom Lane folgendes:
> In 8.0 that might be counterproductively high --- we have seen cases
> where more sort_mem = slower with the older sorting code. I concur
> with Luke's advice that you should update to 8.2 (not 8.1) to get the
> improved sorting code.

Hey, yes, 8.2 is released! Great, and thanks to all the people, which
help to develop this great software.


Andreas
--
Andreas Kretschmer
Kontakt: Heynitz: 035242/47215, D1: 0160/7141639 (mehr: -> Header)
GnuPG-ID: 0x3FFF606C, privat 0x7F4584DA http://wwwkeys.de.pgp.net

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

"Steinar H. Gunderson" <sgunderson@bigfoot.com> writes:
> By the way, is the new sorting code any better for platforms that already
> have a decent qsort() (like Linux)?

It seemed better to us. Linux' qsort() is really mergesort, which is
better sometimes but often worse --- mergesort tends to have a less
CPU-cache-friendly memory access distribution. Another big problem with
the Linux version is that it pays no attention to sort_mem, but will
enthusiastically allocate lots of additional memory, thereby blowing
whatever cross-backend memory budgeting you might have been doing.

If you care there is quite a lot of discussion in the -hackers and
-performance archives from last spring or so.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

Steinar H. Gunderson wrote:
> On Tue, Dec 05, 2006 at 01:02:06PM -0500, Tom Lane wrote:
>> In 8.0 that might be counterproductively high --- we have seen cases
>> where more sort_mem = slower with the older sorting code. I concur
>> with Luke's advice that you should update to 8.2 (not 8.1) to get the
>> improved sorting code.
>
> By the way, is the new sorting code any better for platforms that already
> have a decent qsort() (like Linux)?

yes - especially on-disk sorts will get some tremendous speedups in 8.2.


Stefan

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

Thanks to Luke and Tom for the input.&nbsp; I guess this was good timing given that it looks like
8.2 was just released today.&nbsp;&nbsp; I will upgade to that before doing anything else.

Glenn

Tom Lane wrote:

Glenn Sullivan &lt;glenn.sullivan@varianinc.com&gt; writes:



I am wanting some ideas about improving the performance of ORDER BY in our use. I have a DB on the order of 500,000 rows and 50 columns. The results are always sorted with ORDER BY. Sometimes, the users end up with a search that matches most of the rows. In that case, I have a LIMIT 5000 to keep the returned results under control. However, the sorting seems to take 10-60 sec. If I do the same search without the ORDER BY, it takes about a second.



Does the ORDER BY match an index? If so, is it using the index? (See EXPLAIN.)



I am currently on version 8.0.1 on Windows XP using a Dell Optiplex 280 with 1Gb of ram. I have set sort_mem=100000 set.



In 8.0 that might be counterproductively high --- we have seen cases where more sort_mem = slower with the older sorting code. I concur with Luke's advice that you should update to 8.2 (not 8.1) to get the improved sorting code. regards, tom lane