Hi,

You may remember some thread about data loading performances and
multi-threading support in pgloader:
http://archives.postgresql.org/pgsql...2/msg00081.php

The pgloader code to handle this is now ready to get tested, a more structured
project could talk about a Release Candidate status.
http://pgloader.projects.postgresql.org/dev/TODO.html
http://pgloader.projects.postgresql....rallel_loading
http://packages.debian.org/pgloader --- experimental has the next version

As for the performances benefit of this new version (2.3.0~dev2), all the work
could be reduced to zilch because of the python Global Interpreter Lock,
which I've been aware of late in the development effort.
http://docs.python.org/api/threads.html

This documentation states that (a) using generators you're not that
concerned, and (b) the global lock still allows for IO and processing at the
same time. As pgloader uses generators, I'm still not sure how much a problem
this will be.

I'd like to have some feedback about the new version, in term of bugs
encountered and performance limitations (is pgloader up to what you would
expect a multi-threaded loader to be at?)

Regards,
--
dim

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBHxAFVlBXRlnbh1bkRAiVZAJ91bRBP1xXG0BTia+vNk1 bi1+RaegCfdiml
o9BQ0aNi73NyJnVAHiTVr5Q=
=hlhG
-----END PGP SIGNATURE-----

On Tue, 2008-02-26 at 13:08 +0100, Dimitri Fontaine wrote:

> I'd like to have some feedback about the new version, in term of bugs
> encountered and performance limitations (is pgloader up to what you would
> expect a multi-threaded loader to be at?)

Maybe post to general as well if you don't get any replies here.

New feature is very important for us.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com

PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk


---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

Hi,

Le mardi 26 février 2008, Dimitri Fontaine a écritÂ*:
> You may remember some thread about data loading performances and
> multi-threading support in pgloader:
> http://archives.postgresql.org/pgsql...2/msg00081.php

As people here have asked for the new features implemented into pgloader
2.3.0, I'm happy to post here about the availability of the new version!
http://pgfoundry.org/projects/pgloader
http://pgfoundry.org/forum/forum.php?forum_id=1283

Please consider this as a testbed related to the parallel COPY and pg_restore
improvements which have been discussed here and on -hackers, as that's how
those new features came to life.

Regards,
--
dim

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBH1VzrlBXRlnbh1bkRAn95AJ9n3nXaZLFOgqF8b7V/U6iGfSGDxACeNtfB
YSTKfTs3hlIYwHFYL7qjUzE=
=Elbh
-----END PGP SIGNATURE-----