Unix Technical Forum

Re: Append table

This is a discussion on Re: Append table within the Pgsql Performance forums, part of the PostgreSQL category; --> Can you help me appending two table values into single table without performing INSERT? Note that these tables are ...


Go Back   Unix Technical Forum > Database Server Software > PostgreSQL > Pgsql Performance

FAQ Members List Calendar Search Today's Posts Mark Forums Read
  #1 (permalink)  
Old 04-19-2008, 10:54 AM
Hanu Kurubar
 
Posts: n/a
Default Re: Append table

Can you help me appending two table values into single table without
performing INSERT?
Note that these tables are of same schema.

Is there any sql command is supported?

Thanks,
Hanu


On 5/29/07, Alvaro Herrera <alvherre@commandprompt.com> wrote:
>
> Michal Szymanski wrote:
> > There is another strange thing. We have two versions of our test
> > >>environment one with production DB copy and second genereated with
> > >>minimal data set and it is odd that update presented above on copy of
> > >>production is executing 170ms but on small DB it executing 6s !!!!
> > >
> > >How are you vacuuming the tables?
> > >

> > Using pgAdmin (DB is installed on my laptop) and I use this tool for
> > vaccuminh, I do not think that vaccuming can help because I've tested on
> > both database just after importing.

>
> I think you are misunderstanding the importance of vacuuming the table.
> Try this: on a different terminal from the one running the test, run a
> VACUUM on the updated table with vacuum_cost_delay set to 20, on an
> infinite loop. Keep this running while you do your update test. Vary
> the vacuum_cost_delay and measure the average/min/max UPDATE times.
> Also try putting a short sleep on the infinite VACUUM loop and see how
> its length affects the UPDATE times.
>
> One thing not clear to me is if your table is in a clean state. Before
> running this test, do a TRUNCATE and import the data again. This will
> get rid of any dead space that may be hurting your measurements.
>
> --
> Alvaro Herrera
> http://www.advogato.org/person/alvherre
> "The Postgresql hackers have what I call a "NASA space shot" mentality.
> Quite refreshing in a world of "weekend drag racer" developers."
> (Scott Marlowe)
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org
>




--
With best regards,
Hanumanthappa Kurubar
Mobile: 98 801 800 65

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #2 (permalink)  
Old 04-19-2008, 10:54 AM
Hanu Kurubar
 
Posts: n/a
Default Re: Append table

Any luck on appending two table in PostgreSQL.
Below are two table with same schema that have different values. In this
case EmpID is unique value.

tabelA
------------
EmpId (Int) EmpName (String)
1 Hanu
2 Alvaro


tabelB
------------
EmpId (Int) EmpName (String)
3 Michal
4 Tom


I would be looking below output after appending tableA with tableB. Is this
possible in PostgreSQL?


tabelA
------------
EmpId (Int) EmpName (String)
1 Hanu
2 Alvaro
3 Michal
4 Tom


Thanks,
Hanu


On 5/30/07, Hanu Kurubar <hanu.kurubar@gmail.com> wrote:
>
> Can you help me appending two table values into single table without
> performing INSERT?
> Note that these tables are of same schema.
>
> Is there any sql command is supported?
>
> Thanks,
> Hanu
>
>
> On 5/29/07, Alvaro Herrera <alvherre@commandprompt.com> wrote:
> >
> > Michal Szymanski wrote:
> > > There is another strange thing. We have two versions of our test
> > > >>environment one with production DB copy and second genereated with
> > > >>minimal data set and it is odd that update presented above on copy

> > of
> > > >>production is executing 170ms but on small DB it executing 6s !!!!
> > > >
> > > >How are you vacuuming the tables?
> > > >
> > > Using pgAdmin (DB is installed on my laptop) and I use this tool for
> > > vaccuminh, I do not think that vaccuming can help because I've tested

> > on
> > > both database just after importing.

> >
> > I think you are misunderstanding the importance of vacuuming the table.
> > Try this: on a different terminal from the one running the test, run a
> > VACUUM on the updated table with vacuum_cost_delay set to 20, on an
> > infinite loop. Keep this running while you do your update test. Vary
> > the vacuum_cost_delay and measure the average/min/max UPDATE times.
> > Also try putting a short sleep on the infinite VACUUM loop and see how
> > its length affects the UPDATE times.
> >
> > One thing not clear to me is if your table is in a clean state. Before
> > running this test, do a TRUNCATE and import the data again. This will
> > get rid of any dead space that may be hurting your measurements.
> >
> > --
> > Alvaro Herrera
> > http://www.advogato.org/person/alvherre
> > "The Postgresql hackers have what I call a "NASA space shot" mentality.
> > Quite refreshing in a world of "weekend drag racer" developers."
> > (Scott Marlowe)
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 4: Have you searched our list archives?
> >
> > http://archives.postgresql.org
> >

>
>
>
> --
> With best regards,
> Hanumanthappa Kurubar
> Mobile: 98 801 800 65





--
With best regards,
Hanumanthappa Kurubar
Mobile: 98 801 800 65

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #3 (permalink)  
Old 04-19-2008, 10:54 AM
Arjen van der Meijden
 
Posts: n/a
Default Re: Append table

There are two solutions:
You can insert all data from tableB in tableA using a simple insert
select-statement like so:
INSERT INTO tabelA SELECT EmpId, EmpName FROM tabelB;

Or you can visually combine them without actually putting the records in
a single table. That can be with a normal select-union statement or with
a view, something like this:
SELECT EmpId, EmpName FROM tabelA UNION EmpID, EmpName FROM tabelB;

You can use this query as a table-generating subquery in a FROM-clause,
like so:

SELECT * FROM (SELECT EmpId, EmpName FROM tabelA UNION EmpID, EmpName
FROM tabelB) as emps WHERE EmpId = 1;

Or with the view:
CREATE VIEW tabelC AS SELECT EmpId, EmpName FROM tabelA UNION EmpID,
EmpName FROM tabelB;

And then you can use the view as if it was a normal table (altough
inserts are not possible without applying rules to them, see the manual
for that).

SELECT * FROM tabelC WHERE EmpId = 1;

Best regards,

Arjen

On 2-6-2007 17:52 Hanu Kurubar wrote:
> Any luck on appending two table in PostgreSQL.
> Below are two table with same schema that have different values. In this
> case EmpID is unique value.
>
> tabelA
> ------------
> EmpId (Int) EmpName (String)
> 1 Hanu
> 2 Alvaro
>
>
> tabelB
> ------------
> EmpId (Int) EmpName (String)
> 3 Michal
> 4 Tom
>
>
> I would be looking below output after appending tableA with tableB. Is
> this possible in PostgreSQL?
>
>
> tabelA
> ------------
> EmpId (Int) EmpName (String)
> 1 Hanu
> 2 Alvaro
> 3 Michal
> 4 Tom
>
>
>
> Thanks,
> Hanu
>
>
> On 5/30/07, *Hanu Kurubar* <hanu.kurubar@gmail.com
> <mailto:hanu.kurubar@gmail.com>> wrote:
>
> Can you help me appending two table values into single table without
> performing INSERT?
> Note that these tables are of same schema.
>
> Is there any sql command is supported?
>
> Thanks,
> Hanu
>
>
> On 5/29/07, *Alvaro Herrera* <alvherre@commandprompt.com
> <mailto:alvherre@commandprompt.com>> wrote:
>
> Michal Szymanski wrote:
> > There is another strange thing. We have two versions of our test
> > >>environment one with production DB copy and second

> genereated with
> > >>minimal data set and it is odd that update presented above

> on copy of
> > >>production is executing 170ms but on small DB it executing

> 6s !!!!
> > >
> > >How are you vacuuming the tables?
> > >

> > Using pgAdmin (DB is installed on my laptop) and I use this

> tool for
> > vaccuminh, I do not think that vaccuming can help because

> I've tested on
> > both database just after importing.

>
> I think you are misunderstanding the importance of vacuuming the
> table.
> Try this: on a different terminal from the one running the test,
> run a
> VACUUM on the updated table with vacuum_cost_delay set to 20, on an
> infinite loop. Keep this running while you do your update
> test. Vary
> the vacuum_cost_delay and measure the average/min/max UPDATE times.
> Also try putting a short sleep on the infinite VACUUM loop and
> see how
> its length affects the UPDATE times.
>
> One thing not clear to me is if your table is in a clean
> state. Before
> running this test, do a TRUNCATE and import the data
> again. This will
> get rid of any dead space that may be hurting your measurements.
>
> --
> Alvaro
> Herrera http://www.advogato.org/person/alvherre
> "The Postgresql hackers have what I call a "NASA space shot"
> mentality.
> Quite refreshing in a world of "weekend drag racer" developers."
> (Scott Marlowe)
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org
> <http://archives.postgresql.org/>
>
>
>
>
> --
> With best regards,
> Hanumanthappa Kurubar
> Mobile: 98 801 800 65
>
>
>
>
> --
> With best regards,
> Hanumanthappa Kurubar
> Mobile: 98 801 800 65


---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #4 (permalink)  
Old 04-19-2008, 10:54 AM
Gregory Stark
 
Posts: n/a
Default Re: Append table

"Arjen van der Meijden" <acmmailing@tweakers.net> writes:

> There are two solutions:

....
> Or you can visually combine them without actually putting the records in a
> single table. That can be with a normal select-union statement or with a view,
> something like this:
> SELECT EmpId, EmpName FROM tabelA UNION EmpID, EmpName FROM tabelB;


If you're sure the two sets are distinct or you want to get any duplicates and
not eliminate them then if you went with this option you would want to use
"UNION ALL" not just a plain union.

In SQL UNION has to remove duplicates which often involves gathering all the
records and performing a big sort and lots of extra work. UNION ALL is much
faster and can start returning records right away.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com


---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #5 (permalink)  
Old 04-19-2008, 10:54 AM
Hanu Kurubar
 
Posts: n/a
Default Re: Append table

Thanks for quick answer.

Previsoly I have exported table records into employee.csv file using COPY
command which has 36,00,0000 records.

After that I have added few more entries in database and EmpId is
incremented.

I want put the exported data back into database with re-generating new
EmpId. Like importing back all data without harming existing data.

If I choose INSERT opeartion, it is very time consuming.

I am thinking of creating new table (dummy table) and copying all data (COPY
from command) into that table and maniplate the data so that EmpId is unique
in parent table and dummy table and then append these two tables.

I feel creating views and joins will make things complex.

Do you have inputs on this?

On 6/2/07, Arjen van der Meijden <acmmailing@tweakers.net> wrote:
>
> There are two solutions:
> You can insert all data from tableB in tableA using a simple insert
> select-statement like so:
> INSERT INTO tabelA SELECT EmpId, EmpName FROM tabelB;
>
> Or you can visually combine them without actually putting the records in
> a single table. That can be with a normal select-union statement or with
> a view, something like this:
> SELECT EmpId, EmpName FROM tabelA UNION EmpID, EmpName FROM tabelB;
>
> You can use this query as a table-generating subquery in a FROM-clause,
> like so:
>
> SELECT * FROM (SELECT EmpId, EmpName FROM tabelA UNION EmpID, EmpName
> FROM tabelB) as emps WHERE EmpId = 1;
>
> Or with the view:
> CREATE VIEW tabelC AS SELECT EmpId, EmpName FROM tabelA UNION EmpID,
> EmpName FROM tabelB;
>
> And then you can use the view as if it was a normal table (altough
> inserts are not possible without applying rules to them, see the manual
> for that).
>
> SELECT * FROM tabelC WHERE EmpId = 1;
>
> Best regards,
>
> Arjen
>
> On 2-6-2007 17:52 Hanu Kurubar wrote:
> > Any luck on appending two table in PostgreSQL.
> > Below are two table with same schema that have different values. In this
> > case EmpID is unique value.
> >
> > tabelA
> > ------------
> > EmpId (Int) EmpName (String)
> > 1 Hanu
> > 2 Alvaro
> >
> >
> > tabelB
> > ------------
> > EmpId (Int) EmpName (String)
> > 3 Michal
> > 4 Tom
> >
> >
> > I would be looking below output after appending tableA with tableB. Is
> > this possible in PostgreSQL?
> >
> >
> > tabelA
> > ------------
> > EmpId (Int) EmpName (String)
> > 1 Hanu
> > 2 Alvaro
> > 3 Michal
> > 4 Tom
> >
> >
> >
> > Thanks,
> > Hanu
> >
> >
> > On 5/30/07, *Hanu Kurubar* <hanu.kurubar@gmail.com
> > <mailto:hanu.kurubar@gmail.com>> wrote:
> >
> > Can you help me appending two table values into single table without
> > performing INSERT?
> > Note that these tables are of same schema.
> >
> > Is there any sql command is supported?
> >
> > Thanks,
> > Hanu
> >
> >
> > On 5/29/07, *Alvaro Herrera* <alvherre@commandprompt.com
> > <mailto:alvherre@commandprompt.com>> wrote:
> >
> > Michal Szymanski wrote:
> > > There is another strange thing. We have two versions of our

> test
> > > >>environment one with production DB copy and second

> > genereated with
> > > >>minimal data set and it is odd that update presented above

> > on copy of
> > > >>production is executing 170ms but on small DB it executing

> > 6s !!!!
> > > >
> > > >How are you vacuuming the tables?
> > > >
> > > Using pgAdmin (DB is installed on my laptop) and I use this

> > tool for
> > > vaccuminh, I do not think that vaccuming can help because

> > I've tested on
> > > both database just after importing.

> >
> > I think you are misunderstanding the importance of vacuuming the
> > table.
> > Try this: on a different terminal from the one running the test,
> > run a
> > VACUUM on the updated table with vacuum_cost_delay set to 20, on

> an
> > infinite loop. Keep this running while you do your update
> > test. Vary
> > the vacuum_cost_delay and measure the average/min/max UPDATE

> times.
> > Also try putting a short sleep on the infinite VACUUM loop and
> > see how
> > its length affects the UPDATE times.
> >
> > One thing not clear to me is if your table is in a clean
> > state. Before
> > running this test, do a TRUNCATE and import the data
> > again. This will
> > get rid of any dead space that may be hurting your measurements.
> >
> > --
> > Alvaro
> > Herrera

> http://www.advogato.org/person/alvherre
> > "The Postgresql hackers have what I call a "NASA space shot"
> > mentality.
> > Quite refreshing in a world of "weekend drag racer" developers."
> > (Scott Marlowe)
> >
> > ---------------------------(end of
> > broadcast)---------------------------
> > TIP 4: Have you searched our list archives?
> >
> > http://archives.postgresql.org
> > <http://archives.postgresql.org/>
> >
> >
> >
> >
> > --
> > With best regards,
> > Hanumanthappa Kurubar
> > Mobile: 98 801 800 65
> >
> >
> >
> >
> > --
> > With best regards,
> > Hanumanthappa Kurubar
> > Mobile: 98 801 800 65

>




--
With best regards,
Hanumanthappa Kurubar
Mobile: 98 801 800 65

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #6 (permalink)  
Old 04-19-2008, 10:56 AM
Chander Ganesan
 
Posts: n/a
Default Re: Append table

Arjen van der Meijden wrote:
> There are two solutions:
> You can insert all data from tableB in tableA using a simple insert
> select-statement like so:
> INSERT INTO tabelA SELECT EmpId, EmpName FROM tabelB;
>
> Or you can visually combine them without actually putting the records
> in a single table. That can be with a normal select-union statement or
> with a view, something like this:
> SELECT EmpId, EmpName FROM tabelA UNION EmpID, EmpName FROM tabelB;

Since they both have the same schema, you could also combine them by
creating a parent table and making both tables children. Check out
PostgreSQL's inheritance features. To make an existing table a child
you'll need to be using PostgreSQL 8.2 or newer.

create table emp_rollup (like tabelA);
alter table tabelA inherits emp_rollup;
alter table tabelB inherits emp_rollup;

Now issue your queries against emp_rollup... You could also just make
tabelB a child of tabelA:

alter table tabelB inherits tabelA;

But that would mean that if you wanted to query only tabelA you'd have
to modify your query syntax.

select * from ONLY tabelA;

Would only retrieve records from tabelA ...


You could also allow PostgreSQL to limit its index usage based on the
EmpID field by defining some table constraints and enabling constraint
exclusion.
>
> You can use this query as a table-generating subquery in a
> FROM-clause, like so:
>
> SELECT * FROM (SELECT EmpId, EmpName FROM tabelA UNION EmpID, EmpName
> FROM tabelB) as emps WHERE EmpId = 1;
>
> Or with the view:
> CREATE VIEW tabelC AS SELECT EmpId, EmpName FROM tabelA UNION EmpID,
> EmpName FROM tabelB;
>
> And then you can use the view as if it was a normal table (altough
> inserts are not possible without applying rules to them, see the
> manual for that).
>
> SELECT * FROM tabelC WHERE EmpId = 1;
>
> Best regards,
>
> Arjen
>
> On 2-6-2007 17:52 Hanu Kurubar wrote:
>> Any luck on appending two table in PostgreSQL.
>> Below are two table with same schema that have different values. In
>> this case EmpID is unique value.
>>
>> tabelA
>> ------------
>> EmpId (Int) EmpName (String)
>> 1 Hanu
>> 2 Alvaro
>>
>>
>> tabelB
>> ------------
>> EmpId (Int) EmpName (String)
>> 3 Michal
>> 4 Tom
>>
>>
>> I would be looking below output after appending tableA with tableB.
>> Is this possible in PostgreSQL?
>>
>>
>> tabelA
>> ------------
>> EmpId (Int) EmpName (String)
>> 1 Hanu
>> 2 Alvaro
>> 3 Michal
>> 4 Tom
>>



--
Chander Ganesan
The Open Technology Group
One Copley Parkway, Suite 210
Morrisville, NC 27560
Phone: 877-258-8987/919-463-0999
http://www.otg-nc.com
Expert PostgreSQL Training - http://test.otg-nc.com/training-cour...id=40&cat_id=8


---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On
Forum Jump


All times are GMT. The time now is 05:23 AM.


Powered by vBulletin® Version 3.6.5
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
Search Engine Optimization by vBSEO 3.2.0
www.UnixAdminTalk.com