Hi All

What is different between page ins/outs and paging space page ins/
out ?

vmstat -s
1889201253 total address trans. faults
716070874 page ins
268266358 page outs
702336 paging space page ins
1300804 paging space page outs
0 total reclaims

Below is our machine output, is page ins/outs normal ?

Date/Time Page in Page Out
=================== =========== ===========
2007/08/20_17:27:57 716251158 268334210
2007/08/20_17:28:07 716262327 268338566
2007/08/20_17:28:17 716274397 268343052
2007/08/20_17:28:27 716285384 268347983
2007/08/20_17:28:37 716299228 268353886
2007/08/20_17:28:47 716311282 268360359
2007/08/20_17:28:57 716324433 268365294
2007/08/20_17:29:07 716335859 268370183

ok when a page out wether its a pageout or a paging space page pout
occurs the LRUD daemon finds
a page in memory not least recently use so it frees it!! what do you
mean 'frees it' ? Well, if the page was a computational page
then LRUD forces it to be written to paging space so this is a paging
space pageout. If the page is a file that has been read into memory
then it writes it to disk IF the file changed otherwise it just gives
the page to another process to overwrite.
A paging space page in indicates that a computational page (part of a
process) has been requested by another process and is read off of
paging space into memory. The other page in indicates a file was
requested from a process so it was read from a filesystem into memory.
That vmstat -s data might be 'normal' but only you as a system admin
of that particular system can answer that question.
On AIX systems that run databases you do wantto do virtual memory
tuning so that very little paging space is used.
But on the same well-tuned Database system ytou might see many page in
and page outs because different files are being accessed all the
time.
hope that this helps a little .....

On 8 20 , 10 56 , LarryBoy <larry.tay...@co.travis.tx.us> wrote:
> ok when a page out wether its a pageout or a paging space page pout
> occurs the LRUD daemon finds
> a page in memory not least recently use so it frees it!! what do you
> mean 'frees it' ? Well, if the page was a computational page
> then LRUD forces it to be written to paging space so this is a paging
> space pageout. If the page is a file that has been read into memory
> then it writes it to disk IF the file changed otherwise it just gives
> the page to another process to overwrite.
> A paging space page in indicates that a computational page (part of a
> process) has been requested by another process and is read off of
> paging space into memory. The other page in indicates a file was
> requested from a process so it was read from a filesystem into memory.
> That vmstat -s data might be 'normal' but only you as a system admin
> of that particular system can answer that question.
> On AIX systems that run databases you do wantto do virtual memory
> tuning so that very little paging space is used.
> But on the same well-tuned Database system ytou might see many page in
> and page outs because different files are being accessed all the
> time.
> hope that this helps a little .....

Thank.
Base on below information, why tends to set pi and po to zero value ?
http://www-03.ibm.com/servers/enable...66/page_4.html

It is seem that our machine is not memory-bound, It seem that is cpu-
bound(cpu id time is small)
vmstat -t 1 10
System Configuration: lcpu=3 mem=6689MB
kthr memory page faults cpu
time
----- ----------- ------------------------ ------------ -----------
--------
r b avm fre re pi po fr sr cy in sy cs us sy id wa hr
mi se
3 3 1032198 371 0 0 0 628 569 0 1216 13997 1447 45 7 38
10 09:00:46
14 8 1032255 370 0 0 0 1592 4349 0 2803 25605 4391 65 17 1
17 09:00:47
13 5 1032380 381 0 0 0 1657 4453 0 2751 26284 4350 49 20 1
30 09:00:48
0 12 1032562 486 0 0 0 1915 5115 0 2975 25642 4895 56 19 0
25 09:00:49
12 5 1032585 366 0 0 0 1416 3799 0 2731 22188 4359 51 18 2
29 09:00:50
6 12 1032576 376 0 0 0 1607 5128 0 3049 20635 4449 61 15 0
24 09:00:51
11 12 1033033 385 0 0 0 1926 25340 1 2760 55272 4007 59 23
4 14 09:00:52
8 13 1032485 554 0 0 0 1045 8906 0 2766 24047 3997 68 19 0
13 09:00:53
10 6 1032544 366 0 0 0 1398 115324 0 2759 23395 4178 71 15
0 14 09:00:54
4 15 1032518 450 0 0 0 1667 7211 0 2821 22323 4386 68 19 1
11 09:00:55

Check by "ps -e -o pid,ppid,pcpu,cpu,etime,time,vsz,user,rssize,comm" .
One of process take about 22% CPU.

PID PPID %CPU CP ELAPSED TIME VSZ
USER RSS COMMAND
1753166 1 22.0 78 05:22:40 03:33:04 24280
root 24412 _sqlsrv2
2785318 1 12.6 30 32:46 00:12:22 1368 phx
6300 _progres
3121200 1 7.1 19 02:27:45 00:31:28 2232 phx
7164 _progres

> Thank.
> Base on below information, why tends to set pi and po to zero
> value ?
> http://www-03.ibm.com/servers/enable...66/page_4.html

As stated: Reading from disk is much slower then from ram.
>
> It is seem that our machine is not memory-bound, It seem that is
> cpu-bound(cpu id time is small)

If you have idle>0 i would say that the system has free computing
resources so its not cpu bound.

According to your posts its looks for me that you have trashing of
file cache - Wait i/o in conjunction with a high sr/fr rate where
cy was even above 0 so your system is memory bound.
See man vmstat for details regarding sr/fr/cy

Your ps says that you are running a database ( progress ? ) which lead
to the conclusion that your system is not tuned for database usages
( usage of the CIO mount option and DB related tuning settings ).

So read the documentation which comes for your database and configure
the database and os accordingly or in case you have more memory - put
it in.

hth
Hajo

On 8 21 , 4 32 , Hajo Ehlers <serv...@metamodul.com> wrote:
> > Thank.
> > Base on below information, why tends to set pi and po to zero
> > value ?
> >http://www-03.ibm.com/servers/enable...66/page_4.html
>
> As stated: Reading from disk is much slower then from ram.
>
>
>
> > It is seem that our machine is not memory-bound, It seem that is
> > cpu-bound(cpu id time is small)
>
> If you have idle>0 i would say that the system has free computing
> resources so its not cpu bound.
>
> According to your posts its looks for me that you have trashing of
> file cache - Wait i/o in conjunction with a high sr/fr rate where
> cy was even above 0 so your system is memory bound.
> See man vmstat for details regarding sr/fr/cy
>
> Your ps says that you are running a database ( progress ? ) which lead
> to the conclusion that your system is not tuned for database usages
> ( usage of the CIO mount option and DB related tuning settings ).
>
> So read the documentation which comes for your database and configure
> the database and os accordingly or in case you have more memory - put
> it in.
>
> hth
> Hajo

Hi Hajo
But are paging space just 24% why Memory bound ?

<b>
System Configuration: lcpu=3 mem=6689MB
memory page faults
cpu
-------------- ------------------------------------- -----------------
------------
r b avm fre re pi po fr sr cy in sy cs
us sy id wa
2 3 1354885 372 0 0 1 1812 13693 0 2038 13319 2769
65 9 8 19
4 7 1354901 367 0 4 1 2123 21329 0 2103 11001 2946
53 9 10 28
6 7 1354900 362 0 2 1 1936 16109 0 2214 15161 3058
59 10 6 26
9 3 1355214 492 0 2 0 2282 25674 0 2330 20846 3322
60 15 5 20
8 8 1355123 376 0 2 0 1885 41644 1 2711 15529 3910
51 16 7 25
7 6 1354998 360 0 0 0 1588 26703 0 2867 13146 4247
69 16 0 14
4 7 1354958 383 0 1 0 1717 12439 0 2733 14364 3993
66 19 6 9
4 10 1354958 364 0 1 0 1625 67156 0 2415 12604 3361
73 11 1 15
8 4 1354971 361 0 0 0 1702 9267 0 2205 15116 3086
75 9 2 13
13 7 1354979 358 0 12 0 1781 7483 0 2183 13510 3045
75 8 0 17
8 6 1354979 365 0 6 2 1686 6266 0 2043 14048 2852
74 10 4 13
3 8 1354979 382 0 2 0 1720 7050 0 1919 11524 2702
81 8 0 11
4 8 1354979 373 0 1 0 1596 5723 0 1986 13677 3011
73 7 1 19
3 11 1354979 364 0 0 0 1604 10152 0 1987 13156 2951
74 6 0 19
16 4 1355213 362 0 3 1 2103 8158 0 1900 18582 2773
66 13 1 20
8 7 1354979 375 0 0 1 1771 8105 0 1872 11950 2701
73 6 1 20
4 11 1354979 361 0 0 1 1631 6302 0 1877 15135 2787
76 8 4 11
2 10 1354979 365 0 0 1 1865 5817 0 2187 12215 3177
59 10 0 31
3 9 1354985 370 0 0 3 1981 9215 0 2496 11559 3740
56 13 5 26
4 6 1355036 452 0 0 1 2133 9397 0 2601 13492 4229
57 17 0 26
The average usage of cpu is : 96.95 Included wait
The average id of cpu is : 3.05

2007/08/22,12:20:39,Users,690,paging space,3072MB,17%
2007/08/22,12:30:40,Users,692,paging space,3072MB,18%
2007/08/22,12:40:41,Users,687,paging space,3072MB,18%
2007/08/22,12:50:43,Users,682,paging space,3072MB,18%
2007/08/22,13:00:45,Users,681,paging space,3072MB,18%
2007/08/22,13:10:47,Users,677,paging space,3072MB,18%
2007/08/22,13:20:49,Users,678,paging space,3072MB,18%
2007/08/22,13:30:51,Users,672,paging space,3072MB,18%
2007/08/22,13:40:53,Users,684,paging space,3072MB,18%
2007/08/22,13:50:55,Users,693,paging space,3072MB,19%
2007/08/22,14:00:57,Users,694,paging space,3072MB,19%
2007/08/22,14:10:59,Users,707,paging space,3072MB,20%
2007/08/22,14:21:01,Users,712,paging space,3072MB,21%
2007/08/22,14:31:03,Users,702,paging space,3072MB,21%
2007/08/22,14:41:08,Users,715,paging space,3072MB,23%
2007/08/22,14:51:22,Users,697,paging space,3072MB,23%
2007/08/22,15:01:25,Users,710,paging space,3072MB,23%
2007/08/22,15:11:33,Users,716,paging space,3072MB,24%
2007/08/22,15:21:40,Users,712,paging space,3072MB,24%
2007/08/22,15:31:42,Users,716,paging space,3072MB,24%
2007/08/22,15:41:46,Users,716,paging space,3072MB,24%
</b>

> But are paging space just 24% why Memory bound ?

Because its looks for me that you have >>>>>file cache trashing <<<<!
which cause even some paging space in and outs.

Which means that either:
- The system has not enough ram at all for its load.
- The system and/or application is not properly configured for its
main task.

Since you provide no information about OS, Application and system
usage i can not give any tuning tips as already done.

regards
Hajo

BTW: get nmon to monitor your system and get familiar with the vmm of
AIX

Your vmstat's sr (pages of memory searched) is very high compared to
the fr (pages freed during LRUD's memory search)
that is what Hajo means by 'thrashing'
I have seen this type of behaviour when the VM is not tuned for the
database and when the DBA has kept making the datafile cache
(a generic DB term -- Oracle's specific term is SGA) larger and larger
until this type of condition occurred. When DBA's use a data file
cache
they pin (reserve only for application use) memory and thus decrease
the amount of memory other processes can use.

Remember if a file in memory is paged out then that action will not
increase paging space use --- so you can have paging out occurring
without an increase in the use of paging space.

To answer your question "why tends to set pi and po to zero value ?"
pi means page in and po means page out -- if fr (pages freed during
an LRUD search) is 0 then po will be 0 -- pi will be 0 if a file is
not or 4K pages of a file are not read into memory --- if no pages
are being freed (what you are seeing ) then no pages can be read in so
pi must be 0

hope that this adds clarity

On 8 22 , 11 08 , LarryBoy <larry.tay...@co.travis.tx.us> wrote:
> Your vmstat's sr (pages of memory searched) is very high compared to
> the fr (pages freed during LRUD's memory search)
> that is what Hajo means by 'thrashing'
> I have seen this type of behaviour when the VM is not tuned for the
> database and when the DBA has kept making the datafile cache
> (a generic DB term -- Oracle's specific term is SGA) larger and larger
> until this type of condition occurred. When DBA's use a data file
> cache
> they pin (reserve only for application use) memory and thus decrease
> the amount of memory other processes can use.
>
> Remember if a file in memory is paged out then that action will not
> increase paging space use --- so you can have paging out occurring
> without an increase in the use of paging space.
>
> To answer your question "why tends to set pi and po to zero value ?"
> pi means page in and po means page out -- if fr (pages freed during
> an LRUD search) is 0 then po will be 0 -- pi will be 0 if a file is
> not or 4K pages of a file are not read into memory --- if no pages
> are being freed (what you are seeing ) then no pages can be read in so
> pi must be 0
>
> hope that this adds clarity

Thank. Yes, Our system may not turnning well.
I am not a System Admin. How to check physical memory and Virtual
memory ?
How to check pin memory ?
How to set pi/po by vmo ?
Thank for provide about what is "file cache trashing".

About OS
> prtconf
Processor Type: PowerPC_POWER5
Number Of Processors: 3
Processor Clock Speed: 1656 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
Memory Size: 7968 MB
Good Memory Size: 7968 MB
Firmware Version: IBM,SF230_120
Console Login: enable
Auto Restart: true
Full Core: false

Application
=========
Progress database
19 GB Production database ( Number of records cut-off at 11:17)
Read create Update
----------- ----------- -----------
739576225 92170 336659

11 GB History Database
2.5 GB transaction Database ( Number of records cut-off at 11:17)
Read create Update
----------- ----------- -----------
834936054 1843 17883

( Number of records read 1,500,000,000 to 1,800,000,000 daily ) scan
for updated records in order to update Oracle Database.

700+ users using Production database and History Database
10+ remote users using transaction Database and Production database by
Java connection


>iostat 5 5

System configuration: lcpu=3 disk=5

tty: tin tout avg-cpu: % user % sys % idle %
iowait
48.3 12118.7 47.3 7.4
35.4 9.8

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 9.4 92.3 21.0 4981121 135425925
hdisk1 9.6 89.8 20.6 1285025 135425921
hdisk3 0.6 314.4 8.2 253228228 225212396
hdisk2 22.0 1457.9 180.6 1660079884 558609944
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle %
iowait
87.6 18976.8 64.3 10.8
10.5 14.3

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 37.7 455.9 63.7 20 2264
hdisk1 35.9 454.3 62.9 8 2268
hdisk3 0.0 0.0 0.0 0 0
hdisk2 80.2 2078.2 343.5 8716 1696
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle %
iowait
127.0 25815.2 73.9 4.2
6.8 15.1

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 3.6 24.8 6.4 16 108
hdisk1 2.2 21.6 5.6 0 108
hdisk3 0.0 0.0 0.0 0 0
hdisk2 88.8 2956.0 501.4 13304 1476
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle %
iowait
178.4 46733.2 78.5 11.9
2.9 6.7

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 29.8 284.8 62.4 8 1416
hdisk1 30.2 284.8 62.6 8 1416
hdisk3 0.0 0.0 0.0 0 0
hdisk2 89.2 2448.0 427.4 10748 1492
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle %
iowait
123.2 31273.2 75.9 9.2
5.9 9.0

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 26.2 245.0 61.2 28 1197
hdisk1 27.8 240.2 60.0 4 1197
hdisk3 0.0 0.0 0.0 0 0
hdisk2 96.4 4471.2 533.4 20256 2100
cd0 0.0 0.0 0.0 0 0

...
> Thank. Yes, Our system may not turnning well.
> I am not a System Admin.
But the task is has to be done by a System Administrator and DB
Administrator


> How to check physical memory and Virtual memory ?
> How to check pin memory ?
> How to set pi/po by vmo ?
Determine and name the problem of your user
Tune and configure the applications according the provided guides.

> About OS> prtconf
....
I assume from the output that you are running AIX 5.3

$ man oslevel

So in case its AIX 5.3
and your not using raw devices
and your DB files are on their OWN filesystem with is JFS2

mount the DB fs with the CIO option and you might get some better
performace


A good starting point is
Oracle DB & RAC 10gR2 on IBM AIX 5L: Tips and Considerations
http://www-03.ibm.com/support/techdo...Index/WP100556

Its for oracle but give a clue for other databases as well.

hth
Hajo