Unix Technical Forum

Sudden & massive SHMADD activity; engine went BANG!

This is a discussion on Sudden & massive SHMADD activity; engine went BANG! within the Informix forums, part of the Database Server Software category; --> IDS9.30HC5; HP-UX11i; largely OLTP; 4ge applications on server; about 150 users. Just now our engine suddenly allocated 48 SHMADD ...


Go Back   Unix Technical Forum > Database Server Software > Informix

Register FAQ Members List Calendar Search Today's Posts Mark Forums Read
  #1 (permalink)  
Old 04-20-2008, 07:50 AM
malc_p@btinternet.com
 
Posts: n/a
Default Sudden & massive SHMADD activity; engine went BANG!

IDS9.30HC5; HP-UX11i; largely OLTP; 4ge applications on server; about
150 users.
Just now our engine suddenly allocated 48 SHMADD segments in under 5
minutes and af'd itself into oblivion.
We normally blithely run with just the initial SHMVIRTSIZE virtual
segment and haven't added a SHMADD segment for weeks and weeks; the
system is pretty well-behaved as a rule and hasn't af'd for months.
The SQL showing in the af file is pretty innoccuous ( I just brought
the engine back online and ran the sql; it returns 85 rows so I think I
can discount that SQL as the culprit).
Trouble is, the system is once again quiet and happy and no SHMADDs and
manglement are happily jumping up and down on my squashy DBA bits to
try to extract the reason for this turn of events from me and our
consequent Service Level Interruption (yea verily and let us all pray
from the book of Key Performance Indicators).
I know what the Virtual Segment is for and how it may be required to
expand but I can't come up with an explanation for the sudden SHMADD
activity. All I can do is keep an eye on it and see if it happens
again.
Can anyone share any clues as to what may have caused it (is there an
IDS9.30HC5 bug I need to know about?)
and so on.
Thanks
Malc

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #2 (permalink)  
Old 04-20-2008, 07:51 AM
John Carlson
 
Posts: n/a
Default Re: Sudden & massive SHMADD activity; engine went BANG!

On 4 Mar 2005 04:38:06 -0800, malc_p@btinternet.com wrote:

>IDS9.30HC5; HP-UX11i; largely OLTP; 4ge applications on server; about
>150 users.
>Just now our engine suddenly allocated 48 SHMADD segments in under 5
>minutes and af'd itself into oblivion.
>We normally blithely run with just the initial SHMVIRTSIZE virtual
>segment and haven't added a SHMADD segment for weeks and weeks; the
>system is pretty well-behaved as a rule and hasn't af'd for months.
>The SQL showing in the af file is pretty innoccuous ( I just brought
>the engine back online and ran the sql; it returns 85 rows so I think I
>can discount that SQL as the culprit).
>Trouble is, the system is once again quiet and happy and no SHMADDs and
>manglement are happily jumping up and down on my squashy DBA bits to
>try to extract the reason for this turn of events from me and our
>consequent Service Level Interruption (yea verily and let us all pray
>from the book of Key Performance Indicators).
>I know what the Virtual Segment is for and how it may be required to
>expand but I can't come up with an explanation for the sudden SHMADD
>activity. All I can do is keep an eye on it and see if it happens
>again.
>Can anyone share any clues as to what may have caused it (is there an
>IDS9.30HC5 bug I need to know about?)
>and so on.
>Thanks
>Malc


No one creating large indexes?

JWC
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #3 (permalink)  
Old 04-20-2008, 07:51 AM
rkusenet
 
Posts: n/a
Default Re: Sudden & massive SHMADD activity; engine went BANG!

Did the application software change recently. We had exactly the
same problem on Mar 10 2004 when it added 8 SHMADD in 15 min
and engine died. It was finally traced to sloppy coding in JDBC
where they kept recycling the result set, leading to memory
leaks in Informix session.

To convince everyone that the fault was in our coding and not
in Informix (ver 9.21), I wrote a pretty sophisticated script in perl
using onstat etc which maps out the open cursors and memories
of every session. I put that in an alert script which was mailing
it to me every 15 minutes. Using that it was pretty easy to identify
rogue sessions causing the leak and we eventually identified the
code in question. If u want the script I can mail it to you.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #4 (permalink)  
Old 04-20-2008, 07:51 AM
malc_p@btinternet.com
 
Posts: n/a
Default Re: Sudden & massive SHMADD activity; engine went BANG!

Well we have got some relatively new apps connecting via JDBC, and we
have run into an issue before whereby the connection pools aren't freed
unless they explicitly close the connections....
Sounds plausible. I'd appreciate a copy of the script if you're ok with
that, thanks RK.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On
Forum Jump


All times are GMT. The time now is 04:41 PM.


Powered by vBulletin® Version 3.6.5
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
Search Engine Optimization by vBSEO 3.2.0
www.UnixAdminTalk.com