There are times when an OutOfMemoryError means exactly what it says. Try adding new objects to an ArrayList in a while(true) loop and you'll see what I mean.

However, there are times when it doesn't.

Recently, when I saw a vital supporting application of our system throwing an OutOfMemoryError in production, my first instinct was to increase the -Xmx switch from the existing 2GB. Let's whack on an extra gig, why not. That will give us at least 6 months until we start worrying about the logical 4GB limit of a 32-bit process's addressable space.

I expect I am not alone in having the knee-jerk reaction that any application's memory problems can be solved by cranking up the heap. I blame James Gosling, or whoever decided that the JRE 1.1 JVM's heap should default to 64M. Even at the start of my Java programming career in 1998 I remember quickly running out of heap space, and needed to look up what this non-standard -Xmx switch did. Increasing this value made these problems just disappear.

However, instead of doing the obvious and increasing the -Xmx, I added extra GC debugging output and attempted to replicate the problem. We have plenty of spare memory on our hardware, so any time spent on such an obvious issue is arguably a waste: there was important business functionality I could be delivering instead of messing around with JVM switches. However, being at times more stubborn than my own good, I insisted on understanding exactly what was going on. In particular:

1. Why was similar behaviour not occurring in the test environment?
   I am blessed with comparable hardware, and data volumes, in a test environment as the production environment. A rare treat, I appreciate, but an invaluable one for situations such as this. Well it turns out the answer to this question was straightforward: it was. The flaw was with our monitoring of this environment. Abashed, I made a mental note to improve our application monitoring and moved on to question 2.
2. Why were we running out of memory?
   Data volumes increase in the system on a monthly basis, so the answer to this question may seem self-evident. Without correct monitoring and re-tuning, our JVMs are expected to run out of memory. This isn't necessarily an architectural flaw, it's simply about allocating the right amount of memory for the current data volumes. However, I had to be sure it was the heap that we were running out of.

Depending on the flavour of JVM, an OutOfMemoryError can indicate a shortage of memory in one of several areas. These broader concepts are common to generational GC algorithms across the major JVM vendors including Sun, IBM and BEA, although the specifics I refer to below relate to the Sun Hotspot GC model.

• The first is the tenured generation. This is usually what I mean when I say "the heap". Memory is segmented into several generations, however it is when the tenured generation is full, and cannot be expanded any further, that the JVM considers itself OutOfMemory.
• The second is the permanent generation. This does not resize during the life time of the application, regardless of how much free space may exist in the rest of the heap, but remains at whatever it was originally set to (default is 64K). Should this prove too small for the perm generation, then the JVM will throw an OOME even if there's plenty of heap left. Adding the -XX:+PrintHeapAtGC switch will tell you if this is the case.
• The third possibility is your operating system is out of memory, e.g. you've asked for a 2GB heap on a box with 1GB RAM and 512MB swap space (not a typical server, admittedly, but serves as an example).

• Another possibility is native components are hogging your 4GB ceiling. Native code competes with the JVM to use the 4GB of addressable space in your application. If these components are memory hungry, your app will be starved of addressable space, even if it hasn't actually used up all the heap you've given it yet. This may manifest itself during the workings of the Hotspot JIT compiler, which itself is a native component, as the Just In Time compiler uses some of your process's space to compile methods to native code at runtime. Should these memory requirements push the addressable space required in the process above 4GB, then you get an OOME thrown which the 1.4 JVM logs as:
  Exception in thread "CompilerThread0" java.lang.OutOfMemoryError: requested
Exception in thread "main" java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?


The process hadn't used all the space available to it when I saw this error -- the Java heap had plenty of room left unused. However for addressing purposes this space was considered consumed.

So, what to do about the above error? Increasing the heap allocation actually exacerbates this problem! It decreases the headroom the compiler, and other native components, have to play with.

So the solution to my problem was:

1. reduce the heap allocated to the JVM.
2. remove the memory leaks caused by native objects not being freed in a timely fashion.

Or just use a 64-bit JVM.

Responses[30]

Posted by Kenneth Roper on 14 January 2008 13:39:00 GMT #