JVM Lies: The OutOfMemory Myth

Mon, 14 Jan 2008 13:39:00 GMT

There are times when an OutOfMemoryError means exactly what it says. Try adding new objects to an ArrayList in a while(true) loop and you'll see what I mean.

However, there are times when it doesn't.

Recently, when I saw a vital supporting application of our system throwing an OutOfMemoryError in production, my first instinct was to increase the -Xmx switch from the existing 2GB. Let's whack on an extra gig, why not. That will give us at least 6 months until we start worrying about the logical 4GB limit of a 32-bit process's addressable space.

I expect I am not alone in having the knee-jerk reaction that any application's memory problems can be solved by cranking up the heap. I blame James Gosling, or whoever decided that the JRE 1.1 JVM's heap should default to 64M. Even at the start of my Java programming career in 1998 I remember quickly running out of heap space, and needed to look up what this non-standard -Xmx switch did. Increasing this value made these problems just disappear.

However, instead of doing the obvious and increasing the -Xmx, I added extra GC debugging output and attempted to replicate the problem. We have plenty of spare memory on our hardware, so any time spent on such an obvious issue is arguably a waste: there was important business functionality I could be delivering instead of messing around with JVM switches. However, being at times more stubborn than my own good, I insisted on understanding exactly what was going on. In particular:

Why was similar behaviour not occurring in the test environment?
I am blessed with comparable hardware, and data volumes, in a test environment as the production environment. A rare treat, I appreciate, but an invaluable one for situations such as this. Well it turns out the answer to this question was straightforward: it was. The flaw was with our monitoring of this environment. Abashed, I made a mental note to improve our application monitoring and moved on to question 2.
Why were we running out of memory?
Data volumes increase in the system on a monthly basis, so the answer to this question may seem self-evident. Without correct monitoring and re-tuning, our JVMs are expected to run out of memory. This isn't necessarily an architectural flaw, it's simply about allocating the right amount of memory for the current data volumes. However, I had to be sure it was the heap that we were running out of.

Depending on the flavour of JVM, an OutOfMemoryError can indicate a shortage of memory in one of several areas. These broader concepts are common to generational GC algorithms across the major JVM vendors including Sun, IBM and BEA, although the specifics I refer to below relate to the Sun Hotspot GC model.

The first is the tenured generation. This is usually what I mean when I say "the heap". Memory is segmented into several generations, however it is when the tenured generation is full, and cannot be expanded any further, that the JVM considers itself OutOfMemory.
The second is the permanent generation. This does not resize during the life time of the application, regardless of how much free space may exist in the rest of the heap, but remains at whatever it was originally set to (default is 64K). Should this prove too small for the perm generation, then the JVM will throw an OOME even if there's plenty of heap left. Adding the -XX:+PrintHeapAtGC switch will tell you if this is the case.
The third possibility is your operating system is out of memory, e.g. you've asked for a 2GB heap on a box with 1GB RAM and 512MB swap space (not a typical server, admittedly, but serves as an example).

In my case I was primarily investigating which of the first two above scenarios was occurring (I knew we had enough spare memory on the box itself), so I was somewhat surprised to find out it was neither.

Another possibility is native components are hogging your 4GB ceiling. Native code competes with the JVM to use the 4GB of addressable space in your application. If these components are memory hungry, your app will be starved of addressable space, even if it hasn't actually used up all the heap you've given it yet. This may manifest itself during the workings of the Hotspot JIT compiler, which itself is a native component, as the Just In Time compiler uses some of your process's space to compile methods to native code at runtime. Should these memory requirements push the addressable space required in the process above 4GB, then you get an OOME thrown which the 1.4 JVM logs as:
Exception in thread "CompilerThread0" java.lang.OutOfMemoryError: requested Exception in thread "main" java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?

The process hadn't used all the space available to it when I saw this error -- the Java heap had plenty of room left unused. However for addressing purposes this space was considered consumed.

So, what to do about the above error? Increasing the heap allocation actually exacerbates this problem! It decreases the headroom the compiler, and other native components, have to play with.

So the solution to my problem was:

reduce the heap allocated to the JVM.
remove the memory leaks caused by native objects not being freed in a timely fashion.

Or just use a 64-bit JVM.

The Clash of the Paradigms

Tue, 08 Jan 2008 10:29:00 GMT

I recently came across a subtle anti-pattern which has caused me some pains. This occurred in a successful application used extensively throughout a large financial corporation. It is a library, which is implemented in C++, but usable from several different programming languages, including Java and C#.

The Java port is a thin layer of Java objects which are used as proxies making JNI calls to underlying C++ objects.

At first glance this is an understandable design: the vast majority of the API's source code is written in one language, and the thin layers used to port the API are presumably easy to maintain.

But it all goes to pot when you consider garbage collection. How does a native object know that it is no longer needed? Well, when there's no more references to the proxy, of course! But there's the rub: the only way a proxy can implicitly determine it is not needed is through a finalize method*. This method is clearly documented to offer no guarantees of when it is invoked; this is likely to be at the mercy of the garbage collector, which in itself implies that at best several potentially-infrequent garbage collection cycles are required before such methods are called.

Consider the case when you have many thin Java objects which each create enormous native data structures: the JVM may not invoke a GC cycle on the modest memory usage of the Java objects, and in the meantime zombie native components may have used up all the swap space available.

The only reasonable alternative is to expose a delete() method in the Java API which is to be explicitly called when the application knows that the native object is no longer required, which in turn frees the native memory used. However, for Java programmers used to their memory-managed sandboxes this goes strongly against the grain. The 3rd party native objects, so innocently wrapped up in Java proxies, quickly spread like a virus throughout large tracts of code. The package in question also, for performance reasons, exposes objects at a relatively fine-grained level, further facilitating its own objects' insinuation throughout the application. What's left is a soup of objects, of which some are POJOs, and some hide native objects and must have their lifecycles carefully tracked to be cleaned up appropriately. Get it wrong and free an object prematurely, and you core-dump the JVM and bring down your application spectacularly, irrevocably and catastrophically. Eliminating memory leaks in your Java code has become a major task.

Admittedly, the impact of the library can be reduced by carefully considered abstractions in client code. But I can't help but blame the designers of the API who enforce their paradigm of programming on all clients of their system. What may be an excellent object-oriented API in C++ does not make a good API in Java. The lesson here is for architects not only to consider the functionality and aesthetics of their API, but also the programming paradigm which suits their clients.

*Tony Printezis has published an alternative for finalizers by using weak references. As the author says, it involves implementing an algorithm, running on its own thread, duplicating functionality provided by the garbage collector, in application code.

Coding the Architecture - gc tag

JVM Lies: The OutOfMemory Myth

The Clash of the Paradigms