If you are getting an OOM around a native call including Runtime.exec(), try adding swap space.

My application uses a 3rd party library that uses Runtime.exec() to gather info about the OS. Under a load, this call would generate an OOM, but wrapped in an IOException. (I had to dig to get the real error)

I found this thread in the java programming forums
The following example shows the creation of a 2Gb auxiliary swap file.

 
# dd if=/dev/zero of=/auxswap bs=1M count=2048
# mkswap /auxswap
# swapon /auxswap

Note that this will affect 32-bit or 64-bit JVMs, it doesn't matter. I recently upgraded to a 64-bit JVM & OS and still received the OOM. Adding the swap solved the problem for both 32-bit & 64-bit.