Performance tuning Java systems

Test, tune and document early

One of my current projects involves me performance testing a third party Java system. In essence, it's a distributed/n-tier Java EE web application and we're hitting it with a high simulated load thanks to Apache JMeter. One of the things we found is that performance degraded quite dramatically as load increased, to the point where some requests could take anywhere up to a minute or two to be serviced. By plotting the raw results and calculating (for example) the 95% percentile, we were able to see that only a very small number of the requests suffered in this way.

After some investigation, the suspected cause of these high response times was "stop the world" garbage collection and we eventually proved this by monitoring the JVM during a test run. Thankfully, the new Java VMs include a ton of tuning options and I've found the following references particularly useful in the past.

So then, some advice :

  1. Make sure that you define your non-functional requirements as early in the project as you can, specifying performance in the context of scalability. Once you've done this, you can focus on figuring out how to meet them.
  2. Make sure that you test your key non-functional requirements as soon as possible, preferably with something like a reference architecture. Tuning your JVMs early lets you gauge how much more performance you might get with further tuning in the future.
  3. Make sure that you document your JVM settings if you've had to tune them to meet your performance targets. The system I'm testing is a third party product and there's no way that the customer should even be running, what's basically a "real-time" system, on untuned JVMs.
  4. If you need to start investigating performance problems, this article over at InfoQ (The Box: A Shortcut to finding Performance Bottlenecks) has some good tips for starting out on this route.
  5. Ensure that your Java system is easy to monitor, using something like JMX, because this will really help you prove and diagnose any issues you might come across. More on that later in the week.

If you get the opportunity to do some performance testing then I highly recommend it. Assuring the non-functional requirements is a key part of an architect's role so it's great experience.



Re: Performance tuning Java systems

I'd suggest having a look at BEAs jrockit. The deterministic garbage collection works very well and within a 10 minute setup can give the same response as you'd get after spending days tuning a normal JVM. It is particularly good for getting a smooth, non-jitter performance. The throughput is reduced slightly but the spread of latency becomes much, much narrower.

Re: Performance tuning Java systems

Indeed, and given your recent experience with JRockit, I'm surprised to not come across it more in the city.

So I understand the sort of statistical guarantee that JRockit provides, but what I'd really like to see is a real-world comparison of JRockit vs a Sun JVM that's been configured using the -XX:MaxGCPauseMillis flag. I'm sure JRockit will perform better, but I wonder how close Sun's JVM will get.

Re: Performance tuning Java systems

Similar experience with regards to a 3rd party jar performing un-documented garbage collection. In this case it was identified by FindBugs, not only will findbugs report bugs in code it can also identify potential performance issues. http://findbugs.sourceforge.net/

Sun Real-time JVM

If you want deterministic garbage collection why not take a look at Sun's Java Real-Time System 2.0 (Java RTS).

Sun Real-time JVM

Am I right in thinking that you need to run it on a real-time OS?

Sun Real-time JVM

Only available for Solaris 10 SPARC and Solaris 10 X86. I must say I haven't tried it. Unlike the standard JVM from Sun this one is not free of cost. They have a free 90-day evaluation license.

Re: Performance tuning Java systems

You make a very good point here. I have seen several times, that people do not tune the garbage collector well. People sometimes take initial GC configuration for granted. Real GC Tuning however cannot be done without the required look requiring the application to be readily developed. However using performance management tools also provide insight into object allocation. Sometimes the real problem is that way to much objects are allocated and destroyed. Then even the best garbage collector strategy cannot help you. You just gave me the motivation to post more on this issue on my own blog at blog.dynatrace.com.

Add a comment Send a TrackBack