One of my current projects involves me performance testing a third party Java system. In essence, it's a distributed/n-tier Java EE web application and we're hitting it with a high simulated load thanks to Apache JMeter. One of the things we found is that performance degraded quite dramatically as load increased, to the point where some requests could take anywhere up to a minute or two to be serviced. By plotting the raw results and calculating (for example) the 95% percentile, we were able to see that only a very small number of the requests suffered in this way.
After some investigation, the suspected cause of these high response times was "stop the world" garbage collection and we eventually proved this by monitoring the JVM during a test run. Thankfully, the new Java VMs include a ton of tuning options and I've found the following references particularly useful in the past.
So then, some advice :
If you get the opportunity to do some performance testing then I highly recommend it. Assuring the non-functional requirements is a key part of an architect's role so it's great experience.
Simon is an independent software development consultant specialising in software architecture and the author of Software Architecture for Developers. Simon lives in Jersey and regularly speaks to audiences around the world. You can find Simon on Twitter at @simonbrown and Google+.