Coding the Architecture - testing tag

Using new technology will make it faster

Fri, 15 May 2009 10:57:11 GMT

I heard an interesting throwaway comment recently that I thought I'd share. To paraphrase...

The system is being rewritten with new technology, so it won't be slower than what we have now.

While it's true that new and updated technology *can* improve performance, it's also very easy to misuse that technology and come up with something that performs perceivably slower. I've seen this a few times, predominantly where simple websites have been rewritten using new technologies. One of the biggest contributing factors is that new technologies often provider "better" and more complex ways to solve the same problem. For example, it's easy to see how a simple two-tier web application could be rewritten with a web-MVC framework and a distributed middle-tier added to make the architecture more fashionably SOA-like. Here, the added complexity could result in net performance gains or losses, depending on how the technology is applied through the design and architecture.

There's nothing wrong with refreshing a software (or hardware) system with new technology, but it's naive to assume that performance won't degrade. New technology generally brings added complexity, and this complexity introduces more things that can go wrong. Any software/hardware architecture should undergo *some* non-functional testing before it goes live, and this still holds true for systems that are being rewritten as part of a technology refresh strategy. Don't assume anything ... test it.

London User Group - October 2008

Tue, 14 Oct 2008 13:55:18 GMT

Here are the details of the October London User Group.

Title : Testing as an Architectual Concern
Summary : In this session, Kevin Seal will present some recent experiences of testing a JavaEE project and how coverage was a double-edged sword, unit testing a blessing and a curse, and a pragmatic mix of test approaches was eventually invaluable in refining the architecture and development process.
Date : Wednesday, 29th October 2008
Time : 18:30-20:00
Location : Skills Matter, 1 Sekforde Street
Format : Presentation and discussion, with further discussion in a local pub (The Crown).
Cost : Free, but registration is required.

Many thanks to Skills Matter for sponsoring the user group.

Testing Times

Fri, 26 Sep 2008 13:09:00 GMT

Having recently taken another look at my project's unit testing strategy I thought I'd share some of my thoughts on how I see unit testing as just another force on the system's design. As such it needs to be weighed in the balance with the other forces.

Unfortunately it's a force that tends to get more influence than it perhaps deserves, not least because it's a force typically backed by the development team.

Who should own the non-functional tests?

Mon, 07 Jul 2008 10:03:00 GMT

This is a follow-up to a post entitled Performance tuning Java systems that I made last year, which talked about some performance testing that I was doing for a customer. Last week, I went back to the same customer to make some small tweaks to the test scripts, and this very simple act highlighted something I thought was very interesting.

The key thing about the system being tested is that it's a customized version of an existing vendor product (a high volume, low latency trading platform), with the vendor being responsible for making all of the changes. At some point last year, my customer wanted to run some tests to ensure that the platform could grow with their business, and that's where I got involved. After a round of testing, environment reconfiguration and patches, the system more or less delivered on what it promised.

Fast-forward to the current day and I was bought back in to make some changes to the test scripts, because the vendor had made some changes to their web application. Since our performance tests were JMeter based, those changes had a knock-on effect on the tests. The thing I found interesting is that the performance test pack hadn't been handed over to the vendor. Regardless of the reasons, the benefits of doing this are fairly clear.

Ownership : although it's good that my customer wanted to proactively test some of the non-functional aspects of the software, I can't help feeling that the software vendor should have ownership of the test pack. My understanding is that they do little in the way of non-functional testing themselves, so this is an ideal opportunity to rectify this.
Continuous execution : if the vendor has ownership, they can include the test pack as a part of their integration/build/release process for continuous execution and early warning of performance degrading changes.
Accuracy : I like to think that we did a good job at putting together the performance testing scripts, but we don't have the inside knowledge of the system that the vendor has. Vendor ownership would ensure that the test scripts are as accurate and representative as possible.
Up to date : the test scripts are inherently tied to the AJAX web application, which means that they need to be kept in sync when changes are made to the user experience. Again, the vendor is best positioned to do this, and taking a more proactive approach would ensure that user experience changes don't degrade the non-functional characteristics of the software.

The overriding question here is why doesn't the vendor undertake their own performance testing? They *should* take responsibility for the non-functional characteristics of their software, particularly when they are making claims of how fast and scalable it is. But perhaps more interesting is that the vendor of another product in use in the same space also doesn't do performance testing. Shouldn't we be asking more from commercial product vendors?

What is Significant?

Sat, 15 Dec 2007 21:15:27 GMT

Recently I wrote a quick blog about taking metrics for optimisation. I suggested they should only be included if the improvement was significant, but how do you define significant?

You may think that 'significant' is just a matter of opinion but it actually has a very specific meaning in statistics - Wikipedia 's Description. You can have a read through the maths but it basically comes down to "a result is called statistically significant if it is unlikely to have occurred by chance".

This is really important and something that performance testers and optimisers often forget. For example...

Imagine that you perform some kind of performance test on your system or code. This could be anything e.g. latency response timings, throughput per time unit etc but we'll assume for this that it's units processed in 10 minutes. The figure you get is 20. You spend a day modifying a piece of code you think will affect the performance, retest and get 22. A 10% improvement - pretty good.

You hand the new code over to a colleague who also does a test. She says that it's worse by 5%. Slander! You take it to your boss who says there is no difference...

What we've done is perform three tests on the old and new system. Lets list them and perform seven others as well:

Old: 20 20 22 19 19 21 19 19 19 22
New: 22	19 22 20 20 19 19 19 21 19

Now it's obvious what happened (although it probably was before). Your test does not produce constant figures even without changes. Both have a range of 3 (19-22), an average of 20 and a variance of 1.55

If you had performed ten runs on the original code first you would have realised that a single result of 22 for the new code is not significant as it's within the range of the previous figures.

Performing the test multiple times on the new code would increase your confidence that it's a significant change. You can test this statistically (but the maths is beyond the scope of this blog entry).

Just to leave you with a challenge, the project I'm currently working has a task that we wish to optimise but it takes ten hours to run even on a grid of several hundreds machines. How do we run realistic, pre-production tests that we know are statistically significant?

Performance tuning Java systems

Mon, 05 Nov 2007 12:23:34 GMT

One of my current projects involves me performance testing a third party Java system. In essence, it's a distributed/n-tier Java EE web application and we're hitting it with a high simulated load thanks to Apache JMeter. One of the things we found is that performance degraded quite dramatically as load increased, to the point where some requests could take anywhere up to a minute or two to be serviced. By plotting the raw results and calculating (for example) the 95% percentile, we were able to see that only a very small number of the requests suffered in this way.

After some investigation, the suspected cause of these high response times was "stop the world" garbage collection and we eventually proved this by monitoring the JVM during a test run. Thankfully, the new Java VMs include a ton of tuning options and I've found the following references particularly useful in the past.

So then, some advice :

Make sure that you define your non-functional requirements as early in the project as you can, specifying performance in the context of scalability. Once you've done this, you can focus on figuring out how to meet them.
Make sure that you test your key non-functional requirements as soon as possible, preferably with something like a reference architecture. Tuning your JVMs early lets you gauge how much more performance you might get with further tuning in the future.
Make sure that you document your JVM settings if you've had to tune them to meet your performance targets. The system I'm testing is a third party product and there's no way that the customer should even be running, what's basically a "real-time" system, on untuned JVMs.
If you need to start investigating performance problems, this article over at InfoQ (The Box: A Shortcut to finding Performance Bottlenecks) has some good tips for starting out on this route.
Ensure that your Java system is easy to monitor, using something like JMX, because this will really help you prove and diagnose any issues you might come across. More on that later in the week.

If you get the opportunity to do some performance testing then I highly recommend it. Assuring the non-functional requirements is a key part of an architect's role so it's great experience.