Coding the Architecture - performance tag

Jitter

Tue, 06 Sep 2011 08:02:41 GMT

On CTA we often talk about non-functional requirements and how this can drive the architecture of a system. Most of these cover issues of desired response time and capacity (latency, throughput, storage etc) but I believe that Jitter is a metric which is either forgotten or unknown to some software engineers – even though it's essential for hardware engineers.

The most basic definition would be “variation in response”. In other words, the response, transmission or latency in a system will not be a constant. In many systems this variation goes unnoticed as it is small compared to the response itself. However I believe that as the systems we deal with become more performant and demanding, software engineers will need to understand, measure and tune this.

A quick web-search of jitter shows the term being used to cover performance degradations due to activities such as garbage collection or unexpected user actions. This seems to greatly annoy telecommunications and hardware engineers who would argue that these are predictable, system and user events which could just be turned off – although ignoring your users and asking for a machine with infinite memory may get you fired. They would argue that true Jitter is an unpredictable variation in response whose occurrence frequency follows a normal distribution. Like most effects that are normally distributed it is caused by cumulative random events, most of which are due to the switching actions. Personally, I think you should measure whatever makes sense in your system.

Jitter means that hard limits for latency can be statistically likely but not guaranteed. Specifications such as:

“The system should respond to action X within 200ms”.

Should be challenged and replaced with statements like:

“The systems should response to action X within 200ms for 95% of requests”

Of course we are making the implicit assumption that this is normally distributed and the system WILL, eventually, respond. You might want to explicitly state that all actions will be executed but stating a hard limit for all responses means that if you measure for long enough, you'll break your specification.

Jitter can be quite obvious on messaging based systems that contain many hops. This will also tend to have a nice, Gaussian distribution due to the cumulative random delays.

The best way to measure Jitter is to set up a standard test where you can sequentially fire a large number of requests into your system and measure the response time. However, rather than simply add these up and find an average we need to count how many responses are in a set of timing windows e.g. how many responses fall between 5ms-10ms, 10ms-15ms, 15ms-20ms etc. If we plot the number in each section we should see our expected Normal distribution.

Reducing Jitter is subtly different from reducing latency itself. You are trying to reduce the variation in the response times or 'squeeze' the response profile.

Often when you reduce latency you will reduce the Jitter proportionally however this is not always the case. For example, network hops are a common cause of latency but each hop will increase the Jitter as well – you are accumulating more random delays with each hop. If you change the physical architecture of your system to have more but quicker hops you will almost certainly increase the jitter (remember this is the variation in response and not the actual response) even if the average latency is lower. It IS important to measure it and monitor as the system evolves.

Simon is covering a number of related issues in his Skills Matter tutorial Load Testing for Developers.

Stating the obvious

Tue, 16 Jun 2009 14:08:33 GMT

If you've watched A developer's guide to load testing, you will have seen the graph below already. It shows the results from load testing a website written in .NET with the Apache JMeter load testing tool. Along the bottom of the graph are the actions that make up the load test script, while the vertical axis is the response time in seconds. Each line represents the results from a different test run with a different number of concurrent users.

The interesting thing about this graph is that the average response time for any page request is just a few seconds, with the exception of the spike around a third of the way in. This spike is caused by the .NET web tier having to aggregate information from several calls to a backend system, where some resource locking bottlenecks the incoming requests. From analysing the load testing environment during the test run, we were able to ascertain where the hotspot was because (a) the CPU load for the web-tier dropped off when this call was taking place and (b) we could see the requests being queued up on the backend system. From a technical perspective then, load testing is a fantastic way to find and highlight bottlenecks in the overall architecture, providing an opportunity to present some real evidence about the performance degradation and where the hotspots are.

Furthermore, in this example there was a general feeling that this particular action was slow, but the evidence was only empirical. People had experienced this slowness but had never actually measured it before. Kevin made a comment during the last user group where he mentioned that the architect sits between the technical and business spaces, spanning the two and bringing them together. Raw results from a load test are one thing, but presenting the data in a way that makes sense to non-technical people can tell a completely different story. Sometimes your load testing results state the obvious, but sometimes they don't.

Using new technology will make it faster

Fri, 15 May 2009 10:57:11 GMT

I heard an interesting throwaway comment recently that I thought I'd share. To paraphrase...

The system is being rewritten with new technology, so it won't be slower than what we have now.

While it's true that new and updated technology *can* improve performance, it's also very easy to misuse that technology and come up with something that performs perceivably slower. I've seen this a few times, predominantly where simple websites have been rewritten using new technologies. One of the biggest contributing factors is that new technologies often provider "better" and more complex ways to solve the same problem. For example, it's easy to see how a simple two-tier web application could be rewritten with a web-MVC framework and a distributed middle-tier added to make the architecture more fashionably SOA-like. Here, the added complexity could result in net performance gains or losses, depending on how the technology is applied through the design and architecture.

There's nothing wrong with refreshing a software (or hardware) system with new technology, but it's naive to assume that performance won't degrade. New technology generally brings added complexity, and this complexity introduces more things that can go wrong. Any software/hardware architecture should undergo *some* non-functional testing before it goes live, and this still holds true for systems that are being rewritten as part of a technology refresh strategy. Don't assume anything ... test it.

Discussion from the May 2008 user group

Sat, 30 Aug 2008 06:57:27 GMT

Although this happened a few months ago, we recorded the discussion that followed our London user group where Moudud Ahmed talked about building a high volume, low latency system in Java. Here are some of the topics that we talked about.

Performance and scalability.
Measuring latency.
Domain-driven design.
BEA's real-time Java platform.
Writing applications with large heap sizes that can survive not doing garbage collection during the working day.

The audio levels are very low on the recording, but we think that the interesting discussion was worth rescuing and publishing. You can grab the discussion from here (mp3 format) ... just don't listen on a tube train! If you missed the user group, a streaming video of the presentation is also available.

Who should own the non-functional tests?

Mon, 07 Jul 2008 10:03:00 GMT

This is a follow-up to a post entitled Performance tuning Java systems that I made last year, which talked about some performance testing that I was doing for a customer. Last week, I went back to the same customer to make some small tweaks to the test scripts, and this very simple act highlighted something I thought was very interesting.

The key thing about the system being tested is that it's a customized version of an existing vendor product (a high volume, low latency trading platform), with the vendor being responsible for making all of the changes. At some point last year, my customer wanted to run some tests to ensure that the platform could grow with their business, and that's where I got involved. After a round of testing, environment reconfiguration and patches, the system more or less delivered on what it promised.

Fast-forward to the current day and I was bought back in to make some changes to the test scripts, because the vendor had made some changes to their web application. Since our performance tests were JMeter based, those changes had a knock-on effect on the tests. The thing I found interesting is that the performance test pack hadn't been handed over to the vendor. Regardless of the reasons, the benefits of doing this are fairly clear.

Ownership : although it's good that my customer wanted to proactively test some of the non-functional aspects of the software, I can't help feeling that the software vendor should have ownership of the test pack. My understanding is that they do little in the way of non-functional testing themselves, so this is an ideal opportunity to rectify this.
Continuous execution : if the vendor has ownership, they can include the test pack as a part of their integration/build/release process for continuous execution and early warning of performance degrading changes.
Accuracy : I like to think that we did a good job at putting together the performance testing scripts, but we don't have the inside knowledge of the system that the vendor has. Vendor ownership would ensure that the test scripts are as accurate and representative as possible.
Up to date : the test scripts are inherently tied to the AJAX web application, which means that they need to be kept in sync when changes are made to the user experience. Again, the vendor is best positioned to do this, and taking a more proactive approach would ensure that user experience changes don't degrade the non-functional characteristics of the software.

The overriding question here is why doesn't the vendor undertake their own performance testing? They *should* take responsibility for the non-functional characteristics of their software, particularly when they are making claims of how fast and scalable it is. But perhaps more interesting is that the vendor of another product in use in the same space also doesn't do performance testing. Shouldn't we be asking more from commercial product vendors?

NFRs for system replacements

Wed, 28 May 2008 10:35:00 GMT

As software architects, we tend to write about non-functional requirements a lot; particularly about how they should be defined and challenged because of the influence they have. One of the reasons for talking about NFRs a lot is that, in the majority, project teams don't give them the attention that they should. And here's a great example that's slightly different from the story where none of the NFRs were defined.

As is the case in most organisations, technology has a limited lifespan. Perhaps the business evolves, the technology becomes harder to maintain, etc. Whatever the reason, many organisations regularly look at replacing their existing systems with solutions that are shinier and more buzzword compliant.

In one of these cases, an organisation put together a 125 page specification for the replacement system detailing exactly what the system should do along with some details of the non-functional requirements. However, out of those 125 pages, only 1 of them featured the NFRs, and they can be summarised as follows.

Performance : the system must be fast.
Scalability : the system must be able to support 100 messages per second.
Availability : the system must be available 24x7.
Security : only certain individuals are permitted to access the system.

Credit where credit is due ... at least some thought was put into the non-functional requirements and conversations to explore those requirements aren't starting from scratch. But on the other hand, those requirements are meaningless!

I want to end this post on a positive note, so let me contrast this story with another replacement project that I've been involved with recently. In this instance, the team *had* thought about the non-functional requirements and had detailed definitions for what they should be. This is excellent, particularly considering that the non-functional requirements surrounding the current implementation are vague in nature. System replacements are an opportunity to revisit what's important, and that includes the non-functional aspects too.

Scalability Principles

Wed, 21 May 2008 13:12:25 GMT

At the simplest level, scalability is about doing more of something. This could be responding to more user requests, executing more work or handling more data. While designing software has its complexities, making that software capable of doing lots of work presents its own set of problems. This article presents some principles and guidelines for building scalable software systems.

London User Group - May 2008

Fri, 25 Apr 2008 13:30:52 GMT

Here are the details of the May London User Group.

Title : Building a high volume, low latency system in Java
Summary : This session will provide some insight into the architecture, design and development of a greenfield high volume, low latency trading and pricing system. Topics include :
- The architecture and design itself.
- How the early development of an executable reference architecture was used to help prove the low latency requirements.
- The structure of the team and how the architecture was shared between a team of architects.
- How mainstream technologies such as Java 5, Spring, JMS, and Oracle were used to successfully deliver a solution where other technologies like C++ might be the immediate first choice for low latency applications.
Speaker : Moudud Ahmed, Detica
Date : Tuesday, 6th May 2008
Time : 18:30-20:00
Location : Skills Matter, 1 Sekforde Street
Format : Presentation followed by a breakout for discussion, with further discussion in a local pub (The Crown).
Cost : Free, but registration is required.

Many thanks to Skills Matter for sponsoring the user group.

Podcast #2 : QCon revisited

Tue, 25 Mar 2008 10:23:20 GMT

As promised the 2nd CTA podcast is a roundtable discussion between some of the CTA contributors - namely Simon Brown, Sam Dalton and Kevin Seal. In this podcast we discuss some of the themes emerging from the recent QCon conference held in London and our views on those themes.

The podcast can be downloaded here or you can subscribe. Full show notes follow below:

1. Introduction

QCon Conference London detail can be found here.

2. Monitoring and Management (0:27)

Roll your own vs. "off the shelf" monitoring
Asynchronous monitoring
JXInsight
The need for an System Architect and an overall understanding of WHAT to monitor and a holistic view
Including Monitoring NFRs upfront in SAD
Monitoring the monitoring vs. best effort
eBay and 1.5TB of monitoring and logfile information per day

3. Rehashing Refactoring (11:56)

Refactoring now mainstream, now emphasizing at the micro-level
Code-debt - how can you measure it?
Refactoring is not something to be "saved up"
Lots of rearchitecture projects sold as refactoring
Sometimes starting again is a better solution than large scale refactoring
Why do things end up in such a mess so often?
Need to get back to writing code for humans to understand
Worry about emphasis on low-level code optimizations rather than architecture
Is refactoring its own worst enemy being referred to as a separate discipline to regular software development?
We are our own worst enemies - we (especially the architects) need to stand up for the quality of our code - stop allowing ourselves to be convinced to make shortcuts

4. The Return of the Architect? (25:08)

Thinking about performance, monitoring etc upfront
Terracotta, Coherence, Gigaspaces
Kevin's views on Terracotta
How much performance could you squeeze from a system in 2 weeks with code changes alone?
Real-time VMs and performance improvements
The benefits of continuous performance testing
Premature optimization
Understanding a system's deficiencies can be as valuable as fixing them

Performance tuning Java systems

Mon, 05 Nov 2007 12:23:34 GMT

One of my current projects involves me performance testing a third party Java system. In essence, it's a distributed/n-tier Java EE web application and we're hitting it with a high simulated load thanks to Apache JMeter. One of the things we found is that performance degraded quite dramatically as load increased, to the point where some requests could take anywhere up to a minute or two to be serviced. By plotting the raw results and calculating (for example) the 95% percentile, we were able to see that only a very small number of the requests suffered in this way.

After some investigation, the suspected cause of these high response times was "stop the world" garbage collection and we eventually proved this by monitoring the JVM during a test run. Thankfully, the new Java VMs include a ton of tuning options and I've found the following references particularly useful in the past.

So then, some advice :

Make sure that you define your non-functional requirements as early in the project as you can, specifying performance in the context of scalability. Once you've done this, you can focus on figuring out how to meet them.
Make sure that you test your key non-functional requirements as soon as possible, preferably with something like a reference architecture. Tuning your JVMs early lets you gauge how much more performance you might get with further tuning in the future.
Make sure that you document your JVM settings if you've had to tune them to meet your performance targets. The system I'm testing is a third party product and there's no way that the customer should even be running, what's basically a "real-time" system, on untuned JVMs.
If you need to start investigating performance problems, this article over at InfoQ (The Box: A Shortcut to finding Performance Bottlenecks) has some good tips for starting out on this route.
Ensure that your Java system is easy to monitor, using something like JMX, because this will really help you prove and diagnose any issues you might come across. More on that later in the week.

If you get the opportunity to do some performance testing then I highly recommend it. Assuring the non-functional requirements is a key part of an architect's role so it's great experience.