Coding the Architecture - nonfunctionalrequirements tag

Software architecture is a platform for conversation

Tue, 01 Dec 2009 22:03:13 GMT

If you're writing software as a part of your day-to-day job, then it's likely that your software isn't going to live in isolation. We tend to feel safe in our little project teams, particularly when everybody knows each other and team spirits are high. We've even built up development processes around helping us communicate better, prioritise better and ultimately deliver better software. However, most software projects are still developed in isolation by teams that are locked away from their users and their operational environments.

Read the full essay...

A developer's guide to load testing - video

Mon, 15 Jun 2009 22:27:51 GMT

Just a quickie ... the video from A developer's guide to load testing at Skills Matter last week is now available to view online. The microphone picked up the questions and comments quite well, which is excellent. Enjoy, and feel free to use the Google Group to continue the discussion.

Thanks again to Skills Matter.

A developer's guide to load testing - slides

Thu, 11 Jun 2009 18:40:49 GMT

Thanks to everybody that came along to the CTA user group earlier in the week. The tube strike may had reduced the numbers but we still had some great discussion about load testing. My slides are available to download (~4MB) and if anybody wants to continue the discussion, please feel free to use the Google Group.

A developer's guide to load testing (PDF, ~4MB)

I'll post an update when the Skills Matter podcast/video is available to view online.

A developer's guide to load testing

Tue, 02 Jun 2009 08:39:00 GMT

Here are the details of the June 2009 London User Group.

Title : A developer's guide to load testing
Summary : Load testing is an often forgotten and seemingly difficult task that many people shy away from doing. It doesn't have to be this way though, with a basic level of load testing often enough to give you confidence that you've satisfied your non-functional requirements around performance and scalability. This session will look at load testing a website from a developers perspective. We'll look at the differences between load testing, stress testing and soak testing along with a hands-on demonstration of an open source load testing tool that you can use to get started. If you're building websites in Java, .NET, PHP or indeed any other programming language, this session will show you how easy it is to load test your website.
Date : Tuesday, 9th June 2009
Time : 18:30-20:00
Location : Skills Matter, 1 Sekforde Street
Format : Presentation and discussion, with further discussion in a local pub (The Crown).
Cost : Free, but registration is required.

Many thanks to Skills Matter for sponsoring the user group.

p.s. this session is being run after the second day of our software architecture training at Skills Matter in London ... there are a couple of places left if you want to come along and learn about software architecture.

Who should own the non-functional tests?

Mon, 07 Jul 2008 10:03:00 GMT

This is a follow-up to a post entitled Performance tuning Java systems that I made last year, which talked about some performance testing that I was doing for a customer. Last week, I went back to the same customer to make some small tweaks to the test scripts, and this very simple act highlighted something I thought was very interesting.

The key thing about the system being tested is that it's a customized version of an existing vendor product (a high volume, low latency trading platform), with the vendor being responsible for making all of the changes. At some point last year, my customer wanted to run some tests to ensure that the platform could grow with their business, and that's where I got involved. After a round of testing, environment reconfiguration and patches, the system more or less delivered on what it promised.

Fast-forward to the current day and I was bought back in to make some changes to the test scripts, because the vendor had made some changes to their web application. Since our performance tests were JMeter based, those changes had a knock-on effect on the tests. The thing I found interesting is that the performance test pack hadn't been handed over to the vendor. Regardless of the reasons, the benefits of doing this are fairly clear.

Ownership : although it's good that my customer wanted to proactively test some of the non-functional aspects of the software, I can't help feeling that the software vendor should have ownership of the test pack. My understanding is that they do little in the way of non-functional testing themselves, so this is an ideal opportunity to rectify this.
Continuous execution : if the vendor has ownership, they can include the test pack as a part of their integration/build/release process for continuous execution and early warning of performance degrading changes.
Accuracy : I like to think that we did a good job at putting together the performance testing scripts, but we don't have the inside knowledge of the system that the vendor has. Vendor ownership would ensure that the test scripts are as accurate and representative as possible.
Up to date : the test scripts are inherently tied to the AJAX web application, which means that they need to be kept in sync when changes are made to the user experience. Again, the vendor is best positioned to do this, and taking a more proactive approach would ensure that user experience changes don't degrade the non-functional characteristics of the software.

The overriding question here is why doesn't the vendor undertake their own performance testing? They *should* take responsibility for the non-functional characteristics of their software, particularly when they are making claims of how fast and scalable it is. But perhaps more interesting is that the vendor of another product in use in the same space also doesn't do performance testing. Shouldn't we be asking more from commercial product vendors?

NFRs for system replacements

Wed, 28 May 2008 10:35:00 GMT

As software architects, we tend to write about non-functional requirements a lot; particularly about how they should be defined and challenged because of the influence they have. One of the reasons for talking about NFRs a lot is that, in the majority, project teams don't give them the attention that they should. And here's a great example that's slightly different from the story where none of the NFRs were defined.

As is the case in most organisations, technology has a limited lifespan. Perhaps the business evolves, the technology becomes harder to maintain, etc. Whatever the reason, many organisations regularly look at replacing their existing systems with solutions that are shinier and more buzzword compliant.

In one of these cases, an organisation put together a 125 page specification for the replacement system detailing exactly what the system should do along with some details of the non-functional requirements. However, out of those 125 pages, only 1 of them featured the NFRs, and they can be summarised as follows.

Performance : the system must be fast.
Scalability : the system must be able to support 100 messages per second.
Availability : the system must be available 24x7.
Security : only certain individuals are permitted to access the system.

Credit where credit is due ... at least some thought was put into the non-functional requirements and conversations to explore those requirements aren't starting from scratch. But on the other hand, those requirements are meaningless!

I want to end this post on a positive note, so let me contrast this story with another replacement project that I've been involved with recently. In this instance, the team *had* thought about the non-functional requirements and had detailed definitions for what they should be. This is excellent, particularly considering that the non-functional requirements surrounding the current implementation are vague in nature. System replacements are an opportunity to revisit what's important, and that includes the non-functional aspects too.

Scalability Principles

Wed, 21 May 2008 13:12:25 GMT

At the simplest level, scalability is about doing more of something. This could be responding to more user requests, executing more work or handling more data. While designing software has its complexities, making that software capable of doing lots of work presents its own set of problems. This article presents some principles and guidelines for building scalable software systems.

Architectural Assumptions

Mon, 24 Dec 2007 12:44:59 GMT

Most of the systems I've worked on in the recent past have been latency rather than throughput orientated. However my current project is definitely throughput focused and scales horizontally rather than vertically (this is a simplification but basically correct).

This has lead to me making a few errors based on my incorrect assumptions. As you may have read I've been retrofitting performance metrics, we're trying to discover the level of overhead compared to 'real' work, so I've been running a single node on my machine to determine the outputs required for statistics. I wanted to write the results to a database and the DBA asked me to produce an estimate of the load in production before he would create my table structures. Cursing the formality of DBAs, I made some calculations based on the number of tasks and the metrics for each. The number I came up with was huge and the DBA almost coughed up his breakfast.

I was surprised by the number because I had failed to realise that my local test was NOT representative of the real system. If the application was vertically scaled then the quantity of metrics in production would be (roughly) the same as on my desktop PC but this application runs across hundreds of grid machines and each one would output these metrics.

I had just designed a distributed denial of service attack on our own database server. The solution was quite simple - I got the machines that aggregate the outputs to also aggregate the performance metrics. This reduced the load by a couple of orders of magnitude.

I think this is a good example of what happens when you fail to take into account the architecture of a system when writing code. All developers need to be aware of the architecure and how it effects the code they are writing. Does anyone else have similar experiences?

Monitoring Java systems

Fri, 09 Nov 2007 11:18:45 GMT

Earlier in the week I wrote about performance tuning Java systems and I hinted that being able to monitor a system is a really useful first step in proving and diagnosing the cause of performance issues. Of course, there are times where you need a profiler, but that's a different story.

So back to monitoring and I'm still surprised by the number of mission critical systems, particularly in the financial markets sector, where the only way to monitor the system is by tailing a log file. As for management, well, that's typically non-existent.

One of the things that I now do when I'm architecting a new system is to always spend a little time on thinking about the system from an operations/support perspective, reflected by including a "Monitoring and Management View" in my architecture documents. Ultimately, you need to work out what the non-functional requirements are, and a good way to do this is to sit down with the support team to run through the following sorts of questions.

Who will be supporting the system?
Where are they located?
What is their skillset and experience?
Will remote support be required?
Do you think you'll want to reconfigure the system at runtime?
How much control do you want over the log files?
Do you have a standard monitoring infrastructure that the system should publish information to? (e.g. Tivoli, BMC, etc)
Do you have existing scripts/code/etc for sending (for example) SNMP traps?
What granularity of monitoring do you need?
What granularity of management do you need?
Do you need a single dashboard for the entire system?

Questions like these are important to ask because you really do need to tailor the monitoring and management to the people that will be supporting the application. Of course, like all functionality, you do need to prioritise monitoring and management features because they do have an associated cost. Similarly, you also need to perform some cost-benefit analysis because it's no good building a comprehensive web-based administration system for a tactical solution that only has a 3 month lifepsan, for example.

With Java, a really effective way to monitor and manage applications is using the Java Management Extensions (JMX). As I've said before, JMX is one of those technologies that rarely gets a look in, but once you get into it, you'll become infected. Using JMX is straightforward enough, but you do need to work out what you want to monitor and manage. With this in mind, I would recommend getting those hooks in as early as you can. Finally, even if you *really* don't have the time to instrument your code, you can get a certain level of JVM monitoring for *free*, just by enabling the JMX agent. Try it, you have no excuse not to!

Performance tuning Java systems

Mon, 05 Nov 2007 12:23:34 GMT

One of my current projects involves me performance testing a third party Java system. In essence, it's a distributed/n-tier Java EE web application and we're hitting it with a high simulated load thanks to Apache JMeter. One of the things we found is that performance degraded quite dramatically as load increased, to the point where some requests could take anywhere up to a minute or two to be serviced. By plotting the raw results and calculating (for example) the 95% percentile, we were able to see that only a very small number of the requests suffered in this way.

After some investigation, the suspected cause of these high response times was "stop the world" garbage collection and we eventually proved this by monitoring the JVM during a test run. Thankfully, the new Java VMs include a ton of tuning options and I've found the following references particularly useful in the past.

So then, some advice :

Make sure that you define your non-functional requirements as early in the project as you can, specifying performance in the context of scalability. Once you've done this, you can focus on figuring out how to meet them.
Make sure that you test your key non-functional requirements as soon as possible, preferably with something like a reference architecture. Tuning your JVMs early lets you gauge how much more performance you might get with further tuning in the future.
Make sure that you document your JVM settings if you've had to tune them to meet your performance targets. The system I'm testing is a third party product and there's no way that the customer should even be running, what's basically a "real-time" system, on untuned JVMs.
If you need to start investigating performance problems, this article over at InfoQ (The Box: A Shortcut to finding Performance Bottlenecks) has some good tips for starting out on this route.
Ensure that your Java system is easy to monitor, using something like JMX, because this will really help you prove and diagnose any issues you might come across. More on that later in the week.

If you get the opportunity to do some performance testing then I highly recommend it. Assuring the non-functional requirements is a key part of an architect's role so it's great experience.