Coding the Architecture - monitoring tag

Adding the fuel gauge

Thu, 16 Apr 2009 06:22:15 GMT

One of my talks at the recent DevWeek conference was about the pitfalls software architects face and I covered some of the problems associated with technology selection. Probably *the* biggest problem is vendor marketing and hype, with many project teams simply taking this at face value. Sometimes a piece of technology will do exactly what it says on the tin, but sometimes it won't. There are truly no silver bullets and every technology, large or small, has trade-offs. You've probably seen this yourself at some point. Vendors (open source and commercial) promising features that they haven't yet implemented through to bold claims about performance or scalability. Depending on your project context, these promises can often make or break your project.

One of the analogies that I made during the session was about the fuel consumption figures quoted by car manufacturers in their glossy brochures. Let's imagine that you need to travel from one side of the country to the other, work out the mileage and then buy or rent a car based upon the fuel consumption figures quoted in a brochure. The quoted consumption figures are usually based on some optimum conditions but real world figures will vary according to the way that you drive, the speed you drive, the ambient temperature, the gradient, the road surface and so on. Depending on all of this, you may or may not achieve your goal.

When we undertake a technology selection exercise, we'll typically evaluate candidates against a set of criteria and choose the one that we think best suits our needs before plugging it in to our projects. Not testing the technology before adoption is the same as driving a car across the country - you're relying on somebody else's claims and it might not go as expected. Literally, your mileage may vary!

Of course, the key difference is that you get a fuel gauge in a car that provides you with constant feedback of how much fuel remains in the tank. In addition, newer cars have onboard computers that can provide you with real-time consumption figures and estimate the mileage remaining. This is all information and it provides a way to monitor what is happening so that you can adjust (or fill up!) as necessary. Laptop batteries are the same. The manufacturers quote maximum battery life figures and while you might not get that in real world usage, you do get to see how much battery life is remaining.

With this in mind, it's worth thinking about why we don't usually add fuel gauges to our own software systems. These systems are usually composed of many complex technologies, each of which makes its own claims and has its own trade-offs. Yet we often deploy and run our systems as a black box. Often this will work but sometimes it won't. And worse still, without a fuel gauge you have no idea when your system will grind to a crawl or stop working completely.

Adding a monitoring capability is fairly easy to do and can give you important insight into the health of your software. For example, it might allow you to monitor how many database connections are being used, or how many messages are waiting to be processed, or how many worker threads are busy servicing user requests. Here are some thoughts on how to cater for monitoring in your architecture, and they're particularly relevant if you're building Java applications.

As with cars and laptops, there are benefits to be had by adding some simple feedback devices to our software systems. After all, wouldn't it be great if you could understand the health of your system and proactively deal with problems before they become major issues?

The Other Interface

Sat, 07 Feb 2009 14:53:00 GMT

One of the most succinct definitions of a technical architect is: a technologist who is responsible for a system meeting its Non-Functional Requirements.

What are often perceived as the most interesting NFRs relate to performance, stability and availability. However, recently I've been paying a lot of attention to perhaps the least glamorous of all the non-functionals: supportability. In a mature system, the lion's share of the time it takes to fix a fault is taken up by diagnosing where the fault lies. Once you've diagnosed it, fixing it is often trivial (testing the fix less so, but that's a discussion for another day).

So how do you decrease this diagnosis time? It boils down to logging and monitoring. There are some excellent monitoring tools available, and I've seen some good home-grown applications, which provide a very informative real-time view of what's going on under the hood of a process (for Java systems, JMX greatly facilitates rolling your own, although you get a lot out of the box with Sun's Java distribution these days). Historical concerns about monitoring tools slowing processes down have all but disappeared: such tools are used on the most latency-sensitive of trading systems. While it's relatively easy to recognise a good monitoring tool, a good approach to logging is less self-evident.

I've encountered dramatically different views on application logging: ranging from the view that the log of a healthy long-running process should be short and readable, no bigger than one screen from top to bottom, to the view that a log file should be exhaustive, often gigabytes in size, and carefully designed post-processing scripts (yes, not just grep) can be used to build a picture of what was going on in the process at a given point in time, or in response to a given event.

The best approach will depend on the nature of your system and how it is supported. I'm currently working on a system supported by several different teams; the development team forms a third or even fourth level of support. Therefore what the system dumps out in its logs feeds into human processes: messages logged at Error level should require manual intervention and possibly escalation to the next level of support, whereas warnings and below should be ignorable.

Everyone who can change the code needs to be aware of this, therefore a logging policy needs to be defined, published and enforced. Ideally this policy will make your system as close to self-diagnosing as possible. When this has not been the case, the black art of knowing which errors can be ignored, or where to look if a process fails with no log information at all, can hugely increase the support costs of the system. It affects the speed of resolution of support incidents, increases the learning curve of new joiners in the team, makes testing more difficult, and reduces software quality by hiding or delaying the discovery of bugs.

If there is one approach which is relevant to all logging policies, it is don't cry wolf, and don't die quietly. To put it another way:

Messages logged as Errors / Severe / Fatal should actually be problems with the system, and should not be ignorable.
When the system fails, if there is scope in the code to log the current state, this should be done whenever possible.

This may sound obvious, but I'm finding out, to my expense, that applying even such a simple logging policy to a mature system after the fact can be very costly.

There's perhaps no right answer as to what makes an ideal application log, however there are many wrong answers. The worst of all is to ignore this interface to your system. So define your logging standard at the same time as you define your other non-functional requirements, and similarly enforce it as the system evolves.

Podcast #2 : QCon revisited

Tue, 25 Mar 2008 10:23:20 GMT

As promised the 2nd CTA podcast is a roundtable discussion between some of the CTA contributors - namely Simon Brown, Sam Dalton and Kevin Seal. In this podcast we discuss some of the themes emerging from the recent QCon conference held in London and our views on those themes.

The podcast can be downloaded here or you can subscribe. Full show notes follow below:

1. Introduction

QCon Conference London detail can be found here.

2. Monitoring and Management (0:27)

Roll your own vs. "off the shelf" monitoring
Asynchronous monitoring
JXInsight
The need for an System Architect and an overall understanding of WHAT to monitor and a holistic view
Including Monitoring NFRs upfront in SAD
Monitoring the monitoring vs. best effort
eBay and 1.5TB of monitoring and logfile information per day

3. Rehashing Refactoring (11:56)

Refactoring now mainstream, now emphasizing at the micro-level
Code-debt - how can you measure it?
Refactoring is not something to be "saved up"
Lots of rearchitecture projects sold as refactoring
Sometimes starting again is a better solution than large scale refactoring
Why do things end up in such a mess so often?
Need to get back to writing code for humans to understand
Worry about emphasis on low-level code optimizations rather than architecture
Is refactoring its own worst enemy being referred to as a separate discipline to regular software development?
We are our own worst enemies - we (especially the architects) need to stand up for the quality of our code - stop allowing ourselves to be convinced to make shortcuts

4. The Return of the Architect? (25:08)

Thinking about performance, monitoring etc upfront
Terracotta, Coherence, Gigaspaces
Kevin's views on Terracotta
How much performance could you squeeze from a system in 2 weeks with code changes alone?
Real-time VMs and performance improvements
The benefits of continuous performance testing
Premature optimization
Understanding a system's deficiencies can be as valuable as fixing them

Monitoring Java systems

Fri, 09 Nov 2007 11:18:45 GMT

Earlier in the week I wrote about performance tuning Java systems and I hinted that being able to monitor a system is a really useful first step in proving and diagnosing the cause of performance issues. Of course, there are times where you need a profiler, but that's a different story.

So back to monitoring and I'm still surprised by the number of mission critical systems, particularly in the financial markets sector, where the only way to monitor the system is by tailing a log file. As for management, well, that's typically non-existent.

One of the things that I now do when I'm architecting a new system is to always spend a little time on thinking about the system from an operations/support perspective, reflected by including a "Monitoring and Management View" in my architecture documents. Ultimately, you need to work out what the non-functional requirements are, and a good way to do this is to sit down with the support team to run through the following sorts of questions.

Who will be supporting the system?
Where are they located?
What is their skillset and experience?
Will remote support be required?
Do you think you'll want to reconfigure the system at runtime?
How much control do you want over the log files?
Do you have a standard monitoring infrastructure that the system should publish information to? (e.g. Tivoli, BMC, etc)
Do you have existing scripts/code/etc for sending (for example) SNMP traps?
What granularity of monitoring do you need?
What granularity of management do you need?
Do you need a single dashboard for the entire system?

Questions like these are important to ask because you really do need to tailor the monitoring and management to the people that will be supporting the application. Of course, like all functionality, you do need to prioritise monitoring and management features because they do have an associated cost. Similarly, you also need to perform some cost-benefit analysis because it's no good building a comprehensive web-based administration system for a tactical solution that only has a 3 month lifepsan, for example.

With Java, a really effective way to monitor and manage applications is using the Java Management Extensions (JMX). As I've said before, JMX is one of those technologies that rarely gets a look in, but once you get into it, you'll become infected. Using JMX is straightforward enough, but you do need to work out what you want to monitor and manage. With this in mind, I would recommend getting those hooks in as early as you can. Finally, even if you *really* don't have the time to instrument your code, you can get a certain level of JVM monitoring for *free*, just by enabling the JMX agent. Try it, you have no excuse not to!