Architectural Assumptions

forgetting the architecture can have nasty side effects

Most of the systems I've worked on in the recent past have been latency rather than throughput orientated. However my current project is definitely throughput focused and scales horizontally rather than vertically (this is a simplification but basically correct).

This has lead to me making a few errors based on my incorrect assumptions. As you may have read I've been retrofitting performance metrics, we're trying to discover the level of overhead compared to 'real' work, so I've been running a single node on my machine to determine the outputs required for statistics. I wanted to write the results to a database and the DBA asked me to produce an estimate of the load in production before he would create my table structures. Cursing the formality of DBAs, I made some calculations based on the number of tasks and the metrics for each. The number I came up with was huge and the DBA almost coughed up his breakfast.

I was surprised by the number because I had failed to realise that my local test was NOT representative of the real system. If the application was vertically scaled then the quantity of metrics in production would be (roughly) the same as on my desktop PC but this application runs across hundreds of grid machines and each one would output these metrics.

I had just designed a distributed denial of service attack on our own database server. The solution was quite simple - I got the machines that aggregate the outputs to also aggregate the performance metrics. This reduced the load by a couple of orders of magnitude.

I think this is a good example of what happens when you fail to take into account the architecture of a system when writing code. All developers need to be aware of the architecure and how it effects the code they are writing. Does anyone else have similar experiences?

About the author

Robert Annett Robert works in financial services and has spent many years creating and maintaining trading systems. He knows far more about low latency data systems and garbage collection than is good for anyone. He likes to think of himself as a pragmatist who loves technology but uses what's appropriate rather than what's cool.

When not pouring over data connections or tormenting interviewees with circular reference questions, Robert can be found locked in his shed with an impressive collection of woodworking tools.

E-mail : robert.annett at

Re: Architectural Assumptions

The "architecture" would include consideration of the data volumes and therefore developing in such a way would contravene the architecture. Perhaps it's a semantic point, but I think what you're saying is that the architecture wasn't properly considered or that implementation began before the architecture had been fully defined or that the assumptions were just wrong. In that respect I think it's fair to say that just about every project I've worked on has been in that situation!

IMHO, this is where good hands-on architects can make a difference: when problems like this arise they roll up their sleeves and develop a solution that is in keeping with the architecture, if not (yet) defined by it.

Quantum metrics

Performance analysis always was an excellent paradigm for an infamous concept in Quantum Mechanics - that the act of observation has an effect on the state that you are trying to observe.

I really like the way that you have clearly added the act of gathering the metrics to the list of "opportunities" of turning performance measurement into performance reduction!

Fortunately, from your light-hearted tone, I can see that you avoided this before getting near implementation. Others might have discovered the error of their design in far more career limiting ways ;-)

Add a comment Send a TrackBack