Coding the Architecture - rannett

Designing Maintainable Systems

Fri, 29 Jan 2010 13:04:00 GMT

I'm currently involved in a project to upgrade a third party piece of software and it's apparent that when the software was originally designed, the upgrade process was not considered. This became obvious when we totaled up the time required to perform, configure and post-release test the upgrade - it came to over three days of work. This was not even taking into account any rollback times (which is fortunately simplified these days by the use of virtualisation).

The software is used heavily from Monday to Friday so we wanted to upgrade over a weekend. The vendor suggested we perform an upgrade on a parallel system and then get the users to re-enter all the data into the new system that was missed - you can imagine how well that would have gone down. This would also mean trying to post-release, regression test two systems that are live, being used and not in sync.

Software almost always needs updating/upgrading (unless it's control software for a deep space probe!) The ability and consequence of upgrading should be considered as part of the design and development process. Questions to ask include:

Can an upgrade be performed in parallel to a live, running system and how does a switchover occur?
Will a system need to be taken down for any upgrades and for how long? How does this affect your Service Level Agreements?
How easy will any upgrade be to rollback? Errors occur!
Can you upgrade parts of the systems or does everything have to be done at once?
What is the effect on any users? Will they need to log out first etc? Will they lose any work if they fail to follow your procedures?
How easy will it be to test the upgraded system to determine success? Your notice of failure shouldn't be an angry user phone call.

Some simple tools can make all the difference. Most of my work is on financial applications and I like to run regression reports between systems for important points e.g. End-of-year. However it's often very difficult to get data out of systems to perform simple comparisons!

Sensible configuration management is often missing. If I've upgraded and configured new features in my pre-production environment I really shouldn't have to repeat the process from scratch in production. Manual processes are prone to errors and ideally once I've prepared for an upgrade I should just hit a 'go' button and sit back.

In my experience very few software developers are aware of IT Service Management (ITSM/ITIL). In particular we should be aware of the Change Management, Release Management and Configuration Management roles that support staff have. If you want to read about ITSM/ITIL then the wiki page is a good place to start.

Some of the processes of ITSM may strike agile developers as being heavy-weight but this doesn't stop you developing the system in an agile manner, it just means that it can be deployed within a formal environment.

An architect should be aware of how the software fits into the organisation. So remember that your ‘users’ aren't just the end users but also the support staff who'll be maintaining your system for the next ten years!

Modifying Open Services

Fri, 14 Nov 2008 19:53:38 GMT

There's been a huge push recently towards service oriented architectures - sharing services within an organisation with benefits such as reuse and making information consistent. Take a simple example such as a catalogue of products for a furniture company. As a shared and open service, all of the companies systems - Sales, Marketing, Support, Delivery and Billing applications - can use this information in an open and consistent way.

If a service is very open and easy to use (e.g. services operating via a RESTful interface) then there is a good chance that applications will use it in a way you didn't originally intend and probably by applications you don't even know about. This sounds great but you'll soon come across the issue that you've lost the ability to audit the current use and gauge the effect of any change. As an example let's say you want to add details for 'forest sustainability' to our furniture information. We add a block of xml to describe this and release. However an application that uses our service starts generating errors as it's not expecting this new information. (We could argue that it shouldn't do this but this is what happens in the real world.) Problems are more likely if you have to modify rather than add to your format. Changing an integer to a floating point number could cause strange issues.

You need to be able to get dependent applications to test with your new service before you release but who's using it and how are they using it? You can log the incoming requests to know what is being used but you don't know who is using it - so how do you know who has to test changes?

This is a problem I've been seeing recently and a solution is to use authentication even if you have no intention of restricting access. You can make the credentials easy to obtain but you need to make sure the users of your service are registered and provide sufficient contact information. Of course, actually getting the users to test and adapt to changes are another issue but at least they can't complain they weren't informed.

Has anyone else seen this issue and what were your solutions? Did you just 'publish and be damned' or end up introducing heavyweight process to control releases?

Project Naming

Sat, 01 Mar 2008 21:06:34 GMT

Whenever I've started work on a new project I've had an introduction along these lines:

Wizard feeds into Pluto, which then re-values. It broadcasts changes that are picked up by Puma and recorded by Halo.

but what they really mean is something like:

The Market-data System feeds into the Trade Calculation Engine, which then re-values. It broadcasts changes that are picked up by the Trade Display GUI and recorded by the Audit system.

The IT industry seems to have a peculiar habit of giving projects names that have no relation to what they actually do. Perhaps we feel the need to make a dull system sound more interesting ("We can make the accounts receivable project exciting by calling it Skydive") and this is fine if you have only one system. I deal with architectures that have dozens of subsystems and this can be a real problem by adding to the learning curve and confusing newcomers.

There is an argument that a system's name shouldn't have meaning as the scope of the system will probably grow and change throughout its life. If a project has a very specific name such as "stock-option pricer" it will be inaccurate once you add the ability to price bond-options. A few years down the line it could actually be much confusing than calling it Pluto.

It would be nice if you could change the name of a project/system but this can be very hard. Not because you can't re-factor code and change configuration but because of funding, politics and the users familiarity.

What is your preferred technique for naming projects and systems? Something funky, something meaningful or a compromise? If you are attending the user group meeting this Tuesday you can share your craziest project name with me!

Update - It's been suggested to me that in a SOA system you should name a service very exactly and when the name no longer matches then it's probably time to introduce a new service. Neat!

System Design and Reconciliation

Fri, 08 Feb 2008 22:06:20 GMT

Even those not working in Finance have probably heard about the intriguing events in a large investment bank and the HUGE losses that have occurred.

The interesting comment from an IT and architecture point of view is the following:

The trader responsible for the fraud had "in-depth knowledge of the control procedures resulting from this former employment in the middle-office", the bank said.

I'm sure the story and our perception of it will change as more facts emerge but we can be certain that the banks internal security and control systems failed.

Many security and control terms such as authentication, authorization and auditing are probably familiar what about reconciliation? Consider a simple stock-taking example. The number of items delivered to a warehouse minus the number of items sold should tell you how many are still there. If it's less, you may have a thief. ( If you've ever been involved in a stock-take you may have seen many missing items - they tend to be small and valuable.) Stock taking is a physical example of reconciliation i.e do all the numbers add up.

Should you be doing this in a system of your own? If you have developed an online store do you periodically make sure that the number of orders to your website equals the number of payments received and items sent? It's very common for a system designer to assume this will be the case, as the data normally flows from one component to the next, but consider what happens if someone sends messages to your delivery system directly. They bypass your ordering and payment systems so there is no matching order or payment. This is likely to be an insider that knows the system well and you can lose a lot of money as you send out goods without being paid. These kind of issues are increasingly likely as designers move away from monolithic systems to SOAs.

Apart from internal checks between your own systems you can reconcile with external ones. In securities trading, we usually get a list of trades from the markets and exchanges at the end of he day and reconcile against the banks internal list of trades. This should show if a trader has been placing trades they shouldn't have (or failing to place those they should).

It's important to make sure that the main system and the reconciliation system are separate otherwise an error or fraud in one could effect the other. (Reconciliation is often simple and can be done in something like a script).

Finally, you should remember that regression testing is actually a form of reconciliation between different versions of systems, so you may be doing some already!

Architectural Assumptions

Mon, 24 Dec 2007 12:44:59 GMT

Most of the systems I've worked on in the recent past have been latency rather than throughput orientated. However my current project is definitely throughput focused and scales horizontally rather than vertically (this is a simplification but basically correct).

This has lead to me making a few errors based on my incorrect assumptions. As you may have read I've been retrofitting performance metrics, we're trying to discover the level of overhead compared to 'real' work, so I've been running a single node on my machine to determine the outputs required for statistics. I wanted to write the results to a database and the DBA asked me to produce an estimate of the load in production before he would create my table structures. Cursing the formality of DBAs, I made some calculations based on the number of tasks and the metrics for each. The number I came up with was huge and the DBA almost coughed up his breakfast.

I was surprised by the number because I had failed to realise that my local test was NOT representative of the real system. If the application was vertically scaled then the quantity of metrics in production would be (roughly) the same as on my desktop PC but this application runs across hundreds of grid machines and each one would output these metrics.

I had just designed a distributed denial of service attack on our own database server. The solution was quite simple - I got the machines that aggregate the outputs to also aggregate the performance metrics. This reduced the load by a couple of orders of magnitude.

I think this is a good example of what happens when you fail to take into account the architecture of a system when writing code. All developers need to be aware of the architecure and how it effects the code they are writing. Does anyone else have similar experiences?

What is Significant?

Sat, 15 Dec 2007 21:15:27 GMT

Recently I wrote a quick blog about taking metrics for optimisation. I suggested they should only be included if the improvement was significant, but how do you define significant?

You may think that 'significant' is just a matter of opinion but it actually has a very specific meaning in statistics - Wikipedia 's Description. You can have a read through the maths but it basically comes down to "a result is called statistically significant if it is unlikely to have occurred by chance".

This is really important and something that performance testers and optimisers often forget. For example...

Imagine that you perform some kind of performance test on your system or code. This could be anything e.g. latency response timings, throughput per time unit etc but we'll assume for this that it's units processed in 10 minutes. The figure you get is 20. You spend a day modifying a piece of code you think will affect the performance, retest and get 22. A 10% improvement - pretty good.

You hand the new code over to a colleague who also does a test. She says that it's worse by 5%. Slander! You take it to your boss who says there is no difference...

What we've done is perform three tests on the old and new system. Lets list them and perform seven others as well:

Old: 20 20 22 19 19 21 19 19 19 22
New: 22	19 22 20 20 19 19 19 21 19

Now it's obvious what happened (although it probably was before). Your test does not produce constant figures even without changes. Both have a range of 3 (19-22), an average of 20 and a variance of 1.55

If you had performed ten runs on the original code first you would have realised that a single result of 22 for the new code is not significant as it's within the range of the previous figures.

Performing the test multiple times on the new code would increase your confidence that it's a significant change. You can test this statistically (but the maths is beyond the scope of this blog entry).

Just to leave you with a challenge, the project I'm currently working has a task that we wish to optimise but it takes ten hours to run even on a grid of several hundreds machines. How do we run realistic, pre-production tests that we know are statistically significant?

Metrics Please!

Wed, 12 Dec 2007 09:57:56 GMT

Again it's time to get something off my chest...

Twice in the last week my project has been bitten by the same, basic issue. The first problem was a memory leak in a small but important tool which was caused by introducing faulty caching to a lookup. The second problem was someone trying to solve an out of memory error by getting a client to avoid duplicating data in a request. This was fine, except that the service that received the request did the duplication anyway so the problem remained.

Now these are both the sort of bugs/problems that you see all the time so why am I wound up about it? It's because they both made it into production and really, really shouldn't.

The first change shouldn't have even been made. The lookup in question is rarely done and is likely to be unique when it is - i.e. any cache will never successfully be hit anyway.

Optimization Rule Number 1 - Always take suitable metrics BEFORE making any optimization. Don't make them on gut feel. If your metrics show that a particular piece of code or system is not a bottleneck or performance problem then don't waste time modifying the code at all!

The second change was certainly needed (a complex calculation for a report was never completing). The problem was correctly identified but the solution wasn't complete.

Optimization Rule Number 2 - Always take metrics AFTER making any optimization. These should be compared with the metrics taken before so you know if the changes have had a significant effect.

Measure twice and cut the code once!

Do you need to build it yourself?

Mon, 12 Nov 2007 15:23:40 GMT

I've worked in many organisations and one thing that never ceases to amaze me is how often teams build unnecessary tools and frameworks.

Yet again, I have to use a bespoke defect tracking system. This is the third one I've come across and none of them are as good as a Commercial off the Shelf (COTS) solution and definitely cost more to develop than to buy.

In the past, I've also seen in-house developed: IDEs, email clients, database admin tools, UML tools etc. Perhaps it's more fun to write tools than the boring, business functionality your client/users needs but does it offer any value?

As a software/systems architect it's your job to deliver value to the business you work for. Whenever someone suggests creating a bespoke tool ask yourself the following:

Can I buy something to do this?
Can I use or tailor an open source product?
Can I use or create a template for a standard tool like Eclipse, Word or Excel?

Be pragmatic in your decision and avoid wasting your limited time and resources on things you don't need to do!

Reference Architectures

Wed, 10 Oct 2007 07:54:40 GMT

Some major architectural decisions, for a reasonable sized project, will be made before the coding starts e.g. using an Enterprise Service Bus, data access methods, transaction handling etc. Your ability to achieve your non-functional requirements will depend greatly on these decisions and most will involve a trade off e.g. latency vs throughput or reliability vs performance.

Some of the trade offs will be obvious (or appear to be) but many will not e.g. how much latency does sending a request to a grid introduce? There is also the chance that your chosen technologies will not work together (maybe they refuse to work with the same database or operating system). Sometimes technologies just don't work as advertised - throughput figures are often misleading.

This is where a reference or proof of concept architecture can help. A reference architecture should contain the following:

All the major pieces of software
All the types of hardware
All the hardware/software layers e.g. DB server, application server etc as separate machines (virtualisation can help)
A simple use case that exercises all the pieces of software and hardware.

You are trying to test the major flow of the system so you can keep business logic to a minimum.

We are trying to prove that the architectural design is capable of delivering the NFRs. This doesn't guarantee that your performance won't degrade as the system is fleshed out but if it doesn't work on a simplified system, the full system has no chance.

Also take the opportunity to identify were you should capture metrics. If you fail to achieve the NFRs externally you'll want to discover where in your system you are having issues. Including metric points from the start makes this much easier.

The initial, internal metrics you gather will also serve as a useful base for comparison. As you add more business logic you will probably see your performance decrease. If you regularly collect metrics you can spot when and where this occurs.

In summary, in the same way that you should test your code early and often, you should do the same with your architecture.

Development with Complex Architectures

Sat, 15 Sep 2007 07:37:05 GMT

Most projects I've worked on have had complex architectures and have shared some development challenges. One of the most frustrating is trying to test code on a local machine when the real system comprises of several large and complex services. If the services are small and lightweight you can run them locally but (as is often the way) they are quite substantial this is not possible. On my last project, to run a full end to end functional test would involve running six services, each of which required more than 1/2 gig of memory. I can't run this up on a developer specification machine, especially if I also want Eclipse and a debugger.

I know someone will say "write unit tests" and of course I do but there is a class of problems that you see when services interact that you will miss with unit tests. For example, the coordination of service states and circular, service call references.

In the past I have seen several possible solutions: