Real Systems Develop Organically

Often software isn't designed and built like a bridge. It's not engineered and left in a fixed state. It's more like a city - growing from a small hamlet into a major city like London. Most of London wasn't planned, it just sort of happened. Interestingly, the bits that weren't planned are often better places to live and work than the bits that were. Usually because people built what they wanted, when they wanted it.

A 'typical' software product in financial services (I'm sure this will ring a bell in other businesses) starts out as a spreadsheet. A customer rings a consultant/trader/salesperson and asks for something unusual. Smelling a fat commission they price up the request (adding a huge 'spread') and the customer goes away happy. Often that's as far as it goes but sometimes another customer phones up asking for something similar, so they reuse the spreadsheet.

As the requests increase the spreadsheet gets more complex and other employees (traders on the desk) start using it too. Often the figures the spreadsheet produces are updated automatically and published onto a message bus like Triarch (finance messaging system), chat system or emailed. This in turn is used by other spreadsheets, VBA apps and the like.

Two years later IT is called in, possibly because "an important application failed yesterday". You discover an application with 100 users that's spread across multiple desktop machines. It all sort of hangs together, it's not documented and there is no backup policy. If one of the users running part of this application goes on holiday they have to tell someone their password so they can run it up. If they quit then... All the users love what it does, even if it's nasty to use, and don't want it to change. It generates several million in profit every year. It's now your job to stop this being a disaster waiting to happen while keeping the users happy. Sounds familiar?

To stretch my analogy to breaking point, this is similar to the problems the Victorians had in London when they wanted to supply water, sewage and power to old parts of the city. Do you try and provide the necessary infrastructure to what already exists or do you knock it down and replace it with something designed from scratch?

In theory you should be able to estimate the costs of both (don't forget to include any lost revenue, training time, mistakes with a new system, long term support costs etc) and choose the cheapest. In practice an organically grown IT system tends to have spread much more widely than you'd dream possible. On one occasion I've received an angry phone call from an IT manager in Zurich because a file was no longer being generated that he needed. The file appeared to be a side effect of a system I was replacing and was no longer needed - except his system was automatically doing an FTP for the file and using it as a risk input to a trading system. How was I to know it did that?

Unlike a physically growing thing like a city, it's just not easy to see what you affect. The strategy that seems to work best is to slowly replace systems a little at a time and run the replacement in parallel. You have to test that the new functionality works but also make sure you're not side-effecting something else entirely and unexpectedly. You may (or may not) be surprised at how many systems run for years in undead mode, with no one daring to turn them off. You need to avoid this as the system will become harder to support over time as those who know how it works leave. Disable the system to start with but make sure you can re-enable if you have to. Only remove it when it has been disabled for a reasonable period of time.

Organically grown systems are common and often successful. I'd advise not ripping them out but viewing them as an agile system that needs some process and refactoring. Does anyone else have similar war stories?

About the author

Robert Annett Robert works in financial services and has spent many years creating and maintaining trading systems. He knows far more about low latency data systems and garbage collection than is good for anyone. He likes to think of himself as a pragmatist who loves technology but uses what's appropriate rather than what's cool.

When not pouring over data connections or tormenting interviewees with circular reference questions, Robert can be found locked in his shed with an impressive collection of woodworking tools.

E-mail : robert.annett at codingthearchitecture.com


Re: How Real Systems Organically Develop

I found it interesting that I had a slightly different take on your organically-grown pricing application...

I first saw the positive side in the cobbled-together application: it worked. It wasn't a disaster waiting to happen, it was a storming success in progress!

I guess this backs up your analogy, though; you very rarely tear something down just to build something similar (at least not at the system/city level). Rather, you change the thing that's causing the problem (the component that's reached end-of-life).

Re: Real Systems Develop Organically

This reminds me of the most excellent Big Ball of Mud article.

Re: Real Systems Develop Organically

You've described a problem applicable to the replacement of any legacy system, regardless of its origin. Replacing a little at a time is the conservative approach, but also tends to introduce architectural artefacts in the new system which you would not require if you had the liberty of, or need for, a big-bang system switch-over. At what stage are these artefacts removed? As you say, I've seen plenty of undead components in non-legacy systems!

The migration strategy for decommissioning a critical, complex project is a non-trivial problem to solve. Your points raise the question of how much a conservative migration strategy should affect, and be allowed to detract from or complicate, the ideal architecture of the replacement.

A big-bang change rings alarm bells, but is it always a worse choice?

Re: Real Systems Develop Organically

The difference is that a legacy system *could* have been developed using a methodology and have documentation, backup policy and a well defined interface. This makes it much easier to replace via a big-bang as you know where the boundaries of the system are. In a completely unplanned system you have no idea where it starts or finishes so probably don't even know what you're trying to replace. Of course in the real world (which you're referring to) almost every system will be a combination of the two - I've given extreme examples!

Re: Real Systems Develop Organically

There is also the possibility of having the best of both worlds. In one sense, this is the whole idea of SOA -- having loosely-coupled services, so that if I change or remove a service it shouldn't break the system (in an ideal world) -- and it allows me to gradually add services to the composite application over time.

Your example with an Excel application is especially interesting because it is a very common problem on Wall Street. We at GigaSpaces have joined forces with Microsoft to develop a solution for this. We call it Excel that Scales. It allows users to use the Excel they know and love as front-end, but maintains data and business logic centrally.

I blog about it here and you can find more info on the solution on Nati Shalom's blog - here.

A white paper we co-wrote with Microsoft on this solution should be published any day now on our web site, and on MSDN. So stay tuned.

Re: Real Systems Develop Organically

The phrase "mission critical Excel sheet" still sends a shiver down my spine. I odd because people always imagine banks to be rigourous places with excellent technology solutions whereas in reality it's more like papier mache, duct tape and glue. I once worked on an Excel sheet that was the trading app for a desk...

Re: Real Systems Develop Organically

I actually recommended somebody adopt an Excel solution very recently(!) because the timescales really wouldn't permit much else. Comes down to the options you have available at the time...

Re: Real Systems Develop Organically

Actually, I think almost *all* trading applications start out as an excel spreadsheet.

It's whether they grow like a seed into a beautiful tree or like a mutant cell into a tumor.


Add a comment Send a TrackBack