A 'typical' software product in financial services (I'm sure this will ring a bell in other businesses) starts out as a spreadsheet. A customer rings a consultant/trader/salesperson and asks for something unusual. Smelling a fat commission they price up the request (adding a huge 'spread') and the customer goes away happy. Often that's as far as it goes but sometimes another customer phones up asking for something similar, so they reuse the spreadsheet.

As the requests increase the spreadsheet gets more complex and other employees (traders on the desk) start using it too. Often the figures the spreadsheet produces are updated automatically and published onto a message bus like Triarch (finance messaging system), chat system or emailed. This in turn is used by other spreadsheets, VBA apps and the like.

Two years later IT is called in, possibly because "an important application failed yesterday". You discover an application with 100 users that's spread across multiple desktop machines. It all sort of hangs together, it's not documented and there is no backup policy. If one of the users running part of this application goes on holiday they have to tell someone their password so they can run it up. If they quit then... All the users love what it does, even if it's nasty to use, and don't want it to change. It generates several million in profit every year. It's now your job to stop this being a disaster waiting to happen while keeping the users happy. Sounds familiar?

To stretch my analogy to breaking point, this is similar to the problems the Victorians had in London when they wanted to supply water, sewage and power to old parts of the city. Do you try and provide the necessary infrastructure to what already exists or do you knock it down and replace it with something designed from scratch?

In theory you should be able to estimate the costs of both (don't forget to include any lost revenue, training time, mistakes with a new system, long term support costs etc) and choose the cheapest. In practice an organically grown IT system tends to have spread much more widely than you'd dream possible. On one occasion I've received an angry phone call from an IT manager in Zurich because a file was no longer being generated that he needed. The file appeared to be a side effect of a system I was replacing and was no longer needed - except his system was automatically doing an FTP for the file and using it as a risk input to a trading system. How was I to know it did that?

Unlike a physically growing thing like a city, it's just not easy to see what you affect. The strategy that seems to work best is to slowly replace systems a little at a time and run the replacement in parallel. You have to test that the new functionality works but also make sure you're not side-effecting something else entirely and unexpectedly. You may (or may not) be surprised at how many systems run for years in undead mode, with no one daring to turn them off. You need to avoid this as the system will become harder to support over time as those who know how it works leave. Disable the system to start with but make sure you can re-enable if you have to. Only remove it when it has been disabled for a reasonable period of time.

Organically grown systems are common and often successful. I'd advise not ripping them out but viewing them as an agile system that needs some process and refactoring. Does anyone else have similar war stories?

Responses[8]

Posted by Robert Annett on 19 July 2007 09:55:37 BST #