When things go wrong...

Mon, 06 Aug 2007 10:35:58 GMT

When incidents arise I feel the role of the architect is to mediate: to get the information needed to steer the technical team, to help identify the problem and to ensure that the solution may be deployed without making things worse.

Recently I've been involved in several incidents where concurrency issues, session hijacking, sequence number conflicts, etc. were floated as possible causes for complaints that were being raised on a website.

While all of these might be possible causes for the symptoms, they're not particularly plausible. It reminds me of a tale of a man who drops his keys on a dark street. He keeps walking until he finds a street light and starts looking for his keys there. While they're not likely to be there, he'd be more likely to find them if they were. I've seen people looking for explanations to production issues in outlying places first before trying to shed some light on the problem. I've certainly done the same when the adrenaline's pumping.

In the case of my recent production incidents, a look at the data showed that the most likely explanations were human error and an ISP recycling a user's email address.

Coding the Architecture - issues tag

When things go wrong...