Why do so many technology projects fail?
Does the lack of architecture contribute to failed projects?
The Financial Times published an interesting article earlier in the week entitled Perspectives: Why do so many technology projects fail? (registration required) detailing a court action between BSkyB and EDS. In summary, it's about the failure to deliver a software project, how software projects aren't like construction projects, how a new way of working is needed, etc, etc. It's all stuff we've heard before and, as the article says, are lessons never learned?
Why do so many technology projects fail? Here are my thoughts, some of which overlap with the FT's article.
- Iterative and agile techniques have revolutionized the way that software development is performed, but our industry needs to take a step back and look at the way in which software projects are engaged. Why, when you read about so many high profile big budget software failures, do businesses still initiate software projects with "we want this, tell me how much it will cost"? *We* know that they'll change their mind. *They* know that they'll change their mind. So let's change the engagement model, stop hiding behind fixed price contracts and work *together* to solve problems.
- In my opinion, the view that software development is a commoditized skill is wrong. This is regardless of whether software is developed on-shore, off-shore, etc. Software development is a discipline that's part engineering and part art. If you want decent software built, don't treat it like a commodity. You wouldn't want anybody to build you a Ferrari, would you?
- And finally, of course, there's the lack of a technical architect. Not all architects sit in ivory towers throwing unworkable "solutions" at development teams. Some of us like to work *in* the development team and eat our own dog food. Think of how many software projects you've seen that function correctly but are not fast enough, scalable enough or secure enough. That's why you need an architect.
I was chatting to Sam yesterday and he has some interesting thoughts too, which I'm hoping he'll write about one day. Something that we both agreed on though, is that the software industry generally needs to learn some lessons and wake up to some of the shortcomings that are so prevalent today.
Re: Why do so many technology projects fail?
Re: Why do so many technology projects fail?
Re: Why do so many technology projects fail?
I agree with the notion of being pessimistic - architects need to put the brakes on runaway requirements all the time ("but I want a circular browser window!!"). Architects need to steer the project clear of nasty holes in the road. I think that architects need to be optimistic too; some bumps in the road you can just ride over. Analysis paralysis can kill team morale and the momentum that successful teams build up. By shouldering the risk ("trust me, it'll work! [or I'll fix it]") you can stop projects getting bogged down in the minutiae. Of course, knowing which is which is the trick!
In relation to the FT article, this seems to sound a chord with Kent Beck's recent QCon talk. Developers need to wake up and start seeing what they do as part of an industry, not just a hobby for which they're paid (often extremely well). Accountability, responsibility and transparency are not just tenets of agile methodologies but are the hallmarks of any professional.
Ultimately, success needs to be measured against the drivers for the project, not simply the drivers for the software. Was it on time? Was it within budget? Is it generating the expected revenue stream/cost-saving? The choice of technology, design of the code, even correct formatting, all contribute to the answers but perhaps not always in as big a way as we like to think.
Re: Why do so many technology projects fail?
Developers need to wake up and start seeing what they do as part of an industry, not just a hobby for which they're paid (often extremely well). Accountability, responsibility and transparency are not just tenets of agile methodologies but are the hallmarks of any professional.You still have to question whether we would benefit from a much wider accreditation scheme. Just take a look at some of the other professional industries such as law, accountancy, chartered secretaries, etc.
Re: Why do so many technology projects fail?
The examples you cite aren't exactly fast-moving industries :P
That said, though, chartered status would probably be a good thing. Unfortunately this needs to be driven by a demand for chartered software developers to make it worth pursuing. I'm not sure whether this premium would make chartered developers more or less competitive with their $100-a-day counterparts.
Re: Why do so many technology projects fail?
Clearly some aspects of our industry *are* fast moving. Examples of this include the pace at which new technology is developed and the fact that many business sponsors want their IT solutions built as quickly as possible.
However, on the flip side, you could say that our industry is actually quite slow moving. As a case in point, think about how many people you know who are still building applications with Java 1.x, Struts, etc. Just last week I saw a "technology standards" document that listed Struts as the preferred web MVC framework.
What are your thoughts on the reason for this disparity? Risk management? Professionals that aren't passionate enough about what they do keep up to date? Something else? Perhaps there is a case for accreditation with a regular renewal process?
Re: Why do so many technology projects fail?
While technology isn't fast-moving overall, it is in a constant state of churn. Choosing, say, Struts involves a technology selection process for which the candidate technologies may change frequently (but for a particular environment may always result in the same selection being made - mandating that in a standards doc seems to defeat this, however!).
And I think that it's this type of process that can be accredited or for which chartered status can be awarded and not the technology. How much value do you place on SCJP, for example?
Re: Why do so many technology projects fail?
Re: Why do so many technology projects fail?
Re: Why do so many technology projects fail?
Another interesting question is when do you know a project is failing? I think one of the problems we have in the software world is that it's not obvious.
If you're building a bridge and the struts don't fit in the foundations, you know you've done something wrong and have to correct it properly. But with software you just hack a bit of code in and continue.
It's also easy to scrap a project you don't like (politics has killed more projects I've worked on than bad code). After all, you don't leave behind a half finished structure spanning a river for people to ask about. If you can't finish a project in the lifespan of the sponsor (CTOs last about three years) you might be doomed. ;-)
Re: Why do so many technology projects fail?
Re: Why do so many technology projects fail?
what i see with big projects being designed by developers with 5-6 years experience who have been elevated to the title of architect, is that clients don't understand that these guys are severely limited. there are common symptoms like DNS or mail servers being flooded because they don't have the capacity or clustering traffic being sent over the live traffic interfaces (e.g. session replication). these are problems often occur due to lack of experience.
here is a great book that every aspiring software architect should read [http://www.pragprog.com/titles/mnee]. there are no mention of gartner studies just examples of problems, from one end of the stack to the other, that i have seen over and over again. reading this will help you on your way to successful projects in future!
in relation to politics, i agree that they can kill a project, regardless of the quality of the overall design. as a consultant i saw situations where permie staff just didn't want us around there was little we could do to motivate them into adopting.
Re: Why do so many technology projects fail?
I also think there's a *lot* of consulting work in this area. ;-)
Re: Why do so many technology projects fail?
I'm not sure a "real architect" would specify every element of a system. They ought to ensure that every element is specified and reviewed but they probably won't try to do it all themselves (depending on the size, scope and duration of the project)! Perhaps a minor semantic point, but marshalling experienced specialists is often also required in my experience. This is particularly true when using bleeding-edge technologies - you simply won't be the best person to make these decisions. The point is excellent, though: if projects aren't sufficiently specified they are more likely to fail.
I agree that there are lots of so-called "architects" around but this is possibly more a reflection of how weakly defined the role is rather than delusions of grandeur. Simon's role profile is a useful yardstick here.
I also have to agree with Rob about "architects" only thinking in terms of software design. Many projects aren't just about code but in my experience this still makes up the bulk of my work. This is possibly because the people charged with other aspects of the system (such as the network infrastructure) are more like engineers and are working with more established processes. Perhaps that's a separate post, though!
Re: Why do so many technology projects fail?
For me, I think the jump from application architect (where you only think about a single technology stack) to "system" architect was the biggest. As Rob says, you suddenly find that you have to take a much broader view of the world, particularly when you get asked about hardware, infrastructure, availability, disaster recovery, etc. I'm going to write a separate blog entry about this.
Re: Why do so many technology projects fail?
i can understand why developers become architects within organisations - they usually touch most components at some point, but most don't understand the weaknesses at the integration points and develop apps that cause chaos in production.
as a developer myself i start each project with the mindset that i will be the person causing problems. i need to know how much load other systems can take before i use them, eg messages per second thru a mail server. i need to know how to deal with the mail server running slowly as opposed to the mail server being unavailable, my code needs to know how to fail quickly and not tie up valuable resources. i imagine we've all worked on projects where request threads have been exhausted waiting for another resource ;-)
Re: Why do so many technology projects fail?
As Kevin says, I don't think you necessarily *need* to know everything down to the lowest level of detail. On anything other than very small projects, there's no way that this is viable, so you need to delegate where appropriate (e.g. where somebody has a much higher level of expertise). Having said this, you're right, you do need to consider the impact that your application will have on the broader infrastructure. That, in my opinion, falls squarely underneath the role of an architect.
Re: Why do so many technology projects fail?
No argument here - technical authority has to be based on something more than just vacuous claims! Sometimes experience will suffice. Sometimes exhaustive testing is needed. Sometimes neither is possible so someone else's experience or testing has to be relied upon.
Architects can be bad at what they do just like everyone else. That doesn't necessarily define what an architect is, more how they do it.
Re: Why do so many technology projects fail?
Re: Why do so many technology projects fail?
This title-related discussion is becoming really boring. When can you start referring to yourself as an architect or not? What is an architect vs. application designer vs. anything else? Who cares?! I mean, what really matters is not who you are, it's what you can do. The architect who drew my house and the one who designed the Millau Viaduct are both architects in that they don't build themselves. They just know enough about materials and structures and so on to actually design things, think of it before they exist.
The main problem I see in many projects is that they tend to start with an architect that is not cut for the job (like a house-drawer for the Millau Viaduct). But what's even worse is when they start with an "overkiller", making things more complex than necessary, overbudgetting everything. My point is: we need pragmatism. We should stop worrying about titles and big unrealistic architectures. Let's focus on getting back to the ground. I think that's the whole point of the philosophy behind Agile Methodologies. But apparently we are still entangled in big discussions about "whose one is bigger?"
Re: Why do so many technology projects fail?
Pragmatism is indeed the key and projects need hands-on architects that understand what they are working with rather than industry analysts (to cite a previous comment). The big question, though, is how do we re-educate our industry, our managers, the business sponsors, etc so they understand this. So many projects could be improved by hiring the right team.
Re: Why do so many technology projects fail?
I think that one of the key aspects to re-educate all the actors is to show them the article from the Times, and others like it. IT project failures seem to be easier to hide, so the more we dig up those failures, the more we draw lessons from them, the more lucid we are and the saner the market will be. But then again we're facing some business and political issues here.
So another approach would be to convince all the actors to try out a different, more pragmatic approach, and show them how it's more efficient and less error-prone. And yes, we will lose money on the short-term because the contracts we sign will involve less money. But it's a long-term shot. Even in IT projects, less is more.











