What is software architecture? What is the role of a software architect? How do you define software architecture? How do you share software architecture? How do you deliver software architecture?

Why do so many technology projects fail?

Does the lack of architecture contribute to failed projects?

The Financial Times published an interesting article earlier in the week entitled Perspectives: Why do so many technology projects fail? (registration required) detailing a court action between BSkyB and EDS. In summary, it's about the failure to deliver a software project, how software projects aren't like construction projects, how a new way of working is needed, etc, etc. It's all stuff we've heard before and, as the article says, are lessons never learned?

Why do so many technology projects fail? Here are my thoughts, some of which overlap with the FT's article.

  • Iterative and agile techniques have revolutionized the way that software development is performed, but our industry needs to take a step back and look at the way in which software projects are engaged. Why, when you read about so many high profile big budget software failures, do businesses still initiate software projects with "we want this, tell me how much it will cost"? *We* know that they'll change their mind. *They* know that they'll change their mind. So let's change the engagement model, stop hiding behind fixed price contracts and work *together* to solve problems.
  • In my opinion, the view that software development is a commoditized skill is wrong. This is regardless of whether software is developed on-shore, off-shore, etc. Software development is a discipline that's part engineering and part art. If you want decent software built, don't treat it like a commodity. You wouldn't want anybody to build you a Ferrari, would you?
  • And finally, of course, there's the lack of a technical architect. Not all architects sit in ivory towers throwing unworkable "solutions" at development teams. Some of us like to work *in* the development team and eat our own dog food. Think of how many software projects you've seen that function correctly but are not fast enough, scalable enough or secure enough. That's why you need an architect.

I was chatting to Sam yesterday and he has some interesting thoughts too, which I'm hoping he'll write about one day. Something that we both agreed on though, is that the software industry generally needs to learn some lessons and wake up to some of the shortcomings that are so prevalent today.



Re: Why do so many technology projects fail?

I agree so much with you (and the article you're quoting). That's why we at Axen are currently testing a new approach to combine agile methodologies and fixed-price projects. Maybe there are others, but this one has been working pretty well so far, and the project started almost 3 months ago.

Re: Why do so many technology projects fail?

I believe that first of all it's a matter of discipline. Maybe you will find my blog post "the right amount of pessimism" interesting with this respect. Most projects succeed because they avoid all traps. In order to avoid the traps, you have to know what are them (and there's enough literature on the subject) and to continuously look out for them during the project. Simple, isn't it? :-D

Re: Why do so many technology projects fail?

I agree with the notion of being pessimistic - architects need to put the brakes on runaway requirements all the time ("but I want a circular browser window!!"). Architects need to steer the project clear of nasty holes in the road. I think that architects need to be optimistic too; some bumps in the road you can just ride over. Analysis paralysis can kill team morale and the momentum that successful teams build up. By shouldering the risk ("trust me, it'll work! [or I'll fix it]") you can stop projects getting bogged down in the minutiae. Of course, knowing which is which is the trick!

In relation to the FT article, this seems to sound a chord with Kent Beck's recent QCon talk. Developers need to wake up and start seeing what they do as part of an industry, not just a hobby for which they're paid (often extremely well). Accountability, responsibility and transparency are not just tenets of agile methodologies but are the hallmarks of any professional.

Ultimately, success needs to be measured against the drivers for the project, not simply the drivers for the software. Was it on time? Was it within budget? Is it generating the expected revenue stream/cost-saving? The choice of technology, design of the code, even correct formatting, all contribute to the answers but perhaps not always in as big a way as we like to think.

Re: Why do so many technology projects fail?

Developers need to wake up and start seeing what they do as part of an industry, not just a hobby for which they're paid (often extremely well). Accountability, responsibility and transparency are not just tenets of agile methodologies but are the hallmarks of any professional.
You still have to question whether we would benefit from a much wider accreditation scheme. Just take a look at some of the other professional industries such as law, accountancy, chartered secretaries, etc.

Re: Why do so many technology projects fail?

The examples you cite aren't exactly fast-moving industries :P

That said, though, chartered status would probably be a good thing. Unfortunately this needs to be driven by a demand for chartered software developers to make it worth pursuing. I'm not sure whether this premium would make chartered developers more or less competitive with their $100-a-day counterparts.

Re: Why do so many technology projects fail?

That's true, they might not necessarily be fast moving. But is ours as fast moving as we think?

Clearly some aspects of our industry *are* fast moving. Examples of this include the pace at which new technology is developed and the fact that many business sponsors want their IT solutions built as quickly as possible.

However, on the flip side, you could say that our industry is actually quite slow moving. As a case in point, think about how many people you know who are still building applications with Java 1.x, Struts, etc. Just last week I saw a "technology standards" document that listed Struts as the preferred web MVC framework.

What are your thoughts on the reason for this disparity? Risk management? Professionals that aren't passionate enough about what they do keep up to date? Something else? Perhaps there is a case for accreditation with a regular renewal process?

Re: Why do so many technology projects fail?

While technology isn't fast-moving overall, it is in a constant state of churn. Choosing, say, Struts involves a technology selection process for which the candidate technologies may change frequently (but for a particular environment may always result in the same selection being made - mandating that in a standards doc seems to defeat this, however!).

And I think that it's this type of process that can be accredited or for which chartered status can be awarded and not the technology. How much value do you place on SCJP, for example?

Re: Why do so many technology projects fail?

Here's an interesting blog entry entitled Where Do Experts Come From? that talks about motivation being a key aspect of becoming an "expert". While *everybody* doesn't need to be an expert, I would say that many people in the IT industry lack motivation and passion for what they do. How much do you think this contributes to project failure?

Re: Why do so many technology projects fail?

ok, tried not to bite, but i couldn't resist! real architects study for years studying other people's designs and attempting their own on paper before they will ever get close to designing something for real. very few people in our industry deserve to call themselves architects, in fact between the two of us we can probably count the amount of people that deserve the title on one hand. the rest of people that use the term are either senior developers or industry analysts. the former should be referred to as such and even then, most people that use that title don't deserve it - you don't become more senior as years pass, you become more senior as you develop into a technical mentor. industry analysts are the types of people that read gartner reports or contract work to ibm. they aren't affraid of white-papers but they can't be expected to write any code, even of prototype quality. the problem is that the more valuable the contract the more likely you are to have industry analysts designing systems. a real architect will specify every component, down to the gauge of the screws required. in development, if someone goes to that level then they might as well write it themselves! so, unless you can design a system from the network, operating system installation and configuration, the database installation and configuration, the code itself and the test harness - you aren't an architect! if you call urself an architect, but are unable to do all of that, then its likely that your projects will suffer. however, if you recognise that there are areas where you need help and address those areas sooner rather than later, then your project has much greater chance of succeeding. thats why the industry has adopted CI - it helps us find out where are projects are likely to fail as soon as possible. when the integration fails then we can find others to help. i could keep ranting, but it'd be interesting to hear what you think...

Re: Why do so many technology projects fail?

Another interesting question is when do you know a project is failing? I think one of the problems we have in the software world is that it's not obvious.

If you're building a bridge and the struts don't fit in the foundations, you know you've done something wrong and have to correct it properly. But with software you just hack a bit of code in and continue.

It's also easy to scrap a project you don't like (politics has killed more projects I've worked on than bad code). After all, you don't leave behind a half finished structure spanning a river for people to ask about. If you can't finish a project in the lifespan of the sponsor (CTOs last about three years) you might be doomed. ;-)

Re: Why do so many technology projects fail?

I have a different experience than the one you mention, meaning I starting doing small bits of architecture in a very complex application because my customer saw that I could do it. He was always looking over my shoulder and correcting me when I was wrong. I have to say that I never read other people's designs (except for open source projects), but only because I didn't know they were available. By the way, could you give some directions to finding them? On the other hand, I don't think that one can start doing software architecture only from books and inspiring from other people's work. Practice is essential, and not only on paper, but in real life as well. I had the opportunity to work on small projects where I could try my ideas without risking too much, and then I learned more than by reading. So my point is that it's a matter of how you mix theory and practice together. What do you say?

Re: Why do so many technology projects fail?

i agree - it is best to learn how to develop whilst working, but while you are still learning you cannot call yourself an architect. designing an application with spring and hibernate, for example, is a small part of producing a new site, there are tiers in front, behind and underneath that are often more complicated than the application code itself. by acknowledging that there are areas you don't understand well enough to design you are already helping your client to get closer to what they want.

what i see with big projects being designed by developers with 5-6 years experience who have been elevated to the title of architect, is that clients don't understand that these guys are severely limited. there are common symptoms like DNS or mail servers being flooded because they don't have the capacity or clustering traffic being sent over the live traffic interfaces (e.g. session replication). these are problems often occur due to lack of experience.

here is a great book that every aspiring software architect should read [http://www.pragprog.com/titles/mnee]. there are no mention of gartner studies just examples of problems, from one end of the stack to the other, that i have seen over and over again. reading this will help you on your way to successful projects in future!

in relation to politics, i agree that they can kill a project, regardless of the quality of the overall design. as a consultant i saw situations where permie staff just didn't want us around there was little we could do to motivate them into adopting.

Re: Why do so many technology projects fail?

I couldn't agree more. The problem I see again and again is 'architects' that only think in terms of java/c# etc. They have no idea what a router, firewall or replicated database is - let alone know how to spec their requirements for a data centre.

I also think there's a *lot* of consulting work in this area. ;-)

Re: Why do so many technology projects fail?

I'm not sure a "real architect" would specify every element of a system. They ought to ensure that every element is specified and reviewed but they probably won't try to do it all themselves (depending on the size, scope and duration of the project)! Perhaps a minor semantic point, but marshalling experienced specialists is often also required in my experience. This is particularly true when using bleeding-edge technologies - you simply won't be the best person to make these decisions. The point is excellent, though: if projects aren't sufficiently specified they are more likely to fail.

I agree that there are lots of so-called "architects" around but this is possibly more a reflection of how weakly defined the role is rather than delusions of grandeur. Simon's role profile is a useful yardstick here.

I also have to agree with Rob about "architects" only thinking in terms of software design. Many projects aren't just about code but in my experience this still makes up the bulk of my work. This is possibly because the people charged with other aspects of the system (such as the network infrastructure) are more like engineers and are working with more established processes. Perhaps that's a separate post, though!

Re: Why do so many technology projects fail?

Some really interesting discussion going on here and my experience is similar. I started becoming more and more involved in high level design work, then application architecture for small systems, then application architecture within larger systems, etc. I have to admit that I learnt a lot from working as a senior developer/application architect on projects where there were other, much more experienced architects. I got to see what worked and what didn't.

For me, I think the jump from application architect (where you only think about a single technology stack) to "system" architect was the biggest. As Rob says, you suddenly find that you have to take a much broader view of the world, particularly when you get asked about hardware, infrastructure, availability, disaster recovery, etc. I'm going to write a separate blog entry about this.

Re: Why do so many technology projects fail?

in that situation i don't the title of architect is deserved - designer is much more suitable. in terms of knowing the overall design - a little knowledge is lethal, if you don't know it down to the nuts and bolts then make sure you don't make any claims in the absence of someone who does! i worked on a project with an architect that made claims that cost us weeks to investigate and they were rarely important.

i can understand why developers become architects within organisations - they usually touch most components at some point, but most don't understand the weaknesses at the integration points and develop apps that cause chaos in production.

as a developer myself i start each project with the mindset that i will be the person causing problems. i need to know how much load other systems can take before i use them, eg messages per second thru a mail server. i need to know how to deal with the mail server running slowly as opposed to the mail server being unavailable, my code needs to know how to fail quickly and not tie up valuable resources. i imagine we've all worked on projects where request threads have been exhausted waiting for another resource ;-)

Re: Why do so many technology projects fail?

Are you saying that "application designer" is more applicable than "application architect"?

As Kevin says, I don't think you necessarily *need* to know everything down to the lowest level of detail. On anything other than very small projects, there's no way that this is viable, so you need to delegate where appropriate (e.g. where somebody has a much higher level of expertise). Having said this, you're right, you do need to consider the impact that your application will have on the broader infrastructure. That, in my opinion, falls squarely underneath the role of an architect.

Re: Why do so many technology projects fail?

No argument here - technical authority has to be based on something more than just vacuous claims! Sometimes experience will suffice. Sometimes exhaustive testing is needed. Sometimes neither is possible so someone else's experience or testing has to be relied upon.

Architects can be bad at what they do just like everyone else. That doesn't necessarily define what an architect is, more how they do it.

Re: Why do so many technology projects fail?

Regarding all the above: 1. An architect must think about non-functional requirements, and this will include playing nice with other applications, securing the application perimeter etc. 2. I don't trust accreditations. The industry is fast moving and being accredited once doesn't mean you still have the necessary abilities 10 years later. 3. An architect shouldn't specify all components, but should specify quasi-completely their behavior in terms of contract and non-functional requirements. It's also his job to make sure that his specifications are fulfilled.

Re: Why do so many technology projects fail?

This title-related discussion is becoming really boring. When can you start referring to yourself as an architect or not? What is an architect vs. application designer vs. anything else? Who cares?! I mean, what really matters is not who you are, it's what you can do. The architect who drew my house and the one who designed the Millau Viaduct are both architects in that they don't build themselves. They just know enough about materials and structures and so on to actually design things, think of it before they exist.

The main problem I see in many projects is that they tend to start with an architect that is not cut for the job (like a house-drawer for the Millau Viaduct). But what's even worse is when they start with an "overkiller", making things more complex than necessary, overbudgetting everything. My point is: we need pragmatism. We should stop worrying about titles and big unrealistic architectures. Let's focus on getting back to the ground. I think that's the whole point of the philosophy behind Agile Methodologies. But apparently we are still entangled in big discussions about "whose one is bigger?"

Re: Why do so many technology projects fail?

I agree about the title discussion, but it's still important to be able to differentiate between skill sets so that you know you can properly define responsibilities.

Pragmatism is indeed the key and projects need hands-on architects that understand what they are working with rather than industry analysts (to cite a previous comment). The big question, though, is how do we re-educate our industry, our managers, the business sponsors, etc so they understand this. So many projects could be improved by hiring the right team.

Re: Why do so many technology projects fail?

I think that one of the key aspects to re-educate all the actors is to show them the article from the Times, and others like it. IT project failures seem to be easier to hide, so the more we dig up those failures, the more we draw lessons from them, the more lucid we are and the saner the market will be. But then again we're facing some business and political issues here.

So another approach would be to convince all the actors to try out a different, more pragmatic approach, and show them how it's more efficient and less error-prone. And yes, we will lose money on the short-term because the contracts we sign will involve less money. But it's a long-term shot. Even in IT projects, less is more.


Add a comment Send a TrackBack
Software architecture for developers