Re-evaluating software architecture

Why don't we just get on with the coding?

Somebody asked a question at my recent BCS session that got me thinking about software architecture and whether we need it.

Why do we need this new software architecture stuff? We've managed fine until now and projects *have* been successful.

Some of this has been talked about recently, but of course it *is* possible to have successful software projects without architecture. Not every system needs a heavy dose of architectural input; particularly when you have small systems, simple requirements, luck or naivety on your side. ;-)

Do we really need software architecture?

But just because a team has never undertaken any software architecture doesn't mean that they'll never need to. It's definitely worth questioning the purpose of software architecture, particularly in our financially challenged world with IT sponsors wanting more for their money. Myself and the others on the site talk a lot about our experiences of software development from an architectural perspective, but these are just *our* experiences. Many of the processes, techniques and advice works well for us, but it's always worth asking yourself whether it will work for you too. What value will it bring? How will it make your project team more effective? How will it make the project more successful?

Questions like the one I was asked at my BCS session always force me to stand back in order to understand another point of view. Sometimes, a lot of what I write about seems common sense to me, which again makes me stand back and make sure that I'm not overstating the obvious. But everybody is different, and one person's common sense can be another's revelation, and vice versa. Sometimes I often question whether I should just get on with coding up the system and to stop worrying about all these other things. But then something happens that puts everything back into perspective again.

One of the projects that I've done some work on recently does have an architecture that has been explicitly defined and it's in pretty good shape. It has separate tiers with their own responsibilities, some architectural principles are in place to ensure consistency and the team have taken a good approach to quality through a comprehensive set of automated tests. The problem is this ... none of it has been formalised or shared effectively.

Is the code the architecture?

Many agile proponents will say that the code is the best documentation you can have, because it's always up to date and represents reality. I often joke about the code being the architecture, but architecture actually sits at a higher level of abstraction than the code. To put this into context, I can look at a block of code and understand what it does, but I have to look across *many* similar blocks of code to understand the underlying architectural principles behind it all.

Architecture introduces structure and guidelines

In my talks and training about software architecture, I say that architecture is about introducing structure and guidelines to a software project, and I've come to realise just how true this is. Without formalising these good ideas, a codebase with some otherwise great ideas can appear inconsistent, with the overall system lacking coherence and clarity. While this lack of consistency can make it harder for the project team to deliver a high quality system, it makes it really hard for members that are new to the team. Without formalising all of this good work and explicit architectural thinking, it's difficult for new team members to understand what's going on, how the system works and what's expected of them (i.e. writing architecturally compliant code). In the worst case scenario, this can introduce a whole host of problems around areas such as performance, scalability, security, etc.

The answer is easy ... just stick a stake in the ground, formalise your decisions, strive for consistency and share the architecture. The benefits from software architecture are worth the effort, and even a little architecture can go a long way.



Re: Re-evaluating software architecture

Hi,

"Is the code the architecture?" Behind this question is another one: "Does the code tell me everything I need to know?". I would say in most instances, no it doesn't. But I don't know if this justifies the need for "Architecture" as something seperate. Architecture is one of those "made up words" as far as software is concerned that eludes precise definition. Lets assume that the architecture is the highest level concrete design of your system that eludes capture and represention using your chosen programming language.

One solution is to change your language to one that does allow you to explicitly define higher level "architectural" abstractions.

Newspeak a new language from Gilad Bracha of "Java Language Specification" fame that allows you to do just this. It does this by removing global namespaces and replacing them with explicit objects that act as higher level namespaces. These abstractions can be used to model libraries, modules, architectural layers, etc directly in code:

http://blog.3plus4.org/2008/12/04/a-taste-of-nested-classes-part-1/

So with Newspeak you can plug high level modules together explicitly, just like lego blocks. So the "architecture" is represented in the code.

I think the original question "why do we need this architecture stuff?" Is very valid. If architecture is just high level design, and you can make explicit high level design decisions in your code (like you can with Newspeak). Then architecturing is just coding.

Even without a language like Newspeak, architecturing can still be coding, but code annotated with comments maybe? Or even an "architectural document" as an appendum perhaps? When Thompson and Ritchie sat down to write Unix , did they see themselves as Architects or Programmers? Maybe for some architecturing is just one aspect of programming. Just as listening, testing, coding, communicating etc are just other aspects. But they are all part and parcel of the same thing.

http://pab-data.blogspot.com/2008/10/comprehending-large-software-systems.html

Perhaps it was this point of view that lay behind the question? "Why do we need to separate architecturing out from everything else?"

Food for thought. Paul.

Re: Re-evaluating software architecture

Hi Simon, Kevin,

I feel compelled to comment again. Reading your blog, it appears that you guys understand too well the difficulties that dissecting the "programming role" can lead to.

Reading your articles on "Conways law" and "minding the gap", I find myself in strong agreement with your diagnosis of the problems that teams face. What troubles me though is that whilst you guys have identified organisational structure as a route cause of many problems, you choose only to offer more of the same but only "done better" as an antidote.

If Conways law says that architecture follows team structure, then a team where each individual does not have an appreciation of the whole will create a design where the separate parts are implemented inconsistently. Where false assumptions amongst team members can lead to design flaws at the macro level.

If there is a gap between people tasked with thinking about the significant design decisions and the design rules that govern the whole, and the people focused on the details and the design rules the govern the parts then the answer is clear. Everyone needs to consider everything, the whole and the parts at the same time.

The gap is artificial, and the "conway" effect can be made to disappear. All that is needed is for the team to act as one. In fact all that is needed is for the team to truly acts as a team. If the team acted as one person, one brain with many hands and feet, then the need for an Architect would disappear. Everyone would share the common role of developer. This is the alternative point of view that lay behind the question "Why do we need this architecture stuff?".

If you take a look at the original paper by Winston Royce you will see that he made a number of astute observations about software development. Yet the ideas of his that were put into practice under the label "Water fall" development, were the ideas that were consistent with a Taylorist management mindset.

A lot of what Winston said is the same as what many Agile people are saying today, but these ideas were dropped because they challenged the cultures of the organisations seeking to adopt "best practice". So what are we left with? "Current practice". For may their current practice defines a number of separate roles including an Architect. This approach can work, but it is not the only way.

In the same way that Henry Fords approach to motor car manufacture does work, but is not the only way. In fact we can see Henry Fords approach struggling to compete in the face of Taiichi Ohno and Toyota's approach to motor manufacture.

This is what makes me uncomfortable. You guys talk about architecture and the role of an architect as if it is a given fact of software development, in the same way that bits and bytes are a given fact. You make no mention that such roles are a choice, rendered necessary based on other choices, and what Tim Lister chooses to call "Teamacide" in his book Peopleware.

True given a certain culture than there is much that can be done, to make desperate "roles" work, but I believe you owe it to your readers to explain that everything you say is premised on certain organisational choices that are just that choices.

There are alternatives. Paul.

Re: Re-evaluating software architecture

What a great comment, Paul!

We need to ensure the "architectural requirements" are met in the same way Toyota must ensure their cars are roadworthy, profitable and marketable. How we achieve this is a matter for debate but I'd certainly hope that we'll continue to evolve new techniques.

Your statement that the "Conway effect" can be made to disappear by (essentially) changing the way the team communicates is paradoxical: the effect you mention mimics the communication structure of the team and will necessarily change. This is why I suggest that you can influence the emergent design by influencing the team.

While I appreciate the utopian situation wherein "everyone considers everything" [about the whole], I can't see it being achievable across domains (eg, knowing everything about equity swaps and JavaEE development). Nonetheless, I hope we're doing our part by trying to make developers more architecturally aware and therefore more appreciative of the existence of the whole, if not all of its intricacies!

You're right that we do talk about architecture as a given. Indeed, all systems have an architecture insofar as they may have portions that are costly to change and requirements of a non-functional nature, regardless of whether someone has taken the time to think about them. How these are dealt with is indeed a choice. It's this choice that we try to shed some light on based on our experiences and opinions.

Nonetheless, if organisational or process change is part of a suitable approach to meeting your needs then I'd recommend it!

Empower your people

Hi Kevin,

I believe we have the same goal: Better Software. The alternative I am suggesting is a more holistic approach; an approach that Ritchie and Thompson would recognise. It starts with the assumption that programmers are highly skilled creative professionals with the ability to address both the big picture and the details at the same time. Such programmers are a cut above what we have comes to know as ‘coders’, who have a much more limited outlook.

This expansive role of "Software Creator" becomes even more challenging in a team setting. If such a context the programmer needs to augment his technical skills with ‘soft’ skills such as communication and collaboration. This explains why programming is so hard, and why we often fail. There is no silver bullet, and It takes a multifaceted individual to program well. To create such individuals requires a great deal of investment and mentoring as common in other highly skilled professions. Team members may still have differing strengths (e.g. J2EE or credit swaps domain knowledge, etc), but at a minimum, they must poses the ability to collaborate well in a team setting.

This type of collaboration is a soft skill that can be taught by example. Teams that value these skills and take the time to develop them, gain the ability to act seamlessly as one, giving the appearance of being as strong as their strongest individual in any one area of expertise.

The result is Software that is modeled, architected, designed and coded as though it was produced by one mind. Through practices like common code ownership, pair programming, pair rotation, small teams, and high bandwidth face-to-face communication, a seamless solution can be created where the Conway effect is visible through its absence (or near absence). So a trait of a team operating well in this mode is a Conway effect that is hard to detect. A sure sign that the team is effectively acting in unison.

Extreme Programming goes into some detail on how to achieve this. In such a context many of the problems you describe in earlier articles are avoided, and since the problems do not arise, there is no need for specific roles or activities to resolve them.

Thanks for the opportunity to share this alternative point of view with your readers. Paul.

Re: Re-evaluating software architecture

Hey Simon, see my comments here. Cheers.

Re: Re-evaluating software architecture

What is architecture and design? Merely how code fits together at different levels of abstraction or viewpoint. ... but its not an industry in its own right.
I agree, I don't think that software architecture *is* an industry in it's own right, but calling it merely a different level of abstraction is an oversimplification. Architecture and coding do go hand-in-hand and it doesn't matter who does it. What's important is that one or more people undertake the role so that the bigger picture is explicitly thought about.
If you are a programmer and don't understand these things, you need to hit the classroom or buy some good books.
You've hit the nail on the head, and this is exactly why the architecture stuff doesn't get done on some (most?) software projects. This is also why we started the site ... to get more people thinking about the architecture aspects of software development.

Re: Re-evaluating software architecture

Simon, I agree with much of what you say, but what do you mean by 'bigger picture'? From my perspective, I think I need some concrete examples.

Re: Re-evaluating software architecture

By "bigger picture" I mean aspects such as the non-functional requirements (ensuring the solution meets the performance/scalability/security/availability/etc goals), technology selection (choosing technologies that are right for the job, compatible, etc), quality assurance, technical leadership, integration of the architecture with other teams in the organisation (DBAs, operations/support, etc), architecture constraints (e.g. choice of platform, interfaces, etc) and so on. This all needs to be done and some of the "bigger picture" has a massive influence on the design of the overall solution. As I said, provided it's all done then that's fine. My experience suggests that this isn't the case though.

Re: Re-evaluating software architecture

One might turn the question on its head: when does coding stop and design begin? Architecture? Business analysis ? Programme management? To (correctly) assert that you can't define the boundaries doesn't mean that you can move freely from one to the other. There is a point where your skills (whatever they may be) aren't best suited to the tasks for which you are responsible. So how big (and blurry) must the picture get before it's outside your core skills?

As a concrete example, I'd offer up creating the five-year technical strategy for developing an internet platform that had to be resold to various clients at differing levels of cost and service level. I spend most my time coding, but what was my role when I helped conceive the strategy, checked it was viable, looked at the costs? I think my responsibilities were sufficiently far from coding to be considered outside.

An example that might be closer to the boundary is whether to use distributed transactions or corrective actions. It clearly has immediate impact on development activities and is understood by most (good/experienced) developers. Whether an "architect" makes this design choice or the development team sit in their beanbag room to make it doesn't really matter: it's still architectural*.

* - dependent on context!

Re: Re-evaluating software architecture

Kevin, good comments. The things you mentioned (architecture, business analysis and program management) are all part of the every day activities that developers need to do. Good developers don't just write code all day long, they need to talk to customers, plan their work etc. Doing these things gives them an incredible amount of context and understanding about the business problem they are trying to solve. Moreover, I don't see a developer's job as being just 'coding' I see it as producing a solution to a business problem and a part of that will be to consider costs, service levels etc. The main theme that I see emerging is that the architect is responsible for more of the non-functional/technology selection decisions. I still don't see why this is different. Whether to use distributed or corrective transactions - isn't that a decision the development team should make, based on need? Are you looking for a single decision maker? Someone who has more experience? Do you think that teams don't understand the difference? I'm really not trying to be awkward, just trying to understand :) Thanks

Re: Re-evaluating software architecture

The idea that a developer does a "bit of programme management" might come as a blow to the egos of CTOs with decades of experience who are in charge of millions of pounds of IT spend :P

The example of distributed transactions is a good one: were the failure modes specified in the requirements? Did anyone even notice if they weren't? Will you really be able to unwind an incorrect decision when you (later) discover what your real need is? Of course, the answer depends on the context. Some teams will sail through such a change, others will implement a point solution, others will say the problem can't be fixed, some will have guessed right, some will have planned right, some won't even know what's going on, etc. If the decision is wrong, some companies won't even notice, others will see their customer base evaporate after an embarrassing failure.

So, as you might have guessed, the answer to your questions is, "it depends". One team I've worked with didn't even care about a single transaction. Others could tell me more about XA than I cared to know. It's easy to say that the former team should know more, but since their day rate was less than 20% of the latter you can see how your utopia doesn't necessarily overlap other stakeholders'.

We're certainly not advocating that developers merely code (what's wrong with being specialised?), quite the reverse: we want developers to be architecturally aware. You can extend this to business aware and domain aware too - that's simply not our main area of influence (although we do our best to learn as much as we can).

Like you, we'd like to see teams that see these decisions coming, take responsibility for them and make them based on real information. However, our experience is that teams aren't like this, at least not always at a justifiable cost. Agile's cross-functional team should not be taken to mean a group of equally skilled team members, but a team whose collective skillset can meet the demands of the project.

The back to basics alternative

Hi Kevin,

I believe we have the same goal: Better Software. The alternative I am suggesting is a more holistic approach; an approach that Ritchie and Thompson of Unix fame would recognise. It starts with the assumption that programmers are highly skilled creative professionals with the ability to address both the big picture and the details at the same time. Such programmers are a cut above what we have comes to know as ‘coders’, who have a much more limited outlook.

This expansive role of "Software Creator" becomes even more challenging in a team context. In this setting, the programmer needs to augment his technical skills with ‘soft’ skills such as communication and collaboration. This explains why programming is so difficult, and why we often fail. There is no silver bullet, and it takes a multifaceted individual to program well. Producing such individuals requires investment and mentoring as is common in other highly skilled professions. Team members may still have differing strengths (e.g. J2EE or credit swaps domain knowledge, etc), but they must poses, at minimum, the ability to collaborate well in a team.

This type of collaboration is a soft skill that can be learned by example. Teams that value this skill and take the time to develop it, gain the ability to act seamlessly as one, giving the appearance of being as strong as their strongest link in any one area of expertise.

The result is Software that is modelled, architected, designed and coded as though it was produced by one mind. Through practices like common code ownership, pair programming, pair rotation, small teams, and high bandwidth face-to-face communication, a seamless solution can be created where the Conway effect is visible through its absence (or near absence). So a trait of a team operating well in this mode is a Conway effect that is hard to detect. A sure sign that the team is effectively acting in unison.

Extreme Programming goes into some detail on how to achieve this. In such a context many of the problems you describe in earlier articles are avoided, and since the problems do not arise, there is no need for specific roles or activities to resolve them.

Thanks for the opportunity to share this alternative point of view with your readers. I, and many others think we need to stop the proliferation of roles and get back to basics. Paul.

The back to basics alternative

Hi Paul, I certainly agree that we should empower teams so that they can collaborate and create software as though the team was a single mind, but I think that there are some steps that need to be taken before this can become a common reality.
It starts with the assumption that programmers are highly skilled creative professionals with the ability to address both the big picture and the details at the same time. Such programmers are a cut above what we have comes to know as ‘coders’, who have a much more limited outlook.
My experience of software development suggests that very few people address both the big picture and the details at the same time. As Kevin has said in another comment, some people are more able to do this than others, but ultimately I don't think that many people are pushed to do this on a day-to-day basis. I'd like to go further back to basics and get at least one person on the team addressing the big picture and the details. In essence, I'd like to get software teams doing more "architecture" than they generally do now. Once that's all working successfully, then they can push the rest of the team to start doing the same so that the empowerment and collaboration can follow. I think that this catalyst is necessary for teams to become collectively more architecturally aware.

Re: Re-evaluating software architecture

Kevin, I think we're pretty much in agreement. Developers and architects are really the same thing, its just that there are good and bad/more knowledgeable and less knowledgeable.

There are elements of development that you generally refer to as architectural and I agree that they need to be addressed, I am just not convinced I would refer to them as architectural. After all, it all comes down to code executing on a machine, whether your input in functional or non-functional is beside the point.

I think I now understand your viewpoint - you are experienced developers helping teams less experienced. From this perspective, it all makes sense.

Re: Re-evaluating software architecture

I'm not sure I'd go so far as to say that they're the same thing because that might give the impression that architectural activities and development activities were the same or required the same skills.

There are people who can undertake all these activities, others who are only suited to one type and others suited to none.

My worry is a team that becomes an XP steamroller and ends up thinking that everything is merely an extension of development because "it all ends up as code". This is clearly not the case, and I only have to look around at all the people crucial to my team who aren't developers to see it.

Re: Re-evaluating software architecture

Hi Simon,
In essence, I'd like to get software teams doing more "architecture" than they generally do now. Once that's all working successfully, then they can push the rest of the team to start doing the same so that the empowerment and collaboration can follow. I think that this catalyst is necessary for teams to become collectively more architecturally aware.
I agree. I would call this person a Coach and a Mentor and they would cover other aspects of programming too, like how to collaborate effectively, how to listen to customers and how to do testing. Its not a new idea. Other highly skilled professions provide a long period of mentorship for would be professionals before they are deemed qualified. Like with Doctors and lawyers. Paul.

Re: Re-evaluating software architecture

Hi Kevin,
I'm not sure I'd go so far as to say that they're the same thing because that might give the impression that architectural activities and development activities were the same or required the same skills.
Really? I wouldn't say a programmer was fully qualified to program unsupervised if he didn't have a good appreciation of architectural concerns. Why? Well if he hit an issue that was architecturally significant, he wouldn't know. So architecture is part and parcel of programming IMO.
My worry is a team that becomes an XP steamroller and ends up thinking that everything is merely an extension of development because "it all ends up as code".
Whether you call it a steam roller or not, it does all end up as code. At the highest level of abstraction there are only two roles: people who commission software and people who produce it.

A great number of very successful systems have been produced by teams with just these two roles. In fact the most successful systems have been produced when these roles have been combined into single role: customer/developer. Unix is a good example of this, whether we talk about Thompson and Ritche, Bill Joy or Linus Torvald. Emacs and Richard Stallman is yet another example.

There is a reason. The more roles, the greater the number of hand-offs and the higher the communication overhead. With fewer roles, there is less scope for misunderstanding and shorter feedback paths. With only one role, feedback is instantaneous.

Paul.

Re: Re-evaluating software architecture

This is quickly degenerating into the sort of agile discussion I seem to have down the pub on a worryingly regular basis! It's not so fun sober.

I wouldn't say a programmer was fully qualified to program unsupervised if he didn't have a good appreciation of architectural concerns.

You describe a world where the majority of current developers are no longer fit for employment. It's a worthy goal (really!). Unfortunately many projects and businesses rely on low-cost development or have incumbent teams that, for whatever reason, don't meet your criterion. That it might have been better for them in the long run will come as little comfort as their projects fail due to high cost or shortage of resources.

Simon's article was about achieving the sort of teams you describe. Rather than simply wishing they were better, he suggested "unqualified programmers" could be supervised until they were.

Re: Re-evaluating software architecture

Hi Kevin,

I actually agree with Simon on the problem. I'm just suggesting an alternative solution.

Alternative meaning that both solutions can work.

And given that most software teams do fail (over budget, over schedule, and/or less that desirable quality), then the the majority of teams evidently do fit my description of "not fully qualified".

I never said that most developers "aren't fit for employment", those are your words. I'm saying that they are under trained, and this is what we should focus on.

Calm down a bit Kevin. I'm not attacking you honestly :)

Thanks for the discussion.

Paul.

Re: Re-evaluating software architecture

I'm probably being stupid or overly semantic -- both are likely based on experience -- but I've failed to divine the suggested alternative solution to the problem (which we all seem to agree exists).

Re: Re-evaluating software architecture

Hi Kevin, I thought I described the alternative. Collaboration and Mentoring. I provide a lot more context to the alternative I am suggesting here. Paul.

Re: Re-evaluating software architecture

Then I have been overly semantic as I see that as the same solution that Simon usually advocates. Indeed, it's pretty much the raison d'etre for this site.

Apologies if I have misinterpreted your commentary as implying something else!


Add a comment Send a TrackBack