Why don't some teams think about architecture?
I ran a half-day workshop yesterday at the Software Architecture Summit in Berlin, where we crowdsourced the answers to a number of questions, focussed around why some agile software teams don't think about architecture and how we can change this situation. Of course, much of what we discussed can be applied to *any* team, but the focus was agile here.
1. What is architecture?
2. What is agile?
3. Is architecture important?
4. Why don't some agile teams think about architecture?
5. How can we reintroduce agile teams to architecture?
You can view my slides online. Thanks to everybody that came along, I really enjoyed it.
Most teams don't ... but they should
Most of the roles that we associate with software development teams are relatively well understood ... developers, testers, ScrumMasters, Product Owners, business analysts, project managers, etc. The software architecture role? Not so much. I regularly ask software teams whether they have a defined terms of reference for the software architecture role and the usual answer is along the lines of "no" or "yes, but we don't use it". Often people working for the *same team* will answer the question differently.
Although the need for thinking about software architecture is usually acknowledged, the responsibilities of the software architecture role often aren't clear. In my experience, this can lead to a situation where there is nobody undertaking the role, or where somebody is assigned the role but doesn't really understand how they should undertake it. If the role isn't understood, it's not going to get done and we have little hope of growing the software architects of tomorrow.
Regardless of what you call it (e.g. architect, tech lead, etc), my advice is simple. If you don't have a shared understanding of the role, start by doing this for the role within your team. Here's a simple starting point for what the role could look like. The role is also described in my book and here on InfoQ.com.
Another approach is to define it collaboratively using a workshop or game storming techniques. Here are some photos from a tutorial I did at QCcon London 2012 called "Are you a software architect?".
Make sure that you base any role definition on your own context to ground the role in reality. Then, once you understand what the role should look like on a team, you can start thinking about whether to achieve consistency across your organisation. This may or may not be necessary/desirable.
Creating a shared understanding of the role will help to avoid questions like this and situations where software architecture falls between the gaps.
A fantastic week of software architecture in Minneapolis
I attended the SATURN 2013 conference in Minneapolis a couple of weeks ago, which is an annual practitioner conference about software architecture, organised by the Software Engineering Institute (SEI) of Carnegie Mellon University. The conference is in its 9th year now, but it's not something that's been on my radar to be honest, partly because it's in the US and partly because I've not heard much about it in the past.
SATURN is definitely not your typical hardcore developer conference, although there were a few talks that focussed on technology and code. The majority of the talks were a mix of how software architecture is being applied in the real-world alongside others that were more research focussed. This is unsurprising since the conference is organised by the SEI but the even the research talks were grounded in industry. I didn't really appreciate how much industry work the SEI actually did until I met some of the team. SATURN is certainly not an academic conference ... I don't remember hearing about missile control systems at a software conference in the past!
The keynotes by Stephan Murer, Scott Berkun, Mary Poppendieck and Philippe Kruchten were excellent, as were Mary's and Philippe's tutorials that I attended on the Friday. If you want to see what sort of things were talked about at SATURN 2013, you can download all of the slides. Here are the links to view/download the slides and photos from my own sessions:
- Effective software architecture sketches
- Photos from the sketching workshop
- The Conflict Between Agile and Architecture - Myth or Reality?
It was great to catch-up with old friends and meet many new ones. The conference was fantastic as it was ... but then, during the closing session, I was surprised and honoured to win the IEEE Software sponsored SATURN 2013 "Architecture in Practice" Presentation Award for my talk about the conflict between agile and architecture!
Talk about the "icing on the cake". Once again, a huge thank you to everybody that voted. :-)
A number of people have recently asked me whether there was somewhere they could chat about software architecture after attending my Software architecture for developers training course, so I've created a Google Group. If you want to discuss software architecture, please feel free to join us too.
If you're working in an agile software development team at the moment, take a look around at your environment. Whether it's physical or virtual, there's likely to be a story wall or Kanban board visualising the work yet to be started, in progress and done. Visualising your software development process is a fantastic way to introduce transparency because anybody can see, at a glance, a high-level snapshot of the current progress. As an industry, we've become pretty adept at visualising our software development process over the past few years although it seems we've forgotten how to visualise the actual software that we're building. I'm not just referring to post-project documentation, this also includes communication during the software development process. Agility is about moving fast and this requires good communication, but it's surprising that many teams struggle to effectively communicate the design of their software.
Read the rest of the article over at InfoQ.com.
Bringing software architecture back into the domain of the development team
One of the things I'm currently doing with a number of software teams is teaching them how to draw pictures. As an industry we've got really good at visualising the way that we work using things like Kanban boards and story walls, but we've forgotten how to visualise the software that we're building. In a nutshell, many teams are trying to move fast but they struggle to create a shared vision that the whole team can work from, which ultimately slows them down. And few people use UML nowadays, which just exaggerates the problem. I've written an article about this and it's due for publication soon (I'll come back and add a link) plus it's covered in my Software Architecture for Developers ebook and in a number of talks that I'm doing around Europe (ITARC, IASA UK, Mix-IT) and the US (SATURN) during April. Here are the slides from Agile software architecture sketches - NoUML! that I presented a few weeks ago in Dublin.
The TL;DR version
The TL;DR version of this post is simply this ... if you're building monolithic software systems but think of them as being made up of a number of smaller components, ensure that your codebase reflects this. Consider organising your code by component (rather than by layer or feature) to make the mapping between software architecture and code explicit. If it's hard to explain the structure of your software system, change it.
Decomposition into components
For the purpose of this post, let's assume visualising a software system isn't a problem and that you're sketching some ideas related to the software architecture for a new system you've been tasked to build. An important aspect of "just enough" software architecture is to understand how the significant elements of a software system fit together. For me, this means going down to the level of components, services or modules. It's worth stressing this isn't about understanding low-level implementation details, it's about performing an initial level of decomposition. The Wikipedia page for Component based development has a good summary, but essentially a component might be something like a risk calculator, audit logger, report generator, data importer, etc. The simplest way to think about a component is that it's a set of related behaviours behind an interface, which may be implemented using one or more collaborating classes. Good components share a number of characteristics with good classes. They should have high cohesion, low coupling, a well-defined public interface, good encapsulation, etc.
There are a number of benefits to thinking about a software system in terms of components, but essentially it allows us to think and talk about the software as a small number of high-level abstractions rather than the hundreds and thousands of individual classes that make up most enterprise systems. The photo below shows a typical component diagram produced during the training classes we run. Groups are asked to design a simple financial risk system that needs to pull in some data, perform some calculations and generate an Excel report as the output.
This sketch includes the major components you would expect to see for a system that is importing data, performing risk calculations and generating a report. These components provide us with a framework for partitioning the behaviour within the boundary of our system and it should be relatively easy to trace the major use cases/user stories across them. This is a really useful starting point for the software development process and can help to create a shared vision that the team can work towards. But it's also very dangerous at the same time. Without technology choices (or options), this diagram looks like the sort of thing an ivory tower architect might produce and it can seem very "conceptual" for many people with a technical background.
Talk about components, write classes
People generally understand the benefit of thinking about software as higher level building blocks and you'll often hear people talking in terms of components when they're having architecture discussions. This often isn't reflected in the codebase though. Take a look at your own codebase. Can you clearly see components or does your codebase reflect some other structure? When you open up a codebase, it will often reflect some other structure due to the organisation of the code. Mark Needham has a great post called Coding: Packaging by vertical slice that talks about one approach to code organisation and a Google search for "package by feature vs package by layer" will throw up lots of other discussions on the same topic. The mapping between the architectural view of a software system and the code are often very different. This is sometimes why you'll see people ignore architecture diagrams (or documentation) and say "the code is the only single point of truth".
Auto-generating architecture diagrams
To change tack slightly, I was in Dublin a few weeks ago and I met Chris Chedgey, who is part of the inspiration behind this post. Chris is the co-founder of a company called Headway Software and they have a product called Structure101. You should take a look if you've not seen it before, they have some cool stuff in the pipeline. I won't do their product any justice by trying to summarise what it does, but one of its many features is to visualise and understand an existing codebase.
When I teach people how to visualise their software systems, we create a number of simple NoUML sketches at different levels of abstraction. These are the context, containers and components diagrams. This context, containers and components approach is basically just a tree structure. A system is made up of containers (e.g. a web server, application server, database, etc), each of which is further made up of components. You can see some example diagrams on Flickr and in my book.
Given this is really just a tree structure, it should be fairly straightforward to auto-generate these diagrams from an existing codebase. And perhaps there is a tool out there that can do this, but I've never seen one that has worked really well. Microsoft Visual Studio can generate some layer diagrams but I've never met anybody that really raves about the architecture diagram support. Most tools generate diagrams showing dependencies between packages or classes but they don't tend to show components. And what's a component anyway? Is any class that implements an interface a component? If you're using inversion of control, perhaps everything that you inject is a component?
There are a number of reasons why auto-generating such diagrams is tricky but, once we start coding, much of the semantics associated with "containers" (runtime environments, process boundaries, etc) and "components" becomes lost of the sea of classes that make up the typical codebase. Many developers break their systems up into a number of projects within their IDEs to represent reusable libraries and deployable units but external tools often don't have access to this information if they are solely working from a bunch of JAR files or DLLs (for example). In essence, the information related to the abstract structural elements isn't adequately represented within a codebase. If you take a look at most codebases, I'm fairly sure that you could come up with a set of rules as to what defines a component but perhaps it would be easier to simply make these concepts explicit. Some techniques already exist to do this (e.g. the Architecture Description Language) but I've never seen them used in the corporate world.
Packaging by component
To bring this discussion back to code, the organisation of the codebase can really help or hinder here. Organising a codebase by layer makes it easy to see the overall structure of the software but there are trade-offs. For example, you need to delve inside multiple layers (e.g. packages, namespaces, etc) in order to make a change to a feature or user story. Also, many codebases end up looking eerily similar given the fairly standard approach to layering within enterprise systems. Uncle Bob Martin says that if you're looking at a codebase, it should scream something about the business domain. Organising your code by feature rather than by layer gives you this, but again there are trade-offs. A variation I've been experimenting with is organising code explicitly by component. The following screenshot shows an example of this in the codebase for my techtribes.je website (a content aggregator and portal for Jersey's digital sector). This screenshot only shows the core components; there's a separate Spring MVC project and the controllers use the components illustrated here.
This is similar to packaging by feature, but it's more akin to the "micro services" that Mark Needham talks about in his blog post. Each sub-package of je.techtribes.component houses a separate component, complete with it's own internal layering and Spring configuration. As far as possible, all of the internals are package scoped. You could potentially pull each component out and put it in it's own project or source code repository to be versioned separately. This approach will likely seem familiar to you if you're building something that has a very explicit loosely coupled architecture such as a distributed messaging system made up of loosely coupled components. I'm fairly confident that most people are still building something more monolithic in nature though, despite thinking about their system in terms of components. I've certainly packaged *parts* of monolithic codebases using a similar approach in the past but it's tended to be fairly ad hoc. Let's be honest, organising code into packages isn't something that gets a lot of brain-time, particularly given the refactoring tools that we have at our disposal. Organising code by component lets you explicitly reflect the concept of "a component" from the architecture into the codebase. If your software architecture diagram screams something about your business domain (and it should), this will be reflected in your codebase too.
The structural elements of software
We could create a convention here to say that all sub-packages of je.techtribes.component are components, but it would be much easier to explicitly mark components using metadata. In Java, we could use annotations to do this, attributes in .NET, etc. If we used the same approach for other structural elements of software (e.g. services, layers, containers, etc), tool vendors could use this metadata to generate meaningful and *simple* architecture diagrams automatically. Plus, they could also use this structural information to generate dependency diagrams that focus on components rather than classes. I've started experimenting with annotations as a way to do this and I've created a Github repo to store whatever I come up with.
The major caveat to all of this is that designing a software system based around components isn't "the only way". It's a nice approach to think about software systems that are more monolithic in nature and it's a great stepping stone to designing loosely coupled architectures. But it isn't a silver bullet. Regardless of how you design software, I do hope this post has got you thinking about the mapping between software architecture and how it's reflected in the code.
Software architecture and coding are often seen as mutually exclusive disciplines and there's often very little mapping from the architecture into the code and back again. Effectively and efficiently visualising a software architecture can help to create a good shared vision within the team, which can help it go faster. Having a simple and explicit mapping from the architecture to the code can help even further, particularly when you start looking at collaborative design and collective code ownership. Furthermore, it helps bring software architecture firmly back into the domain of the development team, which is ultimately where it belongs.
Just a quick note to say that I've added some new essays to my Software Architecture for Developers book. They are:
- Software architects should be master builders: this talks about the importance of software architects needing deep technology skills and working as a part of the team.
- Technology is not an implementation detail: this is about why you shouldn't leave technology decisions as a after-thought.
- The need for sketches: this explains the importance of sketches in communicating the overall software architecture, both inside and outside of the immediate team.
- Would you code it that way?: this talks about the key question you need to ask yourself whenever you're designing software and visualising that design through sketches.
Last week I spoke at QConLondon 2013 on the topic of "Modern Legacy Systems". Here are the slides and I'll update with a link to the video when QCon makes it available.
Thank you to everyone that attended and I had a couple of great conversations afterwards about the topic. I even had someone tell me that they were the 'other side' of one of the anecdotes I told - they were on the better of the two sides! The auditorium I spoke in (Churchill) was HUGE and had a maximum capacity of 700. I had under the capacity but this was the largest venue I've spoken in and don't mind admitting that I found it intimidating! Fortunately I had watched Damien Conway's excellent Instantly Better Presentations talk earlier in the day and the tips came in handy. I would really advise anyone giving presentations to watch the video when it is available.
I attended many fascinating presentations and a couple that stand out most were around technology and art - visual problem solving. This was a really enjoyable but I'm struggling to find an immediate use for what I learned in my job!
Also worth a mention was Jesper Richter-Reichhelm's talk on Painful Success. This was a review of an architectural mistake that was made early in a project and its resolution. The problem came down to an incorrectly chosen architectural archetype which became evident when the system was under load. I think this is very common, particularly for on-line applications where it is assumed that a website architecture is suitable. What impressed me was Jesper's honesty about the issue, which is necessary in finding a solution. Many people find admitting these errors impossible and I think we can all learn from the good example.
Just a quick note to say thank you to everybody who came along to my workshop and/or talk at DevWeek 2013 last week. Here are links to the slides and photos.
- Slides from my "Effective architecture sketches" workshop
- Photos from my "Effective architecture sketches" workshop
- Slides from my "Software architecture for developers" talk
Documentation on software projects is often a contentious topic, particularly since the agile manifesto says that we should value working software over comprehensive documentation. My take on lightweight documentation has now been included in my Software Architecture for Developers book and I've had a stream of e-mail requests since publishing the new version to ask whether I would create a sample software guidebook too.
A sample software guidebook
Yes is the answer ... and I've already started putting this together, based upon a side-project of mine called techtribes.je, which is basically a community portal for the IT, tech and digital industry in Jersey*. This is a fun little side-project rather than a huge high-performance distributed system, but it should be enough to illustrate the concepts in the book.
And some sample software architecture sketches
The sample guidebook will also include some software architecture diagrams, which will again serve as an example of the context, containers, components and classes sketches that I talk about. The following context diagram (included in the sample guidebook) provides an overview of techtribes.je:
The introduction and context sections of the sample guidebook are available in the published version of my book now and I'll be adding the other sections over the next few days.
Plus some open source code
As a final note, I'm planning to open-source the code behind techtribes.je on GitHub so that you can see the correlation between the code and the documentation. If there's anything else that you'd like to see in the book, please just drop me a line.
* just to avoid any confusion, I'm referring to Jersey in the Channel Islands, not New Jersey! :-)