Coding the Architecture - sbrown

SATURN 2013

Fri, 17 May 2013 18:29:08 GMT

I attended the SATURN 2013 conference in Minneapolis a couple of weeks ago, which is an annual practitioner conference about software architecture, organised by the Software Engineering Institute (SEI) of Carnegie Mellon University. The conference is in its 9th year now, but it's not something that's been on my radar to be honest, partly because it's in the US and partly because I've not heard much about it in the past.

SATURN is definitely not your typical hardcore developer conference, although there were a few talks that focussed on technology and code. The majority of the talks were a mix of how software architecture is being applied in the real-world alongside others that were more research focussed. This is unsurprising since the conference is organised by the SEI but the even the research talks were grounded in industry. I didn't really appreciate how much industry work the SEI actually did until I met some of the team. SATURN is certainly not an academic conference ... I don't remember hearing about missile control systems at a software conference in the past!

The keynotes by Stephan Murer, Scott Berkun, Mary Poppendieck and Philippe Kruchten were excellent, as were Mary's and Philippe's tutorials that I attended on the Friday. If you want to see what sort of things were talked about at SATURN 2013, you can download all of the slides. Here are the links to view/download the slides and photos from my own sessions:

It was great to catch-up with old friends and meet many new ones. The conference was fantastic as it was ... but then, during the closing session, I was surprised and honoured to win the IEEE Software sponsored SATURN 2013 "Architecture in Practice" Presentation Award for my talk about the conflict between agile and architecture!

Talk about the "icing on the cake". Once again, a huge thank you to everybody that voted. :-)

Software architecture for developers Google Group

Fri, 17 May 2013 17:24:00 GMT

A number of people have recently asked me whether there was somewhere they could chat about software architecture after attending my Software architecture for developers training course, so I've created a Google Group. If you want to discuss software architecture, please feel free to join us too.

Agile Software Architecture Sketches and NoUML

Tue, 16 Apr 2013 21:12:00 GMT

If you're working in an agile software development team at the moment, take a look around at your environment. Whether it's physical or virtual, there's likely to be a story wall or Kanban board visualising the work yet to be started, in progress and done. Visualising your software development process is a fantastic way to introduce transparency because anybody can see, at a glance, a high-level snapshot of the current progress. As an industry, we've become pretty adept at visualising our software development process over the past few years although it seems we've forgotten how to visualise the actual software that we're building. I'm not just referring to post-project documentation, this also includes communication during the software development process. Agility is about moving fast and this requires good communication, but it's surprising that many teams struggle to effectively communicate the design of their software.

Read the rest of the article over at InfoQ.com.

Mapping software architecture to code

Mon, 08 Apr 2013 10:27:59 GMT

One of the things I'm currently doing with a number of software teams is teaching them how to draw pictures. As an industry we've got really good at visualising the way that we work using things like Kanban boards and story walls, but we've forgotten how to visualise the software that we're building. In a nutshell, many teams are trying to move fast but they struggle to create a shared vision that the whole team can work from, which ultimately slows them down. And few people use UML nowadays, which just exaggerates the problem. I've written an article about this and it's due for publication soon (I'll come back and add a link) plus it's covered in my Software Architecture for Developers ebook and in a number of talks that I'm doing around Europe (ITARC, IASA UK, Mix-IT) and the US (SATURN) during April. Here are the slides from Agile software architecture sketches - NoUML! that I presented a few weeks ago in Dublin.

The TL;DR version

The TL;DR version of this post is simply this ... if you're building monolithic software systems but think of them as being made up of a number of smaller components, ensure that your codebase reflects this. Consider organising your code by component (rather than by layer or feature) to make the mapping between software architecture and code explicit. If it's hard to explain the structure of your software system, change it.

Decomposition into components

For the purpose of this post, let's assume visualising a software system isn't a problem and that you're sketching some ideas related to the software architecture for a new system you've been tasked to build. An important aspect of "just enough" software architecture is to understand how the significant elements of a software system fit together. For me, this means going down to the level of components, services or modules. It's worth stressing this isn't about understanding low-level implementation details, it's about performing an initial level of decomposition. The Wikipedia page for Component based development has a good summary, but essentially a component might be something like a risk calculator, audit logger, report generator, data importer, etc. The simplest way to think about a component is that it's a set of related behaviours behind an interface, which may be implemented using one or more collaborating classes. Good components share a number of characteristics with good classes. They should have high cohesion, low coupling, a well-defined public interface, good encapsulation, etc.

There are a number of benefits to thinking about a software system in terms of components, but essentially it allows us to think and talk about the software as a small number of high-level abstractions rather than the hundreds and thousands of individual classes that make up most enterprise systems. The photo below shows a typical component diagram produced during the training classes we run. Groups are asked to design a simple financial risk system that needs to pull in some data, perform some calculations and generate an Excel report as the output.

This sketch includes the major components you would expect to see for a system that is importing data, performing risk calculations and generating a report. These components provide us with a framework for partitioning the behaviour within the boundary of our system and it should be relatively easy to trace the major use cases/user stories across them. This is a really useful starting point for the software development process and can help to create a shared vision that the team can work towards. But it's also very dangerous at the same time. Without technology choices (or options), this diagram looks like the sort of thing an ivory tower architect might produce and it can seem very "conceptual" for many people with a technical background.

Talk about components, write classes

People generally understand the benefit of thinking about software as higher level building blocks and you'll often hear people talking in terms of components when they're having architecture discussions. This often isn't reflected in the codebase though. Take a look at your own codebase. Can you clearly see components or does your codebase reflect some other structure? When you open up a codebase, it will often reflect some other structure due to the organisation of the code. Mark Needham has a great post called Coding: Packaging by vertical slice that talks about one approach to code organisation and a Google search for "package by feature vs package by layer" will throw up lots of other discussions on the same topic. The mapping between the architectural view of a software system and the code are often very different. This is sometimes why you'll see people ignore architecture diagrams (or documentation) and say "the code is the only single point of truth".

Auto-generating architecture diagrams

To change tack slightly, I was in Dublin a few weeks ago and I met Chris Chedgey, who is part of the inspiration behind this post. Chris is the co-founder of a company called Headway Software and they have a product called Structure101. You should take a look if you've not seen it before, they have some cool stuff in the pipeline. I won't do their product any justice by trying to summarise what it does, but one of its many features is to visualise and understand an existing codebase.

When I teach people how to visualise their software systems, we create a number of simple NoUML sketches at different levels of abstraction. These are the context, containers and components diagrams. This context, containers and components approach is basically just a tree structure. A system is made up of containers (e.g. a web server, application server, database, etc), each of which is further made up of components. You can see some example diagrams on Flickr and in my book.

Given this is really just a tree structure, it should be fairly straightforward to auto-generate these diagrams from an existing codebase. And perhaps there is a tool out there that can do this, but I've never seen one that has worked really well. Microsoft Visual Studio can generate some layer diagrams but I've never met anybody that really raves about the architecture diagram support. Most tools generate diagrams showing dependencies between packages or classes but they don't tend to show components. And what's a component anyway? Is any class that implements an interface a component? If you're using inversion of control, perhaps everything that you inject is a component?

There are a number of reasons why auto-generating such diagrams is tricky but, once we start coding, much of the semantics associated with "containers" (runtime environments, process boundaries, etc) and "components" becomes lost of the sea of classes that make up the typical codebase. Many developers break their systems up into a number of projects within their IDEs to represent reusable libraries and deployable units but external tools often don't have access to this information if they are solely working from a bunch of JAR files or DLLs (for example). In essence, the information related to the abstract structural elements isn't adequately represented within a codebase. If you take a look at most codebases, I'm fairly sure that you could come up with a set of rules as to what defines a component but perhaps it would be easier to simply make these concepts explicit. Some techniques already exist to do this (e.g. the Architecture Description Language) but I've never seen them used in the corporate world.

Packaging by component

To bring this discussion back to code, the organisation of the codebase can really help or hinder here. Organising a codebase by layer makes it easy to see the overall structure of the software but there are trade-offs. For example, you need to delve inside multiple layers (e.g. packages, namespaces, etc) in order to make a change to a feature or user story. Also, many codebases end up looking eerily similar given the fairly standard approach to layering within enterprise systems. Uncle Bob Martin says that if you're looking at a codebase, it should scream something about the business domain. Organising your code by feature rather than by layer gives you this, but again there are trade-offs. A variation I've been experimenting with is organising code explicitly by component. The following screenshot shows an example of this in the codebase for my techtribes.je website (a content aggregator and portal for Jersey's digital sector). This screenshot only shows the core components; there's a separate Spring MVC project and the controllers use the components illustrated here.

This is similar to packaging by feature, but it's more akin to the "micro services" that Mark Needham talks about in his blog post. Each sub-package of je.techtribes.component houses a separate component, complete with it's own internal layering and Spring configuration. As far as possible, all of the internals are package scoped. You could potentially pull each component out and put it in it's own project or source code repository to be versioned separately. This approach will likely seem familiar to you if you're building something that has a very explicit loosely coupled architecture such as a distributed messaging system made up of loosely coupled components. I'm fairly confident that most people are still building something more monolithic in nature though, despite thinking about their system in terms of components. I've certainly packaged *parts* of monolithic codebases using a similar approach in the past but it's tended to be fairly ad hoc. Let's be honest, organising code into packages isn't something that gets a lot of brain-time, particularly given the refactoring tools that we have at our disposal. Organising code by component lets you explicitly reflect the concept of "a component" from the architecture into the codebase. If your software architecture diagram screams something about your business domain (and it should), this will be reflected in your codebase too.

The structural elements of software

We could create a convention here to say that all sub-packages of je.techtribes.component are components, but it would be much easier to explicitly mark components using metadata. In Java, we could use annotations to do this, attributes in .NET, etc. If we used the same approach for other structural elements of software (e.g. services, layers, containers, etc), tool vendors could use this metadata to generate meaningful and *simple* architecture diagrams automatically. Plus, they could also use this structural information to generate dependency diagrams that focus on components rather than classes. I've started experimenting with annotations as a way to do this and I've created a Github repo to store whatever I come up with.

The major caveat to all of this is that designing a software system based around components isn't "the only way". It's a nice approach to think about software systems that are more monolithic in nature and it's a great stepping stone to designing loosely coupled architectures. But it isn't a silver bullet. Regardless of how you design software, I do hope this post has got you thinking about the mapping between software architecture and how it's reflected in the code.

Software architecture and coding are often seen as mutually exclusive disciplines and there's often very little mapping from the architecture into the code and back again. Effectively and efficiently visualising a software architecture can help to create a good shared vision within the team, which can help it go faster. Having a simple and explicit mapping from the architecture to the code can help even further, particularly when you start looking at collaborative design and collective code ownership. Furthermore, it helps bring software architecture firmly back into the domain of the development team, which is ultimately where it belongs.

New essays in the book

Thu, 14 Mar 2013 10:24:00 GMT

Just a quick note to say that I've added some new essays to my Software Architecture for Developers book. They are:

Software architects should be master builders: this talks about the importance of software architects needing deep technology skills and working as a part of the team.
Technology is not an implementation detail: this is about why you shouldn't leave technology decisions as a after-thought.
The need for sketches: this explains the importance of sketches in communicating the overall software architecture, both inside and outside of the immediate team.
Would you code it that way?: this talks about the key question you need to ask yourself whenever you're designing software and visualising that design through sketches.

Leanpub does have the ability to e-mail readers whenever a new version of the book is published, but I try to only do this once or twice a month so as not to spam everybody. There is a version history at the back of the book though, which will tell you when new essays have been added plus it hyperlinks right to them. Hope you enjoy the new content.

Photos and slides from DevWeek 2013

Mon, 11 Mar 2013 08:40:10 GMT

Just a quick note to say thank you to everybody who came along to my workshop and/or talk at DevWeek 2013 last week. Here are links to the slides and photos.

Thanks again.

Sample software guidebook and sketches

Tue, 26 Feb 2013 11:06:00 GMT

Documentation on software projects is often a contentious topic, particularly since the agile manifesto says that we should value working software over comprehensive documentation. My take on lightweight documentation has now been included in my Software Architecture for Developers book and I've had a stream of e-mail requests since publishing the new version to ask whether I would create a sample software guidebook too.

A sample software guidebook

Yes is the answer ... and I've already started putting this together, based upon a side-project of mine called techtribes.je, which is basically a community portal for the IT, tech and digital industry in Jersey^*. This is a fun little side-project rather than a huge high-performance distributed system, but it should be enough to illustrate the concepts in the book.

And some sample software architecture sketches

The sample guidebook will also include some software architecture diagrams, which will again serve as an example of the context, containers, components and classes sketches that I talk about. The following context diagram (included in the sample guidebook) provides an overview of techtribes.je:

The introduction and context sections of the sample guidebook are available in the published version of my book now and I'll be adding the other sections over the next few days.

Plus some open source code

As a final note, I'm planning to open-source the code behind techtribes.je on GitHub so that you can see the correlation between the code and the documentation. If there's anything else that you'd like to see in the book, please just drop me a line.

^{* just to avoid any confusion, I'm referring to Jersey in the Channel Islands, not New Jersey! :-)}

Software architecture training in the Czech Republic

Tue, 22 Jan 2013 10:50:28 GMT

I'm pleased to say that my Software Architecture for Developers training course is coming to the Czech Republic in 2013, in conjunction with Aguarra. There are two public courses currently scheduled for the 21st/22nd February and the 23rd/24th September.

Please see the Aguarra website for prices and to book a place.

Software documentation as a guidebook

Tue, 15 Jan 2013 11:09:22 GMT

"Working software over comprehensive documentation" is what the Manifesto for Agile Software Development says and it's incredible to see how many software teams have interpreted those five words as "don't write *any* documentation". The underlying principle here is that real working software is much more valuable to end-users than a stack of comprehensive documentation but many teams use this line in the agile manifesto as an excuse to not write any documentation at all. Unfortunately the code doesn't tell the whole story and not having a source of supplementary information about a complex software system can slow the team down as they struggle to navigate the codebase.

I'm also a firm believer that many software teams have a duty to deliver some supplementary documentation along with the codebase, especially those that are building the software under an outsourcing and/or offshoring contract. I've seen IT consulting organisations deliver highly complex software systems to their customers without a single piece of supporting documentation, often because the team doesn't *have* any documentation. If the original software developers leave the consulting organisation, will the new team be able to understand what the software is all about, how it's been built and how to enhance it in a way that is sympathetic to the original architecture? And what about the poor customer? Is it right that they should *only* be delivered a working codebase?

The problem is that when software teams think about documentation, they usually think of huge Microsoft Word documents based upon a software architecture document template from the 1990's that includes sections where they need to draw UML class diagrams for every use case that their software supports. Few people enjoy reading this type of document, let alone writing it! A different approach is needed. Read more in "Software Architecture for Developers"...

Digitalisation - complex for all the wrong reasons

Fri, 11 Jan 2013 11:29:26 GMT

Here's a short article that I wrote in 2011 for Dansk IT about digitalisation. The TL;DR version is "understand your enterprise architecture".

- - -

In my role as a software developer, Iʼm often asked to automate existing business processes and transform them into computer systems. Sometimes this is about replacing systems that are solely manual in nature and sometimes itʼs about replacing manual systems that have a digital core like an Excel spreadsheet. Digitalising these processes can be as simple as creating a way to capture the essential data, implementing some automated processing and ensuring that the resulting data is stored centrally for others to use. Unfortunately though, life is never that simple and digitalisation becomes a complex task for all the wrong reasons.

The goal of most digitalisation efforts is usually related to reducing cost, improving efficiency or simply making somebodyʼs life easier. Achieving this normally requires centralising some aspects of the data and the associated process, ultimately requiring people to work together who are often in different teams and often in different locations. To complicate matters, you need to deal with the data silos that each of those separate teams have built up over the years. Essentially, itʼs *their* data that they have complete control of and itʼs *their* version of the truth. Iʼve seen this everywhere from large investment banks where each business area has their own definition of what should really be common reference data, through to organisations where each department has a different view of their customers. Creating a single consolidated source of truth typically requires teams to relinquish some of their control and likely some of their data too. Reaching agreement on the definition of common concepts is one thing, but getting teams to give up some of their perceived power is another entirely.

Agile might be the hottest thing in software development at the moment, but having an understanding of the enterprise architecture is crucial for any digitalisation effort, particularly if it spans an organisation and crosses internal boundaries. My advice? Focus on the end-goal and donʼt get caught up on the politics, particularly if youʼre an outsider looking in.