Coding the Architecture

SATURN 2013

Fri, 17 May 2013 18:29:08 GMT

I attended the SATURN 2013 conference in Minneapolis a couple of weeks ago, which is an annual practitioner conference about software architecture, organised by the Software Engineering Institute (SEI) of Carnegie Mellon University. The conference is in its 9th year now, but it's not something that's been on my radar to be honest, partly because it's in the US and partly because I've not heard much about it in the past.

SATURN is definitely not your typical hardcore developer conference, although there were a few talks that focussed on technology and code. The majority of the talks were a mix of how software architecture is being applied in the real-world alongside others that were more research focussed. This is unsurprising since the conference is organised by the SEI but the even the research talks were grounded in industry. I didn't really appreciate how much industry work the SEI actually did until I met some of the team. SATURN is certainly not an academic conference ... I don't remember hearing about missile control systems at a software conference in the past!

The keynotes by Stephan Murer, Scott Berkun, Mary Poppendieck and Philippe Kruchten were excellent, as were Mary's and Philippe's tutorials that I attended on the Friday. If you want to see what sort of things were talked about at SATURN 2013, you can download all of the slides. Here are the links to view/download the slides and photos from my own sessions:

It was great to catch-up with old friends and meet many new ones. The conference was fantastic as it was ... but then, during the closing session, I was surprised and honoured to win the IEEE Software sponsored SATURN 2013 "Architecture in Practice" Presentation Award for my talk about the conflict between agile and architecture!

Talk about the "icing on the cake". Once again, a huge thank you to everybody that voted. :-)

Software architecture for developers Google Group

Fri, 17 May 2013 17:24:00 GMT

A number of people have recently asked me whether there was somewhere they could chat about software architecture after attending my Software architecture for developers training course, so I've created a Google Group. If you want to discuss software architecture, please feel free to join us too.

Agile Software Architecture Sketches and NoUML

Tue, 16 Apr 2013 21:12:00 GMT

If you're working in an agile software development team at the moment, take a look around at your environment. Whether it's physical or virtual, there's likely to be a story wall or Kanban board visualising the work yet to be started, in progress and done. Visualising your software development process is a fantastic way to introduce transparency because anybody can see, at a glance, a high-level snapshot of the current progress. As an industry, we've become pretty adept at visualising our software development process over the past few years although it seems we've forgotten how to visualise the actual software that we're building. I'm not just referring to post-project documentation, this also includes communication during the software development process. Agility is about moving fast and this requires good communication, but it's surprising that many teams struggle to effectively communicate the design of their software.

Read the rest of the article over at InfoQ.com.

Mapping software architecture to code

Mon, 08 Apr 2013 10:27:59 GMT

One of the things I'm currently doing with a number of software teams is teaching them how to draw pictures. As an industry we've got really good at visualising the way that we work using things like Kanban boards and story walls, but we've forgotten how to visualise the software that we're building. In a nutshell, many teams are trying to move fast but they struggle to create a shared vision that the whole team can work from, which ultimately slows them down. And few people use UML nowadays, which just exaggerates the problem. I've written an article about this and it's due for publication soon (I'll come back and add a link) plus it's covered in my Software Architecture for Developers ebook and in a number of talks that I'm doing around Europe (ITARC, IASA UK, Mix-IT) and the US (SATURN) during April. Here are the slides from Agile software architecture sketches - NoUML! that I presented a few weeks ago in Dublin.

The TL;DR version

The TL;DR version of this post is simply this ... if you're building monolithic software systems but think of them as being made up of a number of smaller components, ensure that your codebase reflects this. Consider organising your code by component (rather than by layer or feature) to make the mapping between software architecture and code explicit. If it's hard to explain the structure of your software system, change it.

Decomposition into components

For the purpose of this post, let's assume visualising a software system isn't a problem and that you're sketching some ideas related to the software architecture for a new system you've been tasked to build. An important aspect of "just enough" software architecture is to understand how the significant elements of a software system fit together. For me, this means going down to the level of components, services or modules. It's worth stressing this isn't about understanding low-level implementation details, it's about performing an initial level of decomposition. The Wikipedia page for Component based development has a good summary, but essentially a component might be something like a risk calculator, audit logger, report generator, data importer, etc. The simplest way to think about a component is that it's a set of related behaviours behind an interface, which may be implemented using one or more collaborating classes. Good components share a number of characteristics with good classes. They should have high cohesion, low coupling, a well-defined public interface, good encapsulation, etc.

There are a number of benefits to thinking about a software system in terms of components, but essentially it allows us to think and talk about the software as a small number of high-level abstractions rather than the hundreds and thousands of individual classes that make up most enterprise systems. The photo below shows a typical component diagram produced during the training classes we run. Groups are asked to design a simple financial risk system that needs to pull in some data, perform some calculations and generate an Excel report as the output.

This sketch includes the major components you would expect to see for a system that is importing data, performing risk calculations and generating a report. These components provide us with a framework for partitioning the behaviour within the boundary of our system and it should be relatively easy to trace the major use cases/user stories across them. This is a really useful starting point for the software development process and can help to create a shared vision that the team can work towards. But it's also very dangerous at the same time. Without technology choices (or options), this diagram looks like the sort of thing an ivory tower architect might produce and it can seem very "conceptual" for many people with a technical background.

Talk about components, write classes

People generally understand the benefit of thinking about software as higher level building blocks and you'll often hear people talking in terms of components when they're having architecture discussions. This often isn't reflected in the codebase though. Take a look at your own codebase. Can you clearly see components or does your codebase reflect some other structure? When you open up a codebase, it will often reflect some other structure due to the organisation of the code. Mark Needham has a great post called Coding: Packaging by vertical slice that talks about one approach to code organisation and a Google search for "package by feature vs package by layer" will throw up lots of other discussions on the same topic. The mapping between the architectural view of a software system and the code are often very different. This is sometimes why you'll see people ignore architecture diagrams (or documentation) and say "the code is the only single point of truth".

Auto-generating architecture diagrams

To change tack slightly, I was in Dublin a few weeks ago and I met Chris Chedgey, who is part of the inspiration behind this post. Chris is the co-founder of a company called Headway Software and they have a product called Structure101. You should take a look if you've not seen it before, they have some cool stuff in the pipeline. I won't do their product any justice by trying to summarise what it does, but one of its many features is to visualise and understand an existing codebase.

When I teach people how to visualise their software systems, we create a number of simple NoUML sketches at different levels of abstraction. These are the context, containers and components diagrams. This context, containers and components approach is basically just a tree structure. A system is made up of containers (e.g. a web server, application server, database, etc), each of which is further made up of components. You can see some example diagrams on Flickr and in my book.

Given this is really just a tree structure, it should be fairly straightforward to auto-generate these diagrams from an existing codebase. And perhaps there is a tool out there that can do this, but I've never seen one that has worked really well. Microsoft Visual Studio can generate some layer diagrams but I've never met anybody that really raves about the architecture diagram support. Most tools generate diagrams showing dependencies between packages or classes but they don't tend to show components. And what's a component anyway? Is any class that implements an interface a component? If you're using inversion of control, perhaps everything that you inject is a component?

There are a number of reasons why auto-generating such diagrams is tricky but, once we start coding, much of the semantics associated with "containers" (runtime environments, process boundaries, etc) and "components" becomes lost of the sea of classes that make up the typical codebase. Many developers break their systems up into a number of projects within their IDEs to represent reusable libraries and deployable units but external tools often don't have access to this information if they are solely working from a bunch of JAR files or DLLs (for example). In essence, the information related to the abstract structural elements isn't adequately represented within a codebase. If you take a look at most codebases, I'm fairly sure that you could come up with a set of rules as to what defines a component but perhaps it would be easier to simply make these concepts explicit. Some techniques already exist to do this (e.g. the Architecture Description Language) but I've never seen them used in the corporate world.

Packaging by component

To bring this discussion back to code, the organisation of the codebase can really help or hinder here. Organising a codebase by layer makes it easy to see the overall structure of the software but there are trade-offs. For example, you need to delve inside multiple layers (e.g. packages, namespaces, etc) in order to make a change to a feature or user story. Also, many codebases end up looking eerily similar given the fairly standard approach to layering within enterprise systems. Uncle Bob Martin says that if you're looking at a codebase, it should scream something about the business domain. Organising your code by feature rather than by layer gives you this, but again there are trade-offs. A variation I've been experimenting with is organising code explicitly by component. The following screenshot shows an example of this in the codebase for my techtribes.je website (a content aggregator and portal for Jersey's digital sector). This screenshot only shows the core components; there's a separate Spring MVC project and the controllers use the components illustrated here.

This is similar to packaging by feature, but it's more akin to the "micro services" that Mark Needham talks about in his blog post. Each sub-package of je.techtribes.component houses a separate component, complete with it's own internal layering and Spring configuration. As far as possible, all of the internals are package scoped. You could potentially pull each component out and put it in it's own project or source code repository to be versioned separately. This approach will likely seem familiar to you if you're building something that has a very explicit loosely coupled architecture such as a distributed messaging system made up of loosely coupled components. I'm fairly confident that most people are still building something more monolithic in nature though, despite thinking about their system in terms of components. I've certainly packaged *parts* of monolithic codebases using a similar approach in the past but it's tended to be fairly ad hoc. Let's be honest, organising code into packages isn't something that gets a lot of brain-time, particularly given the refactoring tools that we have at our disposal. Organising code by component lets you explicitly reflect the concept of "a component" from the architecture into the codebase. If your software architecture diagram screams something about your business domain (and it should), this will be reflected in your codebase too.

The structural elements of software

We could create a convention here to say that all sub-packages of je.techtribes.component are components, but it would be much easier to explicitly mark components using metadata. In Java, we could use annotations to do this, attributes in .NET, etc. If we used the same approach for other structural elements of software (e.g. services, layers, containers, etc), tool vendors could use this metadata to generate meaningful and *simple* architecture diagrams automatically. Plus, they could also use this structural information to generate dependency diagrams that focus on components rather than classes. I've started experimenting with annotations as a way to do this and I've created a Github repo to store whatever I come up with.

The major caveat to all of this is that designing a software system based around components isn't "the only way". It's a nice approach to think about software systems that are more monolithic in nature and it's a great stepping stone to designing loosely coupled architectures. But it isn't a silver bullet. Regardless of how you design software, I do hope this post has got you thinking about the mapping between software architecture and how it's reflected in the code.

Software architecture and coding are often seen as mutually exclusive disciplines and there's often very little mapping from the architecture into the code and back again. Effectively and efficiently visualising a software architecture can help to create a good shared vision within the team, which can help it go faster. Having a simple and explicit mapping from the architecture to the code can help even further, particularly when you start looking at collaborative design and collective code ownership. Furthermore, it helps bring software architecture firmly back into the domain of the development team, which is ultimately where it belongs.

New essays in the book

Thu, 14 Mar 2013 10:24:00 GMT

Just a quick note to say that I've added some new essays to my Software Architecture for Developers book. They are:

Software architects should be master builders: this talks about the importance of software architects needing deep technology skills and working as a part of the team.
Technology is not an implementation detail: this is about why you shouldn't leave technology decisions as a after-thought.
The need for sketches: this explains the importance of sketches in communicating the overall software architecture, both inside and outside of the immediate team.
Would you code it that way?: this talks about the key question you need to ask yourself whenever you're designing software and visualising that design through sketches.

Leanpub does have the ability to e-mail readers whenever a new version of the book is published, but I try to only do this once or twice a month so as not to spam everybody. There is a version history at the back of the book though, which will tell you when new essays have been added plus it hyperlinks right to them. Hope you enjoy the new content.

Presentation at QConLondon 2013

Tue, 12 Mar 2013 21:27:58 GMT

Last week I spoke at QConLondon 2013 on the topic of "Modern Legacy Systems". Here are the slides and I'll update with a link to the video when QCon makes it available.

Thank you to everyone that attended and I had a couple of great conversations afterwards about the topic. I even had someone tell me that they were the 'other side' of one of the anecdotes I told - they were on the better of the two sides! The auditorium I spoke in (Churchill) was HUGE and had a maximum capacity of 700. I had under the capacity but this was the largest venue I've spoken in and don't mind admitting that I found it intimidating! Fortunately I had watched Damien Conway's excellent Instantly Better Presentations talk earlier in the day and the tips came in handy. I would really advise anyone giving presentations to watch the video when it is available.

I attended many fascinating presentations and a couple that stand out most were around technology and art - visual problem solving. This was a really enjoyable but I'm struggling to find an immediate use for what I learned in my job!

Also worth a mention was Jesper Richter-Reichhelm's talk on Painful Success. This was a review of an architectural mistake that was made early in a project and its resolution. The problem came down to an incorrectly chosen architectural archetype which became evident when the system was under load. I think this is very common, particularly for on-line applications where it is assumed that a website architecture is suitable. What impressed me was Jesper's honesty about the issue, which is necessary in finding a solution. Many people find admitting these errors impossible and I think we can all learn from the good example.

Photos and slides from DevWeek 2013

Mon, 11 Mar 2013 08:40:10 GMT

Just a quick note to say thank you to everybody who came along to my workshop and/or talk at DevWeek 2013 last week. Here are links to the slides and photos.

Thanks again.

Sample software guidebook and sketches

Tue, 26 Feb 2013 11:06:00 GMT

Documentation on software projects is often a contentious topic, particularly since the agile manifesto says that we should value working software over comprehensive documentation. My take on lightweight documentation has now been included in my Software Architecture for Developers book and I've had a stream of e-mail requests since publishing the new version to ask whether I would create a sample software guidebook too.

A sample software guidebook

Yes is the answer ... and I've already started putting this together, based upon a side-project of mine called techtribes.je, which is basically a community portal for the IT, tech and digital industry in Jersey^*. This is a fun little side-project rather than a huge high-performance distributed system, but it should be enough to illustrate the concepts in the book.

And some sample software architecture sketches

The sample guidebook will also include some software architecture diagrams, which will again serve as an example of the context, containers, components and classes sketches that I talk about. The following context diagram (included in the sample guidebook) provides an overview of techtribes.je:

The introduction and context sections of the sample guidebook are available in the published version of my book now and I'll be adding the other sections over the next few days.

Plus some open source code

As a final note, I'm planning to open-source the code behind techtribes.je on GitHub so that you can see the correlation between the code and the documentation. If there's anything else that you'd like to see in the book, please just drop me a line.

^{* just to avoid any confusion, I'm referring to Jersey in the Channel Islands, not New Jersey! :-)}

More Legacy

Mon, 25 Feb 2013 21:05:00 GMT

In my last blog on legacy systems I talked about what they were and weren't. In this one I'm going to expand on this and how they fit into the business process lifecycle.

Like most developers the centre of my work life is the software development process. This might involve a business analysis phase before and some support afterwards but this is a relatively small percentage of my time.

The 'software development centric' view of the world is below (this is the 'SDLC diagram' and your own process may differ but will share elements). There are two parts that interact with the outside world - requirements and deploy (which may be lightweight and frequent) and the external interactions are wrapped in that.

For some businesses (e.g. public facing web sites) the software IS the business and there is no significant period before or after the software development - it occurs constantly. These businesses are very interesting to us developers for this very reason (and therefore the projects often discussed at conferences) but are actually the exception. For the majority of organisations their individual IT systems perform a function that constitute only a small part of what they do.

Therefore users of our software view the world differently. The software development phase is a very small part of their business processes lifecycle and lifespan. They view the world a little more like this:

Most of the processes they execute will originally have been done ‘manually’ (I include generic software tools such as spreadsheets) and may been done this way from 6 months to 100 years. (If you think I'm exaggerating with '100 years' then you should speak to someone who's worked with an old insurance company!)

Eventually it will be decided that it is cost effective to develop bespoke software (or spend a lot of effort configuring BPM/CRM etc software). This process may be iterative and deliver value quickly but will be mostly complete after a relatively short period of time (compared to the organisation’s lifespan).

It is now fully live and a software team will consider it to now be in its maintenance phase. From the organisation's point-of-view this is where the real value is extracted from the system. Very little money is being spent on it but it is being used and either helping to generate revenue or increasing the efficiency of a process.

As time goes on there will be a decreasing number of changes made to it. Eventually it will be considered ‘legacy’. Ultimately the system will reach end-of-life and replaced when no longer fit for purpose.

If we were going to scale the diagram to indicate time then it might look like this:

These are kind of lifecycle times that I've experienced and yours may differ - this will depend largely on the industries you work within.

Therefore 'legacy' should be viewed as a phase in the lifecycle of the process. Note that I have a small arrow going back to Software Development from the maintenance/legacy phase as it may have upgrades or additions made many years after being deployed.

A comment on my previous blog post said that they refer to 'legacy' systems as 'mature' as this has more positive connotations. I think it also accurately reflects legacy as being a stage of a lifecycle rather than any particular indication of quality.

Lastly I'm just going to quickly plug my talk at QCon again (Modern Legacy Systems) which explores some legacy issues, makes suggestions about approaching them and how to leave a good legacy rather than a bad one. If you haven't booked your place at QCon yet then you can use code ANNE100 to get £100 off.

What is Legacy?

Tue, 05 Feb 2013 21:56:00 GMT

I'm speaking at QCon london this year on the topic Modern Legacy Systems

This was tweeted by @QCon as follows (note this talk isn't specifically about Java but about systems in general).

and within minutes UncleBobMartin replied:

This illustrates a good point about attitudes towards legacy code and systems. Although the term is often used negatively, legacy code is NOT, necessarily, bad code (and a system is more than just code).

In most areas of life a 'Legacy' is a good thing! It might refer to a valuable art collection left to a museum or perhaps the body of work of a famous author. Only in I.T. is something old (but valuable) viewed as bad.

When we refer to a System being Legacy we're really saying that the system is built in a way that differs to how we'd choose to do so now. The system may have been written well, using the best technologies and tools available at the time. A system written in 2001 using Java 1.2 and Oracle 8i etc may have made perfect sense at the time but if you wrote it now you'd at least use the latest versions available or even something different entirely (Scala, mongoDB etc didn't exist then).

A system maintained by professional programmers would be upgraded throughout time and migrated to new technology when required (as Bob suggests). However, this is rarely a decision for the programmers - it's a management decision. To be fair to management, does it really make sense to spend large amounts of money continually upgrading a system if it works? If the programmers and systems team did a good job then no-one might need to make changes for many years. This indicates a high return-on-investment and a good quality system. A Legacy!

Of course when you do have to make changes or additions you have a very different set of challenges compared to modifying a system that is being continually developed. My talk at QCon explores some of these issues, makes suggestions about approaching them and how to leave a good legacy rather than a bad one. If you haven't booked your place at QCon yet then you can use code ANNE100 to get £100 off.