Sketching API Connections

An Interesting Conversation

Daniel Bryant, Simon and I recently had a discussion about how to represent system communication with external APIs. The requirement for integration with external APIs is now extremely common but it's not immediately obvious how to clearly show them in architectural diagrams.


How to Represent an External System?

The first thing we discussed was what symbol to use for a system supplying an API. Traditionally, UML has used the Actor (stick man) symbol to represent a "user or any other system that interacts with the subject" (UML Superstructure Specification, v2.1.2). Therefore a system providing an API may look like this:


Actor


I've found that this symbol tends to confuse those who aren't well versed in UML as most people assume that the Actor symbol always represents a *person* rather than a system. Sometimes this is stereotyped to make it more obvious e.g.


Actor with stereotype


However the symbol is very powerful and tends to overpower the stereotype. Therefore I prefer to use a stereotyped box for an external system supplying an API. Let's compare two context diagrams using Boxes vs Stick Actors.


Actor

In which diagram is it more obvious what are systems or people?

Note that ArchiMate has a specific symbol for Application Service that can be used to represent an API:


Actor (Application Service notation from the Open Group's ArchiMate 2.1 Specification)


An API or the System that Supplies it?

Whatever symbol we choose, what we've done is to show the *system* rather than the actual API. The API is a definition of a service provided by the system in question. How should we provide more details about the API?

There are a number of ways we could do this but my preference is to give details of the API on the connector (line connecting two elements/boxes). In C4 the guidelines for a container diagram includes listing protocol information on the connector and an API can be viewed as the layer above the protocol. For example:


Actor


Multiple APIs per External System

Many API providers supply multiple services/APIs (I'm not referring to different operations within an API but multiple sets of operations in different APIs, which may even use different underlying protocols.) For example a financial marketplace may have APIs that do the following:

  1. Allow a bulk, batch download of static data (such as details of companies listed on a stock market) via xml over HTTP.
  2. Supply real time, low latency updates of market prices via bespoke messages over UDP.
  3. Allow entry of trades via industry standard FPML over a queuing system.
  4. Supply a bulk, batch download of trades for end-of-day reconciliation via FPML over HTTP.

Two of the services use the same protocol (xml over HTTP) but have very different content and use. One of the APIs is used to constantly supply information after user subscription (market data) and the last service involves the user supplying all the information with no acknowledgment (although it should reconcile at EOD).

There are multiple ways of showing this. We could:

  1. Have a single service element, list the APIs on it and have all components linking to it.
  2. Show each service/API as a separate box and connect the components that use the individual service to the relevant box.
  3. Show a single service element with multiple connections. Each connection is labeled and represents an API.
  4. Use a Port and Connector style notation to represent each API from the service provider. Provide a key for the ports.
  5. Use a UML style 'cup and ball' notation to define interfaces and their usage.

Some examples are below:


A Single Service element and simple description


Actor


In the above diagram the containers are stating what they are using but contain no information about how to use the APIs. We don't know if it is a single API (with different operations) or anything about the mechanisms used to transport the data. This isn't very useful for anyone implementing a solution or resolving operational issues.


Single, Service box with descriptive connectors


Actor

In this diagram there is a single, service box with descriptive connectors. The above diagram shows all the information so is much more useful as a diagnostic or implementation tool. However it does look quite crowded.


Services/APIs shown as separate boxes


Actor


Here the external system has its services/APIs shown as separate boxes. This contains all the information but might be mistaken as defining the internal structure of the external system. We want to show the services it provides but we know nothing about the internal structure.


Using Ports to Represent APIs


Actor with


In the above diagram the services/APIs are shown as 'ports' on the external system and the details have been moved into a separate key/table. This is less likely to be mistaken as showing any internal structure of the external service. (Note that I could have also shown outgoing rPorts from the Brokerage System.)


UML Interfaces


Actor

This final diagram is using a UML style interface provider and requirer. This is a clean diagram but requires the user to be aware of what the cup and ball means (although I could have explained this in the key).


Conclusion

Any of these solutions could be appropriate depending on the complexity of the API set you are trying to represent. I'd suggest starting with a simple representation (i.e. fully labeled connections) and moving to a more complex one if needed BUT remember to use a key to explain any elements you use!

Software architecture vs code (DevDay 2014)

I had the pleasure of delivering the closing keynote at the DevDay 2014 conference in Krakow, Poland last month. It's a one day event, with a bias towards the .NET platform, and one of my favourite conferences from this year. Beautiful city, fantastic crowd and top-notch hospitality. If you get the chance to attend next year, do it!

If you missed it, you can find videos of the talks to watch online. Here's mine called Software architecture vs code. It covers the conflict between software architecture and code, how we can resolve this, the benefits of doing so, fishing and a call for donations to charity every time you write public class without thinking. Enjoy!

p.s. I've written about some of these same topics on the blog ... for example, Modularity and testability and Software architecture vs code. My Structurizr project is starting to put some of this into practice too.

Software architecture sketching in Iceland

I'll be in Iceland next month for the Agile Iceland 2014 conference, which I'm really looking forward to as everybody tells me that Iceland is a fantastic country to visit. While in Iceland, I'll also be running my 1-day software architecture sketching workshop on the 6th of November. If you're interested in learning how to communicate the design of your software in a simple yet effective way without using lots of complex UML diagrams, please do join me. Everybody who attends will also get a copy of my Software Architecture for Developers ebook too. :-)

Comparing solutions

A common set of abstractions is much more important than a common notation

Here's a little snippet that my class really picked up on yesterday. During the training course, we get people into groups and ask them to design a solution based upon some simple requirements. The deliverable is "one or more diagrams to describe your solution". Aside from answering a few questions about the business domain and the environment, that's pretty much all the guidance that groups get.

As you can probably imagine, the resulting diagrams are all very different. Some are very high-level, others very low-level. Some show static structure, others show runtime and behavioural views. Some show technology, others don't. Without a consistent approach, these differing diagrams make it hard for people to understand the solutions being presented to them. But furthermore, the differing diagrams make it really hard to compare solutions too.

As I've said before, I don't actually teach people to draw pictures. What I do instead is to teach people how to think about, and therefore describe, their software using a simple set of abstractions. This is my C4 model. With these abstractions in place, groups then redraw their diagrams. Despite the notations still differing between the groups, the solutions are much easier to understand. The solutions are much easier to compare too, because of the consistency in the way they are being described. A common set of abstractions is much more important than a common notation.

Modularity and testability

Designing software requires conscious effort; let's not stop thinking.

I've been writing blog posts covering a number of topics over the past few months; from the conflict between software architecture and code and architecturally-evident coding styles through to representing a software architecture model as code and how microservice architectures can easily turn into distributed big balls of mud. The common theme running throughout all of them is structure, and this in turn has a relationship with testability.

The TL;DR version of this post is: think about modularity, think about how you structure your code, think about the options you have for testing your code and stop making everything public.

1. The conflict between software architecture and code

I've recently been talking a lot about the disconnect between software architecture and code. George Fairbanks calls this the "model-code gap". It basically says that the abstractions we consider at the architecture level (components, services, modules, layers, etc) are often not explicitly reflected in the code. A major cause is that we don't have those concepts in OO programming languages such as Java, C#, etc. You can't do public component X in Java, for example.

2. The "unit testing is wasteful" thing

Hopefully, we've all see the "unit testing is wasteful" thing, and all of the follow-up discussion. The unfortunate thing about much of the discussion is that "unit testing" has been used interchangeably with "TDD". In my mind, the debate is about unit testing rather than TDD as a practice. I'm not a TDDer, but I do write automated tests. I mostly write tests afterwards. But sometimes I write them beforehand, particularly if I want to test-drive my implementation of something before integrating it. If TDD works for you, that's great. If not, don't worry about it. Just make sure that you *do* write some tests. :-)

There are, of course, a number of sides to the debate, but in TDD is dead. Long live testing. (ignore the title), DHH makes some good points about the numbers and types of tests that a system should have. To quote (strikethrough mine):

I think that's the direction we're heading. Less emphasis on unit tests, because we're no longer doing test-first as a design practice, and more emphasis on, yes, slow, system tests. (Which btw do not need to be so slow any more, thanks to advances in parallelization and cloud runner infrastructure).

The type of software system you're building will also have an impact on the number and types of tests. I once worked on a system where we had a huge number of integration tests, but very few unit tests, primarily because the system actually did very little aside from get data from a Microsoft Dynamics CRM system (via web services) and display it on some web pages. I've also worked on systems that were completely the opposite, with lots of complex business logic.

There's another implicit assumption in all of this ... what's the "unit" in "unit testing"? For many it's an isolated class, but for others the word "unit" can be used to represent anything from a single class through to an entire sub-system.

3. The microservices hype

Microservices is the new, shiny kid in town. There *are* many genuine benefits from adopting this style of architecture, but I do worry that we're simply going to end up building the next wave of distributed big balls of mud if we're not careful. Technologies like Spring Boot make creating and deploying microservices relatively straightforward, but the design thinking behind partitioning a software system into services is still as hard as it's ever been. This is why I've been using this slide in my recent talks.

If you can't build a structured monolith, what makes you think microservices is the answer!?

Modularity

Uncle Bob Martin posted Microservices and Jars last month, which touches upon the topic of building monolithic applications that do have a clean internal structure, by using the concept of separately deployable units (e.g. JARs, DLLs, etc). Although he doesn't talk about the mechanisms needed to make this happen (e.g. plugin architectures, Java classloaders, etc), it's all achievable. I rarely see teams doing this though.

Structuring our code for modularity at the macro level, even in monolithic systems, provides a number of benefits, but it's a simple way to reduce the model-code gap. In other words, we structure our code to reflect the structural building blocks (e.g. components, services, modules) that we define at the architecture level. If there are "components" on the architecture diagrams, I want to see "components" in the code. This alignment of architecture and code has positive implications for explaining, understanding, maintaining, adapting and working with the system.

It's also about avoiding big balls of mud, and I want to do this by enforcing some useful boundaries in order to slice up my thousands of lines of code/classes into manageable chunks. Uncle Bob suggests that you can use JARs to do this. There are other modularity mechanisms available in Java too; including SPI, CDI and OSGi. But you don't even need a plugin architecture to build a structured monolith. Simply using the scoping modifiers built in to Java is sufficient to represent the concept of a lightweight in-process component/module.

Stop making everything public

We need to resist the temptation to make everything public though, because this is often why codebases turn into a sprawling mass of interconnected objects. I do wonder whether the keystrokes used to write public class are ingrained into our muscle memory as developers. As I said during my closing session at DevDay in Krakow last week, we should make a donation to charity every time we type public class without thinking about whether that class really needs to be public.

Donate to charity every time you type public class without thinking

A simple way to create a lightweight component/module in Java is to create a public interface and keep all of the implementation (one or more classes) package protected, ensuring there is only one "component" per package. Here's an example of a such a component, which also happens to be a Spring Bean. This isn't a silver bullet and there are trade-offs that I have consciously made (e.g. shared domain classes and utility code), but it does at least illustrate that all code doesn't need to be public. Proponents of DDD and ports & adapters may disagree with the naming I've used but, that aside, I do like the stronger sense of modularity that such an approach provides.

Testability

And now you have some options for writing automated tests. In this particular example, I've chosen to write automated tests that treat the component as a single thing; going through the component API to the database and back again. You can still do class-level testing too (inside the package), but only if it makes sense and provides value. You can also do TDD; both at the component API and the component implementation level. Treating your components/modules as black boxes results in a slightly different testing pyramid, in that it changes the balance of class and component tests.

Rethinking the testing pyramid?

A microservice architecture will likely push you down this route too, with a balanced mix of low-level class and higher-level service tests. Of course there is no "typical" shape for the testing pyramid; the type of system you're building will determine what it looks like. There are many options for building testable software, but neither unit testing or TDD are dead.

In summary, I'm looking for ways in which it we can structure our code for modularity at the macro-level, to avoid the big ball of mud and to shrink the model-code gap. I also want to be able to automatically draw some useful architecture diagrams based upon the code. We shouldn't blindly be making everything public and writing automated tests at the class level. After all, there are a number of different approaches that we can take for all of this, and the modularity you choose has an implication on the number and types of tests that you write. As I said at the start; think about modularity, think about how you structure your code, think about the options you have for testing your code and stop making everything public. Designing software requires conscious effort. Let's not stop thinking.

Firm foundations

My opening keynote at ASAS 2014 in The Netherlands

I truly believe that a lightweight, pragmatic approach to software architecture is pivotal to successfully delivering software, and that it can complement agile approaches rather than compete against them. After all, a good architecture enables agility and this doesn't happen by magic. But the people in our industry often tend to have a very different view. Historically, software architecture has been a discipline steeped in academia and, I think, subsequently feels inaccessible to many software developers. It's also not a particularly trendy topic when compared to [microservices|Node.js|Docker|insert other new thing here].

I've been distilling the essence of software architecture over the past few years, helping software teams to understand and introduce it into the way that they work. And, for me, the absolute essence of software architecture is about building firm foundations; both in terms of the team that is building the software and for the software itself. It's about technical leadership, creating a shared vision and stacking the chances of success in your favour.

I'm delighted to have been invited back to ASAS 2014 and my opening keynote is about what a software team can do in order to create those firm foundations. I'm also going to talk about some of the concrete examples of what I've done in the past, illustrating how I apply a minimal set of software architecture practices in the real world to take an idea through to working software. I'll also share some examples of where this hasn't exactly gone to plan too! I look forward to seeing you in a few weeks.

Read more...

Software architecture vs code

The video from GOTO Chicago 2014

My Software architecture vs code talk has evolved considerably over the past year and I've presented a number of iterations at events in Europe and the US. If you're interested, thanks to the nice people behind the GOTO conferences, you can now watch the video of the version that I presented at GOTO Chicago 2014.

During September I'll be presenting a new version of this session at Foo Café in Malmö, Software Architecture Summit in Berlin and DevDay in Krakow. See you there!

Free and open source example software guidebook

A software guidebook for the techtribes.je website

It needs a little updating (isn't that always the case!), but I've moved the example software guidebook (previously an appendix in my Software Architecture for Developers book) into a separate free and open source book on Leanpub.

techtribes.je - Software Guidebook

techtribes.je is a side-project of mine to create a content aggregator for the tech, IT and digital sector in Jersey, Channel Islands. The code behind the techtribes.je website is open source and available on GitHub. The source for the software guidebook is also open source and available on GitHub.

The techtribes.je software guidebook is based upon the concept of a software guidebook as described in my Software Architecture for Developers book; the software guidebook is a lightweight, pragmatic way to document the "big picture" of a software system. In essence, it's my simplified version of many "software architecture document" templates you'll find out there on the web.

techtribes.je - Software Guidebook is available to download for free from Leanpub. I hope you find it useful.

Why do I use Leanpub?

My perspective as a self-published author

There's been some interesting discussion over the past fews days about Leanpub, both on Twitter and blogs. Jurgen Appelo posted Why I Don't Use Leanpub and Peter Armstrong responded. I think the biggest selling points of Leanpub as a publishing platform from an author's perspective may have been lost in the discussion. So, here's why my take on why I use Leanpub for Software Architecture for Developers.

Some history

I pitched my book idea to a number of traditional publishing companies in 2008 and none of them were very interested. "Nice idea, but it won't sell" was the basic summary. A few years later I decided to self-publish my book instead and I was about to head down the route of creating PDF and EPUB versions using a combination of Pages and iBooks Author on the Mac. Why? Because I love books like Garr Reynolds' Presentation Zen and I wanted to do something similar. At first I considered simply giving the book away for free on my website but, after Googling around for self-publishing options, I stumbled across Leanpub. Despite the Leanpub bookstore being fairly sparse at the start of 2012, the platform piqued my interest and the rest is history.

The headline: book creation, publishing, sales and distribution as a service

I use Leanpub because it allows me to focus on writing content. Period. The platform takes care of creating and selling e-books in a number of different formats. I can write some Markdown, sync the files via Dropbox and publish a new version of my book within minutes.

Typesetting and layout

I frequently get asked for advice about whether Leanpub is a good platform for somebody to write a book. The number one question to ask is whether you have specific typesetting/layout needs. If you want to produce a "Presentation Zen" style book or if having control of your layout is important to you, then Leanpub isn't for you. If, however, you want to write a traditional book that mostly consists of words, then Leanpub is definitely worth taking a look at.

Leanpub uses a slightly customised version of Markdown, which is a super-simple language for writing content. Here's an example of a Markdown file from my book, and you can see the result in the online sample of my book. Leanpub does allow you to tweak things like PDF page size, font size, page breaking, section numbering, etc but you're not going to get pixel perfect typesetting. I think that Leanpub actually does a pretty fantastic job of creating good looking PDF, EPUB and MOBI format ebooks based upon the very minimal Markdown. This is especially true when you consider the huge range of ebook reader software across PCs, Macs, Android devices, Apple devices, Kindles, etc. Plus the readers themselves can mess with the fonts/font sizes too.

Book formatting on Leanpub

It's like building my own server at Rackspace versus using a "Platform as a Service" such as Cloud Foundry. You need to make a decision about the trade-off between control and simplicity/convenience. Since authoring isn't my full-time job and I have lots of other stuff to be getting on with, I'm more than happy to supply the content and let Leanpub take care of everything else for me.

Toolchain

My toolchain as a Leanpub author is incredibly simple: Dropbox and Mou. From a structural perspective, I have one Markdown file per essay and that's basically it. Leanpub does now provide support for using GitHub to store your content and I can see the potential for a simple Leanpub-aware authoring tool, but it's not rocket science. And to prove the point, a number of non-technical people here in Jersey have books on Leanpub too (e.g. Thrive with The Hive and a number of books by Richard Rolfe).

Iterative and incremental delivery

Before starting, I'd already decided that I'd like to write the book as a collection of short essays and this was cemented by the fact that Leanpub allows me to publish an in-progress ebook. I took an iterative and incremental approach to publishing the book. Rather than starting with essay number one and progressing in order, I tried to initially create a minimum viable book that covered the basics. I then fleshed out the content with additional essays once this skeleton was in place, revisiting and iterating upon earlier essays as necessary. I signed up for Leanpub in January 2012 and clicked the "Publish" button four weeks later. That first version of my book was only about ten pages in length but I started selling copies immediately.

Variable pricing and coupons

Another thing that I love about Leanpub is that it gives you full control over how you price your book. The whole pricing thing is a balancing act between readership and royalties, but I like that I'm in control of this. My book started out at $4.99 and, as content was added, that price increased. The book now currently has a minimum price of $20 and a recommended price of $30. I can even create coupons for reduced price or free copies too. There's some human psychology that I don't understand here, but not everybody pays the minimum price. Far from it, and I've had a good number of people pay more than the recommend price too. Leanpub provides all of the raw data, so you can analyse it as needed.

An incubator for books

As I've already mentioned, I pitched my book idea to a bunch of regular publishing companies and they weren't interested. Fast-forward a few years and my book is the currently the "bestselling" book on Leanpub this week, fifth by lifetime earnings and twelfth in terms of number of copies sold. I've used quotes around "bestselling" because Jurgen did. ;-)

Leanpub bestsellers

In his blog post, Peter Armstrong emphasises that Leanpub is a platform for publishing in-progress ebooks, especially because you can publish using an iterative and incremental approach. For this reason, I think that Leanpub is a fantastic way for authors to prove an idea and get some concrete feedback in terms of sales. Put simply, Leanpub is a fantastic incubator for books. I know of a number of books that were started on Leanpub have been taken on by traditional publishing companies. I've had a number of offers too, including some for commercial translations. Sure, there are other ways to publish in-progress ebooks, but Leanpub makes this super-easy and the barrier to entry is incredibly low.

The future for my book?

What does the future hold for my book then? I'm not sure that electronic products are ever really "finished" and, although I consider my book to be "version 1", I do have some additional content that is being lined up. And when I do this, thanks to the Leanpub platform, all of my existing readers will get the updates for free.

I've so far turned down the offers that I've had from publishing companies, primarily because they can't compete in terms of royalties and I'm unconvinced that they will be able to significantly boost readership numbers. Leanpub is happy for authors to sell their books through other channels (e.g. Amazon) but, again, I'm unconvinced that simply putting the book onto Amazon will yield an increased readership. I do know of books on the Kindle store that haven't sold a single copy, so I take "Amazon is bigger and therefore better" arguments with a pinch of salt.

What I do know is that I'm extremely happy with the return on my investment. I'm not going to tell you how much I've earned, but a naive calculation of $17.50 (my royalty on a $20 sale) x 4,600 (the total number of readers) is a little high but gets you into the right ballpark. In summary, Leanpub allows me focus on content, takes care of pretty much everything and gives me an amazing author royalty as a result. This is why I use Leanpub.

Speaking in September

After a lovely summer (mostly) spent in Jersey, September is right around the corner and is shaping up to be a busy month. Here's a list of the events where you'll be able to find me.

It's going to be a fun month and besides, I have to keep up my British Airways frequent flyer status somehow, right? ;-)