Where is the complexity?

You can move it about but it's still there!


Software Developers tend to think in terms of code, Database Administrators in schemas, Operational teams in configuration and Systems Teams in hardware and networks. A software architect needs to consider all of these and in particular where the risks and complexities are.

A (non-trivial) system will involve the following:

  1. Data
  2. Code
  3. Configuration
  4. Virtualisation
  5. Hardware Servers
  6. Network connections
  7. Physical Location attributes
  8. Supporting Infrastructure (from UPS to fire suppressant)

These all interact and affect each other and will each have a certain level of complexity. Complexity is difficult to measure, although many attempts have been made. These include Function points in your requirements, Cyclomatic or Halstead complexity metrics in code, complexity class for algorithms, various statistical measures on data and even the application of graph theory for your networks. Different parts of your systems will use different metrics to measure this complexity so you can't compare them - is the network that connects a grid of servers more complex than the hardware setup of each server and is this less complex than the software running on each of the nodes? The only meaningful metric for comparison is probably based on cost required for design/implementation/maintenance.

Reduction or Displacement?

Sometime tools that reduce complexity in one area simply push it into another. For example, tools that 'magically' create database schemes from classes might make the code simpler (removing the DAO/mapping code) but if the schemas they create are a mess then the complexity has just been moved or been temporarily hidden. The same is often true between configuration and code - dependency injection systems such as Spring may make the code more readable and modular but at the expense of complex xml files.

Some frameworks use convention based, implicit configuration instead of explicit configuration. This reduces the amount of user-defined configuration but has this reduced the complexity or just hidden it within a tool?

Virtualisation is a wonderful technology but again this largely moves the complexity - from hardware into software. There are benefits for modification management and turn-around time but you still need an expert who knows what they are doing (the skill-set has just changed).

How do we react to it?

One of the dangers that software architects face is that their background can determine how they perceive complexity and how they deal with it. It is very common to push complexity out of the area they know and into another because it initially appears to be reduced. In reality it is still there but is now residing in part of the system that is less well understood.

Conversely, there can also be a temptation to try to solve all the systems issues in the part of the system that the software architect understands best. However this can add complexity because the problems are not being solved in the most appropriate place. For example I've seen systems with large amounts of code to do a job that could be achieved with a simple action in a database.

If you are working in a organisation which has strong silos there may be a temptation to push the complexity from your part of the organisation into another. Again the complexity is still present but is now in an area where you may not be able to affect change. I have seen projects where responsibilities and complexities have been pushed out of groups but then hurriedly brought back in when problems need to be solved.

Ultimately we need to accept that complexity exists in valuable IT systems but we should make sure it is in the correct part of the system and not try to hide it.

About the author

Robert Annett Robert works in financial services and has spent many years creating and maintaining trading systems. He knows far more about low latency data systems and garbage collection than is good for anyone. He likes to think of himself as a pragmatist who loves technology but uses what's appropriate rather than what's cool.

When not pouring over data connections or tormenting interviewees with circular reference questions, Robert can be found locked in his shed with an impressive collection of woodworking tools.

E-mail : robert.annett at codingthearchitecture.com

Re: Where is the complexity?

Wise post !
I like your Lavoisier-like approach "Nothing is lost, nothing is created, everything is transformed" ... about complexity. Though Complexity should actually be encapsulated though innovation.I would go futher you in your thoughts.

For 2 years, Microservices 'concept' is a bunch of contradictions and my impression is still that it looks like a naive approach of sofware design. A fairy tale where everything is simple, in contrast with too complicated previous approaches.
Microservices comforts developpers with reassuring success stories that happened in some prestigious companies.
Today it seems like the #1 (and only) software design need is *scalability*.Other aspects sound like noises and contradictions between different authors : An ESB is "eavy" (but messaging is ok), a JEE container is "eavy" (but embeeding a JEE container in a jar is OK) etc.
I read books, posts, and I find no answer to main small-teams software problematics, e.g. data consistency or criteria for deciding a refactoring into a design or an other.

When it comes to discuss to downsides/issues that creates, we can read paternalistics answers are like e.g. "you should pay attention to cross services transaction" / but no real answer. Don't start with this, Don't do that... But what whe should do is never expained.

I understand that scallability issue must be addressed to specific desingn/ technology. But addressing that spcific need, I'm afraid that Microservices enthousiasts are generalizing to the entire software design spectum.

Its just like issues at infratructure / devops level should be quickly moved to somewhere else. Most probalby the "We Cannot" is stronger that "We should be able, but we'll have bigger problems elswhere". We prefer taking risks in opening possibiliies (even when we actually don't need them), and describe conterparts without having clear answers.

I'm afraid I still didn't find today serious reflexion about risks, real costs, and real functionality needs : do you have some good links discussing that points ? There should be a balance between designs.

I have developed JEE software for 15 years. I understand the good and the bad design. Question : does a structured approach with Microservices exist today ?

Re: Where is the complexity?

Personally, I think that microservices require exactly the same design practices you should use with any other architectural template. I'd suggest the classic books such as:

  • "Just Enough Software Architecture by George H. Fairbanks" or

  • "Software Systems Architecture by Nick Rozanski, Eóin Woods".

    Of course, you can use the C4 method to help you design and document them as well!
  • Re: Where is the complexity?

    A good read. ...another meaningful metric I'd like to add is "productivity" i.e. speed of development and operations. 1st gives you the edge of competing with new features faster, 2nd means savings in operations giving you more dough to put in developing new features... :-)

    Re: Where is the complexity?

    Excellent post. I would add that all computer systems have a theoretical minimum complexity that is related to the complexity of the problem that they are trying to solve. The system can be more complex than needed, but cannot be less. Distributing a solution across many components (either through a micro-services architecture or through a granular object-orientated design) makes it easier to understand what individual components do but harder to understand how those components contribute to the overall solution. The complexity has been shifted from the way that the components are implemented to the way that they are wired up. Perhaps, as you say in your blog, this is a case of the programmer shifting complexity away from their area of expertise over to that of the application assembler (which I recall was an actual role in the early days of J2ee)

    Add a comment Send a TrackBack