Coding the Architecture - soa tag

SOA may be failing, but the approaches will live on

Thu, 10 Sep 2009 07:15:00 GMT

Recently there seem to be more mutterings about how "SOA is dead" or "SOA is failing". From my perspective, I think that the rationale behind SOA is sound, as are the design principles (implementation independent loosely coupled services, reuse, etc) but adoption is way harder than anybody anticipated. Here are my key reasons.

Focus on technology: many project teams went through a phase where they'd buy in an Enterprise Service Bus or another Enterprise Application Integration technology and focus their minds on making everything talk in web services. We all know that SOA isn't inherently about web services, but if it wasn't XML it wasn't going to work. Crucially, SOA is about the business and this is the point that many people neglect. One of the first things to do with SOA is to step back from the technology and look at the overall business processes. Then, once you understand this, you can start to review the processes, improve them and decompose them into meaningful steps. Then you can start mapping that onto the technology. With SOA, if you don't focus on the business, you miss out on the reuse opportunities.
Politics: I've seen a lot of large organisations strive to centralise core parts of their business (e.g. reference data) but it never works. Politics always gets in the way, primarily because people don't like giving up control and disparate groups within an organisation rarely agree on anything. I once did some work for a large investment bank and they a fantastic opportunity to build a truly leading edge client-side dealing platform across FX, equities, fixed income, etc. Technically it was very feasible but the massive egos and politics got in the way. Even if an organisation did manage to centralise some of their key business services, there's still the need to appropriately advertise and version them using something like a service catalog.

I regularly use the same approaches from SOA in "non-SOA" projects and find they are a very good way to come up with a well organised service-centric architecture that is easy to build, deploy and test. At the end of the day, everything seems to be one giant cycle. SOA may be going out of fashion now while everybody talks about cloud computing, but its principles will live on and I'm sure it'll be back in another guise before we know it.

System Design and Reconciliation

Fri, 08 Feb 2008 22:06:20 GMT

Even those not working in Finance have probably heard about the intriguing events in a large investment bank and the HUGE losses that have occurred.

The interesting comment from an IT and architecture point of view is the following:

The trader responsible for the fraud had "in-depth knowledge of the control procedures resulting from this former employment in the middle-office", the bank said.

I'm sure the story and our perception of it will change as more facts emerge but we can be certain that the banks internal security and control systems failed.

Many security and control terms such as authentication, authorization and auditing are probably familiar what about reconciliation? Consider a simple stock-taking example. The number of items delivered to a warehouse minus the number of items sold should tell you how many are still there. If it's less, you may have a thief. ( If you've ever been involved in a stock-take you may have seen many missing items - they tend to be small and valuable.) Stock taking is a physical example of reconciliation i.e do all the numbers add up.

Should you be doing this in a system of your own? If you have developed an online store do you periodically make sure that the number of orders to your website equals the number of payments received and items sent? It's very common for a system designer to assume this will be the case, as the data normally flows from one component to the next, but consider what happens if someone sends messages to your delivery system directly. They bypass your ordering and payment systems so there is no matching order or payment. This is likely to be an insider that knows the system well and you can lose a lot of money as you send out goods without being paid. These kind of issues are increasingly likely as designers move away from monolithic systems to SOAs.

Apart from internal checks between your own systems you can reconcile with external ones. In securities trading, we usually get a list of trades from the markets and exchanges at the end of he day and reconcile against the banks internal list of trades. This should show if a trader has been placing trades they shouldn't have (or failing to place those they should).

It's important to make sure that the main system and the reconciliation system are separate otherwise an error or fraud in one could effect the other. (Reconciliation is often simple and can be done in something like a script).

Finally, you should remember that regression testing is actually a form of reconciliation between different versions of systems, so you may be doing some already!

What about the application tier?!

Mon, 24 Sep 2007 15:05:32 GMT

I've recently become involved in a project with a very clear idea of its functional and data requirements - so much so that the presentation technology has been selected and the database vendor and schema determined. The suppliers of these have also been chosen and brought in. So, you've got your data, you've got your user interface, all you've got to do is connect the two.

At first glance the omission of an application tier is just an oversight; the majority of functions can be delegated to the database: little more than a bit of remoting and a bit of transformation. On closer inspection it becomes evident that the hole between the presentation and the data is more than just technical. In fact, it's a dumping ground for all the requirements that didn't fit neatly into either of the other two. It's the region in which misaligned assumptions are going to be magically ironed out. For example, the presentation tier assumes (or at least prefers) synchronous calls whereas the database would be better addressed asynchronously. Sure the app tier could hide the asynchronous nature of the database call from the presentation tier but that's simply patching over the hole.

In reality, the omission of the application tier is a clear sign that the exciting technology choices have led to a project starting without a clear system architecture. It sounds to me like another case of trying to introduce architecture during implementation.

I'm not sure you can architect and implement at the same time - there are always some aspects of the architecture such as technology, team and methodology selection that need prior thought.

Book Review - Understanding Enterprise SOA

Tue, 02 May 2006 15:31:40 GMT

If ever there was an overused three-letter acronym, SOA would be it. Sadly few people that use the acronym actually understand what it really means. Managers think that it is the answer to all of their problems ("Let's just get one of those SOA things in"), and developers assume that it is just exposing everything as web services willy nilly. This book aims to cut through the hype and get to the truth behind the acronym.

Initially it seems that the book is targeted towards management types, but it soon becomes apparent that there is plenty in there for the technologists amongst us. Sure, there are no implementation examples, but this means that the book is able to concentrate on the real issues behind SOA rather than the Hello World examples that so many books turn to.

The book is told as the story of the IT sprawl that is Titan Insurance. Titan face the same problem as a lot of large IT organisations in that they have a lot of disparate systems that all need to work together to be effective. The book presents the case for creating an SOA based on Web services standards to achieve the integration, and presents many compelling arguments as the why this is the right way to go. The use of a recurring example through the book helps to give good context to the explanations which is something that is missing from so many IT books at this level.

As an architect trying to sell the SOA dream, this book presents "ready-canned" arguments that you can use, and plenty of technical information to back it up. This book explores both the technical and organisational issues behind SOA and is not afraid to highlight the parts of the puzzle that are not quite ready for the prime-time yet, and preempts most of the questions that people will typically ask you about SOA.

Rarely is an IT book a compelling read, but some how this book manages to be just that. I believe that I gained something from each and every chapter in this book. I think that the primary thing that I have learnt from this book is that employing an SOA is a great way to reduce coupling between systems (be they legacy or not) and also a good way to avoid the vendor lock-in associated with other EAI techniques. I also learnt that there is a fair way to go in some areas before this is an "easy" way to go, mainly regarding security.

Conclusion

If you are a developer looking to understand the nuts and bolts of web-services, then this book is certainly not for you. If, however, you are a looking to understand the hype behind SOA and to be able to have a reasoned argument for or against it's adoption, then you should give this book a read.

Service catalog

Mon, 27 Mar 2006 19:02:01 GMT

Having been involved in an SOA project, one of the lessons that I've learnt is that we need to be far smarter in the way that we manage, organise and reuse services. One of the key promises of SOA is reuse on a much more granular level than one could ever hope to achieve with pure object oriented techniques, but my experience suggests that reuse isn't something that you get for free. Particularly, reuse problems arise when others within your team|department|enterprise don't realise that you've already defined and/or built a particular service that they could reuse. How do you solve this? Well I think that the answer to this is two-fold.

Define : ensure your service interfaces are agreed upon up-front.
Catalog : document the basics of each service somewhere.

The first part (service definition) won't come as any surprise, but it's a prerequisite for the second part, which is all about building a service catalog. One of the hardest parts of SOA is actually defining the services in the first place and, once this is done, cataloguing them is fairly straightforward. However, without this last step, how do you easily work out what services are available to you and the operations they provide.

First of all, what sort of information is it useful to capture in a service catalog? My initial suggestion would be as follows.

Service name
Description
Type of service (e.g. business, application specific, data, etc)
Service level agreements and other quality of service characteristics (e.g. performance, scalability, security, availability, etc)
For each service operation
- Operation name
- Description
- Request/response messages | input/output parameters and possible errors
- Pre-conditions
- Post-conditions
- Whether the operation is idempotent

So what alternatives do we have for building a service catalog? In the case of web services, one option is to augment the WSDL used to define the services, perhaps performing an XSLT transformation on the document to make it more readable, perhaps in a HTML format. This might work if you have control over the generation of your WSDL, but many tools auto-generate it for you. Another option is to build your service catalog within a UML model. This is the approach that I've seen tried and while it's easy to define services, their operations and the common view of data that those operations work upon, some of the other aspects are harder to document. Well, they're not harder to document because most modeling tools support the notion of adding your own metadata to your classes. It's finding the information that is hard. Ideally it should be be readily available and people shouldn't have to click through lots of dialogs to find this "hidden" metadata. Unless you can get some good reports out of your modeling tool of choice, I'd be tempted not to use it for this purpose.

Ultimately, I think a simple text/HTML file or Word document is the best option because it's more easily accessible to all types of users, including those on the business and technical sides. Of course, this doesn't mean that you need to manually create it. If you can automatically extract information from your UML model|WSDL|annotations in Java implementations|etc then you'll spend less effort keeping the catalog up to date.

What do you think? If you've used something like a service catalog, what information did it capture and how?

Understanding Enterprise SOA

Wed, 08 Mar 2006 21:32:03 GMT

The Register have just published a review of Manning's Understanding Enterprise SOA. Their verdict?

For developers looking to get to grips with the nuts and bolts of SOAP, REST and the, er, rest, this isn't the place to look for details. This is decidedly not a book for those about to cut code. However, if you want to make sense of all of the SOA fuss and how it fits in with the web services you're crafting then this is definitely a good place to start.

I wonder what Sam will think of it.

Service dependencies

Thu, 02 Mar 2006 17:49:58 GMT

Dan Creswell has an interesting take on how he sees service aggregation panning out in the future, particularly with respect to the problems around service dependencies.

There is much talk about aggregating services together with a lot of excitement around the idea of combining services supplied by other enterprises. At the base level this seems like an exciting proposition and it's possible to realize this promise now but I think there are some interesting issues just over the horizon.

...

Assuming that you've managed to wire all these services together successfully, you've now substantially widened your network of dependencies. Many of the factors in question are now out of your control, you are reliant on the service's vendor to "do the right thing". This is an order of magnitude worse than being reliant on some other team's services within a single enterprise.

Personally, this makes me think that (1) it's even more important to define public services at the correct level of granularity and (2) you need to think carefully about designing service interfaces that are seamlessly upgradeable. Certainly some food for thought.

Service-Oriented Architecture Compass

Tue, 07 Feb 2006 10:39:29 GMT

I've just finished reading Service-Oriented Architecture Compass : Business Value, Planning and Enterprise Roadmap for JavaRanch and the review has been posed here. My summary? Here it is.

Overall, if you're an architect looking at SOA now or planning to in the future, this book will help you clarify your thoughts and steer you through the brave new world of a service oriented architecture. It comes highly recommended.

This is a great book and covers a much broader range of topics than how to build an SOA with [Java|.NET|another technology]. It'll be interesting to see how this book stacks up to the one that Sam is reviewing. If you've read this book, what did you think?