I know someone will say "write unit tests" and of course I do but there is a class of problems that you see when services interact that you will miss with unit tests. For example, the coordination of service states and circular, service call references.

In the past I have seen several possible solutions:

1) Don't bother and just use unit tests for the local code and rely on automated integration tests to pick up problems caused by services interacting. This can work well if this category of problem is rare and you make sure you have good functional test coverage (which you should anyway). This is a little fire and forget and isn't good for iterative development where you spot issues and modify the code - if you have a number of these issues and the integration tests run overnight you might have broken the run for a week...

2) Run everything across several developer machines. This allows you to use your development equipment but ties up several people and it's very easy for the versions to get out of sync between boxes (you have to deliver code to a shared branch and sync to it).

3) Buy hugely powerful boxes for all developers. Obviously I like to have a super machine but the cost can be huge (I once worked on a project with over 100 developers) and this can still be slow for testing as even the biggest desktop machine will struggle to bring up 4 gigs worth of services (loading all static data and constructing in memory structures etc) and the contention is often a killer.

4) Have continually running, shared services on server boxes that are started every morning from the nightly build. Therefore the developer only runs up services they have modified and tests against the shared services. It is particularly effective if services need a large quantity of static/cached data as this will only be loaded when the shared services are started. Although this works well for business code contained within a specific service, it is obviously less effective if you change some common, foundation code. Developers also have to be careful that tests don't interact in unexpected ways e.g. two developers changing prices for the same item. Also watch out for serialization/on-the-wire encoding issues between the latest development code and the services built that morning.

5) Use virtualization. All of the above solutions have issues and ideally we want the following:

a) Developer integration tests are sandboxed from other developers.

b) Tests are run against the latest code on all services.

c) Most of the work is not done on a developers machine but performed on powerful, shared server machines (at any one time very few developers will be running integration tests).

One way of achieving this is to run the tests in a sandboxed, virtual environment. Virtualisation allows you to create a number of virtual servers on a single physical machine. As far as both internal applications and external users are concerned these virtual machines are the same as physical ones. What we can do is to create an entire set of machines (that mirrors a production environment) on the shared integration servers for our developer to run their tests on. This is created specifically for this test and can be destroyed afterwards.

This allows a FULL integration test for the code but also the system configuration associated with it. The tests don't know they are running on a virtual system so should be functionally identical. Care needs to be taken if you are running non-functional tests, as you are not guaranteed the full processing power of the machine.

You will need to write a more complex deployment script but this should be a one off cost.

At this point I have to admit that I have never created a full, virtual deployment as described above - does anyone have experiences or issues/ideas they'd like to share?

Responses[3]

Posted by Robert Annett on 15 September 2007 08:37:05 BST #