Who should own the non-functional tests?

Shouldn't we be asking more from commercial product vendors?

This is a follow-up to a post entitled Performance tuning Java systems that I made last year, which talked about some performance testing that I was doing for a customer. Last week, I went back to the same customer to make some small tweaks to the test scripts, and this very simple act highlighted something I thought was very interesting.

The key thing about the system being tested is that it's a customized version of an existing vendor product (a high volume, low latency trading platform), with the vendor being responsible for making all of the changes. At some point last year, my customer wanted to run some tests to ensure that the platform could grow with their business, and that's where I got involved. After a round of testing, environment reconfiguration and patches, the system more or less delivered on what it promised.

Fast-forward to the current day and I was bought back in to make some changes to the test scripts, because the vendor had made some changes to their web application. Since our performance tests were JMeter based, those changes had a knock-on effect on the tests. The thing I found interesting is that the performance test pack hadn't been handed over to the vendor. Regardless of the reasons, the benefits of doing this are fairly clear.

  • Ownership : although it's good that my customer wanted to proactively test some of the non-functional aspects of the software, I can't help feeling that the software vendor should have ownership of the test pack. My understanding is that they do little in the way of non-functional testing themselves, so this is an ideal opportunity to rectify this.
  • Continuous execution : if the vendor has ownership, they can include the test pack as a part of their integration/build/release process for continuous execution and early warning of performance degrading changes.
  • Accuracy : I like to think that we did a good job at putting together the performance testing scripts, but we don't have the inside knowledge of the system that the vendor has. Vendor ownership would ensure that the test scripts are as accurate and representative as possible.
  • Up to date : the test scripts are inherently tied to the AJAX web application, which means that they need to be kept in sync when changes are made to the user experience. Again, the vendor is best positioned to do this, and taking a more proactive approach would ensure that user experience changes don't degrade the non-functional characteristics of the software.

The overriding question here is why doesn't the vendor undertake their own performance testing? They *should* take responsibility for the non-functional characteristics of their software, particularly when they are making claims of how fast and scalable it is. But perhaps more interesting is that the vendor of another product in use in the same space also doesn't do performance testing. Shouldn't we be asking more from commercial product vendors?

About the author

Simon is an independent consultant specializing in software architecture, and the author of Software Architecture for Developers (a developer-friendly guide to software architecture, technical leadership and the balance with agility). He’s also the creator of the C4 software architecture model and the founder of Structurizr, which is a collection of open source and commercial tooling to help software teams visualise, document and explore their software architecture.

You can find Simon on Twitter at @simonbrown ... see simonbrown.je for information about his speaking schedule, videos from past conferences and software architecture training.



Re: Who should own the non-functional tests?

No doubt the vendor should know that their software lives up to any claims they make but have they made any such claims?

Should the vendor be required to prove their software on someone else's infrastructure or with bespoke modifications?

I think it's reasonable to have an expectation that minor upgrades won't negatively impact your non-functionals but you should probably have verified them yourself at some point.

Re: Who should own the non-functional tests?

In this case, the vendor *has* made claims that the software will meet the NFRs (particularly around performance and scalability). Unfortunately they weren't able to offer advice on how best to configure the system components (e.g. which ones to scale-out, how much memory to assign to the processes, etc) so I suspect these claims are fairly shallow! As for the modifications, well the vendor is doing these themselves, so yes, I expect them to underwrite the NFRs here too.

Re: Who should own the non-functional tests?

Your comments indicate that you don't particularly trust the vendor with the non-functionals! It sounds doubtful you'll fully trust the results of the non-functional tests were you to get them to own them.

It seems that one of the unexpected challenges of being an architect is the continual compromise, balancing the ideal with the reality. Is there any benefit in improving the vendor's processes? Regardless of the outcome, litigation over broken promises is a failure. Does the software care who owns the tests? In short, who's paying for this?


Add a comment Send a TrackBack