Who should own the non-functional tests?

Mon, 07 Jul 2008 10:03:00 GMT

This is a follow-up to a post entitled Performance tuning Java systems that I made last year, which talked about some performance testing that I was doing for a customer. Last week, I went back to the same customer to make some small tweaks to the test scripts, and this very simple act highlighted something I thought was very interesting.

The key thing about the system being tested is that it's a customized version of an existing vendor product (a high volume, low latency trading platform), with the vendor being responsible for making all of the changes. At some point last year, my customer wanted to run some tests to ensure that the platform could grow with their business, and that's where I got involved. After a round of testing, environment reconfiguration and patches, the system more or less delivered on what it promised.

Fast-forward to the current day and I was bought back in to make some changes to the test scripts, because the vendor had made some changes to their web application. Since our performance tests were JMeter based, those changes had a knock-on effect on the tests. The thing I found interesting is that the performance test pack hadn't been handed over to the vendor. Regardless of the reasons, the benefits of doing this are fairly clear.

Ownership : although it's good that my customer wanted to proactively test some of the non-functional aspects of the software, I can't help feeling that the software vendor should have ownership of the test pack. My understanding is that they do little in the way of non-functional testing themselves, so this is an ideal opportunity to rectify this.
Continuous execution : if the vendor has ownership, they can include the test pack as a part of their integration/build/release process for continuous execution and early warning of performance degrading changes.
Accuracy : I like to think that we did a good job at putting together the performance testing scripts, but we don't have the inside knowledge of the system that the vendor has. Vendor ownership would ensure that the test scripts are as accurate and representative as possible.
Up to date : the test scripts are inherently tied to the AJAX web application, which means that they need to be kept in sync when changes are made to the user experience. Again, the vendor is best positioned to do this, and taking a more proactive approach would ensure that user experience changes don't degrade the non-functional characteristics of the software.

The overriding question here is why doesn't the vendor undertake their own performance testing? They *should* take responsibility for the non-functional characteristics of their software, particularly when they are making claims of how fast and scalable it is. But perhaps more interesting is that the vendor of another product in use in the same space also doesn't do performance testing. Shouldn't we be asking more from commercial product vendors?

Coding the Architecture - vendor tag

Who should own the non-functional tests?