http://www.codingthearchitecture.com/tags/optimization/ Software architecture for developers en Coding the Architecture Mon, 21 May 2012 09:41:00 GMT Pebble (http://pebble.sourceforge.net) http://backend.userland.com/rss http://www.codingthearchitecture.com/2007/12/15/what_is_significant.html <p> Recently I wrote a quick blog about taking metrics for optimisation. I suggested they should only be included if the improvement was significant, but how do you define significant? </p> <p> You may think that 'significant' is just a matter of opinion but it actually has a very specific meaning in statistics - <a href="http://en.wikipedia.org/wiki/Statistical_significance">Wikipedia 's Description</a>. You can have a read through the maths but it basically comes down to "a result is called statistically significant if it is unlikely to have occurred by chance". </p> <p> This is really important and something that performance testers and optimisers often forget. For example... </p> <p> Imagine that you perform some kind of performance test on your system or code. This could be anything e.g. latency response timings, throughput per time unit etc but we'll assume for this that it's units processed in 10 minutes. The figure you get is 20. You spend a day modifying a piece of code you think will affect the performance, retest and get 22. A 10% improvement - pretty good. </p> <p> You hand the new code over to a colleague who also does a test. She says that it's worse by 5%. Slander! You take it to your boss who says there is no difference... </p> <p> What we've done is perform three tests on the old and new system. Lets list them and perform seven others as well: </p> <p> <pre> Old: 20 20 22 19 19 21 19 19 19 22 New: 22 19 22 20 20 19 19 19 21 19 </pre> </p> <p> Now it's obvious what happened (although it probably was before). Your test does not produce constant figures even without changes. Both have a range of 3 (19-22), an average of 20 and a variance of 1.55 </p> <p> If you had performed ten runs on the original code first you would have realised that a single result of 22 for the new code is not significant as it's within the range of the previous figures. </p> <p> Performing the test multiple times on the new code would increase your confidence that it's a significant change. You can test this statistically (but the maths is beyond the scope of this blog entry). </p> <p> Just to leave you with a challenge, the project I'm currently working has a task that we wish to optimise but it takes ten hours to run even on a grid of several hundreds machines. How do we run realistic, pre-production tests that we know are statistically significant? </p> How do you deliver software architecture? http://www.codingthearchitecture.com/2007/12/15/what_is_significant.html#comments http://www.codingthearchitecture.com/2007/12/15/what_is_significant.html Sat, 15 Dec 2007 21:15:27 GMT