The SquizLog

Finding Software Bugs

Charles makes some interesting points about software complexity, bugs and testing in his article "Where Bugs Come From".

Obviously most software systems are complex. And it is desirable to ship them with as few bugs as possible and, hopefully, no serious bugs. So the question is, how best to find and fix the bugs?

With the advent of XP, automated unit testing with tools like JUnit, has become popular. And, as Charles points out, this is a useful approach for catching some bugs. XP also advocates automating acceptance tests but in my experience this is usually not practical.

So automated testing is good for catching a subset of the bugs. But I think there tends to be an over-reliance on automated unit testing. For one thing, it is a mode of testing that is driven by the developers. Developers are usually focussed on proving that the software they have developed works, rather than finding the bugs. This is why I think it is valuable to involve testers who have not had any part in developing the software. They have a different mindset.

Tom DeMarco and Timothy Lister pointed this out in their book "Peopleware: Productive Projects and Teams" when they described the legendary Black Team of testers, who "were testing code that had been written by someone else, so they were free of the cognitive dissonance that hampers developers when testing their own progams". This team's effectiveness in finding bugs grew gradually until they had cultivated "an image of destroyers. What they destroyed was not only your code but your whole day. They did monstrously unfair things to elicit failures, overloading the buffers, comparing empty files, and keying in outrageous input sequences. Grown men and women were reduced to tears by watching their programs misbehave under the demented handling of these fiends. The worse they made you feel, the more they enjoyed it."

Finding the bugs in a complex software system involves a combination of approaches. There are others that I haven't written about here. But I think destructive testing is one approach that could be utilised more.

Posted to Peopleware, Software Development by Keith Pitty