Maintaining software quality with defect detection
I’ve been reading Steve McConnell’s excellent “Code Complete 2” and wanted to share this section on defect detection rates.
| Removal Step | Lowest Rate | Modal Rate | Highest Rate |
|---|---|---|---|
| Informal design reviews | 25% | 35% | 40% |
| Formal design inspections | 45% | 55% | 65% |
| Informal code reviews | 20% | 25% | 35% |
| Formal code inspections | 45% | 60% | 70% |
| Modeling or prototyping | 35% | 65% | 80% |
| Personal desk-checking of code | 20% | 40% | 60% |
| Unit test | 15% | 30% | 50% |
| New function (component) test | 20% | 30% | 35% |
| Integration test | 25% | 35% | 40% |
| Regression test | 15% | 25% | 30% |
| System test | 25% | 40% | 55% |
| Low-volume beta test (< 10 sites) | 25% | 35% | 40% |
| High-volume beta test (> 1000 sites) | 60% | 75% | 85% |
The most interesting facts that this data reveals is that the modal rates don’t rise above 75 percent for any single technique. Moreover, for the most common kinds of defect detection - unit testing and integration testing - the modal rates are only 30-35 percent.
Whilst important, unit tests aren’t the single ultimate aim, and can only ensure software quality when used in conjunction with other methods of defect detection. It seems obvious that more scrutiny and testing will result in better software quality, but it is very interesting to understand the effectiveness. The difficulty comes in knowing how much to invest and where to get the best return.