Achieving
Software Quality Using Defect Filters
By Randall Rice
The
concept of defect filters is a powerful one that many others have
written about in the past. Michael Fagan is credited by many as a
pioneer in the area of software inspections as an effective method to
perform quality control early in a software project. Others, such as
Vern Crandall and Alka Jarvis have also written on the topic of filters
in their book, Inroads to Software Quality.
My
experience is that many of the defects that people normally find in
testing or even worse, in production, could be more easily and cheaply
found early in a project by using filters.
I
discussed the value of defects and the importance of learning from the
information we can gain from them in my article (Defects are Good). Filters give us a place to gather this
information while reducing the cost of finding them. With an effective
series of filters, you can virtually eliminate some forms of testing
from the end of a project.
What
is a Defect Filter?
Vern
Crandall has an excellent illustration for software testing. Let's say
that we want to brew a pot of coffee. When the pot is finished brewing,
we notice that there are grounds in the coffee. Yuck! So, the task
becomes to get rid of the grounds.
We
could get a pair of tweezers and reach into the dark coffee and squeeze
the tweezers, hoping that we would find a ground. Each ground we find
would represent a successful effort. This effort might work, but would
not be considered effective, in fact, it would look silly to most of us.
Another
approach would be to run the coffee through a filter to remove the
grounds. It's much faster and not as painful on the fingers!
In
testing, each ground represents a defect. The tweezers approach is how
many people approach testing. They're not sure where the defects are,
so testers perform a series of tests. Some tests will pass, others will
fail. The ones that fail are like finding a ground with the tweezers -
a successful test!
The
filter represents a method to screen defects. In a software project, a
filter could be an inspection, review or walkthrough. If one filter
doesn't catch all the grounds (defects), then keep adding filters until
the grounds no longer appear. Ideally, the filtration process would
start early in the process to achieve the lowest cost of defect removal.
The
High Cost of Late Defect Discovery and Repair
People
have been measuring software defect information for over 30 years. One
of the most significant things we have learned is that the longer a
defect stays in a product, the more costly and risky it is to remove.
Although
numbers vary by project and environment, the costs to fix defects
average to what has come to be known as the "1:10:100" rule. This rule
says that a defect that costs $1 to fix in requirements or design costs
$10 to fix in a traditional test phase and $100 to fix after the
product goes into production (live) use.
There
are various reasons for defect removal costs to increase as the project
progresses. These include:
- The
ripple effect of changes throughout the system or application
- Repeating
past tasks in the project (re-design, re-coding, new or changed
documentation, etc.)
-
Notifying people on the project and users of changes
Regardless
of the reasons, late defect removal is costly and time consuming. It is
also a known fact that changes at the end of a project add risk. If all
of these reasons aren't enough, consider the fact that many times the
options to fix a defect may be greatly reduced toward the end of a
project.
All
of these reasons should support the need to find defects as early as
possible in a project. In fact, the emphasis should be on defect
prevention, but that requires defect measurement and analysis, along
with defined processes that can be refined. So, for the purpose of this
article, we will focus on early defect detection as maturity level 2
and leave the higher levels of test maturity for other future articles
on this topic.
How
Do Filters Work?
You
might think of a filter as a gateway or check before going on to
successive stages of a project. in the figure below, an example is
shown of a filter between the process of defining requirements and
performing application design. Such a filter would ensure that the
requirements to be used by application designers met pre-defined levels
of quality.
The
filters could consist of checklists, reviews, inspections, or any other
method that would check for the presence of defects.
How
Are Filters Implemented?
While
people recognize the need and benefits of filters, it's hard sometimes
to get others in the organization to see the same need. Reasons vary,
but many times the people creating the defects aren't impacted enough
to be motivated to change.
The
implementation of filters is a process decision and should be driven by
the senior management of the organization. This means that management
must take an active role in designing and introducing filters to the
organization. Unless this happens, people will not see the importance
of adhering to a process that uses filters.
My
experience is that filters are best introduced gradually into an
organization. The implementation effort can be determined by scope and
intensity.
Scope of Implementation
One
approach to implementing filters is to pilot the process on a small
project where the results can be measured. In addition, the lessons
learned on a small, lesser risk project, are less risky than on a
mission-critical project.
Intensity of Implementation
Since
walkthroughs are less formal than reviews or inspections, they are a
natural entry point for implementing filters. This is especially true
for first-time implementations or when strong resistance is involved. A
word of caution is to be aware that since walkthroughs are less intense
than inspections, the defect discovery rate will be lower
Training is Essential
People
must be trained in the filtering process and interpersonal facilitation
skills that are critical to the success of inspection-based techniques.
If you can't or don't want to design and conduct your own training,
Rice Consulting Services has a course on Implementing Defect Filters.
Many
filter implementation efforts have failed due to damaged relationships.
This usually occurs because the reviewers have not been trained in how
to give interactive criticism. In addition, it is easy to drift into
critiquing the producer instead of the product.
Common
Inhibitors
In
implementing defect filters, you may likely hear one or more of the
following objections. Don't be argumentative, just be prepared to
answer them with facts.
"We
don't have time to perform filtering tasks"
Actually,
experience from companies show that although they may appear to take
more time, filters actually reduce overall project time by as much as
15%! Plus, the lower defect rates will reduce costs and increase the
quality of the release.1
You can
always bring up the point that there is always time to perform rework!
"People
can't get along well enough"
Filtering
activities such as walkthroughs and inspections are certainly affected
by cultural issues. There are some options:
-
Initially focus on non-interpersonal filters, such as checklists.
-
Initially involve people with interpersonal skills in review sessions.
-
Conduct training to increase interpersonal skills.
Make
filters a part of new and challenging projects. If someone wants to be
on the "A" team, they will need to follow the processes management have
in place.
"We
have no processes."
Okay,
that's a big problem. The solution is to define simple and repeatable
processes that can accommodate the filtering concept. The key words
here are simple and repeatable. Once again, management needs to
reinforce the importance of processes or else people will ignore them.
Conclusion
Defect
filters are simple and effective ways to dramatically improve the
quality of software. Not only is overall quality improved, but project
time is reduced. Filters can be implemented gradually and there are
several good resources you can use. The keys to success are to have a
process in place and to stay aware of interpersonal issues.
Resources
Footnotes
1. From Software
Inspection Process
by Susan Strauss and Robert Ebenau. JPL estimates they have saved $7.5
million dollars from performing 300 inspections. AT&T found
that
inspections reduced production defects by 90% (only four production
defects were reported). Bell Northern found that they can find 80% of
all defects by performing inspections, the defects are found two to
four times faster than by testing and they save 33 hours of rework per
defect found in inspections.
All materials on this site
copyright 1996 - 2008, Rice Consulting Services, Inc.
Rice
Consulting Services, Inc.
P.O. Box 892003
Oklahoma City, OK 73189
405-691-8075
"Leaders
are made, they are not born. They are made by hard effort,
which is the price which
all of us must pay to achieve any goal that is
worthwhile." -- Vince Lombardi
This site best
viewed with the Mozilla Firefox
browser!
|
