Randy Rice's Software Testing Site

The Cost of Software Quality - A Powerful Tool to Show the Value of Software Quality


By Randall W. Rice, CSTE, CSQA, CTFL

Joseph Juran, one of the founding fathers of the quality movement, published the idea of a "Cost of Quality" or "CoQ" in his Quality Control Handbook in 1951. Since it was a metric developed in the manufacturing context, it is often seen as the quality-related costs seen in building physical products in an assembly line method.

The CoQ is a little confusing because it can defined in different ways. To some, it is not the total cost to achieve quality, but rather the cost of NOT achieving quality. This is also called the "cost of non-conformance." This was the view of Phillip Crosby, who wrote the book Quality is Free in 1979.

There is also the "Total Quality Costs" (TQC) view that is comprised of more than costs of non-conformance, such as the costs of assessment and prevention. The confusion arises because many people equate TQC to the CoQ. I tend to take that view and include prevention and assessment costs in the CoQ.

As a software quality practitioner and consultant, I have often considered whether adjustments are needed to accurately apply the traditional CoQ to software quality. This article will explain the traditional CoQ metric and also consider additional costs seen in software projects, but not typically seen in manufacturing.

As a point of distinction, in this article I will distinguish between the "traditional CoQ" and the "software CoQ" (which I'll refer to as CoSQ in this article).

I'm not the first person to write about the distinction between CoSQ and CoQ. Cem Kaner wrote an article published in Software QA on this topic in 1996 and Herb Krasner wrote an article for CrossTalk in 1998 on this topic.

However, I hope that this article helps in some small way to continue the thinking processes to help adapt this valuable metric to software projects. My goal is to promote a metric that clearly shows to management in companies why it makes sense to embrace software quality instead of ignoring it.

What is the Cost of Quality?

Like so many things in quality, usages and meaning of this term vary depending on when and where they were learned. I learned the following computation from my days in preparation for the CSQA (Certified Software Quality Analyst) certification:

The traditional CoQ is comprised of:

The cost of defect prevention - This can include process improvement, training, root cause analysis, or any other activities to prevent defects.

The cost of defect detection - This can include inspections, testing, or any quality control (QC) activities designed to find problems before customers find them.

The cost of failures - This includes both internal and external failure costs. Internal failures are the costs associated with the product before the customer receives it. External failures are the costs associated with the product after the customer receives it. In the manufacturing context, this also includes warranty costs. In many ways, both of these are the costs of rework and waste, but also can include the costs a failure may have on other items, people, projects, etc. These costs can also include the clean-up costs of a failure.

Just as a benchmark, the CoQ can typically range from 15% to 40% (source: qualityportal.com) of the cost to make something. For example, in software, that would be a percentage of the total project costs.

What Does the CoQ Indicate?

If you include the costs of prevention and assessment, the CoQ shows how much an organization spends to deliver products of acceptable quality to customers. However, if your view of CoQ includes just the failure costs, it is the cost of not achieving quality.

In 1979, Philip Crosby wrote one the groundbreaking books of the quality movement in the U.S., called Quality is Free.  My first reaction when I saw the book's title was confusion because it seemed that quality costs a lot to achieve. However, as I read the book, Crosby's point was that while it does cost money to achieve quality, it costs more not to achieve it. In other words, quality improvement pays for itself many times over by preventing costly quality problems.

Another way that Crosby explained it was that when something is done right the first time and no rework is needed, then quality costs nothing. However, then something has to be scrapped or reworked, additional costs are seen. (I still think that it costs something extra to do something right the first time, but that's just me!)

As a real-life example, I was at the Oklahoma City airport recently getting a quick meal before my flight. When I opened the sack, I realized I had someone else's order. So, I took the sack back to the counter and explained that I had the wrong order. They were very apologetic and got my order to me, but since I had already walked away from the counter they had to discard the food I had taken the first time. This is where getting something wrong had a tangible cost to the food vendor.

Until you look at each component of the CoQ, it's impossible to know what the percentage number really means. For example, two companies may each have a 30% cost of quality. Company "A" spends almost all of its CoQ on rework and has unhappy customers, while Company "B" spends almost all of its CoQ on prevention and assessment and has happy customers.

An Interesting Dynamic

You might initially think that when you increase the cost of detection or prevention, you are just shifting the allocation of costs. For example, instead of spending 10% of a product development cost in failure cost, you may think of just spending that 10% in better testing or detection. However, there is a dynamic at work that is interesting and powerful.

For example, for every dollar that you spend in prevention and detection, you often save many more dollars in failure costs. The multiplier varies by organization and when a defect is found, but suffice it to say that this is a winning proposition. This is the case Crosby made in "Quality is Free".

Therefore, it is very feasible to reduce your CoSQ and at the same time improve quality and increase profits. Remember that every dollar spent on failure costs is one less dollar of profit. Consider multi-million dollar project failures and you have  not only a costly project failure, but an equivalent profit loss! (Management may say, "Oh well, it's just a write off to the business." I say, "So are bonuses!")

The Impact of Poor Quality on Software Projects

One cost impact that seems to be very applicable to software is the cost of a failed or delayed project. Although this could perhaps be assigned to internal failure costs, the CoQ context is typically seen as the cost of product rework.

An example of this is when a project is delayed by three months because of high levels of defects found during user acceptance testing. This three-month delay causes:
Another example is of the project that is canceled after spending $44 million on requirements specification and system analysis and design. It is determined at this point in the project that the approach is simply not feasible and that spending further money is too risky.1

Security Costs

Consider the cost to both customers and the company when customers' private data is compromised. In the case of Providence Health Systems of Portland, Oregon, the cost of a data breach has been enormous. According to Information Week, "A company official told the Portland Oregonian that he expects the case, not including litigation costs, to cost from $7 million to $9 million. That includes providing affected patients with access to credit monitoring and restoration services. There's also the prospect of civil lawsuits by state agencies or individuals whose data was stolen, potentially costing millions more."2

"ChoicePoint paid dearly earlier this year when it agreed to pay $15 million in fines after unwittingly selling more than 100,000 consumer credit reports to thieves posing as legit customers. The FTC said the company didn't have "reasonable procedures" for screening customers."3

These costs do not include the costs to customers who had to spend many hours trying to contact the credit reporting agencies, monitoring their credit card accounts and resolving fraudulent transactions.

Opportunity Costs

Poor quality software can also place a company in a position where opportunities are missed because of the inability to respond quickly. Consider the example of a bank that identifies a market opportunity. They present the idea to their IT department which provides an estimate of eight months because of high backlogs of fixes needed because of a previous project that is still experiencing problems, plus there is no project life cycle process in place. The result is that projects take longer and experience a high number of defects. Yet, nothing is ever done to improve the situation because "we don't have the time or the money" or "we're still making a profit, aren't we?"

However, what if the cost of losing the business opportunity is over $20 million dollars in lost profit? You can improve a lot of processes with that kind of money!

Why are the CoQ and CoSQ Helpful?

The main reason these metrics are helpful is because they are focused on money. I often tell people in my training classes on software testing that if you look deep into the eyes of management, you will see dollar signs and calendars.

What I mean is that money and time are at the forefront of many software managers' and executives' minds. It's hard to make a case on the goodness of quality, increased customer satisfaction, or even risk. The executives and managers ask "Where's the profit in it?"  That's the question you must be prepared to answer about software quality and the CoSQ can help you do that.

Juran noted, "Because the main language of [corporate management] was money, there emerged the concept of studying quality-related costs as a means of communication between the quality staff departments and the company managers."4

How to Use This Metric

The CoSQ is a powerful metric to show management that software quality is a business opportunity, not a cost center. Yes, "quality" is a fuzzy concept and fails to resonate many times, but the costs are not fuzzy or trivial. In fact, the costs are very measurable and the companies that understand their CoQ often see higher profits because waste is reduced and opportunities are realized.

I suggest taking a baseline snapshot of your organization's cost for the major cost areas (The items shown are a suggested starting point. You may think of others.):

Prevention costs:
 Detection costs:
 Internal failure costs
 External failure costs
 Another good thing to do is to start measuring the average cost per defect. I know of situations where people were surprised to learn that the average cost of a defect exceeded $5,000 in their organization. You can improve a lot of things for the cost of just 4 or 5 defects!

Use Examples from Your Organization

These shouldn't be hard to find. Try to find situations where problems caused high dollar costs, or where problems have been prevented by improving processes or by inspections and testing.

Be careful with this! People can get very defensive when you use their projects as your poster child. Before using any example as a case study, I always talk to the people involved and get their blessing as well as their feedback on the accuracy of the numbers.

A Word of Warning

Don't overreach in your calculations or oversell the value of software quality. The first thing that will happen when you start discussing the CoSQ metric is that your numbers will be scrutinized and discounted. If your numbers turn out to be inflated and your claims appear to be too high, you will lose both your message and your credibility.

Instead, be conservative in your cost assumptions and don't over-promise value. You can do this and still show the powerful value of software quality.

About Your Customers

Dr. Juran was big on the idea that we have both internal and external customers. I agree that we must consider both perspectives.

I have heard some people downplay the importance of internal customers because they don't buy products or services. True, but I would argue that internal customers also generate profits because they sell things, they keep valuable customers, they develop new products and services to sell, etc.

While the external customers are very, very important, the internal customers are the ones that make the decision whether or not to outsource IT function

Many companies measure customer satisfaction. I used to think this was great until I read a book by Jeffrey Gittomer called,Customer Satisfaction is Worthless, Customer Loyalty is Priceless. His point is that a satisfied customer thinks your product or service is OK, but could be easily swayed to a competitor. A loyal customer believes in your product or service so much that they are with you for a long time. The only reason they leave is because they are forced to, by lowering the quality of your products and services.

This reminds me of a time I was presenting the findings and recommendations of a software testing assessment to an executive team at a financial services company. This company was spending over $250,000 a year just on the salary costs of the developers who were resolving daily software problems. The system was responsible for moving billions of dollars daily.

At one point in my presentation in which I described the risks in their current methods, the VP of Marketing spoke up and said, "The only metric I care about is how many customers we have lost, and we haven't lost any." Therefore, he saw no need for improvement.

About six months later, the CIO that had hired me for the assessment called. She was working for a different company and told me that a major "incident" had occurred and they did lose customers. The problem was that all the people who could have fixed the chronic nature of the defective software had left the company in burnout and frustration (I suppose that could be a metric as well).

Conclusion

Dr. W. Edwards Deming taught that a business is comprised of functions and activities, which can be identified, measured and improved. The goal of this effort is not to score higher on an assessment or to reach a certain level or to win an award. The goal is to improve the way a business does its work so that it becomes more profitable and healthy.

One of the key metrics that can drive this type of improvement is the Cost of Quality (CoQ). For software, we need to consider a wider view to include the costs that are typically not seen in manufacturing.

The CoSQ can be a valuable tool for conveying the value of software quality for the purpose of improving the business. Don't oversell it, but use it as a way to show the value of quality and a way to benchmark your progress.


1. This happened in California when a vehicle registration and driver's license database was never deployed after $44 million in development costs--three times the original cost estimate. Source: Making IT Better:
Expanding Information Technology Research to Meet Society's Needs
2. Information Week, "The High Cost of Data Loss", March 20, 2006
3. ibid
4. Gryna, F. M. (1988) "Quality Costs" in Juran, J.M. & Gryna, F. M. (1988, 4th Ed.), Juran's Quality Control Handbook, McGraw-Hill, page 4.2.


Bio

Randall Rice is a leading author, speaker and consultant in the field of software testing and software quality. Randy has 30 years experience building and testing mission-critical projects in a variety of environments and is co-author of the book, Surviving the Top Ten Challenges of Software Testing. He is the author and instructor of Testing SOA and Structured User Acceptance Testing courses, presented by Rice Consulting Services. You can contact Randall through his web site at www.riceconsulting.com.

All materials on this site copyright 1996 - 2008, Rice Consulting Services, Inc.

Rice Consulting Services, Inc.
P.O. Box 892003
Oklahoma City, OK  73189
405-691-8075