Without Defined Requirements
W. Rice, CSTE, CSQA, CTFL
the past few years in teaching and consulting with testers and test
managers worldwide, I have noticed something interesting. On one hand,
testers complain they hardly ever get user requirements adequate for
testing. On the other hand, when discussing what to base tests upon,
the main response is "user requirements."
When I suggest there are
other ways to design and evaluate tests, the response is similar to the
time I accidentally walked into the glass of the revolving door at the
Portland airport- surprise, followed by intense laughter.
I'm serious, though. There really are other ways to design and evaluate
tests besides requirements. First, let's look at why you may need to do
By the way, let me be quick to say that I am a big believer in having
solid, testable requirements for any project. Without them, you are at
risk for a multitude of problems, including not knowing if an
acceptable solution has been delivered. However, I also know we must
anticipate real-world situations in which we may not (or should I say
"probably not") get testable requirements.
Here are just a few of the situations that may require other techniques
than requirements-based testing.
"We don't do requirements well" -
The title of this article is intentional. The word "defined" explicitly
indicates that requirements are in a documented form that can be
discussed, reviewed, managed and all the other things you do with
things that are written - including filing them away and forgetting
them. Hopefully, we'll not do the filing away and forgetting
Failing to create and manage requirements well is a cultural and a
process problem. No matter how important the testers see requirements
as being, unless the project team and customers sees them as important,
you're not going to get them or at least to the degree you need them.
So, even though you may rant and rave, at the end of the day you still
don't have testable requirements.
Commercial Off-the-Shelf Software -
Let's take the example of buying a commercial software product. You
should have defined requirements, but not for the purpose of building
the software. Your concerns relate to the fitness of use. At the least,
your requirements will be a list of things the software should do for
you. Hopefully, the requirements will also define desired or needed
attributes such as performance and usability. However, these
requirements will not describe the processing rules of the software or
of your organization.
Your task then becomes to design
tests that validate the software or system will do what you need it to
do, how you need to do it.
Stale Requirements -
Perhaps at one time in the past you had a set of current and correct
requirements. However, the passing of time was not kind to them.
Business has changed, laws and technology have changed and now those
requirements are not even close to how the system or your organization
Lack of User or Customer Input -
The project team, including testers, may want testable requirements.
However, the users and customers simply will not provide them. Perhaps
they can't reach agreement on what is desired. Or, they will not give
it their time - "We have a business to run here. You do your job and
we'll do ours." Sure, they'll tell you when you've missed the mark -
they just have a problem telling you where the mark is. That's very
People Think They are Writing Good
To some people, requirements are sufficient even though they contain
vague words and omit important details. These people seldom review
these kind of requirements and in reality, little can be done with
them. They are created as a step in a process to say they have defined
requirements for the purpose of compliance.
Incorrect Requirements -
Remember the pie charts and research data that shows that most of the
defects on a project originate in requirements? The problem is that
even in well-defined requirements there are still errors.
Requirements Convey "What" not "How" -
This is the correct nature of requirements. They should not convey how
to implement a feature in the software. In addition, they should not
convey how a user performs a task - that's what a use case does. So,
when you design requirement-based tests you are focusing on the
processing rules, not on the sequence of performing those
This means that you will need more information than contained in
requirements to design a complete test in any event.
Testing Using a Lifecycle Approach-
In a lifecycle testing approach, items are reviewed and tested as they
are created. In addition, the tests and their evaluation criteria are
based on previously defined items such as concept of operations,
requirements, use cases and design (both general and detailed).
For example, when testing code at a unit level, you can base some tests
on requirements, but you will also need to use the detailed design for
things such as interface testing, integration testing and detailed
functional testing. You will also need to know the coding constructs
for structural testing.
Other Methods of Test Design and Evaluation
Taking our eyes off requirements for a moment can reveal some
interesting things. Here are some other places we can look for test
This is probably my favorite source of non-requirement based tests and
is very well suited for user acceptance testing. You may be thinking,
"Isn't this a good application for use cases?" You would be correct,
except you may not have adequately defined use cases.
User scenarios are based on business or
operational processes and how they are performed. Of course, this
requires that you or someone else in the organization understand those
processes. Even better, it would be great if those processes could be
in a documented form someplace.
It is amazing that when the entire functional process for a task is
defined, the individual paths or scenarios can become very apparent.
Each of these paths or scenarios can be the basis for a test script. If
test scripts don't appeal to you (and that's okay), then you can think
of a series of test cases or conditions that will accomplish the
testing of the process based on user scenarios.
The big question is where to find the knowledge of the processes. There
are several possible sources of process information in most
User knowledge -
You will have to interview them and work with them to map the process.
It helps to speak with multiple users to get the full picture.
Training materials - Sometimes you will find
process flow charts, etc. in training documents.
Business process re-engineering documentation -
At some time in the past there may have been an effort to re-engineer
processes. This is one deliverable from that type of effort.
System documentation - Yes, you may find
documented processes there as well. Just make sure it is still current.
It takes a little practice to get good at mapping out the major
process. I prefer to use flowcharting as the technique, since there are
so many good flowcharting tools on the market. One of my favorite tools
is SmartDraw (www.smartdraw.com). If you visit their web site, you will
see some examples of defining processes with flowcharts.
One important thing to remember is to keep the level of detail general
enough to manage the complexity. My guideline is to limit the major
decisions shown in the chart to about five or six. Even this low number
of decisions can yield twenty-five or more scenarios.
I do not advocate the need to test all scenarios unless the risk is
high. You will need to prioritize and choose the scenarios that are the
most likely and most critical.
Here is how a typical process flow might look.
1 - Applying a Late Payment Penalty
If we identify some of the scenarios paths, we can see:
Figure 2 - Distinct Scenarios in Applying a Late Payment
Usage patterns are just ways that people naturally use a system or
software. For example, some people are partial to using a mouse for as
many actions as possible. Other people use the mouse and keystrokes.
Some people make maximum use of shortcuts, while others ask, "What's a
The problem with this
source of tests is that if you try to test all the software functions
in all the usage patterns, you won't have time to finish the test. For
this reason, I usually recommend this type of testing be done in
Generic Test Sources
There are at least fourteen sources of tests:
1. Field and data edits - Are the field edits and attributes correct?
2. Record handling - Is information processed correctly from point to
point in the application without data loss?
3. File processing - Are files accessed and processed correctly?
4. Field and data relationships - When two or more data items are
related, are the relationships correctly processed?
5. Matching and merging of data - When two or more data sources are
involved, are the data merged together correctly?
6. Security - Are the data protected from malicious access, both
internal and external to the organization?
7. Performance - Does the application have sufficient performance to
meet user needs in terms of response times, throughput, handling of
load, etc.? This can also include stress testing to simulate the
conditions that cause the application to fail due to excessive load.
8. Control flows (processes and code) - Do the various control flows in
the code perform functions as intended?
9. Procedures and documentation - Do procedures and documentation
accurately describe the application's behavior and function?
10. Audit trails - Can transactions and functions be traced to a
specific time and user?
11. Recovery - Can processing be recovered from an error state?
12. Ease of Use - Can someone in the intended user audience use the
application without extensive training or additional skills? Can the
functions in the application be performed easily?
13. Error Handling - Are errors (both expected and unexpected) handled
14. Search - Can data be queried and found in the application?
This is functionality that is contained in most applications, such as
adding, updating, deleting and querying data. The catch is that these
functions are performed differently from one application to the next.
In the absence of any pre-defined knowledge of system or software
behavior, exploratory testing can be a way to both learn and to test.
You may find behavior that is obviously incorrect and you may also
observe behavior that leads to further research to determine
correctness. Although I think exploratory testing is a helpful method,
my main concern is how to know if I'm seeing a bug or a "feature." Be
prepared to spend significant time in research or reporting defects
that are not really issues.
What is Correct Behavior?
We have looked at ways to define the test, but how do we know if the
observed results are correct? In a requirements-based approach, the
answer would be to look at the requirements. However, we need to
consider other sources for our oracle.
Here are a few sources of expected application behavior:
An Independent Processing Source -
Let's say you are testing a computation in a new system that is
replacing an older system. To check the result you could perform the
same computation on the older system and compare the results. Or, you
could create another source, such as a spreadsheet to have an
independent source of comparison. Your approach would depend on what is
available and what it would take to create something new.
In some cases, you can ask a user to assess the correctness of your
results. This could even be a part of acceptance testing or sign off.
Of course, this depends on the knowledge, willingness and availability
Other Subject Matter Experts - Your oracle
may be a person who is not a user, but understands the processing rules
enough to evaluate the results.
System Documentation -
The quality (accuracy, detail and clarity) of the documentation is the
key factor for its use in evaluating test results. This documentation
can include user guides as well as technical design documents.
Training Materials -
These may be detailed enough for test evaluation. At the very least,
you should be able to verify that the documentation is accurate.
Before totally dismissing the above methods in favor of requirements,
consider that even good requirements can have errors. So these methods
can be another source of expected results even when requirements are
For many years people have understood that in dealing with software and
systems there may well be a difference between how a system is
described on paper and how it is actually intended to behave. This
difference is addressed in two views of test and evaluation:
verification and validation.
based on project deliverables and evaluates something to determine if
it has been built correctly according to specifications or the "paper
world." This is primarily the developer perspective. Validation is
based on evaluating something based on user need or simulated real
world conditions. The intent is to determine fitness for use from the
customer or user perspective. Without both perspectives, you are at
risk of building or acquiring something that meets specifications, but
is unfit for use.
Requirements-based testing is verification. It is a solid and valuable
technique if you have clear testable requirements. The techniques
discussed in this article are ways to test from the real-world usage
perspective and can be considered validation.
Unfortunately, testers have been trained for so many years that the
only way to test is based on requirements that many have forgotten or
never learned other methods of testing. I something refer to this
situation as "The Lost Art of Validation."
there are several phases and many types of testing. Requirements can
define what should be tested and what defines correct behavior for some
of these test phases and types, but not all of them. Requirements are
best applied at the system phase of testing. The danger of basing user
acceptance testing on requirements is that they may not accurately or
completely define what is needed or desired.
Hopefully, this article has helped expand or reinforce your views of
validation. In addition, I hope it gives you some new techniques to use
in your test design and evaluation.
materials on this site
copyright 1996 - 2009, Rice Consulting Services, Inc.
Consulting Services, Inc.
P.O. Box 892003
Oklahoma City, OK 73189