What is User Acceptance
Testing?
by
Randall
W. Rice, CSTE. CSQA, CTFL
Over
the past few years, there have been a variety of definitions applied to
User Acceptance Testing (UAT). Your success in validating that a system
or application is "fit for use" by the intended user depends on how you
define this phase of testing.
For example, if you see UAT as a functional test based solely on user
requirements, you will likely miss the same things in testing that were
missed in defining the requirements. Another example is that if you see
UAT as the tests that can be automated in agile testing, you may miss
the "hands on" assessment of the actual user to determine how the
application actually meets their needs.
I need to be clear that I am not saying that you must use my definition
of UAT or else you will be hopelessly doomed to project failure. What I
am saying is that there are a variety views of UAT that may or may not
meet your needs - and that you had better be sure you know the
differences in the ways UAT is defined.
Who Should Perform UAT?
I'm a radical, so I suggest the actual current and/or future users do
the planning, testing and evaluation for acceptance. There are those
who see differently. Some people prefer to have testers take the role
of users. Others have a UAT team composed of former users who now do
nothing but testing. In other organizations UAT might be performed by
business analysts.
I like to have actual users perform UAT because 1) they are going to be
using the system eventually anyway, 2) they know the current ways of
doing their jobs and therefore can tell when something won't work for
them, and 3) they should know what they will be getting in terms of
system features and quality.
This is not without challenges. Here are some of the reasons that are
typically given for not involving users in UAT:
1) Not enough time due to performing normal job duties
2) No training on the new system to be tested
3) No interest
4) Not enough testing knowledge or experience
These are all significant challenges, but can typically be managed.
What is "Acceptance?"
In reality, contracts have been signed and money has been spent, so
"acceptance" is typically not an "accept or reject" proposition. UAT is
more about finding gaps between how the system works and how
operational processes are performed.
A Word About Validation
It seems to me that the distinction between verification and validation
has been lost in recent years. It's important that we understand the
difference between these two types of testing so that we can get a
complete and accurate evaluation of what we are testing.
I'll refer to the
ISTQB glossary at this point, which references ISO 9000:
"verification: Confirmation by examination and through provision of
objective evidence that specified requirements have been fulfilled.
[ISO 9000]"
"validation: Confirmation by examination and through provision of
objective evidence that the requirements for a specific intended use or
application have been fulfilled. [ISO 9000]"
In IEEE 1012-1998 for Software Verification and Validation, the
identical definitions are given as referenced in ISO 9000, but there is
some additional notation:
Validation
"1 - In design and development, validation concerns the process of
examining a product to determine conformity with user needs."
"2 - Validation is normally performed on the final product under
defined operating conditions. It may be necessary in earlier stages."
Verification
"1 - In design and development, verification concerns the process of
examining the result of a given activity to determine conformity with
the stated requirement for that activity."
So, allow me to paraphrase a bit. Verification determines if something
has been built according to specifications. Validation determines if
something works as intended in the user's environment and meets their
needs.
Those are two greatly differing activities!
UAT is generally considered to be validation. In fact, it is typically
the only time validation is performed in a project. System testing,
integration testing, unit testing, as well as reviews are all examples
of verification because they are based on specifications and
requirements. Therefore, it is very important that in the one
validation opportunity we have, we get it right.
Now let's look at some of the differing views of UAT.
The Beta Test
In this definition, software is given (or sold) to users for them to
try as they do their normal activities. Some beta testers go beyond
that and actually try to break the software.
The problem with beta testing is that you never know how much testing
was actually done. Even worse, you never know in advance how much
testing will be done. If you're relying on people to beta test your
software, you're probably going to miss a lot of things.
Beta testing does serve a useful purpose in finding configuration
problems that might not otherwise not be found in your own testing. It
also provides the opportunity to get feedback about the product early.
However, this still doesn't really meet the bar of validation because
beta testing does not imply acceptance and it often lacks the rigor of
a controlled test.
Agile Acceptance Testing
In agile development, acceptance testing is the functional testing that
is based on a user's stated needs. Functional tests are designed based
on those needs. Some of the functional test are automated, while others
are performed manually. In agile methods, the developer may be the one
actually performing these tests. This means that the user may or may
not be observing the results of the test.
In the absence of defined requirements (at least to the extent as they
are seem traditionally) these acceptance tests are close to the
functional tests that would be considered system testing in other
development approaches. It's good that these tests are being done, but
these are still more verification than validation.
By the way, the other major forms of testing seen in agile are test
driven development (TDD) and exploratory testing. Both of these types
of testing focus on finding defects as opposed to validating fitness
for use.
The Traditional View
There is also much confusion in the traditional view of acceptance
testing. Some take UAT to mean that the system or application is tested
by users to confirm that documented requirements have been met. Others,
such as myself, see UAT as a real-world validation of the system or
validation performed by users.
The difference is what the test is based upon for the purpose of the
analysis of results. The problem with basing UAT on requirements is
that 1) many times we don't have well defined requirements on projects,
and 2) even when we do have well defined requirements they can have
defects in them.
At this point, many people ask, "Then, what do you base your test upon?"
Actually, there are many ways to test without defined requirements. You
can refer to my article,
Testing Without Defined Requirements to see a further listing
of those methods.
For the purposes of UAT, a very effective way to design tests is to
base them on user processes. These can be workflows or other process
driven activities that people will use the software to accomplish. Not
only is this a different basis for testing, but it is a process driven
view that is typically not achievable based on requirements. In fact,
this may be the only opportunity to perform any type of business
process validation. One of the most critical concerns in any system
deployment is that the system will work immediately after deployment to
support what the users do. Business process validation helps assure
that business processes are supported by the application or system.
I also combine process-driven test design with data-driven test design
to achieve a test that models the real-world. To illustrate this, I use
an analogy of plumbing. Without water flowing through the pipes, all
you can tell is whether or not the pipes are in the right place. When
water is flowing through the pipes, then you can detect leaks,
determine if the pressure is adequate, the color and odor of the water,
the correctness of hot and cold, etc. The pipes represent processes,
the water represents the data that flows through the processes and the
faucets represent the controls in a system.
I'm a big believer in requirements-based testing. I just believe that
the users need a second, independent perspective of testing to
compensate for the errors and gaps that are typically seen in
requirements -- even when they have been reviewed.
In any event, UAT should be based on a defined set of acceptance
criteria that are defined at the project inception.
Unfortunately, UAT is often performed at the worst possible time in a
project -- at the very end. If you wait until this point to find where
the system fails to meet user needs, there is a huge risk that the
project will be late or not accepted at all. To mitigate this risk, I
advise involving users in reviews and testing that precede UAT.
In my experience, there really isn't much value in a surprise factor of
testing. In other words, it's fine for users to see what's coming their
way long before they have to test it.
Manual vs. Automated UAT
Traditional UAT is also performed manually. I prefer this approach
because 1) UAT is typically performed just once which means there is
little return on investment for test automation, and 2) users need to
be seeing what the software actually looks like and how it performs.
Automation takes this perspective away from people.
There is perhaps a role for automation in UAT. There may be mundane
testing that can be easily automated. However, the user needs to first
understand how the application is performing those functions. Also,
users may be called upon to perform some level of performance and
regression testing, but it is the rare instance that this can be
performed without assistance from people who know how to do this type
of testing. So I consider these to be the exceptions instead of the
rule.
There is also the possibility of automating repeated UAT. This is
sometimes seen when multiple releases are delivered. A user may need to
test new functions in addition to all the functions previously tested
in a prior release, such as in a regression test. This is typically
seen in the agile world as well as any type of iterative development
approach.
So, there can be an opportunity for automating some acceptance tests.
The main concern is that users get to experience first-hand how the
system supports their needs.
Summary
Regardless of which perspective of UAT is performed on your projects,
keep in mind that the need for actual users to be testing the software
in their own world, using their own processes, is not only helpful, but
also needful.
Regardless of your methodology or development approach, there is a
critical need to both verify that the system meets requirements and to
validate that the system meets user needs. Always keep in mind that
defined requirements may not reflect actual user needs. Therefore, you
need both verification and validation.
I hope this brief distinction of some major forms of UAT helps define
your own view of UAT in a way that will add value to your testing
efforts.
All
materials on this site
copyright 1996 - 2008, Rice Consulting Services, Inc.
Rice
Consulting Services, Inc.
P.O. Box 892003
Oklahoma City, OK 73189
405-691-8075