Managing
Expectations in Testing Dirty Systems
By
Randall W. Rice
One of
the most important activities in a software project is to manage
people's expectations. Even in projects that are well-managed, people
are prone to bring varying and sometimes unrealistic expectations to
the table. The thing that often distinguishes well-managed project from
those that aren't is how the project manager is able to effectively
deal with a variety of expectations and guide the project to a
successful completion. There is a fine art to managing expectations and
still meeting user needs. For a more detailed discussion of this topic
in general, we recommend Naomi Karten's book, Managing Expectations,
from Dorset House.
William
E. Perry and I discussed the Expectation Gap in our previous book, Surviving the Top Ten Challenges
of Software Testing, but feel that the
concept bears repeating here.
The
expectation gap is the difference between what the user expects and
what the user gets. Ideally, the gap should be zero. Realistically, it
is a challenge to keep the gap small.
The best
way to keep the expectation gap small is to involve users and customers
from the outset of the project. This can be done in a variety of ways,
including reviews, prototyping and walkthroughs.
In
testing dirty systems, expectation management becomes important because
of the following characteristics of dirty systems:
- Testing
is often seen and practiced as an end-project activity.
- Certain
people tend to influence project decisions to a greater degree than
others.
- People
tend to override processes.
- Early
project assumptions and requirements are seldom documented in writing.
We will
deal with each of these dirty system characteristics in more detail,
along with solution strategies to help you overcome some of the
problems associated with them.
Testing is often seen and practiced as an
end-project activity.
When
testing is performed as an end-project activity, there is a lot of risk
accumulated at a very critical point in time. Problems, which have not
been seen or realized throughout the project, become evident.
Sometimes, the resolution of these problems are very difficult due to
the ripple effect seen when fixing one problem requires changes to many
other areas of the system.
Big-bang
testing is risky because options to fix problems are reduced at the end
of the project. This is often seen in the case of performance problems
where the options to a) tune the application software, b) tune the
database and c) upgrade to more powerful hardware. The problems or
reduced options appear in losses of time and money. The project is
essentially held hostage to whichever solution must be delivered to
make the project a success. People who sponsor projects do not expect
to have months of work depend on the success or failure of one aspect
of the project, which may become apparent at the last-minute and which
may be a very expensive solution.
End-project
testing is also the time when end-user see the application for the
first time. If the application looks good to them, then life is good.
However, if the users feel that the application is not what they
expected, then the project is at risk of user non-acceptance. Some IT
managers have the attitude, "If the users don't like the new system,
then that's their problem." The reality is that if the users don't like
the system, they will be less efficient in their work. Although there
are cases where the users are on different political agendas, there are
also valid reasons for non-acceptance. These situations must be
considered objectively to avoid a last-minute project failure. These
types of last-minute expectation problems can be avoided by involving
the users and customers throughout the project.
Solution
Strategies
- Perform
walkthroughs and reviews of project deliverables, such as requirements
documents and prototypes.
- Involve
users in early testing and system testing.
- Perform
performance and usability testing early in the development and
maintenance cycles.
Certain
people tend to influence project decisions to a greater degree than
others.
A casual
observation of most organizations will reveal the strong personalities.
To hear these people describe their function in the organization, you
would think the organization revolved around them only. These are the
people who tend to influence project decisions to their benefit, with
little regard to the big picture.
In
testing dirty systems, these highly influential people may try to exert
pressure on the test team leader to test in a way that will prove the
person's point to management. Dirty systems are more susceptible to
this kind of influence because of a lack of process and standards for
test execution and reporting.
Solution
Strategies
- Define
objective criteria to manage test activities
- Define
objective test reporting criteria
- Define
objective test reporting standards
People tend to override processes.
In dirty
systems, people tend to bypass processes (if processes exist) to
achieve an objective more quickly, or to achieve an objective outside
of the scope of the project or to meet an expectation. This is a major
reason why dirty systems become that way!
These
shortcuts almost always leave something undocumented, undone, or added
to the project, which will adversely affect the overall quality of the
project. Over a period of time these shortcuts accumulate into a mass
of undocumented features and additional scope that leaves people
wondering "How did this happen?"
In
reality, processes are in place for a reason. The process adds
consistency, control and repeatability to the work being done - all the
things that are missing in dirty systems! To have a process defined but
not in use is to virtually not have a process at all.
The
reason people tend to bypass steps in the process due to human nature.
Some processes get complicated to follow and it's easier to take the
path of least resistance. A true quality assurance group is one that
manages quality and does not perform the actual testing of a product.
The value of QA is that it assures compliance to processes. If defects
are to be prevented, the job of QA becomes to fix the process to
prevent defects. This does not happen if there is no process or there
is no one to monitor compliance to the process.
Solution
Strategies
- Establish
a QA function to assure compliance to processes.
- Emphasize
the importance of process to management and staff.
- Get
management to make the message of the importance of following processes.
Early project assumptions and requirements are
seldom documented in writing.
If user
requirements are not documented in writing, do they still exist? The
answer is "yes, they exist in the expectations of everyone on the
project." It's a sure thing that each person's expectations will differ
in some way because they view the project from their own perspective.
For a customer service person, the expectation is to be able to serve
the customer more efficiently. For the technical support person, the
expectation is a system easy to maintain. The fact is that these
assumptions and expectations exist on every project. When the initial
project objectives, assumptions and requirements are not defined in
writing, there is no common starting point, so you already have an
expectation gap from the start! Therefore, you are assured of an
expectation gap at the end of the project as well.
Since
the lack of documented requirements is a foundational symptom of dirty
systems, testers in these environments must be aware that they will be
battling the expectation gap throughout the project. These issues will
become apparent especially as users and customers become involved in
testing.
Process
From the
process perspective, the main process to be performed is that of
requirements management. Like CM, there is a growing body of knowledge
in requirements management. These processes for requirements management
can be expressed at a high level as:
1.
Define the problem to be solved.
2.
Involve the correct people
3.
Interview stakeholders
4.
Define the requirements as a high level
5.
Review the high level requirements
6.
Define requirements at a detailed level
7.
Review the detailed requirements
8.
Finalize and publish requirements
QC
The
following questions can be the basis of a checklist for requirements
management:
1. Is
there a requirements management process in place?
2. Is
there a standard for the type and amount of information to be
documented for requirements?
3. Are
standards or guidelines in place to prioritize requirements?
4.
Have requirement categories been defined and published?
5. Is
there a specific person responsible for requirements administration?
6.
Have people been trained in how to gather, verify and manage
requirements?
7. Is
there a contact person in the user and development groups responsible
for developing and reviewing requirements?
8. Is
there a tool in place to document and manage requirements?
People
Project
Manager -
Oversees the requirements gathering effort at a management level. For
example, the project manager would ensure that the requirements do not
creep outside of the intended scope of the project.
Systems
Analyst - Works with the users to elicit and
define user requirements. The systems analyst is the person who takes a
variety of needs, ideas and technical options and creates a workable
technical and business solution. The systems analyst must have a blend
of skills, which include working with users, understanding business
needs, and understanding technical issues and solutions. Many times,
the systems analyst it is the bridge between users and the development
staff.
Users
- Provides input to the analysts during the interview
process. the users are an essential group to the requirements gathering
process, as they provide the critical information needed to meet real
world needs. It is important to understand that to each individual user
will have a slightly different perspective on what is needed in solving
a problem or delivering a new application. In gathering user input,
it's necessary to talk to a variety of users representing a variety of
levels, such as senior management, middle management, and " in the
trenches" workers.
Customers
- Provides input to the analysts during the interview process. The
customers are the people who are sponsoring a project and may or may
not be the actual users of the delivered solution. Therefore, the
customers' needs and agendas may likely differ from those of users. It
may take a variety of creative techniques to obtain input from users
and customers, such as focus group sessions, facilitated group
interviews, questionnaires, and surveys.
QA
- A true QA group facilitates the requirements gathering process,
although they may not be directly responsible for documenting the
obtained requirements. One of the main responsibilities of the QA group
is to ensure that the requirements gathering process is being followed
correctly. An example of this would be for a QA analyst to be part of
the requirements interview process and to follow a checklist of
requirements attributes that should be present in the requirements that
are to being gathered.
Test
Team Leader - The test team leader can also
play a leadership role in gathering requirements. The test team leader
is very familiar with what is needed in a general nature in terms of
requirements and can help keep the requirements gathering and
documentation process on track. The test team leader is a good judge of
scope and feasibility while requirements are being defined.
Testers
- Since testers work with requirements as much as developers, they know
better than anyone else the level of detail and scope that is needed
for workable requirements. Therefore, testers are an ideal group of
people to be involved in writing requirements. When testers are
involved in the requirements gathering process, they see gaps and
inconsistencies very early in the process. In addition, testers also
get an idea very early of what the test objectives will be.
Tools
The
primary tools for this topic are requirements management tools.
Representative examples of these tools are listed in the tools section
of this chapter, but include:
- Requisite
Pro by Rational Software
- Caliber
RM by Technology Builders, Inc.
- DOORS
by Telelogic
Conclusion
Projects
fail for a variety of reasons, but a leading cause is missed
expectations on the part of users and customers. It is possible with
the tools and techniques available to IT professionals to manage
expectations effectively. RCS has a new course on requirements
management, which is a key activity for managing expectations.
All materials on this site
copyright 1996 - 2008, Rice Consulting Services, Inc.
Rice
Consulting Services, Inc.
P.O. Box 892003
Oklahoma City, OK 73189
405-691-8075
"Leaders
are made, they are not born. They are made by hard effort,
which is the price which all of us must pay to achieve any goal that is
worthwhile." -- Vince Lombardi
This site best
viewed with the Mozilla Firefox
browser!
|
