Strategies for Integration
and Interoperability Testing
By
Randall W. Rice
In a
world where it seems that just about every application interfaces with
a number of other applications, integration and interoperability
testing (IIT) is often not addressed adequately in a project. In this
article, I deal with some of the issues and risks of testing highly
integrated applications and will outline a test strategy for such
applications.
You do
not need to look far to see examples of software interoperability
problems in the real world. Most organizations have a variety of
hardware and software, often from a slate of several competing vendors
whose products do work and play well together. You may be thinking that
web-based applications enhance portability, and in some ways they do.
However, web-based applications also introduce a new set of
interoperability and compatibility concerns, such as browser
compatibility and data interoperability, such as in the case of
exchanging data via XML.
The
Risks of Poor Integration and Interoperability
We've
all felt the frustration of installing a software application on a PC
and suddenly everything starts behaving very strangely ? all the way
from minor annoyances to system hangs and crashes. In some ways it
makes you wish that there were rigorous compatibility and
interoperability standards that software and hardware manufacturers
must follow. On the other hand, if there were such industry-wide
standards and enforcement, we know our choices would be limited and the
products on the market would be expensive. So, we are in a grand
dilemma between reliability and flexibility in which other tradeoffs
also abound.
Lack of
reliability and flexibility are risks that are seen in diverse, highly
integrated systems. It is up to the vendor to verify and validate
applications to the extent the customer will not experience
compatibility problems, however this is the rub. The possible number of
computing platform combinations are endless and, therefore, impossible
to test completely. In this article, we will examine the risks of such
applications and some strategies to deal with testing a domain where
there are no easy answers and no guarantees.
The
risks often seen in integration and interoperability are:
As
data is passed between applications, there is a risk the data can be
lost or misdirected.
Bottlenecks
can be inadvertently built in shared applications which results in poor
performance.
As
software operates together on the same platform, a problem in one
application can cause the entire system to crash.
As
data and control are passed between applications, are the results
correct? Does the application operate as the user has been trained?
Although
the goal of interoperability is to promote maintainability, that is not
always the case. The application must be designed with maintainability
in mind.
If the
application is built on any specific technology, there is a risk that
two years from now that technology will be obsolete.
These
risks can also be seen as the drivers for an IIT strategy. For example,
to test the risk of data loss you would want to include passing data
across defined interfaces and then using the data in other related
applications.
Key Point #1 - IIT should not be confused with
basic functional testing. The purpose of IIT is to test interfaces and
interoperability. It must be assumed that other forms of testing such
as functional and compatibility testing have already been performed.
First,
we need to define some basic terms:
Comparing
an application against a set of established operating standards to
determine operational compatibility.
Testing
to validate that an application functions correctly on a particular
computer configuration.
Testing
the points of integration between units and systems
Testing
with other applications that have met compliance standards; the ability
of two or more systems or components to exchange information and to use
the information that has been exchanged.
Testing
from an external perspective of what the application should do.
Functional testing is performed from a "black box" perspective, with no
knowledge of the inner workings of the application. Functional testing
is cause/effect in nature.
Key Point #2 - There must be a way to build and
maintain an integrated test environment (ITE) that closely resembles
the production environment. Your test results are only as reliable as
the environment in which you perform the test.
Figure
1 - Increasing Complexity of the Test Environment
The
scope of IIT can be wide, including:
These
are the separate software units or segments that must work together
correctly.
These
are separate major systems or applications that must work together
correctly.
Software
applications may reside on a variety of hardware platforms.
Most
hardware platforms have many compatibility issues with internal
components.
- Disparate
data sources and structures
Data
does not come from universal sources and will be seen in many forms.
Applications that work together must be able to accept and convert data
to a usable form.
This
introduces issues such as language and time zones, which add a new
dimension to a project.
- Internal
and external software sources
This
speaks to the issue of control. We can control the test internal
applications, but when we deal with other organizations, we must be
able to communicate and get cooperation to achieve a total test.
Figure
2 - CM in IIT
With
this wide scope of testing, there must be a way to control and
co-ordinate this type of testing, which is where the co-process of
configuration management (CM) comes into play. CM is the process of
tracking all components of an application or system throughout its
operational lifespan. CM is how the development, test, and production
environments are controlled.
In
Figure 2, a typical CM process is used to manage the integrated test
environment and migrate applications to deployment. This picture shows
an ongoing view of systems maintenance where units or components are
checked out of a production source library. This is to prevent multiple
people from working on the same component at the same time. As
low-level tests are performed and passed in the development
environment, they are migrated to the system test environment for
integration testing. Finally, after integration and interoperability
tests are passed, the application is deployed and the source code
becomes the new production version. In the event of problems, the
application is restored from the old version or fixed on the fly and
reinstalled.
Key Point #3 - Communication is a primary success
factor in performing IIT.
In
other forms of testing, such as compatibility, configuration and basic
functional testing, it is possible to perform the tests in relative
isolation. True, you may need to speak with other people to get the job
done, but not to the extent of coordinating tests between systems and
organizations. In IIT, communication and coordination are paramount. If
people aren't talking, then the test isn't going to be successful.
Organizations
have learned an important lesson in two recent real-life technology
efforts: Y2K and e-commerce. Both of these efforts required integration
and interoperability testing to validate that organizations could send
data and control through multiple points to achieve an overall
successful completion. In e-commerce, we also learned that systems are
sometimes closed loops back to the customer, with several trails to be
followed (Figure 3). These types of integration needs have been present
in the past as well as applications have had to handle ever-increasing
levels of integration.
Figure
3 - E-commerce Integration with the Closed Loop Back to the Customer
This
means that to test integration and interoperability you have to pick up
the phone and call people to coordinate the effort. E-mail is great,
but don't depend on it. In fact, never confuse an e-mail or a memo with
effective communication. Effective communication requires both a
sending and receiving party that is actively engaged in the
conversation. This means sending the message clearly and listening
intently!
The
communication factor becomes especially important when the test spans
time zones and multiple languages.
Test
Strategies for IIT
We see
that IIT is often complex and wide-ranging. In fact, most test (and
development) processes fail on scalability when taken to the level of
integration and interoperability. Casual test methods often fail to
achieve the needed rigor and coverage. Rapid prototyping and other
iterative methods are often focused at a single application level which
minimizes IIT. So, what can be done to test effectively for IIT?
Key Point #4 - Understand that exhaustive testing
is not possible in just about all software applications. This applies
especially to IIT. Leverage the risk of deployment by limiting your
exposure. Then, take action on the information learned from problems
found in small scope deployments.
What
this means is that systems that are highly integrated and disparate are
by their very nature risky. It's like eating spicy Mexican food and
then complaining about the heartburn. If you want to still eat spicy
food, you'll need to find a way to deal with the heartburn. In highly
integrated systems, we can mitigate some of the risk by the following
strategies.
- Test
Interfaces and Interoperability the Best You Can, Then Deploy to a
Pilot Group
This is essentially a beta
test, but there are some important considerations.
1. Beta testing has risks, such
as the possibility of production defects, the low quality of feedback
and the high risk of bad PR.
2.
Limited deployment will not mitigate risk in all situations. There are
some cases where limited deployment to a small audience is absolutely
acceptable, such as when the risk is low to moderate. However, in high
risk applications like medical devices, avionics, nuclear power
controls, etc., a defect even in one site can cause catastrophic
results!
- Build
a Robust and Restorable IIT Environment and Perform IIT in That
Environment
There
are still risks due to the reality of non-exhaustive testing, but you
may be able to perform a pilot or other form of limited deployment.
Of
course, the problem with this is that a simulator is a contrived world.
However, for initial tries simulators help reduce some risks. Just ask
any drivers education teacher.
- Test
Using a Live, but Segmented, Environment
This
is often called an Integrated Test Environment. This technique has the
risk of accidentally introducing test data into real operational
processes, but the concept is that test data is processed right along
with real data in production, but the test data is identified in such a
way as to never mingle with production data. For example, test accounts
may span from 600000000 to 699999999. Everything else is live data. The
upside is that you can test transactions as they will flow in live
processing ? just be careful not to do things like mail out bills with
test addresses, etc. Hey, it happened during Y2K testing in some
companies.
Conclusion
There is
much more that can be said about this topic of integration and
interoperability testing. We cover many of these topics in our new
three-day course, Integration and Interoperability Testing. The main
point I wish to convey is that the integration and interoperability
aspects of systems should not be minimized.
As
platform possibilities continue to increase and global business needs
drive more interoperability between critical supply chain system, this
topic will only get more important to testers. The integrated domain is
inherently risky and there is a tension between interoperability
standards and the freedom to innovate and deploy new technologies
quickly.
I
predict that IIT will be a key concern as global B2B application become
reality and the sooner you develop your test strategies for IIT, the
better.
All materials on this site
copyright 1996 - 2008, Rice Consulting Services, Inc.
Rice
Consulting Services, Inc.
P.O. Box 892003
Oklahoma City, OK 73189
405-691-8075
"Leaders
are made, they are not born. They are made by hard effort,
which is the price which
all of us must pay to achieve any goal that is
worthwhile." -- Vince Lombardi
This site best
viewed with the Mozilla Firefox
browser!
|
