Glossary
These are a few of the terms you will encounter frequently in software testing. This glossary will continue to expand, so if you see a term missing, or would like to have a term defined, e-mail me.
Where applicable, the source of the definition is shown. One of the problems in testing is that the common body of knowledge is not standard, but is drawn from a variety of sources. For that reason, you may see a term defined differently here than you may use it. I will be happy to accommodate explanations of alternate usage where appropriate.
A
algorithm
(IEEE) (1) A finite set of well-defined rules for the solution of a problem in a finite number of steps. (2) Any sequence of operations for performing a specific task.
algorithm analysis
(IEEE) A software V&V task to ensure that the algorithms selected are correct, appropriate, and stable, and meet all accuracy, timing, and sizing requirements.
analysis
(1) To separate into elemental parts or basic principles so as to determine the nature of the whole (2) A course of reasoning showing that a certain result is a consequence of assumed premises. (3) (ANSI) The methodical investigation of a problem, and the separation of the problem into smaller related units for further detailed study.
anomaly.
(IEEE) Anything observed in the documentation or operation of software that deviates from expectations based on previously verified software products or reference documents. See: bug, defect, error, exception, fault.
application software
(IEEE) Software designed to fill specific needs of a user; for example, software for navigation, payroll, or process control. Contrast with support software; system software.
architecture
(IEEE) The organizational structure of a system or component.
assertion
(NIST) A logical expression specifying a program state that must exist or a set of conditions that program variables must satisfy at a particular point during program execution.
assertion checking
(NIST) Checking of user-embedded statements that assert relationships between elements of a program. An assertion is a logical expression that specifies a condition or relation among program variables. Tools that test the validity of assertions as the program is executing or tools that perform formal verification of assertions have this feature.
audit
(1) (IEEE) An independent examination of a work product or set of work products to assess compliance with specifications, standards, contractual agreements, or other criteria. (2) (ANSI) To conduct an independent review and examination of system records and activities in order to test the adequacy and effectiveness of data security and data integrity procedures, to ensure compliance with established policy and operational procedures, and to recommend any necessary changes.
audit trail
(1) (ISO) Data in the form of a logical path linking a sequence of events, used to trace the transactions that have affected the contents of a record. (2) A chronological record of system activities that is sufficient to enable the reconstruction, reviews, and examination of the sequence of environments and activities surrounding or leading to each event in the path of a transaction from its inception to output of final results.
B
baseline
(NIST) A specification or product that has been formally reviewed and agreed upon, that serves as the basis for further development, and that can be changed only through formal change control procedures.
basis path testing
batch
(IEEE) Pertaining to a system or mode of operation in which inputs are collected and processed all at one time, rather than being processed as they arrive, and a job, once started, proceeds to completion without additional input or user interaction. Contrast with conversational, interactive, on-line, real time.
batch processing
Execution of programs serially with no interactive processing. Contrast with real time processing.
benchmark
A standard against which measurements or comparisons can be made.
black-box testing
(1) Testing that ignores the internal mechanism or structure of a system or component and focuses on the outputs generated in response to selected inputs and execution conditions. (2) Testing conducted to evaluate the compliance of a system or component with specified functional requirements and corresponding predicted results. Syn. functional testing, input/output driven testing. Contrast with white-box testing.
boolean
Pertaining to the principles of mathematical logic developed by George Boole, a nineteenth century mathematician. Boolean algebra is the study of operations carried out on variables that can have only one of two possible values, i.e., 1 (true) and 0 (false). As ADD, MULTIPLY, and DIVIDE are the primary operations of arithmetic, AND, OR, and NOT are the primary operations of Boolean Logic.
boundary value
(1) (IEEE) A data value that corresponds to a minimum or maximum input, internal, or output value specified for a system or component. (2) A value which lies at, or just inside or just outside a specified range of valid input and output values.
boundary value analysis
(NBS) A selection technique in which test data are chosen to lie along "boundaries" of the input domain [or output range] classes, data structures, procedure parameters, etc. Choices often include maximum, minimum, and trivial values or parameters.
branch
An instruction which causes program execution to jump to a new point in the program sequence, rather than execute the next instruction. Syn: jump.
branch analysis
(Myers) A test case identification technique which produces enough test cases such that each decision has a true and a false outcome at least once. Contrast with path analysis.
branch coverage
(NBS) A test coverage criteria which requires that for each decision point each possible branch be executed at least once. Syn: decision coverage. Contrast with condition coverage, multiple condition coverage, path coverage, statement coverage.
bug
A fault in a program which causes the program to perform in an unintended or unanticipated manner. See: anomaly, defect, error, exception, fault.
C
cause effect graphing
(1) (NBS) Test case selection technique in which the input and output domains are partitioned into classes and analysis is performed to determine which input classes cause which effect. A minimal set of inputs is chosen which will cover the entire effect set. (2) (Myers) A systematic method of generating test cases representing combinations of conditions. See: testing, functional.
change control
The processes, authorities for, and procedures to be used for all changes that are made to the computerized system and/or the system's data. Change control is a vital subset of the Quality Assurance [QA] program within an establishment and should be clearly described in the establishment's SOPs, See: configuration control.
change tracker
A software tool which documents all changes made to a program.
client/server
A term used in a broad sense to describe the relationship between the receiver and the provider of a service. In the world of microcomputers, the term client-server describes a networked system where front-end applications, as the client, make service requests upon another networked system. Client-server relationships are defined primarily by software. In a local area network [LAN], the workstation is the client and the file server is the server. However, client-server systems are inherently more complex than file server systems. Two disparate programs must work in tandem, and there are many more decisions to make about separating data and processing between the client workstations and the database server. The database server encapsulates database files and indexes, restricts access, enforces security, and provides applications with a consistent interface to data via a data dictionary.
code
See: program, source code.
code audit
(IEEE) An independent review of source code by a person, team, or tool to verify compliance with software design documentation and programming standards. Correctness and efficiency may also be evaluated. Contrast with code inspection, code review, code walkthrough. See: static analysis.
code inspection
(Myers/NBS) A manual [formal] testing [error detection] technique where the programmer reads source code, statement by statement, to a group who ask questions analyzing the program logic, analyzing the code with respect to a checklist of historically common programming errors, and analyzing its compliance with coding standards. Contrast with, code audit, code review, code walkthrough. This technique can also be applied to other software and configuration items. Syn: Fagan Inspection. See: static analysis.
code review
(IEEE) A meeting at which software code is presented to project personnel, managers, users, customers, or other interested parties for comment or approval. Contrast with code audit, code inspection, code walkthrough. See: static analysis.
code walkthrough
(Myers, NBS) A manual testing [error detection] technique where program (source code] logic [structure] is traced manually [mentally] by a group with a small set of test cases, while the state of program variables is manually monitored, to analyze the programmer's logic and assumptions. Contrast with code audit, code inspection, code review. See: static analysis.
coding standards
Written procedures describing coding [programming] style conventions specifying rules governing the use of individual constructs provided by the programming language, and naming, formatting, and documentation requirements which prevent programming errors, control complexity and promote understandability of the source code. Syn: development standards, programming standards.
comparitor
(IEEE) A software tool that compares two computer programs, files, or sets of data to identify commonalities or differences. Typical objects of comparison are similar versions of source code, object code, data base files, or test results.
compatibility testing
completeness
(NIST) The property that all necessary parts of the entity are included. Completeness of a product is often used to express the fact that all requirements have been met by the product. See: traceability analysis.
complexity
(IEEE) (1) The degree to which a system or component has a design or implementation that is difficult to understand and verify. (2) Pertaining to any of a set of structure based metrics that measure the attribute in (1).
computer aided software engineering (CASE)
An automated system for the support of software development including an integrated tool set, i.e., programs, which facilitate the accomplishment of software engineering methods and tasks such as project planning and estimation, system and software requirements analysis, design of data structure, program architecture and algorithm procedure, coding, testing and maintenance.
computer system audit
(ISO) An examination of the procedures used in a computer system to evaluate their effectiveness and correctness and to recommend improvements. See: software audit.
computer system security
(IEEE) The protection of computer hardware and software from accidental or malicious access, use, modification, destruction, or disclosure. Security also pertains to personnel, data, communications, and the physical protection of computer installations.
configurable, off-the-shelf software (COTS)
Application software, sometimes general purpose, written for a variety of industries or users in a manner that permits users to modify the program to meet their individual needs.
configuration control
(IEEE) An element of configuration management, consisting of the evaluation, coordination, approval or disapproval, and implementation of changes to configuration items after formal establishment of their configuration identification. See: change control.
configuration management
(IEEE) A discipline applying technical and administrative direction and surveillance to identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verifying compliance with specified requirements. See: configuration control, change control
consistency
(IEEE) The degree of uniformity, standardization, and freedom from contradiction among the documents or parts of a system or component.
consistency checker
A software tool used to test requirements in design specifications for both consistency and completeness.
control flow analysis
(IEEE) A software V&V task to ensure that the proposed control flow is free of problems, such as design or code elements that are unreachable or incorrect.
control flow diagram.
(IEEE) A diagram that depicts the set of all possible sequences in which operations may be performed during the execution of a system or program. Types include box diagram, flowchart, input-process-output chart, state diagram. Contrast with data flow diagram. See: call graph, structure chart.
corrective maintenance
(IEEE) Maintenance performed to correct faults in hardware or software. Contrast with adaptive maintenance, perfective maintenance.
correctness
(IEEE) The degree to which software is free from faults in its specification, design and coding. The degree to which software, documentation and other items meet specified requirements. The degree to which software, documentation and other items meet user needs and expectations, whether specified or not.
coverage analysis
(NIST) Determining and assessing measures associated with the invocation of program structural elements to determine the adequacy of a test run. Coverage analysis is useful when attempting to execute each statement, branch, path, or iterative structure in a program. Tools that capture this data and provide reports summarizing relevant information have this feature See: testing, branch; testing, path; testing, statement.
crash
(IEEE) The sudden and complete failure of a computer system or component.
critical control point
(CA) A function or an area in a manufacturing process or procedure, the failure of which, or loss of control over, may have an adverse affect on the quality of the finished product and may result in a unacceptable health risk.
critical design review
(IEEE) A review conducted to verify that the detailed design of one or more configuration items satisfy specified requirements; to establish the compatibility among the configuration items and other items of equipment, facilities, software, and personnel; to assess risk areas for each configuration item; and, as applicable, to assess the results of producibility analyses, review preliminary hardware product specifications, evaluate preliminary test planning, and evaluate the adequacy of preliminary operation and support documents. See: preliminary design review, system design review.
criticality
(IEEE) The degree of impact that a requirement, module, error, fault, failure, or other item has on the development or operation of a system. Syn: severity.
criticality analysis.
(IEEE) Analysis which identifies all software requirements that have safety implications, and assigns a criticality level to each safety-critical requirement based upon the estimated risk.
cyclic redundancy [check] code (CRC)
A technique for error detection in data communications used to assure a program or data file has been accurately transferred. The CRC is the result of a calculation on the set of transmitted bits by the transmitter which is appended to the data. At the receiver the calculation is repeated and the results compared to the encoded value. The calculations are chosen to optimize error detection. Contrast with check summation, parity check.
cyclomatic complexity
(1) (McCabe) The number of independent paths through a program. (2) (NBS) The cyclomatic complexity of a program is equivalent to the number of decision statements plus 1.
D
data analysis
(IEEE) (1) Evaluation of the description and intended use of each data item in the software design to ensure the structure and intended use will not result in a hazard. Data structures are assessed for data dependencies that circumvent isolation, partitioning, data aliasing, and fault containment issues affecting safety, and the control or mitigation of hazards. (2) Evaluation of the data structure and usage in the code to ensure each is defined and used properly by the program. Usually performed in conjunction with logic analysis.
data corruption
(ISO) A violation of data integrity. Syn: data contamination.
data dictionary
(IEEE) (1) A collection of the names of all data items used in a software system, together with relevant properties of those items; e.g., length of data item, representation, etc. (2) A set of definitions of data flows, data elements, files, data bases, and processes referred to in a leveled data flow diagram set.
data flow analysis
(IEEE) A software V&V task to ensure that the input and output data and their formats are properly defined, and that the data flows are correct.
data flow diagram
(IEEE) A diagram that depicts data sources, data sinks, data storage, and processes performed on data as nodes, and logical flow of data as links between the nodes. Syn: data flowchart, data flow graph.
data integrity
(IEEE) The degree to which a collection of data is complete, consistent, and accurate. Syn: data quality.
data validation
(1) (ISO) A process used to determine if data are inaccurate, incomplete, or unreasonable. The process may include format checks, completeness checks, check key tests, reasonableness checks and limit checks. (2) The checking of data for correctness or compliance with applicable standards, rules, and conventions.
dead code
Program code statements which can never execute during program operation. Such code can result from poor coding style, or can be an artifact of previous versions or debugging efforts. Dead code can be confusing, and is a potential source of erroneous software changes.
debugging
(Myers) Determining the exact nature and location of a program error, and fixing the error.
decision coverage
(Myers) A test coverage criteria requiring enough test cases such that each decision has a true and false result at least once, and that each statement is executed at least once. Syn: branch coverage. Contrast with condition coverage, multiple condition coverage, path coverage, statement coverage.
decision table
(IEEE) A table used to show sets of conditions and the actions resulting from them.
defect
Noncomformance to requirements. See: anomaly, bug, error, exception, fault. defect analysis. See: failure analysis.
defect analysis
design of experiments
A methodology for planning experiments so that data appropriate for [statistical] analysis will be collected.
design phase
(IEEE) The period of time in the software life cycle during which the designs for architecture, software components, interfaces, and data are created, documented, and verified to satisfy requirements.
design specification
(NIST) A specification that documents how a system is to be built. It typically includes system or component structure, algorithms, control logic, data structures, data set [file] use information, input/output formats, interface descriptions, etc Contrast with design standards, requirement. See: software design description.
development methodology
(ANSI) A systematic approach to software creation that defines development phases and specifies the activities, products, verification procedures, and completion criteria for each phase. See: incremental development, rapid prototyping, spiral model, waterfall model.
diagnostic
(IEEE) Pertaining to the detection and isolation of faults or failures. For example, a diagnostic message, a diagnostic manual.
different software system analysis.
(IEEE) Analysis of the allocation of software requirements to separate computer systems to reduce integration and interface errors related to safety. Performed when more than one software system is being integrated. See: testing, compatibility.
disaster recovery
dynamic analysis
(NBS) Analysis that is performed by executing the program code. Contrast with static analysis. See: testing.
E
embedded software
(IEEE) Software that is part of a larger system and performs some of the requirements of that system; e,g., software used in an aircraft or rapid transit system. Such software does not provide an interface with the user. See: firmware,
end user
(ANSI) (1) A person, device, program, or computer system that uses an information system for the purpose of data processing in information exchange. (2) A person whose occupation requires the use of an information system but does not require any knowledge of computers or computer programming. See: user.
entity relationship diagram
(IEEE) A diagram that depicts a set of real-world entities and the logical relationships among them. See: data structure diagram.
environment
(ANSI) (1) Everything that supports a system or the performance of a function. (2) The conditions that affect the performance of a system or function.
equivalence class partitioning
(Myers) Partitioning the input domain of a program into a finite number of classes [sets], to identify a minimal set of well selected test cases to represent these classes. There are two types of input equivalence classes, valid and invalid. See: testing, functional.
error
(ISO) A discrepancy between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition. See: anomaly, bug, defect, exception, fault.
error analysis
See: debugging, failure analysis.
error detection
Techniques used to identify errors in data transfers. See: check summation, cyclic redundancy check [CRC], parity check, longitudinal redundancy.
error guessing
(NBS) Test data selection technique. The selection is to pick values that are likely to cause errors.
error seeding
(IEEE) The process of intentionally adding known faults to those already in a computer program for the purpose of monitoring the rate of detection and removal, and estimating the number of faults remaining in the program. Contrast with mutation analysis.
event table
A table which lists events and the corresponding specified effect[s] of or reaction[s] to each event.
exception
(IEEE) An event that causes suspension of normal program execution. Types include addressing exception, data exception, operation exception, overflow exception, protection exception, underflow exception.
exception conditions/responses table
A special type of event table.
execution trace
(IEEE) A record of the sequence of instructions executed during the execution of a computer program. Often takes the form of a list of code labels encountered as the program executes. Syn: code trace, control flow trace. See: retrospective trace, subroutine trace, symbolic trace, variable trace.
exception.
(IEEE) An event that causes suspension of normal program operation. Types include addressing exception, data exception, operation exception, overflow exception, protection exception, underflow exception. See: anomaly, bug, defect, error, fault.
extremal test data
(NBS) Test data that is at the extreme or boundary of the domain of an input variable or which produces results at the boundary of an output domain. See: testing, boundary value.
F
Fagan inspection
See: code inspection.
fail-safe
(IEEE) A system or component that automatically places itself in a safe operational mode in the event of a failure,
failure
(IEEE) The inability of a system or component to perform its required functions within specified performance requirements. See: bug, crash, exception, fault.
failure analysis
Determining the exact nature and location of a program error in order to fix the error, to identify and fix other similar errors, and to initiate corrective action to prevent future occurrences of this type of error. Contrast with debugging.
Failure Modes and Effects Analysis
(IEC) A method of reliability analysis intended to identify failures, at the basic component level, which have significant consequences affecting the system performance in the application considered.
Failure Modes and Effects Criticality Analysis
(IEC) A logical extension of FMEA which analyzes the severity of the consequences of faiIure
fault
An incorrect step, process, or data definition in a computer program which causes the program to perform in an unintended or unanticipated manner. See: anomaly, bug, defect, error, exception.
fault seeding
See: error seeding.
Fault Tree Analysis
(IEC) The identification and analysis of conditions and factors which cause or contribute to the occurrence of a defined undesirable event, usually one which significantly affects system performance, economy, safety or other required characteristics.
feasibility study
Analysis of the known or anticipated need for a product, system, or component to assess the degree to which the requirements, designs, or plans can be implemented.
flowchart or flow diagram
(2) (ISO) A graphical representation in which symbols are used to represent such things as operations, data, flow direction, and equipment, for the definition, analysis, or solution of a problem. (2) (IEEE) A control flow diagram in which suitably annotated geometrical figures are used to represent operations, data, or equipment, and arrows are used to indicate the sequential flow from one to another. Syn: flow diagram. See: block diagram, box diagram, bubble chart, graph, input-process-output chart, structure chart.
formal qualification review
(IEEE) The test, inspection, or analytical process by which a group of configuration items comprising a system is verified to have met specific contractual performance requirements. Contrast with code review, design review, requirements review, test readiness review.
function
(1) (ISO) A mathematical entity whose value, namely, the value of the dependent variable, depends in a specified manner on the values of one or more independent variables, with not more than one value of the dependent variable corresponding to each permissible combination of values from the respective ranges of the independent variables. (2) A specific purpose of an entity, or its characteristic action.
functional analysis
(IEEE) Verifies that each safety-critical software requirement is covered and that an appropriate criticality level is assigned to each software element.
functional configuration audit
(IEEE) An audit conducted to verify that the development of a configuration item has been completed satisfactorily, that the item has achieved the performance and functional characteristics specified in the functional or allocated configuration identification, and that its operational and support documents are complete and satisfactory. See: physical configuration audit.
functional decomposition
See: modular decomposition.
functional design
(IEEE) (1) The process of defining the working relationships among the components of a system. See: architectural design. (2) The result of the process in (1).
functional requirement
(IEEE) A requirement that specifies a function that a system or system component must be able to perform.
G
graph
(IEEE) A diagram or other representation consisting of a finite set of nodes and internode connections called edges or arcs. Contrast with blueprint. See: block diagram, box diagram, bubble chart, call graph, cause-effect graph, control flow diagram, data flow diagram, directed graph, flowchart, input-process-output chart, structure chart, transaction flowgraph.
graphic software specifications
Documents such as charts, diagrams, graphs which depict program structure,, states of data, control, transaction flow, HIPO, and cause-effect relationships; and tables including truth, decision, event, state-transition, module interface, exception conditions/responses necessary to establish design integrity.
H
hazard
(DOD) A condition that is prerequisite to a mishap.
hazard analysis
A technique used to identify conceivable failures affecting system performance, human safety or other required characteristics. See: FMEA, FMECA, FTA, software hazard analysis, software safety requirements analysis, software safety design analysis, software safety code analysis, software safety test analysis, software safety change analysis.
hazard probability
(DOD) The aggregate probability of occurrence of the individual events that create a specific hazard.
hazard severity
(DOD) An assessment of the consequence of the worst credible mishap that could be caused by a specific hazard.
I
inspection
A manual testing technique in which program documents [specifications (requirements, design, source code or user's manuaIs are examined in a very formal and disciplined manner to discover errors, violations of standards and other problems. Checklists are a typical vehicle used in accomplishing this technique. See: static analysis, code audit, code inspection, code review, code walkthrough.
installation and checkout phase
(IEEE) The period of time in the software life cycle during which a software product is integrated into its operational environment and tested in this environment to ensure that it performs as required.
Institute of Electrical and Electronic Engineers (IEEE)
345 East 47th Street. New York, NY 10017. An organization involved in the generation and promulgation of standards. IEEE standards represent the formalization of current norms of professional practice through the process of obtaining the consensus of concerned, practicing professionals in the given field.
instruction
(1) (ANSI/IEEE) A program statement that causes a computer to perform a particular operation or set of operations. (2) (ISO) In a programming language, a meaningful expression that specifies one operation and identifies its operands, if any.
instruction set
(1) (IEEE) The complete set of instructions recognized by a given computer or provided by a given programming language. (2) (ISO) The set of the instructions of a computer, of a programming language, or of the programming languages in a programming system. See: computer instruction set.
instrumentation
(NIBS) The insertion of additional code into a program in order to collect information about program behavior during program execution. Useful for dynamic analysis techniques such as assertion checking, coverage analysis, tuning.
interface
(1) (ISO) A shared boundary between two functional units, defined by functional characteristics, common physical interconnection characteristics, signal characteristics, and other characteristics, as appropriate The concept involves the specification of the connection of two devices having different functions. (2) A point of communication between two or more processes, persons, or other physical entities. (3) A peripheral device which permits two or more devices to communicate.
interface analysis
(IEEE) Evaluation of: (1) software requirements specifications with hardware, user, operator, and software interface requirements documentation, (2) software design description records with hardware, operator, and software interface requirements specifications, (3) source code with hardware, operator, and software interface design documentation, for correctness, consistency, completeness, accuracy, and readability. Entities to evaluate include data items and control items.
interface requirement
(IEEE) A requirement that specifies an external item with which a system system component must interact, or sets forth constraints on formats, timing, or other factors caused by such an interaction.
invalid inputs
1 (NBS) Test data that lie outside the domain of the function the program represents. (2) These are not only inputs outside the valid range for data to be input, i.e., when the specified input range is 50 to 100, but also unexpected inputs, especially when these unexpected inputs may easily occur; e,g., the entry of alpha characters or special keyboard characters when only numeric data is valid, or the input of abnormal command sequences to a program.
J
JCL.
job control job job control language.
job.
(IEEE) A user-defined unit of work that is to be accomplished by a computer. For example, the compilation, loading, and execution of a computer program. See: job control language.
job control language.
(IEEE) A language used to identify a sequence of jobs, describe their requirements to an operating system, and control their execution.
L
latent defect.
See: bug, fault. life cycle. See: software life cycle.
life cycle methodology.
The use of any one of several structured methods to plan, design, implement, test. and operate a system from its conception to the termination of its us. See: waterfall model.
logic analysis. (IEEE) Evaluates the safety-critical equations, algorithms, and control logic of the software design. (2) Evaluates the sequence of operations represented by the coded program and detects programming errors that might create hazards.
low-level language. See: assembly language. The advantage of assembly language is that it provides bit-level control of the processor allowing tuning of the program for optimal speed and performance. For time critical operations, assembly language may be necessary in order to generate code which executes fast enough for the required operations. The disadvantage of assembly language is the high-level of complexity and detail required in the programming. This makes the source code harder to understand, thus Increasing the chance of introducing errors during program development and maintenance.