table of Contents
1 Introduction 0
1.1 Code coverage analysis 0
1.2 Structured Test and Functional Test (Structural Testing & Functional Testing) 1
1.3 assumption 1
2 Basic metrics 1
2.1 Statement Coverage 1
2.2 Determination Cover 2
2.3 Condition Coverage 3
2.4 Multiple Condition Coverage 3
2.5 Branch Condition Combination Coverage (4) 4
2.6 Correction Condition / Decision Covered (Modified Condition / Decision Coverage) 4
2.6.1 Calculation formula for coverage: 5
2.7 Path coverage (Path coverage) 5
3 Other metrics 6
3.1 Function Coverage 6
3.2 Function EXITS COVERAGE 6
3.3 Call Cover 6
3.4 Linear code sequence and jump overlay (Linear Code Sequence and Jump (LCSAJ) Coverage 7
3.4.1 Calculation Formula of Coverage: 7
3.5 Data Flow Cover (Data Flow Coverage) 8
3.6 Object Code Branch Coverage 8
3.7 Loop coverage (LOOP COVERAGE) 8
3.8 Competition coverage (RACE COVERAGE) 8
3.9 Comparison Operator Coverage 8
3.10 Weak change coverage (Weak Mutation Coverage) 9
3.11 Table Coverage 9
4 Compare a variety of coverage 9
4.1 Overlay Target for Release Version 9
4.2 Intermediate version coverage target 9
5 summary 10
6 Reference 10
7 Glossary 11
1 Introduction
1.1 Code coverage analysis
This article gives a complete concept of code coverage analysis.
Code coverage analysis is such a process:
· Find some code that the program has been taken through a series of tests without performing
· Create an additional test case to increase coverage
· Decide the quantitative measure of the code coverage.
One effective aspect of code coverage analysis is:
· Identify an invalid test case without increasing coverage.
Coverage analysis requires the source code of the test program and often needs to be recompiled with a special command.
This article discusses how you should consider how to increase coverage analysis in your test plan. Coverage analysis has certain benefits and weaknesses. You should choose a range of measurement methods. You should establish a minimum percentage of coverage to determine when you stop overlay analysis. Coverage analysis is only one of many test technology, you can't just rely on it.
1.2 Structured Test and Functional Test (Structural Testing & Functional Testing)
Code coverage analysis is a structured test technology (Aka Glass Box Testing and White Box Testing). Structured Test is a comparison of the destination of the active behavior and source code. And functional tests (Aka Black-Box Testing), functional testing is compared to the behavior and determination of the test program. Structured test check procedures work, consider logical defects that may exist in the structure. Function test checks the ability to complete the requirements of the test program, do not consider how it works. Structured Test is also called path test because you select a test case to pass the path of the program structure. Don't be confused with path coverage metrics, which will be described below.
Roughly, structured tests do not seem to be unsafe, structured tests cannot find erroneous errors, but demand definitions do not exist, and is incomplete. This phenomenon actually exists. When the timeline developed is asked, when the demand definition is rarely updated, the product itself replaces the role of the demand definition.
1.3 assumption
Some basic principles assume that the following list:
· FAULTS --- and control flow related flaws, you can find these defects through change control flow [Beizer1990 P.60]. For example, a program is written as "if (c)" better than "if (! C)".
· You can find defects without having to know the consequences of this defect possible and all tests.
· Other assumptions include definitions that can be completed, no negligent defects and code without codes, etc.
2 basic metrics
There are many coverage measures, which introduces some basic measures and weaknesses.
2.1 Statement coverage (Statement coverage)
This metric reports whether each executable statement is executed.
Also known as: line cover, segment cover [NTAFOS1988], C1 [Beizer1990 P.75] and basic block coverage (Basic block cover). Basic Blocks Override When each sequence is the statement that is not branchless, the statement coverage is the same.
The main benefit of this coverage metric is that it can be applied directly on the target code and do not need to process the source code. This measure is completed in order to perform the outline.
The main disadvantage of this coverage metric is that some control structures are very slow. For example, consider the following C / C code:
INT * P = NULL;
IF (condition)
P = & variable;
* P = 123;
If the statement coverage is overwritten when the Condition is fake, the code is executed is failed. This is a serious flaw of a statement coverage, and if the IF statement is a common situation.
Statement coverage cannot report whether the loop reaches its termination condition - only whether the loop is executed.
Since the Do-While loop is usually at least once, the statement covers the same as the branchless statement.
The statement coverage is a slow logical operator (|| and &&).
Statement coverage cannot distinguish between consecutive Switch statements.
Test cases are usually related to judging rather than in the statement. You may not have to use 10 separate test cases to test a statement with 10 non-branchful statements, you may use only one test case, consider an IF-ELSE statement, including a statement in the clause, 99 The statement is in the ELSE clause, when performing two possible paths, the statement coverage is obtained: 1 or 99 coverage. Basic block coverage ignore this problem.
2.2 Decision Cover This metric Report does the BOOL type expression value true and false are tested in the control structure (eg if-statement and while-statement). The entire BOOL type expression is considered to be a TRUE and FALSE instead of whether or not the internal-and or logical-or operator is included. In addition, this metric includes overwriting Switch-Statement Cases, Exception Handlers, and And Interrupt Handlers.
Also known as: Branch cover, all-edges coverage [ROPER1994 P.58], basic path coverage [ROPER1994 P.48], C2 overlay [beizer1990 p.75 ], It is determined to the determination path coverage (Decision-Decision-Path or DDP Testing [ROPER1994 P.39].) "Basis Path" test is the selection path to reach all judgment overrides.
This metric has the simplicity of the statement coverage, but there is no problem with the statement.
Disadvantages is that this metric is ignored by BOOL in the BOOL type expression. For example, consider the following C / C / Java code:
IF (Condition1 && (Condition2 || Function1 ())))
STATEMENT1;
Else
STATEMENT2;
This metric can be completely free to call Function1. Test expression can take Condition1 to true and condition2 to true when the test is true, and the test expression can take Condition1 to False.
2.3 Condition Cover (CONDition coverage)
Conditions Overview Report True or False of the results of each sub-expression. Logical-and Logical-OR stands independent. Condition cover independent metrics for each child expression.
This metric is similar to Decision Coverage, but is sensitive to control of control.
However, complete condition coverage does not guarantee a complete determination overlay. For example, consider the following C / Java code.
BOOL F (BOOL E) {Return False;}
BOOL A [2] = {False, False};
IF (f (a && b)) ...
IF (A [Int (A && B)]) ...
IF ((A && B)? false: false) ...
All three IF statements regardless of the value of A and B, it is determined that the coverage can only reach 50%. However, the conditions coverage can reach 100%.
2.4 Multiple Coverage (Multiple Coverage)
Multi-conditional coverage reports the combination of each possible BOOL type child. Compared to the conditional coverage, the test case required for multi-conditional coverage is determined by the truth of the logical operator with a conditional logical operator by Logical-and Logical-OR.
For languages with Short Circuit Operators, C, C , and Java, the benefit of multi-conditional coverage is that it requires a thorough test.
A disadvantage is that it may be very lengthy to determine the minimum setting of a needed test case, especially for very complex BOOL type expressions. Another disadvantage is that this measure of test cases requires very large changes for the conditions of similar complexity. For example, consider the following C / C / Java conditions: To achieve a complete multi-conditional coverage, the first one requires 6 test cases,
The second requires 11 test cases.
However, two conditions have the same operands and operators.
For Visual Basic and Pascal, multiple-condition overrides have the same advantages and weaknesses for logical expressions for logical expressions. Consider the following Visual Basic code:
IF a and b1en
...
Multi-conditional coverage requires four test cases, A and B values True and False, respectively.
2.5 Branch Condition Combination Cover (CONDition / Decision Coverage)
Branch Condition Combined Coverage is a mixed-branch of condition coverage and branch coverage (Decision covege). It has the simplicity of both but there is no shortcomings.
2.6 Correction Condition / Decision Covered (Modified Condition Coverage)
Also known as MC / DC and MCDC.
This metric requires enough test cases to determine that each condition can affect the result of the included determination [chilenski1994]. This measure is the first to be created by Boeing is used in RCTA / DO-178B in aviation software.
This metric is originally designed to have a language without Short Circuit Operators. The role of Short Circuit Logical Operators in C, C , and Java is merely when their results can affect the assessment conditions of the included decisions.
When the following needs are encountered, consider coverage with MCDC:
• Requirements 1: Each of the program modules should be considered at least once, and each program is determined to all convert at least once.
• Demand 2: The determination of the program is decomposed into Bool conditions via a logical operator (AND, OR, ETC.). Each condition is independent of the result value of the decision, or the change in single condition will result in the change of the decision.
Therefore, according to the above definition, the following three sets of data must be. (Changes in single conditions will result in changes in judgments)
The T1 & T2, or T1 & T3 has been satisfied with the requirements 1, but requires the condition 2, there must be both T1 & T2 & T3.
T4: (f, f) is redundant because a conditional change cannot cause the decision to change.
T1 (t, t) and T2 (t, f) indicate Y independently affecting the independent pair of Y.
T1 (t, t) and T3 (f, t) indicate that X independently affects the independent pair of X.
Rethlean:
Test use example 1 (t, t, t) and test case 5 (f, t, t), for X independent.
Test case 2 (t, t, t) and test case 6 (f, t, t), for X independent.
Test case 3 (t, t, t) and test cases 7 (f, t, t), for X independent.
Test case 2 (t, t, t) and test case 4 (f, t, t), for Y independent.
Test use example 3 (t, t, t) and test case 4 (f, t, t), for Z independence.
As a result, the test case group {1, 5, 2, 4, 3} meets the requirements of the MCDC covering the expression X, Y, and Z.
Obvious, this is not the only combination, for example, {6, 2, 4, 3}. 2.6.1 Calculation formula for coverage:
or:
2.7 Path Cover (Path coverage)
This metric reports whether each possible branch of the function is executed. A path is a unique series of branches from the entrance to the entry to the function.
Also called as an assertion coverage (Predicate Coverage). As a result, it can be seen that the path of the logical conditions that can be combined [Beizer1990 P.98].
Because the loop introduces a large number of paths, this metric only considers a limited number of cyclic possibilities. There are a lot of variants to handle cycles, the Boundary-Interior Path Testing only considers two possibilities of the loop. Repeat zero and more zero repeats [NTAFOS1988]. For Do-While cycles, two, zero repetitions, and more zero repetitions.
One advantage of the path coverage is: need to be thorough test.
But there are two shortcomings:
First, the path is increasing with the branch's index, for example: a function contains 10 IF statements, with 1024 paths to test. If an IF statement is added, the number of paths reaches 2048.
Second, many paths cannot be independent of the data executed. E.g
IF (SUCCESS)
STATEMENT1;
STATEMENT2;
IF (SUCCESS)
STATEMENT3;
The path coverage is that the above statement contains four paths, in fact, only two are feasible: Success = false and Success = True.
Researchers inven invented many path coverage techniques to avoid a large number of paths. For example, N-Length Sub-Path Coverage reports whether you exercised Each Path of Length n branches. Other variants include linear code sequence and jump (lcsaj) Coverage and Data Flow coverage.
3 Other metrics
Here, some of the other basic rare benefits and weaknesses are introduced.
3.1 Function coverage
This metric reports whether you call each function or process. For preliminary tests to ensure that there is at least in all software, there is no total shortage.
Most coverage tools are supported.
3.2 Function Exit Coverage (Function EXITS COVERAGE)
Report the entrance, exit and termination instructions of the function. Coverage statistics.
As far as I know, TestRT supports this coverage.
3.3 Call Cover (Call coverage)
This metric is reporting that you perform each function call. The premise is that the defects typically occur at the interface of the module.
Also called the call pair coverage.
As far as I know, TestRT supports this coverage.
3.4 Linear code sequence and jump cover (LINEAR CODE SEQUENCE AND JUMP (LCSAJ) COVERAGE
This is a change in path coverage (Path Coverage). Considering that only sub-paths in the source code can be easily replaced, there is no need for a flowchart. A LCSAJ is a series of sequences performed by a series of source code lines.
The advantage is that this metric is more thorough than the determination of the overlay test, and avoids the difficulty of the path coverage. The disadvantage is that it cannot avoid the path of not being implemented.
As far as I know, LDRA Testbed supports this coverage.
3.4.1 Calculation formula for coverage:
As shown below:
A LCSAJ is determined by the number of four features.
A Start Point: It can be the start of the start of the program or any target of any control flow jump. A linear code sequence: The code body that can be processed. It can be composed of several consecutive basic blocks.
An end point: The first line encountered from Which a jump is Made Which Has Been reached from the start point by the unbroken linear sequence of code.
A Target Point: The point to which the End Points "control flow jump is made This will be the Start Point of the next LCSAJ Therefore, since the start point of the linear code sequence is a line which is the target of another jump.. , These Fragments Are Also Called Jump-to-Jump Paths.
This example calculates that this LCSAJ covered is 11.
3.5 Data flow coverage (Data Flow coverage)
This is a variant of Path Coverage, just considering the sub-branch of the reference from the variable to the subsequent variable.
The advantage is the data processed directly from the appropriate report. A disadvantage is that another disadvantage is too complicated without determining the overlay. Many variables are used to calculate, for determination, partial, globally, if there is a pointer more complicated.
3.6 Object Code Branch Cover (Object Code Branch coverage)
This metric reports whether the machine language condition branch passes the branch.
This measure given more reports more dependent on compilers instead of the program structure. Because the structure of the target code and the structure of the source program are very different. Moreover, the branch destroys the instruction channel, and the compiling is sometimes avoided to generate a branch, and replaced with a series of non-filled instructions.
3.7 loop coverage (Loop coverage)
This metric reports whether you execute each cycle zero, only once or more (continuous). For the Do-While loop, the loop coverage report does not perform only once a cycler is once (continuous).
The valuable aspect of this measure is to determine if more than once for the While loop and the FOR cycle, this information is not in other coverage reports.
As far as I know, the GCT and Rational TestRT can perform this measure.
3.8 Competition coverage (RACE coverage)
This metric report is that multiple threads perform the same piece of code at the same time. It helps detect the synchronous failure of resources. It is very useful for multi-threaded programs such as operating system programs.
As far as I know, GCT can perform this measure.
3.9 Compare Operator Cover (Relational Operator Coverage)
This metric reports whether a boundary condition has occurred for comparison operators (<=,>,> =). Assuming the boundary condition test case discovers Off-by-one errors, and the error uses a comparison operator, for example, it is used as <=. For example, consider the following C / C code:
IF (a STATEMENT; The comparison operator reports whether the condition A == B has occurred. If IF A == B has occurred and the program is executed normally, then. . . . . . As far as I know, GCT can perform this measure. 3.10 Weak Mutation Coverage This metrics are similar to the Relational Operator Coverage, but more common [howden1982]. It reports whether the test case has an incorrect reference operator and operand. As far as I know, GCT can perform this measure. 3.11 Table Coverage (Table coverage) This metric shows that each entry of a special array is referenced. This measure is very useful for programs control through a limited state machine. 4 Compare a variety of coverage · Decision coverage contains Statement Coverage, · Condition / Decision Coverage contains Decision Coverage and CONDition Coverage • Path Coverage contains Decision coverage. · Predicate coverage contains Path Cover and Multiple Condition coverage Academia said strong coverage metrics contain weak coverage measures. Various coverage metrics cannot be compared from quantity. 4.1 Overlay Target for Release Versions Each project should choose a minimum coverage goal, and software that strictly requires safety standards should make a high goal. The goal of the unit test should be higher than the system test because a Failure at a low level code can affect the caller of multiple high levels of code. With Statement Coverage, Decision Coverage or Condition / Decision Coverage, over 80 -90 or more coverage before the release. Some people may feel that the coverage of less than 100 is not guaranteed by software, but you consume more FAULTS if you are used for other test technologies, such as code reading, etc., you will be found to avoid The target is less than 80. 4.2 Overlay Targets in the Intermediate Version Choosing a good intermediate coverage target can greatly improve the productivity of the test. When you find the most Failures with the least effort, then you have the highest test productivity. How do you measure? Create a test case, join them to your test set and run their total time is. 5 summary Cover analysis is a structured test technology to help make up for the gap in the test sleeve. There are many helpful projects that are missing, without updating demand definition projects. Condition / decision coverage technology I think is the best for C, C and Java languages. Setting a coverage of a target 100 (for any coverage) will prevent the productive productive forces, before the release, a 80 -90 or more target for statements, branches, and condition coverage are better. 6 reference Beizer1990 Beizer, Boris, "Software Testing Techniques", 2nd Edition, New York: Van Nostrand Reinhold, 1990 Chilenski1994 John Joseph Chilenski and Steven P. Miller, "Applicability of Modified Condition / Decision Coverage to Software Testing", Software Engineering Journal, September 1994, Vol. 9, No. 5, pp.193-200. RTCA / DO-178B, "Software Considerations in Airborne Systems and Equipment Certification", RCTA, December 1992, pp.31, 74.Howden1982 "Weak Mutation Testing and Completeness of Test Sets", IEEE Trans. Software Eng., Vol.SE -8, No.4, July 1982, PP.371-379. McCabe1976 Mccabe, Tom, "A Software Complexity Measure", IEEE TRANS. Software Eng., Vol.2, No.6, December 1976, PP.308-320. Morell1990 Morell, Larry, "A THEORY OF FAUT-BASED TESTING", IEEE TRANS. Software Eng., Vol.16, No.8, August 1990, PP.844-857. NTAFOS1988 NTAFOS, SIMEON, "A Comparison of Some Structural Testing Strategies", IEEE Trans. Software Eng., Vol. 14, No.6, June 1988, PP.868-874. ROPER1994 ROPER, MARC, "Software Testing", London, McGraw-Hill Book Company, 1994 7 Glossary Fault --- a bug. A Defect. A bug, a fault. Error --- a person-oriented error, the result is a FAULT. Failure --- A real-time running performance of FAULT.
