Tools and techniques of code coverage testing

Tools and techniques of code coverage testing

1. International Journal of Computer Engineering and Technology (IJCET), ISSN 0976-6367(Print),
ISSN 0976 - 6375(Online), Volume 5, Issue 9, September (2014), pp. 165-171 © IAEME
165











TOOLS AND TECHNIQUES OF CODE COVERAGE TESTING


R. M. Sharma

Assistant Professor, Department of Computer Application,
MCNUJC, Bhopal Madhya Pradesh, India



ABSTRACT

Poor quality software is the result of deficiency in test measurement process. Code coverage
analysis provides a measure of how much source is being covered by test execution. The
completeness of Software testing can be measured by requirement coverage, structural coverage, and
architectural coverage. This paper focus on techniques of structural coverage testing that is also
known as code coverage testing. The code coverage testing includes the statement coverage, path
coverage, function coverage and condition coverage. The code coverage analysis can be done by
some special analysis tools are known as code coverage tools. This paper also present some
commonly used coverage analysis tools based on some coverage criteria, this may help to select
particular code coverage tool.

Keywords: Branch Testing, Code Coverage Testing, Code Coverage Tools, Loop Testing,
Path Testing, Statement Testing.

1. INTRODUCTION

Software testing is a critical element of software quality assurance and represents the ultimate
review of specification, design and coding. Testing is the process of executing a program with the
intent of finding error [1]. The software maintenance costs are increase because few pieces of
software can be changed with any degree of confidence that new bugs aren’t being introduced. For
example if we write a bunch of tests scripts and when we execute test cases we don’t know how
much code covered by the test cases, it requires some measurement of code coverage. The code
coverage metrics provide how much code of program has been tested. Now a day’s there are so many
code coverage tools are available which provide automated code coverage testing and give analysis
reports of test coverage.


INTERNATIONAL JOURNAL OF COMPUTER ENGINEERING &
TECHNOLOGY (IJCET)



ISSN 0976 – 6367(Print)
ISSN 0976 – 6375(Online)
Volume 5, Issue 9, September (2014), pp. 165-171
© IAEME: www.iaeme.com/IJCET.asp
Journal Impact Factor (2014): 8.5328 (Calculated by GISI)
www.jifactor.com

IJCET
© I A E M E

2. International Journal of Computer Engineering and Technology (IJCET), ISSN 0976-6367(Print),
ISSN 0976 - 6375(Online), Volume 5, Issue 9, September (2014), pp. 165-171 © IAEME
166

2. CODE COVERAGE TESTING

Since any software product realized in terms of source code, if we execute test cases to
exercise the different parts of the source code, then only that parts of the source code has been tested.
Code coverage metrics are used to determine the percentage of source code covered by testing. Code
coverage testing is a method of designing and executing test cases and finding out the percentage of
source code covered by testing. Now a day’s there are so many automated code coverage testing
tools are available, they involves “dynamic testing” methods of executing the software with pre-
written test cases, and finding out how much of the code has been covered [3]. If a better coverage of
code is required, several iteration of testing may be required, for each iteration tester has to check the
statistics and write a new set of test cases for the section of code that is not covered by the earlier test
cases [4]. To do this type of testing tester need to understand the program code and logic but also
need to understand how to write effective test cases. The automated and specialized tools are
designed to monitor and keep records of what portions of code covered by the testing, and what
portion is not covered [6]. To achieve maximum coverage the code coverage testing involves
following types of coverage.

2.1 Statement Coverage
2.2. Path Coverage
2.3 Condition Coverage
2.4 Function Coverage

2.1 Statement coverage: In this testing method test cases are designed in such a way to execute
every statement in a program at least one time during testing. The statement coverage can be defined
as Statement Coverage = (Total Statement Exercised /Total Number of executable Statements in
program)*100
The statements can be classified into four types

2.1.1 Simple statement (sequence flow)
2.1.2 Two way statement (if then else)
2.1.3 Multi way statement (switch)
2.1.4 Loop statements (while, do, until, repeat, for)

2.1.1 Testing simple statement: A statement (without branch and loop) is known as simple statement.
For a section of code that consists of statements that are sequentially executed, test cases can be
designed to run through each statement at least once from top to bottom. However, this may not
always be true. For example while testing exception like divide by zero encounters, and then the test
cases may not cover all the statements.

2.1.2 Testing two way statement: For testing two way statement like (if then else), we should write
case for each (if then else), (at least) one test case for (then) part and (at least) one test case for
(else) part.

2.1.3 Testing Multi way statement: The multi way statement can be reduced to multiple two way
statement to cover all possible switch cases.

3. International Journal of Computer Engineering and Technology (IJCET), ISSN 0976-6367(Print),
ISSN 0976 - 6375(Online), Volume 5, Issue 9, September (2014), pp. 165-171 © IAEME
167

2.1.4 Testing looping statement: Loops are cornerstone for the vast majority of all algorithm
implements in software [1]. As we know most of the defects in a program due to the loops. In most
of the case loops fail because of “boundary conditions”[5]. One of the common looping errors is that
termination condition of the loop not properly stated. Basically loops can be define in four types

2.1.4.1 Simple loops
2.1.4..2 Concatenated loops
2.1.4.3 Nested loops
2.1.4.4 Unstructured loop


Figure (1): Types of loops

2.1.4.1 Simple loop testing: For better coverage of statement within the loop there should be test
cases that

1. Skip loop entirely, so that the situation of the termination condition true before starting
the loop is tested.
2. Exercise the level (only one pass through the loop, two pass through the loop, or m pass
through the loop where m < n and n is total iteration) to check all possible “ normal “
operations of the loop.
3. Try to cover all boundary conditions (n-1, n , n+1 pass through the loop)

2.1.4.2 Nested loop testing: For better coverage of statement with in the loop Bezier suggest the
following approach [1].

1. Start at inner most loop. Set all other loops to minimum values.
2. Conduct simple loop test for the inner most loop, while holding the outer loop at
(minimum iteration parameters) . Add other tests for out of range or excluded the values.
3. Work outward conducting test for the next loop but keeping all other outer loop at
minimum values and other loops to “typical” values.
4. Continue until all loops have been tested.

4. International Journal of Computer Engineering and Technology (IJCET), ISSN 0976-6367(Print),
ISSN 0976 - 6375(Online), Volume 5, Issue 9, September (2014), pp. 165-171 © IAEME
168

2.1.4.3 Concatenated loop testing:

1. If each loop is independent from other loop, loops can be tested using simple loop
method.
2. If two loops are concatenated and the loop counts for loop 1, is used as the initial value
for loop 2, then the loops are not independent, in this situation nested loop testing method
can be applied.

2.1.4.4 Unstructured Loop Testing: In this situation the loop can be redesigned for better
statement coverage with in the loop.

2.2 Path Coverage: Path coverage testing used to test various logical paths in the program. In path
coverage testing, we split a program into a number of distinct paths [5]. We identify the number of
independent paths in a program. An independent path through the program that introduces at least
one new set of processing statements or new condition [1]. Path coverage can be defined by

Path coverage = (Total paths exercised/Total number of path in program)*100
Four Steps for Path testing

a) Compute the program graph.
b) Calculate the cyclomatic complexity.
c) Select a basis set of paths.( independent path)
d) Generate test cases for each of these paths

The cyclomatic complexity [7] define the number of independent paths in a program, the
cyclomatic complexity can be computed as


Fig (2): Flow Graph

(a) Number of regions in the flow graph
V(G)= R (regions )
Where V(G) is a cyclomatic complexity
As shown in figure(2) there are 2 regions, so the cyclomatic complexity [7] for flow graph
is V(G)=2
1
5
2
3 4
6




s

5. International Journal of Computer Engineering and Technology (IJCET), ISSN 0976-6367(Print),
ISSN 0976 - 6375(Online), Volume 5, Issue 9, September (2014), pp. 165-171 © IAEME
169

(b) V(G) = E-N+2 where E is the total number of edges in the flow graph and N is the total
number of nodes in the flow graph. In the figure 2 the total node =6 and total edges=6 the
cyclomatic complexity is

V(G)= E-N+6=6-6+2=2

The two independent paths for given flow graph are
Path (1) 1-2-3-5-6
Path (2) 1-2-4-5-6
Therefore we can say that cyclomatic complexity V (G) [1] provides the upper bound for
number of independent paths. If we design the test cases for each independent path, it will increase
code coverage.

2.3 Condition Coverage: Condition testing method test all the logical condition contained in the
program module testing [1]. A simple condition is a Boolean or a relational expression like ( “<” “>”
“<=” AND(“&”) OR (“|”) NOT). The Condition coverage can be defined as
Condition coverage= (Total Decision Exercised/Number of decision in a program)*100

2.4 Function coverage: A function is a logical unit of program, it is designed for special task. The
function coverage can be defined as

Function coverage= (Total function Exercised/ Total functions in program)*100

The advantages of Function coverage

• Functions are easier to identify hence it is easy to write test cases for better function
coverage.
• Functions are directly related to user requirement to the test coverage of the product.

3. CODE COVERAGE ANALYSIS AND TESTING TOOLS

This section focus on, commonly used coverage tools with their features and the support
they provide.

3.1 CodeCover: CodeCover is a free white-box testing tool, it measures statement, branch, loop,
term (Boolean) coverage. It provides the open language interface for Java and COBOL [13]. It
Remove (redundant) test cases from a test suite to reduce regression testing effort significantly
without decreasing testing effectiveness. It provides reports of program elements which were not
executed. CodeCover run on Command line (Linux, Windows, Mac OS). It also provide
synchronization coverage, Synchronized statements are used to synchronize critical code sections. If
a synchronized statement is locked, other threads will wait until the locked section is set free. The
synchronized coverage metric shows whether a synchronize statement has caused waiting effects. In
particular the synchronized coverage is helpful to quantify the effects of load testing.

3.2 Cobertura: Cobertura is a free Java code coverage tool, that calculates the percentage of code
covered by tests. It can be used to identify which parts of your Java program are lacking test
coverage. It is based on jcoverage. It provide statement and branch coverage for each class, each
package, and for overall project [4]. It also calculates cyclomatic complexity of each class [11]. It

6. International Journal of Computer Engineering and Technology (IJCET), ISSN 0976-6367(Print),
ISSN 0976 - 6375(Online), Volume 5, Issue 9, September (2014), pp. 165-171 © IAEME
170

generates reports of coverage (statement and branch) in HTML or XML format. It Instruments Java
bytecode after it has been compiled. Cobertura Shows the cyclomatic complexity of each class, and
the average cyclomatic code complexity for each package, and for the overall product [3]. It can
sort HTML results by class name, percent of lines covered, percent of branches covered, etc

3.3 Emma: EMMA is an open-source java tool that is used for measuring and reporting Java code
coverage. It supports class, method, line, basic block (statements) coverage. EMMA can detect when
a single source code line is covered only partially. It provides analysis report in plain text, HTML,
XML format [12]. The HTML report supports source code linking. EMMA is quite fast: the runtime
overhead of added instrumentation is small about (5-20%) and the byte code instrumentation is very
fast [3]. Memory overhead is a few hundred bytes per Java class. EMMA is 100% pure Java, has no
external library dependencies, and works in any Java 2 JVM, and can be integrate with Maven.

3.4 GCT: GCT is a C Code Coverage Tool. GCT was the third coverage tool by Brian Marick. It
instruments C code in a source-to-source translation, and then passes the instrumented code to a
compiler [14]. It is suitable for measuring the coverage of embedded systems as it was first used on a
UNIX kernel. In addition to branch and multiple-condition coverage, it also has boundary-condition
and loop coverage facility.

3.5 Coverlipse: Coverlipse is an Eclipse plugin that visualizes the code coverage of JUnit Tests. It is
unique because it integrates seamlessly in Eclipse. The coverage results are given directly after a
JUnit run. This makes it the perfect tool for developers to recognize their tests and fulfill their task
[15]. It provides statement coverage and branch coverage metrics to tester.

3.6 Testwell CTC++: Testwell CTC++ is a powerful and easy-to-use code coverage analyzer for
C/C++. It provides function coverage, statement coverage branch coverage, and condition coverage
[8]. The coverage/execution profile data is can be reported in textual, HTML (hierarchical, color-
coded, information shown along with the actual source code), XML, and in Excel format.

3.7 Quilt: Quilt is a Java software development tool that measures code coverage, the extent to
which unit testing exercises the software under test. It is used for branch and statement coverage
analysis. It is optimized for use with the JUnit test package, the Ant Java build facility, and the
Maven project management toolkit [9]. The programmer simply sets a switch at test time. Then when
program modules are loaded for testing, Quilt automatically generates coverage reports.

3.8 JCover: It is a code coverage testing tool for JAVA. It provide user friendly interface. It is used
for statement and branch Coverage analysis, and it computes Method, Class, File, and Package
coverage. It is also used for sever side testing. Jcover can gather test coverage measures of
applications whose source code is available, or to work with compiled class files (bytecode) [10]. Its
distribution includes several examples of applications to show you how to use the tool effectively. It
provides Coverage Comparison, to see how coverage improves across several test runs. This tool
generates a variety of reports and charts in HTML, XML, and CSV standard. We can even export the
data to MDB format for subsequent analysis. The unique feature of JCover is coverage differencing,
to understand how two sets of test runs differ from each other that enable a tester to optimize test
cases by minimizing redundancy.

3.9 Squish Coco: Squish Coco utilizes source code instrumentation to analyze the applications
source code. Squish Coco is available for C, C++, C#, and Tcl programming languages. It finds and
highlights the untested code; it also finds unreachable code and reduces the duplicate tests. It

7. International Journal of Computer Engineering and Technology (IJCET), ISSN 0976-6367(Print),
ISSN 0976 - 6375(Online), Volume 5, Issue 9, September (2014), pp. 165-171 © IAEME
171

computes optimal order of test execution that maximizes the overall coverage. Squish Coco can run
on Linux, Windows, Mac OS X and other platforms. Squish Coco can be used with every testing
method like unit tests, automated tests, and manual tests. Squish Coco supports function coverage,
line coverage, branch coverage and condition/decision coverage [16]. Furthermore, Squish Coco can
merge multiple execution reports to provide advance analysis.

CONCLUSION

The amount of code coverage depends on understanding code logic and writing effective test
cases. Code coverage up to 40-50 percent is easily achieved by tester, code coverage of more than 80
percent requires enormous amount of effort and understanding the code. This paper described
various code coverage testing techniques and coverage analysis tools to understand the method and
process of code coverage testing.

[1] Roger s. Pressman “software engineering A Practitioner’s Approach” 4th edition.
[2] Ms. L.Shanmuga Priya, Ms.A.Askarunisa, Dr. N.Ramaraj "Measuring The Effectiveness of
Open Coverage Based Testing Tools" Journal of Theoretical and Applied Information
Technology © 2005 - 2009 JATIT. pp. 499-514.
[3] Muhammad Shahid, Suhaimi Ibrahim, "An Evaluation of Test Coverage Tools in Software
Testing" International Conference on Telecommunication Technology and Applications Proc
.of CSIT vol.5 (2011) © (2011) IACSIT Press, Singapore pp.216-222.
[4] Dr. Neetu Dabas, Mrs. Kamna Solanki "Comparison of Code Coverage Analysis Tools: A
Review" International Journal of Research in Computer Applications & Information
Technology Volume 1, Issue 1, July-September, 2013, pp. 94-99, © IASTER 2013.
[5] Zhu, H., Hall, P.A.V. and May, J.H.R. (1997), “Software unit test coverage and adequacy”.
ACM Comput. Surv. 29, pp. 366–427.
[6] Malaiya, Y.K., Li, M.N., Bieman, J.M. and Karcich, R. (2002), “Software reliability growth
with test coverage” IEEE Trans. Reliab., 51, pp. 420–426.
[7] McCabe, T. (1976) “ A complexity measure ”. IEEE Trans. Softw. Eng., 5, pp. 45–50.
[8] Testwell CTC++, www.testwell.fi.
[9] Quilt, http://www.codecoverage.com
[10] JCover, http://www.mmsindia.com/JCover.html.
[11] Cobertura, http://cobertura.sourceforge.net/.
[12] Emma, http://emma.sourceforge.net/.
[13] Code Cover, http://www.codecover.org.
[14] GCT http://www.exampler.com/testing-com/tools.html.
[15] Coverlipse http://coverlipse.sourceforge.net/.
[16] Squish Coco: http://www.froglogic.com/squish/coco/.
[18] Deepak.S.Sakkari and Dr.T.G.Basavaraju, “Optimized Coverage and Connectivity for
Randomly Deployed Wireless Sensor Network for Lifetime Conservatory”, International
Journal of Computer Engineering & Technology (IJCET), Volume 4, Issue 6, 2013,
pp. 247 - 255, ISSN Print: 0976 – 6367, ISSN Online: 0976 – 6375.
[17] Anurag, “Energy Efficient K-Target Coverage in Wireless Sensor Network”, International
Journal of Computer Engineering & Technology (IJCET), Volume 4, Issue 3, 2013,
pp. 254 - 259, ISSN Print: 0976 – 6367, ISSN Online: 0976 – 6375.