Software reliability

A Brief description of Software reliability

1. NAVAL EMC CENTRE
LT CDR PABITRA KUMAR PANDA
M TECH (RE), IIT KGP
11 AUG 2010

2. NAVAL EMC CENTRE
INTRODUCTION
RELIABILITY
HARDWARE VS SOFTWARE RELIABILITY
SOFTWARE RELIABILITY GROWTH MODELS
STATISTICAL TESTING
CONCLUSION
SCOPE OF PRESENTATION

3. NAVAL EMC CENTRE
SOFTWARE RELIABILITY
•Reliability is usually defined in terms of a
statistical measure for the operation of a
software system without a failure occurring
•Software reliability is a measure for the
probability of a software failure occurring
•Two terms related to software reliability
–Fault: a defect in the software, e.g. a bug in the
code which may cause a failure
–Failure: a derivation of the programs observed
behavior from the required behavior

4. NAVAL EMC CENTRE
SOFTWARE REL IABILITY CONTD.
•Software Reliability is an important attribute of
software quality, together with
–functionality,
–usability,
–performance,
–serviceability,
–capability,
–installability,
–maintainability,
–documentation.

5. NAVAL EMC CENTRE
What is Software Reliability
•Software Reliability is hard to achieve,
because the complexity of software tends
to be high.
•While the complexity of software is
inversely related to software reliability, it is
directly related to other important factors
in software quality, especially functionality,
capability.

6. NAVAL EMC CENTRE
•Cannot be defined objectively
–Reliability measurements which are quoted
out of context are not meaningful
•Requires operational profile for its
definition
–The operational profile defines the expected
pattern of software usage
•Must consider fault consequences
–Not all faults are equally serious. System is
perceived as more unreliable if there are more
serious faults
Software reliability

7. NAVAL EMC CENTRE
HARD WARE VS SOFTWARE HAZARD RATE

8. NAVAL EMC CENTRE
SOFTWARE FAILURE MECHANISM
•Failure cause: Design Defects.
•Repairable system concept: Periodic restarts
may fix software problems.
•Time dependency and life cycle : Not a
function of operational time.
•Environmental factors: Do not affect
•Reliability prediction: Software reliability can
not be predicted from any physical basis, since it
depends completely on human factors in design.

9. NAVAL EMC CENTRE
•A failure corresponds to unexpected run-time
behaviour observed by a user of the software.
•A fault is a static software characteristic which
causes a failure to occur.
•Faults need not necessarily cause failures. They
only do so if the faulty part of the software is
used.
FAILURES AND FAULTS

10. NAVAL EMC CENTRE
INPUT/OUTPUT MAPPING
I
e
Input set
OeOutput set
Program
Inputs causing
erroneous
outputs
Erroneous
outputs

11. NAVAL EMC CENTRE
RELIABILITY PERCEPTION
Possible
inputs
User 1
User 3
User 2
Erroneous
inputs

12. NAVAL EMC CENTRE
FAILURE CLASSIFICATION
Transient: failures occur only for certain inputs.
Permanent: failures occur for all input values.
Recoverable: when failures occur the system
recovers with or without operator intervention.
Unrecoverable: the system may have to be
restarted.

Cosmetic:May cause minor irritations. Do not
lead to incorrect results.
.

13. NAVAL EMC CENTRE
Measuring Software Reliability
•Errors do not cause failures at the same frequency and severity.
–measuring latent errors alone not enough
•The failure rate is observer-dependent
•No simple relationship observed between system reliability and the
number of latent software defects.
•Removing errors from parts of software which are rarely used
makes little difference to the perceived reliability.
•removing 60% defects from least used parts would lead to only
about 3% improvement to product reliability.
•Reliability improvements from correction of a single error depends
on whether the error belongs to the core or the non-core part of the
program.
•The perceived reliability depends to a large extent upon how the
product is used. In technical terms on its operation profile.

14. NAVAL EMC CENTRE
SOFTWARE FAILURE MECHANISM

15. NAVAL EMC CENTRE
AVERAGE FAILURE RATE OF A MS
PRODUCT
Failure intensity
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
1234567891011
Months frm release
Failures/month/unit

16. NAVAL EMC CENTRE
REASONS FOR THIS PHENOMENON
•Users learn with time and avoid failure
causing situation.
•Users start with exploring more, then limit
to some part of the product
–Most users use a few product features
•Configuration related failures are much
more in the start.
•These failures reduce with time

17. NAVAL EMC CENTRE
Measuring Software Reliability
Don’t define what you won’t collect..
Don’t collect what you won’t analyse..
Don’t analyse what you won’t use..

18. NAVAL EMC CENTRE
MEASURING SOFTWARE REL IABILITY
•Measuring software reliability remains a difficult
problem because we don't have a good
understanding of the nature of software
•Even the most obvious product metrics such as
software size have not uniform definition.
•Level of reliability required for a software product
should be specified in the SRS document.

19. NAVAL EMC CENTRE
SOFTWARE RELIABILITY
MODELING

20. NAVAL EMC CENTRE
SOFTWARE REL IABILITY MODELS
•Models have emerged as people try to
understand the characteristics of how and why
software fails, and try to quantify software
reliability
•Over 200 models have been developed since
the early 1970s, but how to quantify software
reliability still remains largely unsolved
•No single model completely represent software
reliability.
•Assumption : reliability is a function of the defect
level and as defects are removed, reliability
improves.

21. NAVAL EMC CENTRE
SOFTWARE REL IABILITY MODELS
•Software modeling techniques can be
divided into two subcategories:
–prediction modeling
–estimation modeling.
•Both kinds of modeling techniques are
based on observing and accumulating
failure data and analyzing with statistical
inference.

22. NAVAL EMC CENTRE
SOFTWARE REL IABILITY MODELS
ISSUES PREDICTION MODELS ESTIMATION MODELS
DATA REFERENCE Uses historical data Uses data from the
current software
development effort
WHEN USED IN
DEVELOPMENT
CYCLE
Usually made prior to
development or test
phases; can be used
as early as concept
phase
Usually made later in
life cycle(after some
data have been
collected); not
typically used in
concept or
development phases
TIME FRAME Predict reliability at
some future time
Estimate reliability at
either present or
some future time

23. NAVAL EMC CENTRE
SOFTWARE REL IABILITY MODELS
•Two main types of uncertainty renders any
reliability measurement inaccurate:
•Type 1 uncertainty:
–our lack of knowledge about how the system
will be used
•Type 2 uncertainty:
–reflects our lack of knowledge about the effect
of fault removal.

24. NAVAL EMC CENTRE
SOFTWARE REL IABILITY MODELS
•Most software models contain the following
parts:
–assumptions,
–factors,
–a mathematical function
•relates the reliability with the factors.
•is usually higher order exponential or
logarithmic.

25. NAVAL EMC CENTRE
SOFTWARE REL IABILITY MODELS
•Jelinski and Moranda Model
–Realizes each time an error is repaired
reliability does not increase by a constant
amount.
–Reliability improvement due to fixing of an
error is assumed to be proportional to the
number of errors present in the system at that
time.

26. NAVAL EMC CENTRE
SOFTWARE REL IABILITY MODELS
•Littlewood and Verall’s Model
–Assumes different fault have different sizes,
thereby contributing unequally to failures.
–Large sized faults tends to be detected and
fixed earlier
–As number of errors is driven down with the
progress in test, so is the average error size,
causing a law of diminishing return in
debugging

27. NAVAL EMC CENTRE
EQUAL-STEP RELIABILITY GROWTH
t1 t2 t3 t4 t5
Reliability
(ROCOF)
Time

28. NAVAL EMC CENTRE
RANDOM-STEP RELIABILITY GROWTH
t1 t2 t3 t4 t5
Time
Note different
reliability
improvements
Fault repair adds new fault
and decreases reliability
(increases ROCOF)
Reliability
(ROCOF)

29. NAVAL EMC CENTRE
MUSA’S MODEL
•Assumptions:-
- Faults are independent and distributed
with constant rate of encounter.
- Well mixed types of instructions,
execution time between failures is
large compared to instruction
execution time.
- Set of inputs for each run selected
randomly.
- All failures are observed, implied by
definition.
- Fault causing failure is corrected
immediately, otherwise reoccurrence
of that failure is not counted.

30. NAVAL EMC CENTRE
MUSA’S BASIC MODEL
•Assumption: Decrement in failure intensity
function is constant.
•Result: Failure intensity is function of average
number of failures experienced at any given
point in time (= failure probability).
Fl(m): failure intensity.
Fl
0
: initial failure intensity at start of execution.
Fm: average total number of failures at a given point in
time.
–v
0
: total number of failures over infinite time.
ú
û
ù
ê
ë
é
-=
0
0
1)(
v
m
lml

31. NAVAL EMC CENTRE
EXAMPLE
•Assume that we are at some point of time t time units in
the life cycle of a software system after it has been
deployed.
•Assume the program will experience 100 failures over
infinite execution time. During the last t time unit interval
50 failures have been observed (and counted). The
initially failure intensity was 10 failures per CPU hour.
•Compute the current (at t) failure intensity:
ú
û
ù
ê
ë
é
=
ú
û
ù
ê
ë
é
-=
ú
û
ù
ê
ë
é
-=
HourCPU
failures
v
5
100
50
110)50(
1)(
0
0
l
m
lml

32. NAVAL EMC CENTRE
MUSA/OKUMOTO LOGARITHMIC
MODEL
•Decrement per encountered failure decreases:
q : failure intensity decay parameter.
•Example 2
Fl
0
= 10 failures per CPU hour.
Fq =0.02/failure.
–50 failures have been experienced (m = 50).
–Current failure intensity:
qm
lml
-
=e
0
)(
68.31010)50(
1)5002.0(
===
-´-
eel

33. NAVAL EMC CENTRE
Model Extension (1)
•Average total number of counted experienced
failures (m) as a function of the elapsed
execution time (t).
•For basic model
•For logarithmic model
ú
ú
û
ù
ê
ê
ë
é
-=
-t
l
tm
0
0
1)(
0
v
ev
( )1ln
1
)(
0
+=qtl
q
tm

34. NAVAL EMC CENTRE
Example 3 (Basic Model)
"l
0
= 10 [failures/CPU hour].
•v
0
= 100 (number of failures over infinite
execution time).
"t = 10 CPU hours:
"t = 100 CPU hours:
ú
ú
û
ù
ê
ê
ë
é
-=
-t
l
tm
0
0
1)(
0
v
ev
[ ] failuresee 6311001100)10(
1
10
100
10
=-=ú
û
ù
ê
ë
é
-=
-
-
m
[ ] failuresee 10011001100)100(
10
100
100
10
»-=ú
û
ù
ê
ë
é
-=
-
-
m

35. NAVAL EMC CENTRE
Example 4 (Logarithmic Model)
"l
0
= 10 [failures/CPU hour].
"q = 0.02 / failure.
"t = 10 CPU hours:
"t = 100 CPU hours:
( )5511002.010ln
02.0
1
)10( =+´´=m
( )1ln
1
)(
0+=qtl
q
tm
( )152110002.010ln
02.0
1
)100( =+´´=m
(63 in basic model)
(100 in basic model)

36. NAVAL EMC CENTRE
Model Extension (2)
•Failure intensity as a function of execution time.
•For basic model:
•For logarithmic model
÷
÷
ø
ö
ç
ç
è
æ
-
=
t
l
ltl
0
0
0)(
v
e
1
)(
0
0
+
=
qtl
l
tl

37. NAVAL EMC CENTRE
Example 5 (Basic Model)
"l
0
= 10 [failures/CPU hour].
•v
0
= 100 (number of failures over infinite
execution time).
"t = 10 CPU hours:
"t = 100 CPU hours:
ú
ú
û
ù
ê
ê
ë
é
===
-
÷
ø
ö
ç
è
æ
´-
hourCPU
failures
ee 68.31010)10(
1
10
100
10
l
÷
÷
ø
ö
ç
ç
è
æ
-
=
t
l
ltl
0
0
0
)(
v
e
ú
ú
û
ù
ê
ê
ë
é
===
-
÷
ø
ö
ç
è
æ
´-
hourCPU
failures
ee 000454.01010)100(
10
100
100
10
l

38. NAVAL EMC CENTRE
Example 6 (Logarithmic Model)
"l
0
= 10 [failures/CPU hour]. q = 0.02 / failure.
"t = 10 CPU hours:
"t = 100 CPU hours:
ú
ú
û
ù
ê
ê
ë
é
=
+´´
=
hourCPU
failures
33.3
11002.010
10
)10(l (3.68 in basic model)
(0.000454 in basic model)
1
)(
0
0
+
=
qtl
l
tl
ú
ú
û
ù
ê
ê
ë
é
=
+´´
=
hourCPU
failures
467.0
110002.010
10
)100(l

39. NAVAL EMC CENTRE
MODEL DISCUSSION
•Comparison of basic and logarithmic model:
–Basic model assumes that there is a 0 failure intensity, logarithmic model
assumes convergence to 0 failure intensity.
–Basic model assumes a finite number of failures in the system, logarithmic
model assumes infinite number.
•Parameter estimation is major problem: l
0
, q, and
v
0
. Usually obtained from:
–system test,
–observation of operational system,
–by comparison with values from similar projects.

40. NAVAL EMC CENTRE
APPLICABILITY OF SOFTWARE
RELIABILITY MODELS
–There is no universally applicable reliability
growth model.
–Reliability growth is not independent of
application.
–Fit observed data to several growth models.
•Take the one that best fits the data.

41. NAVAL EMC CENTRE
STATISTICAL TESTING

42. NAVAL EMC CENTRE
•Testing for reliability rather than fault detection
•Test data selection should follow the predicted
usage profile for the software
•Measuring the number of errors allows the
reliability of the software to be predicted
•An acceptable level of reliability should be
specified and the software tested and amended
until that level of reliability is reached
STATISTICAL TESTING

43. NAVAL EMC CENTRE
STATISTICAL TESTING
•Different users have different operational profile:
–i.e. they use the system in different ways
–formally, operational profile:
•probability distribution of input
•Divide the input data into a number of input
classes:
–e.g. create, edit, print, file operations, etc.
•Assign a probability value to each input class:
–a probability for an input value from that class
to be selected.

44. NAVAL EMC CENTRE
•Determine operational profile of the software
•Generate a set of test data corresponding to
this profile
•Apply tests, measuring amount of execution
time between each failure
•After a statistically valid number of tests have
been executed, reliability can be measured
STATISTICAL TESTING PROCEDURE

45. NAVAL EMC CENTRE
STATISTICAL TESTING DIFFICULTIES
•Uncertainty in the operational profile
–particular problem for new systems with no
operational history.
•High costs of generating the operational profile
–Costs dependent on what usage information
is collected by the organisation which requires
the profile
•Statistical uncertainty when high reliability is
specified
–Usage pattern of software may change with
time

46. NAVAL EMC CENTRE
AN OPERATIONAL PROFILE
Number
of inputs
Input
classes

47. NAVAL EMC CENTRE
RELIABILITY PREDICTION
Reliability
Required
reliability
Fitted reliability
model curve
Estimated
time of reliability
achievement
Time
= Measured reliability

48. NAVAL EMC CENTRE
CASE STUDY

49. NAVAL EMC CENTRE
BANK AUTO-TELLER SYSTEM
•Each machine in a network is used 300 times a
day
•Bank has 1000 machines
•Lifetime of software release is 2 years
•Each machine handles about 200, 000
transactions
•About 300, 000 database transactions in total
per day

50. NAVAL EMC CENTRE
EXAMPLES OF A RELIABILITY SPEC.
Failure class Example Reliability metric
Permanent,
non-corrupting.
The system fails to operate with
any card which is input. Software
must be restarted to correct failure.
ROCOF
1 occurrence/1000 days
Transient, non-
corrupting
The magnetic stripe data cannot be
read on an undamaged card which
is input.
POFOD
1 in 1000 transactions
Transient,
corrupting
A pattern of transactions across the
network causes database
corruption.
Unquantifiable! Should
never happen in the
lifetime of the system

51. NAVAL EMC CENTRE
SPECIFICATION VALIDATION
•It is impossible to empirically validate very high
reliability specifications
•No database corruptions means POFOD of less
than 1 in 200 million
•If a transaction takes 1 second, then simulating
one day’s transactions takes 3.5 days
•It would take longer than the system’s lifetime to
test it for reliability

52. NAVAL EMC CENTRE
COSTS OF INCREASING RELIABILITY

53. NAVAL EMC CENTRE
CONCLUSIONS
•Software reliability is a key part in software
quality
•Software reliability improvement is hard
•There are no generic models.
•Measurement is very important for finding the
correct model.
•Statistical testing should be used but it is not
easy again…
•Software Reliability Modelling is not as simple as
it looks.

54. NAVAL EMC CENTRE
REFERENCES
•Musa, JD, Iannino, A. and Okumoto, K., “Software Reliability: Measurement,
Prediction, Application”, McGraw-Hill Book Company, NY, 1987.
•A. V. Aho, R. Sethi, and J. Ullman, "Compilers: Principles,Techniques, and
Tools", Addison-Wesley, Reading, MA, 1986.
•Reliability Engineering Hand Book, by Bryan Dodson and Dennis Nolan.
•Software Reliability Prediction Model, White Paper, By M Thangarajan
•Software Reliability Engineering – A Roadmap by Michael R Lyu

55. NAVAL EMC CENTRE
THANK YOU!