Managing Software Defects
Defect analysis and traceability
Craig Henderson, June 2006. Published online May 2007
An irony of Software Engineering is that it is an inexact science using very exact technology. Nothing could be more precise than a unit of two states, 0 or 1; yet creating the sequence of states that can get a machine to do something useful or interesting – at least to do it correctly and consistently – is very difficult. Defects will exist within software for as long as software itself exists, and that is set to be a very long time.
So we can resign ourselves to the inevitable doom of failure, or we can accept that the software we produce will contain defects and manage the situation as best we can. I am a realist – some may say a pessimist – but also a perfectionist. Accepting failure is difficult for me to do, so I look to the alternative of managing the situation and looking for improved methods of doing so.
Defect Management
The expectation of having defects is so great that all companies producing software commercially will run some form of defect management system. There is a vast choice of commercial and freeware products available to choose from, and I’m not going to condemn or condone any here. I have used a variety over the years and all have served the purpose – it’s just a case of efficiency. I would summarize the key requirements of a useful defect management system to be:
· Defect lifetime management
· User and Role based workflow
· Defect segregation (multiple products supported)
· Defect cross-referencing – preferably across segregation
· Root cause analysis (incomplete/inaccurate requirements, coding error, unit testing, system testing)
Defect Analysis
Table 1 shows a tabular analysis of defect statistics collected during a short release cycle of four software builds.
|
Build |
Defects Reported |
Defects Fixed |
Known Defects |
KLOC |
Defect Density |
|
Baseline |
-- |
-- |
460 |
78 |
5.9 |
|
0 |
26 |
78 |
408 |
84 |
4.9 |
|
1 |
14 |
31 |
391 |
84 |
4.7 |
|
2 |
5 |
28 |
368 |
85 |
4.3 |
|
3 |
2 |
9 |
361 |
85 |
4.2 |
|
|
47 |
146 |
|
|
|
Table 1: Defect analysis
The columns are defined as:
Build. The build number is an incremental number that uniquely identifies the software build. In this example, the build number starts at zero and increments by one each time the software is built. This build number can be used in a variety of ways to add further value. For example, one company I worked for used odd number builds internally within the software engineering team and even numbers to release a software build to the test team. This segregation of purpose helps to control the distribution of software internally within the company. The numbering scheme also extended naturally so that General Availability (GA) releases are only ever an even build number.
Defects Reported identifies how many defects have been reported to date (see Managing the Statistics) in each build of software. This figure should be a very simple metric to collect via a query from the defect management system in use in the organisation.
Defects Fixed identifies how many defects were claimed fixed (see Fixed or not fixed) in each build of software. This figure should also be a very simple metric to collect via a query from the defect management system.
Known Defects is calculated from the number of Known Defects in the previous build, plus the number of Defects Reported in the build less the number of Defects Fixed in the build. For example, for Build 1 in Table 1, 408+14-31=391.
KLOC. Thousands of lines of code in the module being analysed (see Scope).
Defect Density. This is a very widely used metric computed by dividing the number of defects by the KLOC to yield an index figure. How this figure is interpreted and used is then a much wider field, and interpretation must be defined in drawing conclusions using the defect density. The figure standing alone is near useless, and can even be dangerous if its purpose and meaning is left to the reader to define.
Each row in Table 1 represents a build of software. The top row is labelled Baseline and the purpose of this is to relate the analysis within this table back to the previous round of analysis. The baseline row is useful to enable segregation of defect analysis between software releases.
Scope
To get most out of the analysis, it is important to define the scope at a manageable and appropriate level. In a complex software project consisting of many applications with shared components, for example, collecting and analysing metrics across the entire product will yield little benefit as the defect density will be diluted and it will be impossible to quantify the quality of each application. Defining the scope on a granularity that is too fine will cause an excessive amount of information to be generated and the analysis will become too time-consuming to be beneficial, and the process will become disproportionate to the value gained from doing it. A fair compromise is to calculate and analyse the Defect Density at the module level, but the scope will vary from one project to the next.
Managing the Statistics
The statistics cannot be frozen once they are published. As testing continues with previous builds, or as customers report issues with releases, the figures for Defects Reported and Defect Density need to be updated. For a General Availability software release, it is important to be able to capture the defect analysis before the release and periodically after it. This will be helpful in analysing the trend of the defect density pre- and post-release, which will give an indication to the efficacy of the system testing within the software production lifecycle.
Fixed or not Fixed?
In nearly every organisation in which I have worked, there has been a lot of debate over what to do with a defect report if system testing demonstrates that the defect is not fixed, or is only partially fixed. The fundamental contentious issue is whether the defect should be reopened and re-assigned to the software engineering team, or if a new defect should be raised to describe the defect again, or the remaining issues not addressed by the original issue. During my career, I have changed my opinion over which is the best way to address such an issue several times.
From a developer’s perspective, re-opening the defect is preferable, as it keeps all the detail together and the discussion through the defect report can help to document the decisions made and the final conclusion over what the fix should be. There can be occasions where a very subjective defect is discussed to such an extent that the defect becomes confused, in which case I’d advocate closing the original defect as a non-issue and create a new defect report (against the same build number) to summarise the discussion. Cross-references between the two defects – in both directions – is an absolute must though, to ensure traceability.
However, from a management perspective, my view is different. While I still appreciate the benefits of the developer’s preference, the overall picture of software quality becomes skewed if defects are re-opened – particularly if the defect is partially fixed, or the fix has introduced further problems. If a defect is only partially fixed, then re-opening the defect will cause the partial fix to be omitted from the release information for the next build, and the regression test and subsequent system testing will suffer. Where new defects have been introduced by a fix, then if the original defect is simply re-opened and re-assigned to the software engineering team, then the new defects are missed from the statistics, and the defect density figures are inaccurate. I therefore now insist that defects are never re-opened, but a new defect must be raised to cover the remaining problem, or new problems that have been introduced, and if I can get a metric of defects claimed fixed but proved not fixed, then that helps to identify areas for improvement within my software engineering team.
Analysis Chart
Figure 1: Radial analysis chart
Table 1 is sufficient in providing the detail for each build and a trend can be seen across the builds by analysing the column contents, but the information is flat and difficult to comprehend. Even in this four-build example, the numbers are beginning to blur and lose any impact. To try to breathe some life into the statistics and to avoid the number blindness of staring at a table of figures, I have devised a radial analysis chart representation of the data, as shown in Figure 1. The intention with this chart is to provide information about the build quality in a way that cannot be presented in a two-dimensional table. Each software build is represented by a circular band – a build band. Thin black lines are drawn outwards from the centre point for each statistic element (column) from Table 1, and where the line passes through the build band, the numeric statistic is written adjacent to the line.
This in itself is fairly unremarkable and simply an alternative representation of the information. However, with a careful design in placing the statistic elements, I can begin to add more data in a third dimension of the graph.
A key piece of information missing from Table 1 is a traceability of the build numbers where defects are reported and fixed. It gives us totals of Defects Reported and Defects Fixed per build, but there is no correlation between the two. This additional information is however represented in the radial analysis chart in Figure 1. By orienting Defects Reported vertically at the top of the chart, we can introduce a Defects Fixed statistic which represents the build where defects are fixed and relate them to where they are reported. Note how the Defects Fixed statistic from Table 1 has been labelled Defects Fixed per Build in the chart for clarity.
The vertical Defects Reported statistic remains unchanged from Table 1 and records the number of defects that are reported in a given build number. From the per-build value at the intersection of the build band and the vertical statistic bar, I have drawn a horizontal statistic bar which crosses the build bands for subsequent builds. At each intersection I have entered the number of those defects that were fixed in each build. This gives a statistic of quality improvement through the build/test cycle prior to a software release to a customer. At the end of each horizontal bar is a total representing the number of defects that were reported in each build and fixed in subsequent builds. The sum of these totals will then equal the sum of Defects Fixed per Build;
Root Cause Analysis
Figure 2: Root Cause Analysis
Collecting defect statistics is invaluable for assessing the efficiency of the development team and can help to identify areas for improvement. With the addition of the traceability statistic in Figure 1, we begin to get real value by seeing a trend of figures through the release cycle. A logical addition to the statistics already discussed is to be able identify where in the software development lifecycle a defect was introduced; the root cause. Identifying the root cause can be a controversial and subjective issue. The idea is not to cause rifts between members of the development team, or even between the development team and other teams that contribute to the software development lifecycle, but to build a long-term picture of the performance of the group and to be able to identify where attention could be concentrated for the biggest improvement.
I have extended the radial analysis chart that I devised for defect analysis for use in Root Cause Analysis (Figure 2). The basic principle is the same, but this time each radial band represents an element to which a defect can be attributed. In my example, I have chosen to use these stages; Requirements, Architecture, Design, Implementation, Testing.
The vertical statistic in this chart is the Defects Attributed figure, which represents the number of defects that have been attributed to each element (band). The horizontal statistic bar coming off of each of these figures crosses subsequent elements in the software development lifecycle and the intersection of this bar with each band shows a figure for the number of defects later in the lifecycle that were ultimately caused by a particular defect; knock-on defects. The totals at the end of the horizontal statistic bars summarise the number of defects that were cause.
The figure shown in Figure 2 demonstrates what I would consider to be a typical trend. A low number of defects were present in the requirements and the design, but the knock-on defects caused by these few are significant.
Conclusion
The radial charts presented here can be used to represent derivative information in a two-dimensional form. I have used defect analysis and root cause analysis to demonstrate their use in presenting complex numeric information in an easy to understand form.
This form if representation can be useful in a number of situations where there are dependencies or contributing values which influence subsequent values. Without any specific time line requirement, the dependencies are captured and represented such that influences can be identified addressed to reduce the knock-on effects.
CRAIG HENDERSON holds a B.Sc. Honours Degree in Computing for Real Time Systems, has been a professional software engineer for over 11 years. His interests include systems programming, enterprise software and modern generic programming. He works in the UK as a Software Development Manager. He can be contacted at cdm.henderson@googlemail.com.
See other papers by Craig.