1995, ANNALS OF SOFTWARE ENGINEERING, VOL. 1

Software Process and Product Measurement

Editors:

James D. Arthur and Sallie M. Henry
Virginia Tech, Blacksburg, Virginia, USA

PREFACE

In the late 1800s Lord Kelvin recognized the crucial role of measurement in the management process. Stated in a paraphrased form, his contention is that "you cannot manage what you cannot measure." Today, metric-based analysis appears to be the most promising approach to controlling the software development process and for assessing the quality of the resulting product. Appropriately chosen, metrics can provide an objective approach to assessing and predicting quality. Assessment capabilities are derived from an examination of product attributes. Prediction, on the other hand, is based on an examination of process artifacts and trends. Moreover, guided by the proper framework, a measurement program can be tailored to provide insights into the development process that reveal why specific problems have occurred and which suggest corrective actions.

To establish an effective software quality measurement program one must recognize those objectives underlying software measurement and implement a measurement process that stresses their achievement. To help focus in on those objectives four crucial questions must be addressed.

Why does an organization pursue the establishment of a measurement program? More specifically, what goals underlie the implementation of a measurement program? The ones touted most often are those of increased productivity, enhanced product quality and an improved development process. Increased productivity is measured on both the individual and group levels. It is often related to short term goals focusing on the speedy development of individual software units. Enhanced quality, on the other hand, is often a tenuous claim because substantiating measures are typically subjective and controversial, and lacks a universally accepted set of standards for either measuring or comparing/contrasting quality. The claim of enhanced quality is often associated with more intermediate- or long-term goals of an organization. An improved process more directly reflects an organizational view and is almost always tied to the longer-term goals.

What is software quality? The meaning of software quality can vary, depending on a person's (or a group's) perspective. For the software engineer, quality characteristics are often stated in terms of attributes associated with individual software components, e.g., high code cohesion and low coupling among modules. A project manager's view of product quality, however, is often focused on the achievement of project level objectives, e.g., maintainability and reliability. Because software quality must reflect characteristics of the product as a whole, and not just components thereof, we maintain that the proper framework for expressing product quality must ultimately accommodate the project manager's perspective, i.e., that associated with project level goals and objectives. Based on a survey of current literature and focusing on product development from a software engineering perspective, we have identified the seven of the most widely accepted project level objectives attributable to software quality: adaptability, correctness, maintainability, portability, reliability, reusability and testability.

Notably missing from the above list is any mention of cost, schedule, and efficiency. Their purposeful omission, however, is not intended to imply that they have no impact on product quality--they can, and do. More precisely, we view cost, schedule and efficiency as constraints which are imposed at the systems engineering level, not as quality objectives of the software engineering process. Effectively, cost, schedule and efficiency are systems engineering objectives that are assumed to be "givens", and which, bound the limits of flexibility afforded to the software developers in producing a quality product.

How is software quality measured? An effective software quality measurement program cannot be developed using semi-related measures combined in an ad hoc, unnatural fashion. To the contrary, those measures, and the framework within which they are applied, must embody a realistic characterization of the software development process. In particular, both the definition and application of software quality measures must be guided by a systematic process that recognizes inherent linkages between the software development process and the achievement of desirable software engineering qualities. More specifically,

• a set of project level objectives should be identified which form the basis for defining a quality product,
• to achieve those objectives, specific principles must be recognized and employed to guide the process by which software is created, and finally
• adherence to a process governed by those principles should result in a product (programs and documentation) that possesses attributes considered desirable and beneficial.

When is software quality measured? To predict product quality, measurement must begin early in the software development life cycle. Typically, predictive measurement begins after the software requirements are fixed. Product quality assessment, on the other hand, entails an examination of the product (code and documentation) itself. Within the classical waterfall life cycle model, assessment is performed after the coding phase. If the development process follows an incremental approach, assessment can be performed on pieces of the product, and from which, those attendant measurement values can also be used to predict the quality of components yet to be developed.

In concert with the above comments and observations, we submit that the following preliminary steps are critical to effective software quality measurement:

1. Establishing an emphasis on software quality as the major goal of the underlying measurement process, and
2. Focusing on the definition of measures that
   • are objective,
   • directly related to software quality, and
   • reflect inherent characteristics of the software engineering process.

Today, software development processes and practices are being structured to improve the likelihood of achieving goals and objectives set forth in the systems and software engineering activities. Most approaches include a well-defined sequence of activities that embody requirements definition, design, implementation and unit/integration/system testing. Associated with each of these is a structured process or methodological approach that outlines how one carries out each activity. These activities, structured processes and methodologies have evolved over time and reflect a wealth of experience. Concomitantly, the articles appearing in this premier volume of the Annals of Software Engineering reflect the authors' experiences and "lessons learned" gained through research and the application of fundamental principles to practical problems. The articles represent a broad spectrum of topics describing many aspects of software quality measurement:

• To set the stage for discussion, the first set of articles compare and contrast software design, product complexity and user comprehension measures. The first two are survey in nature while the third draws more from personal research and experiences. The article by Chris Kemerer describes a number of empirical studies focusing on complexity metrics and comprehension. His survey provides a characterization of the major differences and similarities among modularity and structure metrics, and measures of program comprehension. Complementing Kemerer's work, Yu and Lamb discuss ten measures focusing primarily on design complexity, and then present a database schema from which most of the measures can be computed. Finally, Zage, Zage and Wilburn relate their work on design metrics to the identification of several pitfalls inherent to the quantitative analysis of software. They also present an approach to formulating design metric models that avoids those pitfalls.
• The second set of articles describe several models that can be applied to and guide software quality prediction and assessment. Boehm, Clark, Horowitz, Westland, Madachy and Selby propose a tailorable family of software sizing models, designated COCOMO 2.0, that is attuned to and accommodates the diversity of current software development philosophies. As described in their article, COCOMO 2.0 is currently serving as a framework for an extensive data collection and analysis effort to support further refinement and calibration of the model's estimation capability. Troster and Tian focus their attention on measuring and modeling defects in legacy software systems. They motivate the need for and introduce tree-based models that more accurately captures the relationships between defect occurrences and selected metrics which are intended to measure them. Troster and Tian present the tree-based approach as one which supports a natural decision process suitable for change solicitation and is useful in guiding remedial actions for quality improvement. Bob Tausworthe's work is motivated by the perceived need to develop cost/attribute relationships inherent to recognized quality factors. In particular, he presents a unifying metric for quality that is defined to be the ratio of cost expended at a given point in time to that which will be required to satisfy the requirements placed on the achievement of a quality attribute. Tausworthe's cost-based approach is intended as an additional "tool" for managing software quality throughout the software development process. Finally, Khoshgoftaar, Pandya and Lanning present a unique approach employing a neural network model to predict program faults. Most notably, their model is not susceptible to the violations of assumption that often wreak havoc on predictive models employing regression-based analysis. In their article, Khoshgoftaar, Pandya and Lanning motivate the need for and develop a neural model for predicting program faults. They apply the model to two commercial systems and then compare the results to those derived from multiple linear regression methods.
• Significant results stemming from novel applications of existing technologies are constantly being reported. The third set of articles fall in this domain. In particular, Murine and Murine describe several current initiatives undertaken by Rome Laboratories to evolve a supporting environment for implementing software quality measurement. Concerns expressed by many government agencies and companies are identified and addressed within the framework of the Rome Laboratory initiatives. Several case studies are presented to substantiate the enunciated concerns and to confirm their observations and research findings. The second article in this set, authored by Welch, Yu, Verhoosel, Haney, Samuel and Ng, explores reengineering technologies to assist in the identification and transformation of candidate procedures for concurrent execution. Their article defines an intermediate representation that: (a) exploits a tier approach to detecting program structure, (b) supports metric computations revealing potential concurrency, and (c) enables a systematic restructuring of existing software that reflects the beneficial qualities of the target language.
• The first three sets of articles present findings that are based on observation and experimental studies. Fundamental to synthesizing information that is meaningful to the researcher and to the software engineering community is knowing: (a) how to design an experiment, (b) what problems to avoid, and (c) how to validate the results. Consequently, the fourth set of articles address issues surrounding experimental design, implementation and validation. The article by Shari Pfleeger presents the key activities necessary for designing and analyzing an experiment in software engineering. She begins by describing how to formulate an hypothesis and how to effect the necessary control over the experimental variables. The article then presents a comprehensive description of the six stages of an experiment: conception, design, preparation, analysis and dissemination. John Munson's article then focuses the reader's attention on the principal problems surrounding the measurement process and the use of metrics. In his experiment, measures of static and dynamic program complexity are used to identify regions of software that are likely to contain faults. Munson approaches the measurement of computer software as multivariate problem, and employs an avionics study to support and help clarify his observations and conjectures. Norm Schneidewind provides the final article in this section and addresses the importance and practicalities of validating software metrics. In particular, he establishes the feasibility of and outlines one particular approach to validating metrics for controlling and predicting software during the design process. Using space shuttle flight software as an example, Schneidewind discusses the important roles of quality factors (e.g., reliability) and statistical techniques (e.g., discriminative power analysis, principal component analysis) in the validation process.
• The final article, authored by Watts Humphrey, introduces the "personal software process." In this article Humphrey focuses on improving the individual's productivity and ability to produce a quality product. His contention is that an improved personal process leads to an overall improvement in the team effort, and consequently, a higher quality product. Humphrey outlines a multi-stage process that motivates software engineers to adopt effective strategies to improve their personal software process. Key elements of the Capability Maturity Model, tailorable to the individual, serve as a basis for Humphrey's personal software process.

As the complexity and sophistication of today's software systems continue to increase, so must the knowledge base employed by the software engineering community. This premiere volume of the Annals of Software Engineering is devoted to software process and product measurement, and is a concerted effort to expand that base. We would like to thank the authors and referees for their meticulous attention to details during the initial submission and revision processes, and for their substantial contributions to making this volume part of that necessarily expanding body of knowledge.

James D. Arthur and Sallie M. Henry

Editors

Annals of Software Engineering Home Page