Acquisition Management Metrics

Definition: According to, metrics are defined as standards of measurement by which efficiency, performance, progress, or quality of a plan, process, or product can be assessed. Acquisition management metrics are specifically tailored to monitor the success of government acquisition programs.

Keywords: acquisition metrics, leading indicators, program success

MITRE SE Roles & Expectations: Within the role of providing acquisition support, MITRE systems engineers (SEs) are tasked with understanding technical risk and assessing success. Management metrics are used as a mechanism to report progress and risks to management. MITRE staff should understand how these metrics influence and relate to acquisition systems engineering.

The use of metrics to summarize a program's current health, identify potential areas of concern, and ability to be successful are common practice among government departments, agencies, and industry. Metrics range from detailed software metrics to more overarching program-level metrics. Some of the following examples are derived primarily from the Department of Defense (DoD) program practice, but the principles are applicable to any program.

Probability of Program Success Metrics

As an aid in determining the ability of a program to succeed in delivering systems or capabilities, the military services developed the Probability of Program Success (PoPS) approach. PoPS standardizes the reporting of certain program factors and areas of risk. Each service measures a slightly different set of factors, but all the tools use a similar hierarchy of five factors at the top level. These factors are Requirements, Resources, Execution, Fit in Vision, and Advocacy. Associated with each factor are metrics, as indicated in Figure 1. These tools are scoring methodologies where metric criteria are assessed by the program office, a metric score/point is determined, and metric scores are weighted and then summed for an overall program score. The summary score is associated with a color (green, yellow, or red), which is the primary way of communicating the result. It is at the metric level and the criteria used to assess the metric where the biggest differences between the tools exist. Each tool weighs metrics differently, with Air Force and Navy varying these weights by acquisition phase. Furthermore, each service uses different criteria to assess the same or similar metric. Consult with the references [1, 2, 3, 4] for each service tool at the end of this paper to better understand how metrics are scored.

Best Practices and Lessons Learned: Determining the value of each metric is the responsibility of the acquisition program team. System engineering inputs are relevant to most of the reporting items; some are more obvious than others. With respect to staffing/resources, it is important to understand the right levels of engineering staffing for the program management office and the prospective development contractor to ensure success. At the outset of an acquisition, a risk management process should be in place (see the Risk Management section within this guide); the ability of this process to adequately identify and track risks is a major component of the PoPS tool. All technical risks should be incorporated in this assessment, including those that may be included in the technical maturity assessment. Immature technology can be a considerable risk to program success if not managed appropriately; it also can be scheduled for insertion into the program delivery schedule upon maturation. For more detail on technology maturity, see the article Assessing Technical Maturity in this section.

Generic Representation of Metrics Considered in PoPS Tools

Figure 1. Generic Representation of Metrics Considered in PoPS Tools

Note: This structure is generic and meant to closely represent what the services capture in their respective PoPS tools and where.

Although a metric name may be different or absent when comparing one tool to another, same or similar qualities may be captured in a different metric. Conversely, metrics may have the same or similar name but capture different qualities of the program. Refer to the individual service's PoPS operations guide for details [1, 2, 3, 4].

Earned Value Management (EVM) Metrics

A subset of program management metrics is specific to contractor earned value. Typical EVM metrics are the Cost Performance Index and the Schedule Performance Index; both are included in each of the service's PoPS tool [5, 6]. Although EVM is mostly considered a monitoring tool for measuring project performance and progress, it is also a planning tool. Using EVM effectively requires the ability to define, schedule, and budget the entire body of work from the ground up. This is something to be considered in the planning phase of an acquisition program (see the article Integrated Master Schedule/Integrated Master Plan Application in this section) because it is closely linked to the Work Breakdown Structure (WBS).

Best Practices and Lessons Learned: Fundamental to earned value is linking cost and schedule to work performed. However, work performed is often specified at too high a level to identify problems early. This is linked back to the generation of the WBS during the initial program planning and whether it was created at a detailed enough level (i.e., measureable 60 day efforts) to clearly define work performed or product developed. In cases where the detail is insufficient, EVM is unlikely to report real problems for several months. It is usually program engineers and acquisition analysts who are able to identify and report technical and schedule problems before the EVM can report them. Another method is the use of Technical Performance Measures (TPMs). TPMs are metrics that track key attributes of the design to monitor progress toward meeting requirements [7, 8]. More detailed tracking of technical performance by contractors is becoming popular as a way to measure progress and surface problems early using Technical Performance Indices at the lowest product configuration item (i.e., Configuration Item, Computer Software Configuration Item) [9].

Appropriate insight into evaluating the work performed for EVM can be challenging. It often requires close engineering team participation to judge whether the EVM is accurately reporting progress. A case where this is particularly challenging is in large programs requiring cross-functional teams and sub-contracts or associate contracts. Keeping the EVM reporting accurate and timely is the issue. To do this, the contractor's team must have close coordination and communication. Check that forums and methods are in place to accurately report EVM data for the entire program.

Systems Engineering Specific Metrics—Leading Indicators

Several years ago, MITRE engineers assisted in the development of a suite of "leading indicators" to track more detailed systems engineering activities for a DoD customer. Table 1 summarizes some of these metrics (expanded to generically apply here), which were to be assessed as red, yellow, or green, according to specified definitions. An analysis of the overlap of the leading indicators with PoPS metrics was conducted. Although several metrics have similar names, the essence of what is captured is different, and there is very little overlap.

Table 1. Leading Indicators

Leading Indicator Area

Detailed Metrics

Measurement Comments

Program Resources

Required Staffing Applied

Appropriate Skills Applied

Churn Rate

Training Available

Staffed to at least 80% according to plan


Ideally churn is less than 5% per quarter



Low volatility (< 5%) can still be a problem if the changing requirements have a large impact to the cost and schedule (like 10–15% cost growth)

Risk Handling





Appropriate Priority Applied

Are the risks coming to closure without significant impact? Alternatively, are there more risks being created over time (+ or – slope)?

Are resources being applied to the risk?

Is there appropriate engineering and PM oversight?


Community of Interest Established and Active

Data Sharing Addressed

Network Accessibility

Ideally, a data sharing plan exists for members of the COI and addresses the data formats, visibility / discovery (to include metadata), and plan for exposure.

Software Development





(These are fairly standard metrics—for more information on software metrics, see NIST Special Publication 500-234.)

Verification & Validation

Complete Requirements Documentation

Requirements Test Verification Matrix (RTVM)

Verification and Validation (V&V) Plan

The degree of completeness and volatility of these items is the measurement.

Technology Readiness

Technology Readiness

TRL of at least 6

Risk Exposure

Cost Exposure

15% deviation indicates high risk.


Schedule Exposure


Watchlist Items (Risks)


Closure Rate

As opposed to the risk handling metric above, this one looks at the severity of the risks—which ones are likely to occur with a med-high impact to the program.

Although this suite of "leading indicators" was not officially implemented by the customer, they make a good set of items to consider for metrics; they also capture areas not covered in other models.

Observations and Lessons Learned

  • Although the metric names in the three cited tools (and others) may be similar, they may be assessed differently. When comparing metrics across different tools, you need to understand the details of metric definitions and assessment scales. Make sure this is conveyed when reporting so that the intended audience gets the right message; the assessment needs to stand on its own and not be misinterpreted.
  • Understand what a metric is supposed to be measuring: trends, current status, and the ability to be successful when resolution plans are in place. This will ensure that results are interpreted and used properly.
  • Use the metrics that are most appropriate for the phase of the program you are in; if the program has overlapping phases, use multiple metrics. When a program in overlapping phases is assessed as if it were in a single program phase (as in PoPS), the resulting report is usually not an accurate representation of the program status.
  • Be cautious of methodologies where subjective assessments are converted to scores. Developing scoring methodologies can appear to be simple, yet mathematical fundamentals must still be followed for results to be meaningful [10]. This is particularly a concern when the resulting single score is taken out of context of the analysis and used as a metric in decision making.

References & Resources

  1. U.S. Army, May 2004, Probability of Program Success Operations Guide.
  2. U.S. Air Force, July 2008, Probability of Program Success (PoPS) Model, SMART Integration, Operations Guide, Version 1.0.
  3. Department of the Navy, September 2008, Naval PoPS Guidebook, Guidance for the Implementation of Naval PoPS, A Program Health Assessment Methodology for Navy and Marine Corps Acquisition Programs, Version 1.0.
  4. Department of the Navy, September 2008, Naval PoPS Criteria Handbook, Supplement for the Implementation of Naval PoPS, Version 1.0.
  5. Meyers, B., Introduction to Earned Value Management, EVM 101, ESC/AE (Acquisition Center of Excellence).
  6. Meyers, B., Analysis of Earned Value Data, EVM 401, ESC/AE (Acquisition Center of Excellence).
  7. Ferraro, M., "Technical Performance Measurement," Defense Contract Management Agency, Integrated Program Management Conference, November 14-17, 2001.
  8. Pisano, Commander N.D., Technical Performance Measurement, Earned Value, and Risk Management: An Integrated Diagnostic Tool for Program Management, Program Executive Office for Air ASW, Assault, and Special Mission Programs (PEO [A]).
  9. June 28, 2007, "Statement of Work (SOW) for Development, Production, Deployment, and Interim Contractor Support of the Minuteman Minimum Essential Emergency Communications Network (MEECN) Program," MMP Upgrade.
  10. Pariseau, R. and I. Oswalt, Spring 1994, "Using Data Types and Scales for Analysis and Decision Making," Acquisition Review Quarterly, p. 145.

Additional References & Resources

"Earned Value Management MITRE Community Share site," accessed January 20, 2010.

Massachusetts Institute of Technology, International Council on Systems Engineering, Practice Software and Systems Measurement, June 15, 2007, Systems Engineering Leading Indicators Guide, Version 1.0.

March 29, 1996, NIST Special Publication 500-234, Reference Information for the Software Verification and Validation Process, (Appendix A.1 Metrics).


Download the SEG

MITRE's Systems Engineering Guide

Download for EPUB
Download for Amazon Kindle
Download a PDF

Contact the SEG Team