Posted by Comprehensive Assessment of Contract Performance using Earned Value Management Data
Management Track
MT18_Presentation_ContractPerformanceUsingEVMData_Laing
Abstract:
Purpose: This presentation outlines a set of data views that provide a comprehensive assessment of contract performance using Earned Value Management (EVM) data. The proposed views provide alternatives to traditional EVM metrics and are intended to provide early-warning indicators of contract performance.
Background: EVM reporting systems, when properly implemented, are designed to provide program managers and decision makers with critical insight into cost and schedule performance. However, oftentimes there are inherent weaknesses in the EVM system, underlying data, and/or commonly applied analysis methods that prevent program managers and decision makers from understanding the full magnitude of contract problems. In some cases EVM data may provide information that is misleading or biased. A few of the typical weaknesses found in EVM systems/data include the following:
• Unrealistic baselines
• Frequent re-baselining and/or baseline changes
• Inappropriate methods applied for measuring earned value
• Data anomalies
Typical analysis methods apply a “Gold Card” approach, using traditional EVM metrics such as Cost Performance and Schedule Performance Indices (CPI and SPI), Cost Variance (CV), and Schedule Variance (SV). Among the problems with these traditional metrics is that they tend to be backwards-looking and can be easily distorted by baseline changes. Improving on this requires a set of views generated from EVM data that both illustrate the changes that occur and provide more accurate assessments of cost and schedule performance in the context of those changes.
Discussion: We offer a set of views that illuminate contract performance by enhancing traditional EVM views with the application of new analytical methods. The views provide a comprehensive assessment of contract performance by addressing the following dimensions: budget growth, estimate growth, risk realization, budget absorption, cost variance, schedule deviations, personnel, and a macro level program view (i.e. “the big picture”). This set of graphical views exposes performance along multiple dimensions, enabling decision makers to recognize poor program performance earlier in the acquisition cycle. By providing views that cover multiple dimensions of contract performance it increases the likelihood of identifying poorly performing programs. A poor performer may be able to hide problems along a single dimension, but is unlikely to be able to do so across multiple dimensions.
To assess budget growth, the amount of contract growth to date and management’s approach to financing the growth (e.g., external budget growth, Management Reserve, budget shifting between WBS elements) is displayed. Estimate growth is addressed by illustrating the annualized and six month rates of growth (and the consequences of growth) at the total contract level and at key WBS elements. A view of management reserve (MR) consumption addresses how quickly risks are realized, and thus how quickly MR is depleted. A view of undistributed budget displays how quickly new budget is absorbed into the WBS. To address cost variance, the To-Complete Performance Index of the total contract and key WBS elements are used to illustrate the consequences of declining cost performance. Schedule performance is addressed using Weibull curve analysis and by measuring the amount of schedule pressure imposed by budget growth and work deferment. Personnel issues are addressed by comparing actual staffing to the planned level. Finally, macro level contract performance is displayed in an annual spending profile. This profile compares the original plan with actual performance (from both the contractor and government perspectives) to illustrate cost and schedule growth from a big picture perspective.
Author(s):
William Laing
Technomics, Inc.
Mr. Laing is a senior cost analyst at Technomics with eleven years of experience. In his four years at Technomics, he has supported the Naval Center for Cost Analysis, the Air Force Cost Analysis Agency (AFCAA), Marine Corps Systems Command, USA TACOM Cost and Systems Analysis Division, the Defence Cost and Resource Center (DCARC), OSD Cost Assessment and Program Evaluation (CAPE), and NAVSEA. For these customers, he has supported Independent Cost Estimates and Assessments, estimating methodology development, an Analysis of Alternatives, program Life-Cycle Cost Estimates, data collection plan development, data validation, and EVM View development. He is currently the project lead for Technomics’ support to OSD CAPE for Space Industrial Base analysis. The focus of this project has been developing EVM data views intended to provide early-warning indicators of contract performance. Prior to this, Mr. Laing was the project lead for Technomics’ support to USA TACOM’s Program Office Estimate for the Joint Lightweight Tactical Vehicle and the lead SRDR analyst for Technomics’ support to the DCARC. Prior to joining Technomics in 2006, Mr. Laing was employed at SAIC for seven years. His primary customer during this time was the Office of the Deputy Assistant Secretary of the Army for Cost and Economics. He performed numerous tasks relating to the Wheeled and Tracked Vehicles Automated Cost Database, including database development, data collection and normalization, methodology development, and life-cycle cost estimating. Other projects during this time included an Economic Impact Analysis for the National Nuclear Security Administration, an econometric analysis of installation efficiency for the Commander of Naval Installations, risk model assessment for the US Coast Guard, and various analyses for AFCAA.
Mr. Laing is ABD in Economics and has M.A. and B.A. Degrees in Economics from George Mason University in Fairfax, VA.
Colleen Craig
Technomics
Ms. Craig is a cost analyst at Technomics with experience in cost research, cost estimation and operations research. In her time at Technomics, she has supported the Air Force Cost Analysis Agency (AFCAA), OSD Cost Assessment and Program Evaluation (CAPE), Deputy Assistant Secretary of the Army-Cost & Economics (DASA-CE) and Department of Homeland Security’s Cost Analysis Division (CAD). For these customers, she has supported database development, EVM analysis, cost research, discrete event simulation, Independent Cost Estimates and development of a technical baseline document, with application areas including defense weapon systems (space systems, missiles, munitions, aircraft and electronics) and telecommunication systems. Prior to joining Technomics in 2009, Ms. Craig was an Industrial Engineer co-op at Walt Disney World Resort where she performed cost analyses and designed and implemented efficiency studies and process improvements. Ms. Craig holds a B.S. in Industrial and Systems Engineering from Virginia Tech and is a member of the Society of Cost Estimating and Analysis (SCEA).
Richard C. Lee
Technomics, Inc.
Richard C. Lee received his B.S. in Mechanical Engineering in 2007 from Virginia Polytechnic Institute and State University. He joined the Advanced Vehicle Dynamics Laboratory at the same institution and started his graduate studies on modeling and simulation of terrain profile models using numerical methods and vehicle dynamics. He received his M.S. in Mechanical Engineering in the spring of 2009. He is a member of the Society of Cost Estimating and Analysis (SCEA), the American Society of Mechanical Engineers (ASME), and the Institute For Operations Research and the Management Sciences (INFORMS). A Cost Analyst at Technomics, Inc., he has supported the Office of the Secretary of Defense, Cost Assessment and Program Evaluation (OSD CAPE), the Office of the Deputy Assistant Secretary of the Army for Cost and Economics (ODASA-CE), and the Naval Center for Cost Analysis (NCCA on projects spanning Earned Value Management (EVM) analysis, data collection and cost research, and risk analysis.
Scott DeNegre
Technomics, Inc
Scott DeNegre has seven years of experience in operations research analysis, with a particular focus in the development of effective solutions for large-scale, real-world systems. Scott’s main application areas include supply chain management, network design, urban planning and security, interdiction problems, and hierarchical decision-making systems. In addition, Scott has over two years of cost analysis experience, during which his methodological focus has been on earned value management, competitive contract analysis, and project scheduling, with application areas including satellite design and development, launch systems, and wheeled and track vehicles. Prior to joining the Technomics team in September 2009, Scott has held positions as a visiting researcher in the Interventional Guidance Technology group at Philips Research North America, an actuarial analyst at Watson Wyatt, and a visiting researcher in the mathematics department at Ecole Polytechnique Federal de Lausanne in Switzerland. Scott received a B.S. in Mathematical Sciences from Johns Hopkins University in 2002, and a M.S. in Management Science from Lehigh University in 2005. He recently defended his doctoral research on critical infrastructure defense and discrete bilevel programming, and will be receiving his Ph.D. from Lehigh University in May 2011. Scott has coauthored several journal papers, presented at a wide variety of professional conferences, and is a member of the Society of Cost Estimating and Analysis (SCEA), the Institute for Operations Research and the Management Sciences (INFORMS), and the Society for Industrial and Applied Mathematics (SIAM).