Posted by The Perils of Portability: CGFs and CVs
Risk Track
RS08_Presentation_PerilsofPortabilityCGFs&CVs_Braxton
Abstract:
With the provisions of the Weapon Systems Acquisition Reform Act of 2009, or WSARA, raising the importance and visibility of risk analysis, it is an opportune time to revisit how risk benchmarks based on historical data are both derived and applied. This paper will focus on two key summary risk metrics: (1) the cost growth factor, or CGF, which captures historical cost growth for individual programs and is used collectively to capture the pervasive tendency to underestimate cost; and (2) the coefficient of variation, or CV, as a measure of the uncertainty in both historical and new estimates. In particular, it will examine the central question of how best to derive historical CVs to apply to new programs. It will build on work developing an S-Curve Tool for the Naval Center for Cost Analysis (NCCA ) and recent papers by the authors on that subject: “Development and Application of CV Benchmarks” (DoDCAS 2011) and “Testing S-Curves for Reasonableness: The NCCA S-Curve Tool” (SCEA 2011).
The paper is divided into three sections. The first section briefly summarizes the basis for the historical benchmarks developed, namely Selected Acquisition Report (SAR) Summary Sheets for Department of Navy (DON) Major Defense Acquisition Programs (MDAPs). We discuss the development of a SAR database, compare SARs to Contract Data, and examine the provenance of the SAR Baseline Estimates.
The second section presents issues in SAR data analysis affecting how the benchmarks are derived. The CGF calculation is often dependent on a quantity adjustment, and we explore the Laspeyres, Paasche, and Fisher index approaches. We show the standard deviation, percentiles, and CV of the CGFs, and present an alternate method of deriving benchmark CVs using maximum likelihood estimation (MLE) regression.
The third section discusses some of the algorithms by which the benchmarks are applied in the NCCA S-Curve Tool. These include specification of baseline S-curves via various combinations of distribution parameters; the application of historical CGF and/or CV benchmarks to adjust the baseline S-curve; and solving an ancillary issue of graphing “noisy” empirical probability density functions (pdfs).
Throughout, we examine the so-called “perils of portability” and what information about the underlying historical data is maintained, distorted, or lost when we excerpt only summary metrics like CGF and CV. We illustrate these discrepancies numerically and graphically using the Milestone B and C subsets of the DON SAR data.
Author(s):
Peter J. Braxton
Technomics, Inc.
Peter J. Braxton holds an AB in Mathematics from Princeton University and an M.S. in Applied Science (Operations Research) from the College of William and Mary. A Senior Cost Analyst at Technomics, Inc., he is a Certified Cost Estimator/Analyst (CCEA) and currently serves as Director, Body of Knowledge, for the Society of Cost Estimating and Analysis (SCEA). He was named SCEA’s 2007 Estimator of the Year for contributions in Education and received both a TASC President’s Award for Operational Excellence and a Northrop Grumman Corporate Contracts and Pricing Award in 2008. He served as SCEA’s Training Chair from 2004 to 2009 and as a Northrop Grumman Technical Fellow from 2006 to 2009.
He currently supports the Naval Center for Cost Analysis (NCCA) and Air Force Cost Analysis Agency (AFCAA) on data collection and cost research efforts, and the Naval Postgraduate School (NPS) and Defense Acquisition University (DAU) on curriculum development. Throughout more than a dozen years in the community, he has worked to advance the state of knowledge of cost estimating and risk analysis, Cost As an Independent Variable (CAIV), and Target Costing on behalf of the Navy Acquisition Reform Office (ARO), the DD(X) development program, and other ship and intelligence community programs. He has conducted independent cost evaluations (ICEs) on all of Northrop Grumman’s major shipbuilding programs, and on several IT systems and services programs. He served as managing editor for development and maintenance of the acclaimed Cost Estimating Body of Knowledge (CEBoK) and its predecessor, Cost Programmed Review Of Fundamentals (CostPROF), and as the Training Track chair for the last seven SCEA international conferences. He has taught extensively at government, corporate, and society training events throughout the United States, Europe, and Australia. He is lead author or co-author of over a dozen professional papers on cost, risk, and CAIV, including two SCEA Best Paper winners.
Richard L. Coleman
TASC
Dick Coleman is a Naval Academy graduate, received an M. S. with Distinction in Operations Research from the U. S. Naval Postgraduate School, where he received the Chief of Naval Operations Award for Excellence in Operations Research. He retired from active duty as a Captain, USN, in 1993. His service included tours as Commanding Officer of USS Dewey (DDG 45), and as Director, Naval Center for Cost Analysis. At TASC and then Northrop Grumman, he worked extensively in cost, CAIV, and risk for the Missile Defence Agency (MDA), Navy ARO, the intelligence community, NAVAIR, and the DD(X) Design Agent team. He was the Director of Independent Cost Estimation for Northrop Grumman Information Systems and conducted Independent Cost Evaluations on over 150 Northrop Grumman programs in ships, IT, electronics, mission and space systems. He supported numerous ship programs including the DDG 1000 class, DDG 51 class, Deepwater, NSC 1 class, LHD 8 and LHA 6, LPD 17 class, Virginia class SSNs, CVN 77, and CVN 78. At Technomics, Inc., he was a Senior Technical Fellow and supported the Naval Center for Cost Analysis, the Department of Homeland Security, the Naval Postgraduate School, and the Defence Acquisition University in cost estimation and risk analysis. He retired from full-time employment in 2011. He has over 80 professional papers to his credit, including five ISPA/SCEA and SCEA Best Paper Awards and two ADoDCAS Outstanding Contributed Papers. He was a senior reviewer for CostPROF and CEBoK and lead author of the Risk Module. He has served as Regional and National Vice President of SCEA and is currently a Board Member. He received the SCEA Lifetime Achievement Award in 2008.
Richard C. Lee
Technomics, Inc.
Richard C. Lee received his B.S. in Mechanical Engineering in 2007 from Virginia Polytechnic Institute and State University. He joined the Advanced Vehicle Dynamics Laboratory at the same institution and started his graduate studies on modeling and simulation of terrain profile models using numerical methods and vehicle dynamics. He received his M.S. in Mechanical Engineering in the spring of 2009. He is a member of the Society of Cost Estimating and Analysis (SCEA), the American Society of Mechanical Engineers (ASME), and the Institute For Operations Research and the Management Sciences (INFORMS). A Cost Analyst at Technomics, Inc., he has supported the Office of the Secretary of Defense, Cost Assessment and Program Evaluation (OSD CAPE), the Office of the Deputy Assistant Secretary of the Army for Cost and Economics (ODASA-CE), and the Naval Center for Cost Analysis (NCCA on projects spanning Earned Value Management (EVM) analysis, data collection and cost research, and risk analysis.
Brian Flynn
Naval Center for Cost Analysis
Dr. Brian Flynn serves as special assistant and advisor to the Deputy Assistant Secretary of the Navy(Cost and Economics). He was appointed to his present position as a Highly Qualified Expert by the Secretary of the Navy in 2007.
Dr. Flynn was awarded the DON Superior Civilian Service Medal in 2005; the DON Meritorious Civilian Service Medal in 2001; and a meritorious promotion while on active duty in the United States Marine Corps in 1974.
Dr. Flynn is Study Director of a NATO effort to generate independent cost estimates of several weapon system acquisition programs and to examine the role played by life-cycle cost analysis in managing national defense enterprises within the Alliance.
Dr. Flynn has led special projects such as financial analyses of defense contractors; re-engineering the defense health care system; economic analyses of personnel manning of Navy ships; cost estimating of Marine Corps housing and pay systems; analyses of the defense industrial base; and portfolio analysis of the Navy’s investment programs.
Dr. Flynn earned a bachelor’s degree and a master’s degree in Economics from Virginia Tech and a Ph.D. in Economics, with Distinction, from Georgetown University. His academic areas of specialty are theoretical econometrics and mathematical economics.
Dr. Flynn has undying pride in having once worn the uniform of a United States Marine. Semper Fi.
Megan Guild
Technomics, Inc.
Megan E. Guild received her B.A. in Mathematics and French with a concentration in Economics in 2010 from Cornell University. She is a member of the Society of Cost Estimating and Analysis (SCEA). As a Cost Analyst at Technomics, Inc., she has supported the Deputy Assistant Secretary of the Army for Cost and Economics (ODASA-CE) and the Naval Center for Cost Analysis (NCCA) on projects spanning software schedule analysis, data collection and cost research, and SAR risk analysis using maximum likelihood estimators and statistical analysis. She currently supports the Office of the Secretary of Defense, Cost Assessment and Program Evaluation (OSD CAPE) as an analyst for the Defense Cost and Resource Center.
Jack Smuck
Technomics, Inc.
Jack Smuck is a senior cost analyst with Technomics, Inc., where he has been employed since 2003. He has over 30 years experience in cost analysis, 22 of which was with the Naval Center for Cost Analysis (NCCA) and its predecessor organizations. He started as a missile cost analyst, prior to becoming Head of the Missile Division. Subsequent assignments were as Head of both the Ships and Electronics Divisions. He completed his civilian Navy career in 2003 as the Deputy Director of NCCA. Mr. Smuck developed many independent cost estimates and analyses and supervised the development of numerous others.
Prior to entering the cost analysis field, he served as an operations research analyst on the staff of several divisions within the Chief of Naval Operations. He first worked in what is now N8, developing the Program Objective Memorandum (POM), then in N4 generating non-nuclear ordnance requirements, and, finally, in what is now N78, generating aircraft requirements.
At Technomics, Mr. Smuck has supported the Army, Navy, and OSD cost centers, in addition to NASA, the Defense Information Systems Agency (DISA), the Defense Acquisition University (DAU), and the Government Accountability Office (GAO), where he co-authored the initial version of the GAO Cost Guide.
Mr. Smuck is a native of Baltimore. He earned a BS in physics from Loyola University, Maryland and an MBA in operations analysis from American University. He is also a graduate of the Industrial College of the Armed Forces (ICAF). Mr. Smuck is DAWIA Level III certified and a member of the Society of Cost Estimating and Analysis (SCEA). He received both the Superior Civilian Service award and the Meritorious Civilian Service award. He is a veteran of the US Army Signal Corps.
Kevin Cincotta
Technomics
Kevin Cincotta is a Senior Cost Analyst at Technomics. His primary areas of expertise are cost analysis, database creation and management, and statistics. Mr. Cincotta leads projects for the Defense Acquisition University (DAU) and Naval Center for Cost Analysis (NCCA). The projects focus on development and maintenance of training materials for cost analysts at all levels; and analysis of cost growth in time series data relating to contract obligations, respectively. In addition, he serves as Director of the Technomics Training Institute, which trains junior and mid-level costs analysts with the aim of building core knowledge and facilitating professional certification.
From 2003 to 2009, Mr. Cincotta was a Research at Fellow at LMI (formerly the Logistics Management Institute). He led myriad projects for clients in the Departments of Defense and Homeland Security. These include analysis of cost per flying hour calibration factors for the Air Force Cost Analysis Agency (AFCAA), development of the Program-Budget/Joint Capability Area (P-B/JCA) data structure for the Office of the Secretary of Defense, Capability Assessment and Program Evaluation (OSD CAPE), and various applied estimates and comparative analyses for the Coast Guard Research and Development Center (RDC). He was also a lead instructor for LMI’s internal cost estimating and analysis training.
From 2001 to 2003, Mr. Cincotta served as a Senior Cost Analyst at MCR, LLC. He worked closely with government clients at the Missile Defense Agency (MDA) to develop a radar cost model, which was presented by MCR at the 2004 Society of Cost Estimating and Analysis (SCEA) conference.
Mr. Cincotta also led several cost analysis-related tasks at the New Vectors (formerly Vector Research, Incorporated and the Altarum Institute) from 1997 to 2001. As a Senior Cost Analyst and Systems Developer, he assisted in creating life cycle cost estimates (LCCEs) for myriad DOD projects, including the Standard Procurement System (SPS), the Defense Occupational Health Readiness System (DOHRS), and the Simplified Tax and Wage Reporting System (STAWRS). He is a frequent presenter at both the Department of Defense Cost Analysis Symposium (DODCAS) and SCEA conferences. Mr. Cincotta is a (SCEA)-Certified Cost Estimator/Analyst (C/CEA). He created several model questions for the current C/CEA exam, and currently serves at the SCEA Training Chair. Known as Crystal City’s original Nature Boy, Cincotta holds a master’s degree in economics and philosophy from the London School of Economics and Political Science, and a bachelor’s in the same fields from the University of Virginia.