Posted by Testing S-Curves for Reasonableness: The NCCA S-Curve Tool
Risk Track
RS09A_Paper_NCCASCurveTool_Coleman
RS09_Presentation_NCCASCurveTool_Coleman
Abstract:
The requirement of the Weapon Systems Acquisition Reform Act of 2009 to ” … disclose in accordance with paragraph (2) the confidence level used in establishing a cost estimate for a major defense acquisition program or major automated information system program…” and the austerity of current Department of Defense (DoD) budgets has brought about an increased interest in risk analysis and widespread use of the S-curve (the cumulative distribution function) of cost. As this interest intensified, experience in the use of S-curves widened.
As is becoming clear with experience so far gained, there is a natural evolution from naïve trust in S-curves; to the realization that not all S-curves are created equal; to the understanding that one of the quickest and surest ways to detect a suspicious S-curve is the coefficient of variation (CV); to the temptation for (if not the actuality of) gaming of CVs; to a mature practice of risk. The authors were centrally placed during the development and evolution of a company-wide risk analysis of the very sort that DoD is now undergoing. They sat on the Risk Working Group that determined policy and process from inception to completion; led implementation of accompanying tools in three different sectors of the company; and served on the committee that approved (and disapproved) the various candidate tools for use. In short, they were intimately involved in the life cycle of the risk evolution within the company, from the earliest stages until the practice had become mature. The briefing will outline the steps (and some missteps) that the Risk Working Group went through and apply the lessons learned to the current situation in the DoD.
Finally, a prototype tool has been developed for practitioners to display the S-curve as developed by the estimator and to compare it to a historically-based, commodity-specific, phase-appropriate S-curve. It can be used to benchmark estimates, to compare current and prior estimates, and to reconcile between two estimates, with a variety of historically based adjustments to either or both. This tool will guide the practitioner in judging the S-curve and will produce output intended for briefings to decision makers. The briefing will describe and demonstrate the tool.
Author(s):
Richard L. Coleman
TASC
Dick Coleman is a Naval Academy graduate, received an M. S. with Distinction in Operations Research from the U. S. Naval Postgraduate School, where he received the Chief of Naval Operations Award for Excellence in Operations Research. He retired from active duty as a Captain, USN, in 1993. His service included tours as Commanding Officer of USS Dewey (DDG 45), and as Director, Naval Center for Cost Analysis. At TASC and then Northrop Grumman, he worked extensively in cost, CAIV, and risk for the Missile Defence Agency (MDA), Navy ARO, the intelligence community, NAVAIR, and the DD(X) Design Agent team. He was the Director of Independent Cost Estimation for Northrop Grumman Information Systems and conducted Independent Cost Evaluations on over 150 Northrop Grumman programs in ships, IT, electronics, mission and space systems. He supported numerous ship programs including the DDG 1000 class, DDG 51 class, Deepwater, NSC 1 class, LHD 8 and LHA 6, LPD 17 class, Virginia class SSNs, CVN 77, and CVN 78. At Technomics, Inc., he was a Senior Technical Fellow and supported the Naval Center for Cost Analysis, the Department of Homeland Security, the Naval Postgraduate School, and the Defence Acquisition University in cost estimation and risk analysis. He retired from full-time employment in 2011. He has over 80 professional papers to his credit, including five ISPA/SCEA and SCEA Best Paper Awards and two ADoDCAS Outstanding Contributed Papers. He was a senior reviewer for CostPROF and CEBoK and lead author of the Risk Module. He has served as Regional and National Vice President of SCEA and is currently a Board Member. He received the SCEA Lifetime Achievement Award in 2008.
Peter J. Braxton
Technomics, Inc.
Peter J. Braxton holds an AB in Mathematics from Princeton University and an M.S. in Applied Science (Operations Research) from the College of William and Mary. A Senior Cost Analyst at Technomics, Inc., he is a Certified Cost Estimator/Analyst (CCEA) and currently serves as Director, Body of Knowledge, for the Society of Cost Estimating and Analysis (SCEA). He was named SCEA’s 2007 Estimator of the Year for contributions in Education and received both a TASC President’s Award for Operational Excellence and a Northrop Grumman Corporate Contracts and Pricing Award in 2008. He served as SCEA’s Training Chair from 2004 to 2009 and as a Northrop Grumman Technical Fellow from 2006 to 2009.
He currently supports the Naval Center for Cost Analysis (NCCA) and Air Force Cost Analysis Agency (AFCAA) on data collection and cost research efforts, and the Naval Postgraduate School (NPS) and Defense Acquisition University (DAU) on curriculum development. Throughout more than a dozen years in the community, he has worked to advance the state of knowledge of cost estimating and risk analysis, Cost As an Independent Variable (CAIV), and Target Costing on behalf of the Navy Acquisition Reform Office (ARO), the DD(X) development program, and other ship and intelligence community programs. He has conducted independent cost evaluations (ICEs) on all of Northrop Grumman’s major shipbuilding programs, and on several IT systems and services programs. He served as managing editor for development and maintenance of the acclaimed Cost Estimating Body of Knowledge (CEBoK) and its predecessor, Cost Programmed Review Of Fundamentals (CostPROF), and as the Training Track chair for the last seven SCEA international conferences. He has taught extensively at government, corporate, and society training events throughout the United States, Europe, and Australia. He is lead author or co-author of over a dozen professional papers on cost, risk, and CAIV, including two SCEA Best Paper winners.
Richard C. Lee
Technomics, Inc.
Richard C. Lee received his B.S. in Mechanical Engineering in 2007 from Virginia Polytechnic Institute and State University. He joined the Advanced Vehicle Dynamics Laboratory at the same institution and started his graduate studies on modeling and simulation of terrain profile models using numerical methods and vehicle dynamics. He received his M.S. in Mechanical Engineering in the spring of 2009. He is a member of the Society of Cost Estimating and Analysis (SCEA), the American Society of Mechanical Engineers (ASME), and the Institute For Operations Research and the Management Sciences (INFORMS). A Cost Analyst at Technomics, Inc., he has supported the Office of the Secretary of Defense, Cost Assessment and Program Evaluation (OSD CAPE), the Office of the Deputy Assistant Secretary of the Army for Cost and Economics (ODASA-CE), and the Naval Center for Cost Analysis (NCCA on projects spanning Earned Value Management (EVM) analysis, data collection and cost research, and risk analysis.
Brian Flynn
Naval Center for Cost Analysis
Dr. Brian Flynn serves as special assistant and advisor to the Deputy Assistant Secretary of the Navy(Cost and Economics). He was appointed to his present position as a Highly Qualified Expert by the Secretary of the Navy in 2007.
Dr. Flynn was awarded the DON Superior Civilian Service Medal in 2005; the DON Meritorious Civilian Service Medal in 2001; and a meritorious promotion while on active duty in the United States Marine Corps in 1974.
Dr. Flynn is Study Director of a NATO effort to generate independent cost estimates of several weapon system acquisition programs and to examine the role played by life-cycle cost analysis in managing national defense enterprises within the Alliance.
Dr. Flynn has led special projects such as financial analyses of defense contractors; re-engineering the defense health care system; economic analyses of personnel manning of Navy ships; cost estimating of Marine Corps housing and pay systems; analyses of the defense industrial base; and portfolio analysis of the Navy’s investment programs.
Dr. Flynn earned a bachelor’s degree and a master’s degree in Economics from Virginia Tech and a Ph.D. in Economics, with Distinction, from Georgetown University. His academic areas of specialty are theoretical econometrics and mathematical economics.
Dr. Flynn has undying pride in having once worn the uniform of a United States Marine. Semper Fi.