A Next Generation Software Cost Model (SP-1)
Jairus Hihn – Principal, NASA Jet Propulsion Laboratory
Tim Menzies – Professor, Computer Science, West Virginia University
James Johnson – NASA
The cost estimation of software development activities is increasingly critical at NASA as software systems being developed in support of NASA missions are becoming larger and more complex. As an example MSL (Mars Scientific Laboratory) launched with over 2 million lines of code. Software development activities are also notorious for their cost growth, with NASA flight software averaging over 50% cost growth. Even more important is the threat of a schedule slip that could result in missing a launch date. All across the agency, estimators and analysts are increasingly being tasked to develop reliable cost estimates in support of program planning and execution. NASA analysts currently employ a wide variety of models and tools in order to produce software cost estimates. Some models are available as COTS software applications, such as SEER-SEM(R). Other models such as COCOMO and COCOMO(R) II can be readily used in a variety of platforms such as Microsoft Excel.
While extensive literature exists on software cost estimation techniques, industry “best practice” continues to rely upon standard regression-based algorithms. These industry wide models necessarily take a one size fits all approach. This results either in models with large estimation variance or the need for a large number of inputs that are frequently not known in the early stages of the software lifecycle. One of the more significant advances in cost estimation has been the development of the Joint Confidence Level (JCL) methods and models. JCL is working well for NASA at PDR but there are challenges with applying this method earlier in the lifecycle. The detailed JCL approach is also less driven by parametric models and historical datasets, becoming more of an extension of network scheduling and resource analysis making the approach challenging to use effectively early in the lifecycle.
In this paper we will summarize our findings on effort/cost model estimation and model development based on ten years of software effort estimation research using data mining and machine learning methods. We will then describe the methodology being used in the development of a NASA Software Cost Model that provides an integrated effort, schedule, risk estimate, as well as identifying the changes in the project characteristics that are most likely to improve a given projects cost-schedule performance and risk exposure.
NASA’s Phasing Estimating Relationships (SP-2)
Chad Krause – Burgess Consulting
Erik Burgess – President, Burgess Consulting Inc.
Darren Elliott – NASA and Commercial Projects Operations Manager, Tecolote Research, Inc.
Cost and schedule estimating in support of budget formulation is limited when cost phasing is not considered. As a result, NASA’a Office of Evaluation (OE) Cost Analysis Division (CAD) initiated a review of historic mission funding profiles for the purpose of corroborating current phasing profiles and optimizing future budgeting performance. Actual expenditures by year, technical parameters, and programmatic information were compiled and normalized from NASA’s extensive library of CADRe (Cost Analysis Data Requirment) documents for programs since 1990. Regression analysis on the normalized data was used to develop Weibull-based models that estimate expenditures and NASA Obligation Authority as a function of time from SRR to launch. Models for total project cost (excluding launch) and spacecraft/instrument cost only are presented, and front/back-loading is shown to be a function of total project cost, mission class, foreign participation, and other factors. Accuracy metrics derived from the historical data and the regression models are explained and incorporated in a phasing toolkit available to the cost-estimating community. Application of these models toward understanding phasing’s ramification on cost and schedule is also discussed.
NASA Instrument Cost Model (NICM) (SP-3)
Hamid Habib-Agahi – Manager, Systems Analysis & Model Development Group, NASA Jet Propulsion Laboratory
Joseph Mrozinski – Systems Engineer, NASA Jet Propulsion Laboratory
George Fox
The NASA Instrument Cost Model (NICM) includes several parametric cost estimating relationships (CERs) used to estimate NASA’s future spacecraft’s instrument development cost. This presentation will cover the challenges associated with creating cost models in an environment where data on previously built instruments is 1) sparse, 2) heterogeneous and 3) book-kept differently by the various NASA centers and support institutions. It will also cover how these challenges were met to create a suitable instrument database which then was used to develop the CERs using Cluster Analysis, Principal Component Analysis and Bootstrap Cross Validation for different types of instruments, such as optical, particles detectors and microwave instruments.
NICM is sponsored by NASA HQ, with the primary NICM team operating at the Jet Propulsion Laboratory (JPL) in Pasadena, California. The first version of NICM was released in 2005. The latest version, NICM VI, was released in January, 2014.
The NASA Project Cost Estimating Capability (SP-4)
Andy Prince – Manager, Cost Engineering Office, NASA/Marshall Space Flight Center
Brian Alford – Operations Research Analyst, Booz Allen Hamilton
Blake Boswell – Analytic Tool Developer, Booz Allen Hamilton
Matt Pitlyk – Operations Research Analyst, Booz Allen Hamilton
The NASA Air Force Cost Model (NAFCOM) has long been the standard NASA capability for estimating the cost of new spaceflight hardware systems during concept exploration and refinement. The software instantiation of NAFCOM was conceived during the early 1990’s during a time of stand-alone programs performing dedicated functions. Despite numerous improvements over the years, the NAFCOM software continued to be failure prone and suffer from performance issues. Decreasing Agency resources meant that the NASA cost community could not support the software engineering effort needed to bring NAFCOM up to an acceptable level of performance.
In addition to the software engineering problems, several other model limitations are directly related to the NAFCOM structure and software architecture. Chief among these is the difficulty in aligning the NAFCOM Work Breakdown Structure (WBS) with the NASA Standard WBS. Other issues include concerns with data security, the approach to risk analysis, insight into the functioning of the model, and clarity into the development of the Cost Estimating Relationships (CERs).
Given the issues summarized above and the improvements in Commercial off-the-Shelf (COTS) software over the last 20 years, NASA has decided to move forward with the development of a new estimating environment: the Project Cost Estimating Capability (PCEC). PCEC is an Excel based architecture that combines a user interface running VBA with WBS and CER libraries. This structure provides a high degree of flexibility and openness while reducing the resources required for software maintenance, thus allowing more effort to put into improving our models and estimating capabilities. The NASA cost community is also taking advantage of existing Information Technology (IT) systems to provide security. COTS and special purpose tools now provide capabilities such as risk analysis and cost phasing, functions previously contained in the NAFCOM software.
The paper begins with a detailed description of the capabilities and shortcomings of the NAFCOM architecture. The criteria behind the decision to develop the PCEC are outlined. Then the requirements for the PCEC are discussed, followed by a description of the PCEC architecture. Finally, the paper provides a vision for the future of NASA cost estimating capabilities.
Developing Space Vehicle Hardware Nonrecurring Cost Estimating Relationships at the NRO CAAG (SP-5)
This paper builds on our 2012 SCEA conference briefing that described the NRO CAAG approach to developing Space Vehicle (SV) hardware Cost Estimating Relationships (CERs) for Nonrecurring (NR) engineering. These CERs are developed from the NRO CAAG’s cost database of more than 2300 space hardware boxes, and can stand as alternatives to other popular parametric tools, like the nonrecurring CERs in USCM or NAFCOM. We will briefly cover our box level estimating method, CER development approach, and the types of hardware (equipment groups) being estimated. We will describe the functional forms and different scale and complexity variables selected for each equipment group. We will also highlight some of the issues encountered and lessons learned during CER development including:
1. Striking a balance between data homogeneity and data quantity in equipment groups
2. Selecting average unit cost (AUC), theoretical first unit cost (T1), or weight as a primary scale variable when developing NR SV CERs
3. Handling incidental nonrecurring costs and points with low % New Design values
4. Determining the impact of production quantity on nonrecurring cost
5. Accounting for cost of prototype units produced such as engineering units, qualification units, and other prototypes
6. Lessons Learned: merits of some statistical measures and methods used to evaluate, compare and select CER candidates
NASA JCL: Process and Lessons (SP-6)
Steve Wilson – PP&C Analyst, NASA
Mike Stelly – Cost and Schedule Analyst, NASA
‘Joint Confidence Level’ (JCL), from its inception, has proven throughout NASA to be much more than a rote framework of mathematical nuances, but, rather, a mechanism for capturing intra-program/project complexity, program control process synergy, and other disparate effects — all faces of a nascent analytical abstraction whose implications touch almost all salient issues that comprise the picture of program health and trajectory.
Our paper will describe JCL implementation and address the creation, implementation, evolution, inherent benefits, inherent issues, its ultimate place among program management’s decision-making toolset, and hard recommendations for organizations hoping to wage successful JCL campaigns. Real-world examples will be referenced, including those from the Constellation, Commercial Crew, and Orion spacecraft development programs.
Issues discussed will include, but may not be limited to:
~The benefits of joint confidence level as a cost-schedule-risk ‘stovepipe merging’ agent within organizations
~The role of risk in JCL and program management and the challenges quantification of risk pose on future analysis
~Cost estimating approaches (parametric cost estimating, build-up estimating, etc) and their varied appropriateness for inclusion in a JCL model
~Schedule Do’s and Don’ts, integration issues and solutions, and an overview of schedule health and confidence level metrics
~The role of uncertainty and its implication on the overlap among cost, schedule, and risk
~Hard recommendations for the future implementation of JCL: consideration of performance, annual risk results, and other process-specific lessons for creating a defensible analysis