Moving Beyond Technical Parameters in our CERs (PA-1)
Eric Druker – Senior Associate, Booz Allen Hamilton
Charles Hunt – Galorath Incorporated
One of the frequent criticisms of JCL analysis (integrated cost and schedule risk analysis) has been that the results typically exhibit coefficients of variation (CV) that are orders of magnitude less than those seen in parametric estimates of similar scope. During a recent NASA research task examining how parametrics estimates can be linked to program management artifacts, the research team stumbled upon a characteristic of our Cost Estimating Relationships (CERs) that almost certainly leads our parametric estimates to have higher than necessary CVs. In particular, today’s CERs, with their focus on technical parameters tend to ignore programmatic attributes likely to drive cost. This presentation will focus on how this feature of CERs, and the fact that they likely use samples from multiple populations representing programmatic attributes, likely drives higher than necessary CVs in our parametric estimates. The presentation will review previous research and current best practices on including programmatic attributes, investigate the challenges of incorporating programmatic attributes, and then propose possible solution spaces to the parametric estimating community to increase focus on modeling key programmatic attributes. Including programmatic attributes not only has the opportunity to reduce CVs, but also could make our estimates more valuable to the program management community by giving them the ability to see how their decisions impact cost.
Using Dummy Variables in CER Development (PA-2)
Shu-Ping Hu – Chief Statistician, Tecolote Research, Inc.
Alfred Smith – General Manager, Tecolote Research, Inc.
Dummy variables are commonly used in developing cost estimating relationships (CER). It has become more popular in recent years to stratify data into distinct categories by using dummy variables. However, many analysts specify dummy variables in their CERs without properly analyzing the statistical validity of using them. For example, the dummy variable t-test should be applied to determine the relevance of using dummy variables, but this test is often neglected. Consequently, the fit statistics can be misleading.
The dummy variable t-test is useful for determining whether the slope (or exponent) coefficients in different categories are significantly different. This is directly applicable to the dummy variable CER where we assume distinct categories in the data set share the same sensitivity for the ordinary independent variable; the only difference is in the response levels.
This paper explains the reasons for using dummy variables in regression analysis and how to use them effectively when deriving CERs. Specific guidelines are proposed to help analysts determine if the application of dummy variables is appropriate for their data set. This paper also demonstrates some common errors in applying dummy variables to real examples. An application using dummy variables in splines (to derive the fitted equation as well as the intersection) is also discussed.
Bayesian Parametrics: Developing a CER with Limited Data and Even Without Data (PA-3)
Christian Smart – Director of Cost Estimating and Analysis, Missile Defense Agency
When I was in college, my mathematics and economics professors were adamant in telling me that I needed at least two data points to define a trend. You may have been taught this same dogma. It turns out this is wrong. You can define a trend with one data point, and even without any data at all. A cost estimating relationship (CER), which is a mathematical equation that relates cost to one or more technical inputs, is a specific application of trend analysis. The purpose of this paper is to discuss methods for applying parametrics to small data sets, including the case of one data point and the case of no data.
The only catch is that you need some prior information on one or more of the CER’s parameters. For example, consider a power CER with one explanatory variable: Y=aX^b . The slope of the equation, b, can be interpreted as an economies of scale factor. As such, it is typically between 0 and 1. When using weight as the explanatory variable, rules of thumb are 0.5 for development cost, and 0.7 for production cost. Bayes’ Theorem can be applied to combine the prior information with the sample data to produce CERs in the presence of limited historical data.
This paper discusses Bayes’ Theorem, and applies it to linear and nonlinear CERs, including ordinary least squares and log-transformed ordinary least squares.
Tactical Vehicle Cons & Reps Cost Estimating Relationship (CER) Tool (PA-4)
Cassandra Capots – Cost Analyst, Technomics, Inc.
Jeffery Cherwonik – Cost Analyst, Technomics, Inc.
Adam James – Cost Analyst, Technomics, Inc.
Leonard Ogborn – Cost Analyst, Technomics, Inc.
When estimating Operating and Support (O&S), it is reasonable to assume that as reliability increases, consumable and reparable parts (“cons and reps”) cost should decrease (less frequent repairs), while as vehicle price increases, parts cost should increase (more expensive parts). Developing a dataset to support cost estimating relationships (CERs) for the Army’s Tactical Vehicle fleet is a significant challenge. Therefore, rather than supplying a single CER for all tactical vehicle parts cost estimating, this study sought an Excel-based tool that would allow cost analysts to select data relevant to their specific vehicle and build tailored CERs. While the foregoing assumptions are certainly logical hypotheses, the topic poses several challenges for cost analysts. This paper will discuss these challenges and detail a three-step process for quantifying the relationship between tactical vehicle reliability and costs of cons and reps.
A lack of consistent data sources and definitions for the leveraged data types posed a challenge in the data definition phase. These data types are vehicle reliability, average unit price (AUP), and average annual parts cost. Quantifying reliability was complicated, as various organizations use different metrics with varying definitions, making meaningful comparison difficult. Additionally, obtaining a consistent vehicle AUP posed an issue, as it was initially difficult to find data from the same source and life cycle phase. Lastly, selecting a consistent source for parts costs was a challenge, as sources collect this data in varying ways, leading to certain distinctions.
Additional challenges were experienced in the data collection phase. The Army Material Systems Analysis Activity (AMSAA) Sample Data Collection (SDC) was targeted for reliability metrics; specifically, the study focused on mean miles between non-mission capable visits (MMBNMC Visits). Tactical vehicle production price and corresponding quantities were pulled from the Wheeled and Tracked Vehicle (WTV) Automated Cost Database (ACDB) and used to calculate vehicle AUP, while the Operating and Support Management Information System (OSMIS) was the source of parts costs. Upon investigation, it was seen that these three sources contained varying amounts of data, making it necessary to determine a subset of vehicles with the critical amount of information to support CER development.
Additional challenges were met during data analysis. As the data and ensuing relationships were analyzed, it was noted that the data experienced an inherently large amount of variability, even when analyzing within-series relationships. Therefore, as opposed to developing a single CER to be used for all tactical vehicles, an Excel-based tool was developed to allow for optimal flexibility in the creation of CERs. In addition to outputting uniquely-developed CERs, the tool provides appropriate statistics to diagnose and assess the level of fit for the selected CERs.
Due to the ability to easily change any selections?and, therefore, the resulting equations and statistics?users may quickly analyze various relationships and perform a variety of in-depth analyses. The result of this study is a robust tool allowing cost analysts to effectively quantify the relationship between a tactical vehicle’s reliability and parts cost.
Unmanned Aerial Vehicle Systems Database and Parametric Model Research (PA-5)
Bruce Parker – Naval Center for Cost Analysis
Rachel Cosgray – Cost Analyst, Technomics, Inc.
Anna Irvine – Technomics, Inc.
Brian Welsh – Technomics, Inc.
Patrick Staley – Naval Center for Cost Analysis
Praful Patel – Operation Research Analyst, Naval Center for Cost Analysis
This handbook documents the first two years of research sponsored by NCCA and ODASA-CE. With the inclusion of UAS in the United States’ (U.S.) military arsenal, the government has a desire to understand the components of a UAS including the air vehicle, GCS and payloads, the development and production process, and the O&S implications of these systems. The goal of this research was to support early stage cost estimating for UAS programs where there are limited data and immature designs. Equations include data from Army, Navy, and Air Force programs, and reflect as broad a range of UAV types, with varied propulsion, mission, size, and shape, as was available for this study. The CERs are intended to support Analysis of Alternatives (AoA), Independent Cost Analysis (ICA), and similar analyses
Building a Complex Hardware Cost Model for Antennas (PA-6)
David Bloom – Senior Engineering Manager, Raytheon Space and Airborne Systems
Danny Polidi – Raytheon
This paper discusses the development of a Complex Antenna Cost Model based on quantifiable sizing mechanisms which are designed to quickly and accurately calculate the “top-down” cost for all engineering and operations disciplines and functions required for antenna development and test.
Previous methods of antenna cost estimation were not based key sizing metrics (KSMs). So, although cost estimates were based on historical cost, scaling factors used to determine cost for new programs were frequently Engineering estimates. Often, previous methods used a bottoms-up approach where each discipline independently bids their contribution to the Program. With that method, each contribution, independently determined, would need to be added together for total antenna development and test cost with a high likelihood of overlap (double dipping cost) or omissions (missing costs). Previous methods would refer back to a parametric Cost Model to provide rationale of new costs, rather than a “similar-to” program. Previous methods of cost estimation would require an independent test of reasonableness. Previous methods of cost estimation would not provide any indication of cost drivers, or sensitivity factors.
The new cost estimation tool uses historical data, and through analytical comparison of requirements/specifications, quantitative effective size factors (or key size metrics) were determined. The KSMs are used as scaling factors along with actual cost for a specific historical program to calculate costs for new programs. The new cost estimation tool uses a top-down approach where all costs for a prior program are considered and inherently all disciplines are then included in the new estimate. This guarantees that there is no overlap or omission of cost. The new cost estimation tool displays graphically all loaded data to allow the user to select the most “similar-to” Program. The new Program cost can be related through KSMs to any of the loaded data. Because all loaded data is graphed along with the new Program cost, the tool provides a test of reasonableness. Each KSM in the new cost estimation tool provides some amount of impact to the total cost. That amount of impact, or sensitivity, is displayed in the tool so that the user has the opportunity to make technology trade-offs to provide the customer with the cost options available.
What makes this cost estimating tool significant is that in any Radar development, the antenna is often the most expensive piece of hardware and it is also the least well characterized in terms of development costs. Many in the Radar industry have described antenna development in terms of a “secret sauce”. This tool removes the “secret sauce” recipe to antenna development and allows the user and the customer the ability to make meaningful cost benefit trade-offs.
ESA Project Office Cost Model (PA-7)
Herve Joumier – Head of ESA Cost Engineering, European Space Agency
Abstract: The Project Office price is traditionally one of the most difficult negotiation areas between space agencies and industrial contractors and covers a significant part of the project cost. It is therefore a domain that requires all the attention of the estimators to better support the negotiations.
Space projects costs are mainly made of manpower costs aggregated through contractual layers ranging from simple structures such as single layer for small low cost project up to highly entwined multiple layers for very large projects involving many international partners. The Project Office traditionally covers Management, Product Assurance and Engineering. In our case, the model is expanded to cover Assembly Integration and Testing activities.
This paper describes the definition and the implementation of a Project Office parametric cost model aimed at defining a reference manpower allocation based on fair judgement and rational fair modelling. It has been developed to improve the cost estimations capability of ESA providing outputs that are used by agencies for comparison with contractors’ proposals. In particular the model focuses on Project Office cost for all the main industrial actors involved in the design and development of a satellite taking into account various possible scenarios depending on quality standards applied to the project and the sub-contracting approach retained. A wide variety of sub-contracting approaches have been observed, not simply driven by technical considerations but usually resulting from political demands to include some participating countries at an appropriate level of participation in line with their level of financial contribution.
This paper describe the steps and approaches of the model development with the intention to be a inspiring source for any organization willing to develop such competencies.
Improving the Accuracy of Cost Estimating Relationship (CER) for Software Systems (PA-8)
David Wang – Director of Integrated Program Management, The Aerospace Corporation
Software plays an important role in many aspects of a modern space system, (e.g. real-time software control for onboard subsystems, networked connectivity, multi-source data fusion, pre-processing and post-processing of information, telemetry, tracking & control (TT&C) software, mission management, mission data, integrated mission applications and services, ground control, etc.). In order to develop useful and predictive cost estimating relationship (CER) for space systems, it is necessary to develop a predictive CER for software systems.
CER is a parametric cost estimating methodology often used by cost analysts to quantify the development costs of a software system. CER expressed Cost as a function of one or more independent variables (i.e. cost drivers). In space system software and ground software, the key cost driver is the size of the software (measured in the number of lines of code). The difference between actual and predicted cost represents the estimation error in the parametric cost model. Sophisticated mathematical models for Predictive Interval (PI) analysis have been proposed to analyze and bound the predictive error. The PI equation can then be used to generate an S-curve to predict the cumulative probability of the cost of a system. Numerous studies using actual cost performance data have shown that CER predictions using the traditional technique are much more optimistic than actual cost performance.
In this paper, we leverage recently published results on the statistical characterization of schedule and cost risks to analyze the prediction accuracy of CER for software systems. Our analytical analysis and empirical statistical analysis of actual code size growth data suggest that the statistics of code size estimate can also be characterized by fat-tail distributions. This suggests that predictive error of CER for large software development programs may be significantly larger than predicted by conventional PI analyses. We show in this paper a practical method for improving the accuracy of the prediction interval estimate, and thereby improving the prediction accuracy of the resulting S-curve.
Hybrid Parametric Estimation for Greater Accuracy (PA-9)
William Roetzheim – CEO, Level 4 Ventures, Inc.
When discussing early stage estimation, estimation by analogy and parametric estimation are often compared and contrasted. But a new hybrid parametric approach that combines these two approaches typically yields significantly greater accuracy. With hybrid parametric estimation, a high-level-object, or HLO, catalog is created based on historic data to represent estimation components at different levels of granularity. At the most abstract level, this catalog may represent an entire project, in which case the HLO catalog results will match traditional estimation by analogy results. However, the HLO catalog will also support a much more granular representation of the items to be delivered, supporting representations all the way down to extremely fine representations such as a line of code (SLOC models) or something like an External Output (EO) or Internal Logical File (ILF) in a function point based environment. The real power of an HLO catalog based approach is in between these two extremes, where we have better granularity and accuracy than a project, but we require less specificity than that required by function points or SLOC based models.
Parametric estimation typically applies a cost estimating relationship (CER) to model a parameter that is often only incidentally related to the item being delivered (e.g., satellite weight) to cost. In this example, the goal is normally not simply lifting a certain amount of weight into orbit, but rather, accomplishing some specific mission. The fact that weight and cost are sufficiently related to allow prediction may be regarded as a fortunate coincidence. With hybrid parametric estimation we apply the statistical analysis and modeling techniques used for parametric estimation, but we look specifically for functional outcomes as our independent variables. These hybrid parametric CERs are, in fact, derived from our HLO catalog.
This talk will discuss hybrid parametric estimation based on HLO catalogs, and give examples of the application and accuracy of this technique within organizations including the State of California, Halliburton, IBM, Procter and Gamble, and multiple top 25 financial institutions.
Linking Parametric Estimates to Program Management Artifacts (LPEPM) (PA-10)
Mike Smith – Booz Allen Hamilton
Ted Mills – Operations Research Analyst, NASA
John Swaren – Solutions Architect, PRICE Systems
A common fate of parametric cost and schedule estimates is that they fall into disuse as a Project’s own artifacts (e.g. Work Breakdown Structure (WBS), budget, schedule, risk lists, etc.) are created and mature. Parametric estimates typically do not map cleanly to WBS or schedule-derived artifacts, allowing a sense among Project Managers (PMs) ? rightly or wrongly ? that “parametric estimates are fine, but they don’t reflect my project.” As a result of this bias, parametric estimates and the estimators that generate them find themselves relegated to obscurity after passing the first project milestone. The problem lies in that dismissing parametric estimates on these grounds, PMs lose the benefit of the historic realities captured in Cost Estimating Relationships (CERs) that drive the models. Conversely, cost estimators have observed that the recent Joint Confidence Level (JCL) analyses required by NASA policy to occur at PDR/KDP-B, have yielded suspiciously narrow Coefficients of Variation in JCL cost S-curves. This gives rise to concerns within the cost community that projects, overly reliant on their own SMEs to provide uncertainty ranges, are missing opportunities to incorporate significant uncertainties into their estimates.
NASA’s Cost Analysis Division (CAD), Booz Allen Hamilton and PRICE Systems collaborated to conduct research into linking parametric estimates to programmatic artifacts in a manner that would elevate parametric estimates and allow Programs and Projects to apply the historical lessons that make parametric estimates so powerful and accurate. This research brought together parametric and programmatic cost estimators, model developers, software developers, schedulers, risk analysts and practitioners ranging from junior analysts to Ph.D thought-leaders to think through and articulate a process by which parametric cost estimates could be linked to programmatic artifacts in a manner that takes maximum advantage of the best each has to offer. Specifically, the collaborative research evaluated the feasibility of a parametric cost model “informing” a JCL model and vice-versa via iterative methodology. This research resulted in a practical, clearly-articulated process for performing this cross-informing linkage, as well as the development of standardized enabling tools (data collection templates and a dashboard tool) through which to visualize and perform iterative comparative analyses. The research used as a test-case a contemporary, real-world NASA project which needed only to meet two conditions: that a recent parametric estimate have been performed; and it had been through a JCL analysis. This ensured that a requisite set of comparable programmatic and parametric products existed. With those paired data sets, the LPEPM research team deconstructed the models and developed a process for linking parametrics to programmatic artifacts and proved that the concept can be executed and has merit. The team encountered challenges resulting in lessons-learned designed to benefit any analyst in the field attempting such a linkage.
Impact of Full Funding on Cost Improvement Rate: A Parametric Assessment (PA-11)
Brianne Wong – Consultant, Booz Allen Hamilton
Erik Burgess – President, Burgess Consulting Inc.
The NRO Cost and Acquisition Assessment Group (CAAG) currently houses data collected from various U.S. Government organizations, including the Department of Defense and NASA. These data points are pooled with NRO data and used in Cost Estimating Relationships for space hardware, which underpin CAAG estimates for major system acquisition programs, aiding in the development of programs and budgets. Various funding rules have been in effect over the years for the different procurement agencies, and these rules may have an impact on cost. This study addresses the DoD policy of Full Funding, in particular, and its impact on recurring cost improvement for multi-unit buys. The NRO, which is not subject to full-funding rules, has historically found much steeper cost-improvement rates (averaging 85% cumulative-average) than the DoD has claimed on their programs. In this study we assess the recurring costs of almost 1,700 unit-level data points dating back to the 1970s and conclude that while funding rules certainly can impact cost in specific cases, the Full Funding rule doesn’t result in a statistically significant difference in cost-improvement rate across the data set.
Developing R&D and Mass Production Cost Estimating Methodologies for Korean Maneuver Weapon System (PA-12)
Doo Hyun Lee – Korean Defense Acquisition Program Administration
Sung-Jin Kang – Professor Emeritus, Korea National Defense University
Suhwan Kim – Assistant Professor, Korea National Defense University
Today cost estimates for the government acquisition programs are important in supporting decisions about funding, as well as evaluating resource requirements at key decision points. Parametric cost estimating models have been extensively used to obtain valid cost estimates in the early acquisition phase. However, these models have many restrictions to obtain valid cost estimates in the Korean defense environment because they are developed to be used in the U. S. environment. In order to obtain reliable and valid R&D cost estimate, it has been important for us to develop our own Cost Estimation Relationships (CER), using historical R&D data. Nevertheless, there has been little research on the development of such model.
In this research, therefore, we have attempted to establish a CER development process to meet the current need, and found certain cost drivers for the Korean historical maneuver weapons system data, using Forward selection, Stepwise Regression and R square selection. We have also developed a CER model for production labor costs, using Learning rate which has been generally applied to estimate valid production labor costs. Learning effects are obtained from repetitive work during the production period under three assumptions; homogeneous production, same producer, and quantity measure in continuous unit.
While developing our own CER, we have used Principle Component Regression (PCR) method to avoid multi-collinearity and restriction of insufficient numbers of samples. As a result, we are able to overcome the multi-collinearity and develop a reliable CER. But many important results in statistical analysis follow the assumption that the population, being sampled or investigated, is normally distributed with a common variance and additive error structure. So, in this research, we have used the parametric power transformation proposed by Box & Cox (Box-Cox transformation) in order to reduce anomalies such as non-additivity, non-normality, and hetero-scedasticity.
This study is the first attempt to develop a CER for the Korean historical maneuver weapons system data for the Korean defense industry environment. This is significant because it will be an important methodology applied to the CER development for the future Korean weapons system.