Analytical Methods Track
Using TRLs to Predict the Future of Nuclear Weapons
Analytical Methods Track (AM01)
M. Michael Metcalf
Technology Readiness Levels (TRL) are used to measure and assess technology maturity. This paper presents innovative research demonstrating how historical TRL data can be used to perform credible, data-driven schedule analysis for programs early in development. Originally designed to help schedule analysis for the nuclear weapons stockpile, the resultant methodology combines historical milestone data and statistical methods to generate a Monte Carlo simulation of a risk-adjusted schedule for complex programs.
Keywords: monte carlo, TRL, schedule risk, MRL
AM01 – Schendt – Paper – Using TRLs to Predict the Future
AM01 – Schendt – Using TRLs to Predict the Future
Software Licenses: A Bill You Can’t Pay?
Analytical Methods Track (AM02)
Cheryl L. Jones
For the past 20 years the DoD has emphasized the purchase and use of commercial software licenses as an approach to cut overall software development costs. Has this approach achieved its desired effect? How can one tell? This presentation will show the challenges and results of analyzing over 4,500 license records collected for the Army Software Maintenance Initiative. Implications to DoD’s future maintenance costs as well as implementing its DevSecOps objectives will be discussed.
Keywords: Cost Management, Decision Analysis, Parametrics, Performance Management
AM02 – JonesCheryl – Paper – Software Licenses
AM02 – JonesCheryl – Software Licenses
8D Cost Trades With Entanglement
Analytical Methods Track (AM04)
Many modern goods have elements that have substantial costs or create hazards if unavailable. If a component begins to add more cost than value to a final product or becomes hard to get, the project’s viability may be at risk. Here, we study the connections between a subcomponent and an ultimate product across interrelated market pairs. We look at their seven primary dimensions to visualize market mechanisms and add time as an eighth dimension.
Keywords: 4D, 7D, 8D, Hypernomics, Multidimensional Economics
AM04 – Howarth – Paper – 8D Cost Trades With Entanglement
AM04 – Howarth – 8D Cost Trades With Entanglement
Sustainment Analysis Methodology for Cost Models and Business Cases
Analytical Methods Track (AM05)
Government infrastructure systems often require detailed sustainability analyses – historical parts failures analysis, parts’ procurement, end-of-life, and economic analyses – to (1) forecast the optimum date for an acquisition to replace aging infrastructure with a New Investment, (2) conduct a cost-benefit analysis to justify further F&E investment, and (3) provide a standard sustainment cost estimate (corrective and preventative maintenance) for acquisitions. Using parametrics, research, and software algorithms, the team demonstrates an optimum approach for sustainment analysis.
Keywords: Cost/Benefit Analysis, Data-Driven, Decision Analysis, Government, Life Cycle, Methods, Parametrics, Regression, Risk, Statistics, Business Case, Python, Optimization, Process, Critical Thinking
AM05 – Bayer – Sustainment Analysis Methodology
AM05 – Bayer – Paper – Sustainment Analysis Methodology
Implementing Quantile Regression for Capital Acquisition Project Estimating
Analytical Methods Track (AM06)
The National Nuclear Security Administration (NNSA) is investigating the use of quantile regression on NNSA capital acquisition data to determine the cost drivers of project costs at values above the median, instead of via use of Log-Ordinary Least Squares. Utilizing this methodology will provide a better understanding and quantification of the uncertainty inherent in the NNSA’s capital project budgets, which are required to be budgeted at between the 70% and 90% confidence level.
Keywords: Budgeting, Parametrics, Regression, Uncertainty
AM06 – Matheson – Implementing Quantile Regression
The Progression of Regressions
Analytical Methods Track (AM07)
Do you lose sleep at night over not being able to find the best regression for your data? Well fear no more, for this brief will alleviate those concerns. We review a case study creating a CER with a limited data set exploring multiple forms of regression (Linear, Nonlinear, Cubic, Polynomial), segmenting data into applicable portions while comparing predicted vs. actual results, and using predictive statistics and guidelines to assess best fit.
AM07 – Aguirre – The Progression of Regressions
Predicting and Minimizing Industrial Base Risks to Emerging Technologies
Analytical Methods Track (AM08)
The global pandemic has highlighted the fragility of the U.S. industrial base relied on to develop the next emergent technologies. Programs are routinely impacted by industrial base issues causing schedule delays and cost growth without the analytical means to predict and prepare for them. An analytical framework was developed that leverages cost estimating methodologies to predict these industrial base issues, quantify their impact, establish a new risk adjusted baseline, and reduce risk through targeted investments.
Keywords: Cost/Benefit Analysis, Risk, Uncertainty
AM08 – Ray – Predicting and Minimizing Industrial
Data Science Track
Developing Conceptual Cost Estimates for Good Ideas
Data Science Track (DS01)
This paper outlines the use of a super learner method for generating conceptual estimates of Australian Department of Defence facilities and infrastructure products. Many organisations require quick turnaround cost estimates for good ideas looking for budget funding. Developing estimates in the absence of firm requirements and a full understanding of proposal scope is challenging. A transformative approach to cost and duration estimation is needed to solve this problem.
Keywords: Modeling, Super learner, conceptual estimates
DS01 – Mitchell – Paper – Developing Conceptual Cost
DS01 – Mitchell – Developing Conceptual Cost
Supercharging Machine Learning with Cloud Computing
Data Science Track (DS02)
Modern cloud computing environments can speed up machine learning models by up to 100X! In this case study with the DoD Joint Artificial Intelligence Center (JAIC), see how AWS and Databricks can be utilized to improve existing machine learning (ML) models for detecting fraud correcting unmatched transactions. When combined with robotic process automation (RPA), this effort saved 100,000 manual hours in FY20, and identified potential fraud with over 90% accuracy.
Keywords: Data-Driven, Statistics, Machine Learning, Cloud Computing, Robotic Process Automation
DS02 – Lawston – Supercharging Machine Learning with Cloud Computing
Statistical Techniques to Improve Software Effort Estimation Data Quality for Cost Engineers
Data Science Track (DS03)
Since the topic of improving data quality has not been addressed for the U.S. defense cost estimating discipline beyond changes in public policy, the goal of the study was to close this gap and provide empirical evidence that supports expanding options to improve software cost estimation data matrices for U.S. defense cost estimators. The purpose of this quantitative study was to test and measure the level of predictive accuracy of missing data theory techniques that were referenced as traditional approaches in the literature, compare each theories’ results to a “ground truth” data set.
Keywords: Engineering Economics, Statistical Imputation, Cost Engineering, Missing Data Theory, Data Cleansing
DS03 – Williams – Paper – Statistical Techniques
DS03 – Williams – Statistical Techniques
Integrating Data Science & Cost Analysis
Data Science Track (DS04)
Dakotah W. Hogan
Data is more abundant than ever before. How can we as cost analysts leverage the full power of the data science world to get the most out of our data and utilize things like automation, machine learning, interactive dashboards and more? The author will demonstrate the innovative ways that AFCAA is utilizing a cloud based platform and various data science tools to transform our entire process from data ingestion/ storage to end products and even end to end cost modeling.
DS04 – Green – Integrating Data Science and Cost Analysis
Managing Data Science: A Stacked Approach to Integrating Advanced Data Analytics
Data Science Track (DS05)
John W. Maddrey
Eric J. Hagee
In this presentation, we will define the levels of the data science stack, what success in the stack means, and how to organize the work force with in the stack. Furthermore, we will discuss global processes for collecting, managing, and analyzing data including desired skill sets and required technologies.
Keywords: Communication, Cost Management, Data Collection, Data-Driven, Functional Requirements, Government, Methods, Modeling, Operations, Performance Management, Program Management, Risk, Scheduling, Statistics, Data Science, Stack, Data Analytics
DS05 – Hagee – Managing Data Science
Cost Estimating Maturity and a Vision for the Future
Data Science Track (DS06)
The science of cost estimating has matured. From the Egyptians to modern methods, cost analyses advances were progressive, recent methods further the science. This paper reflects on rich heritage and looks to the future of cost estimating and analysis using emerging and maturing techniques. We look at postmodern tools and methods like Natural Language Processing and other methods enhance our ability accurately forecast costs of more complex systems. Last, we discuss future research and trends.
Keywords: Bayesian, Data-Driven, Decision Analysis, Methods, Program Management, Risk, Space, Advanced Tools
DS06 – Malone – Cost Estimating Maturity
DS06 – Malone – Paper – Cost Estimating Maturity
Machine Learning/NLP Track
Utilizing Artificial Intelligence to Evaluate and Estimate Requirements
Machine Learning/NLP Track (ML01)
Business Analysts face the challenge of parsing through hundreds/thousands of requirements to define system needs or to accurately estimate the software development cost. This is a grueling process and mistakes are still an inevitability. However, AI can serve analysts in automating requirements analysis, producing high-level concepts, and informing total lifecycle estimates in a short period of time. This presentation provides an overview of select AI capabilities and shows their applications to requirements analysis and cost estimation.
Keywords: Budgeting, Cost/Benefit Analysis, Functional Requirements, Process Engineering, Software, Artificial Intelligence (AI), Systems Analysis, Requirements Analysis, Software applications, Information systems, Software Sizing, Function Point, Function Point Analysis, Automation, Natural Language Processing (NLP)
ML01 – Zabrah – Utilizing AI to Evaluate Requirements
Estimation of Application Maintenance by Means of Machine Learning
Machine Learning/NLP Track (ML02)
Eric van der Vliet
This presentation is about an estimation model for application maintenance. The model calculates the required effort and costs for maintaining software application based on incidents with different service levels and priorities. The calculation is based on historical data collected of different business domains. To manage the large amount of data, application characteristics and domains, advanced data analytics and machine learning techniques are required. The development approach as well as the prototype results will be shown in this presentation.
Keywords: Data-Driven, Machine Learning, Data Analytics, Managed Services
ML02 – van der Vliet – Estimation of Application Maintenance
Linear Regression: How to Make What’s Old New Again
Machine Learning/NLP Track (ML03)
With the booming popularity of machine learning techniques, data scientists may have you believe there is no place for linear regression. Though many cost estimating applications are nonlinear, when there are linear relationships among features, ordinary least squares regression often performs better than the most powerful machine learning techniques. Scenarios for which linear regression should be chosen over more complicated algorithms are presented, as well as techniques such as regularization, gradient descent, and Bayesian methods.
ML03 – Roye – Paper – Linear Regression
ML03 – Roye – Linear Regression
Adopting a Data Science Paradigm
Machine Learning/NLP Track (ML04)
The availability of large unstructured datasets accelerated via autonomous data collection and warehousing, makes the use of advanced analytical skillsets a necessity. The incorporation of data science methodologies can yield meaningful improvements to cost estimating practices through effective data governance, streamlined data collection/normalization, and increased opportunities for exploratory analysis. This presentation will evaluate the software and skillsets required for advanced modeling and analysis, as well as planning considerations for data science training development.
Keywords: Communication, Data Collection, Data-Driven, Decision Analysis, IT, Methods, Modeling, Software, Data Science, Data Analysis, Programming
ML04 – Ferris – Adopting a Data Science Paradigm
Advanced Natural Language Processing for Work Breakdown Structures
Machine Learning/NLP Track (ML05)
The National Nuclear Security Administration (NNSA) collects Work Breakdown Structure (WBS) data for Capital Asset Projects. Implementing the structure across projects grows exponentially in complexity due to varying scope, contextual changes, and vendor requirements. This paper will demonstrate how Natural Language Processing can automate the process of identifying and classifying WBS elements used for Capital Asset Projects. NLP can improve the Cost Analyst’s workflow by reducing time-consuming aspects of their work.
Keywords: Data-Driven, Government, Methods
ML05 – Lapoff – Advanced Natural Language
ML05 – Lapoff – Paper – Advanced Natural Language
Strategic Investment Planning using Machine Learning Techniques
Machine Learning/NLP Track (ML06)
Wendy E. Robello
Early identification of anticipated customer needs is a major component to winning new business in competitive environments. This allows bidders to make strategic investment decisions that will improve core competencies to meet anticipated customer needs. Current approaches to predict customer needs are subjective, costly, and time-consuming. This research will provide bidders an economical and objective approach using supervised machine learning to predict customer needs deduced from prioritized customer selection criteria to strengthen their proposed solution.
Keywords: Decision Analysis, strategic investment, machine learning, competitive positioning, customer needs
ML06 – Robello – Strategic Investment Planning
A Deep Look into Optimistic Bias based on a NASA’s STEM 4th Grade Activity
Machine Learning/NLP Track (ML07)
Beyond this 4th, Grade NASA STEM activity found on NASA JPL’s web site, this paper is a direct result based on independent research being conducted by Steve A. Sterk who works in the Program Planning and Control (PPC) Branch at NASA Armstrong Flight Research Center. It is a believed optimistic cognitive bias, is the number one problem while developing cost and schedule estimates and thus digging into human behavioral, a heuristic technique in completing the task at hand. This presentation will include; How people think, the start of an Optimistic Cognitive Bias data base for Machine Learning.
Keywords: Bias, Learning Curve, How People Think
ML07 – Sterk – A Deep Look into Optimistic Bias
Management & Risk Track
Delusions of Success: Overcoming Optimism Bias in Schedule Forecasting
Management & Risk Track (MR02)
Schedule performance of major defense acquisition programs remains a challenge, leading to an average capability delay of more than two years and substantial cost growth, according to the latest U.S. Government Accountability Office assessment. Stakeholders must de-risk schedule estimates and improve performance through data-driven approaches based on realized prior program histories to prevent optimism bias. The authors evaluate the merits of taking an ‘outside view’ to mitigate risk through reference class forecasting, utilizing empirical distributional information from 116 programs across six commodity classes to develop more realistic and reliable front-end schedule estimates.
Keywords: Bias, Data-Driven, Methods, Scheduling, Statistics, Uncertainty
MR02 – Voth – Delusions of Success
MR02 – Voth – Paper – Delusions of Success
Real-World Data Transformation Challenges in the DoD Supply Chain
Management & Risk Track (MR03)
Merger and acquisitions have driven tremendous growth of the top one hundred government contractors. This growth has also embedded land mines in the historical data of these “compete-mates.” The presenter will discuss real-world lessons from the extraction and transformation of programmatic, cost, schedule, and effort data in support of processes from Should Cost to Basis of Estimate for bid and proposal pursuit.
MR03 – Ahearn – Real World Data Transformation
Cost and Schedule Risk Analysis of Megaprojects
Management & Risk Track (MR04)
Cost and schedule risk analysis of megaprojects can be complex, time consuming, and may not give meaningful results. The presentation demonstrates a practical and easy to implement steps how to perform risk analysis of projects of any size. They include schedule diagnostics consolidation. The large megaproject schedule can be separated on subprojects, which can be analyzed individually. The analysis can be performed for any milestone and multiple times during the course of project.
Keywords: Cost Management, Monte Carlo, Risk, Scheduling, Uncertainty, Megaprojects
MR04 – Trumper – Cost and Schedule Risk Analysis of Megaprojects
Uncertainty of Expert Judgment in Agile Software Sizing
Management & Risk Track (MR05)
Agile software estimating and planning often rely on expert judgment to assess the size of the development effort at various levels of granularity and stages of maturity. Previous research by the author quantified the inherent risk and uncertainty of the self-similar scales (e.g., T-shirt sizing) commonly used in these assessments. This paper expands those a priori mathematical results and empirically tests the accuracy of experts in applying those scales. It elucidates the ideal ratio to align with the desired confidence interval, and recommends feedback mechanisms to improve consistency.
Keywords: Early Cost, Functional Requirements, Risk, Software, Uncertainty, Agile, Story Points
MR05 – Braxton – Paper – Uncertainty of Expert Judgment
MR05 – Braxton – Uncertainty of Expert Judgment
Alternative EAC Methodologies – Calculating EACs without Standard EVM Data
Management & Risk Track (MR06)
Developing an Estimate at Complete (EAC) typically involves establishing a performance-based estimate founded in the best practices of Earned Value Management (EVM) Gold Card analysis. But what if you are missing EVM data or do not have a performance baseline? This presentation explores alternative approaches to developing an EAC by leveraging burn rate profiles. Estimates are then developed by applying regression analysis and/or linear interpolation techniques to the historical burn rates observed.
MR06 – Lee – Alternative EAC Methodologies
Decision Support and Operation Design in Mission Critical Applications
Management & Risk Track (MR07)
This talk covers key concepts of decision making under uncertainty. Building on applications in soldier-robot teaming, we explain how to balance exploration and exploitation tradeoffs using multi-armed bandits and reinforcement learning. The second half of the talk is concerned with challenges of operation design in costly and limited-information environments. Using examples from adversarial information operations, we explain how to balance costs and benefits of data collection and how to mitigate risks to data acquisition resources.
MR07 – Rahimian – Decision Support and Operation Design
Modeling & Analysis Track
Second Source Manufacturing: Lessons from the Second World War
Modeling & Analysis Track (MA01)
Brent M. Johnstone
Manufacturing defense systems at different sites is increasingly common due to foreign coproduction and international cooperative ventures. These situations challenge estimators, posing questions about the transfer of learning and relative efficiency of multiple production sites. This paper examines cost history from World War II, when U.S. bomber production lines were shared across multiple companies. The conclusions are tested against modern experience and guidance provided to estimators seeking help.
Keywords: Data-Driven, Labor, Learning Curves, Manufacturing, Methods, Modeling
MA01 – Johnstone – Paper – Second Source Manufacturing
MA01 – Johnstone – Second Source Manufacturing
Model-based Cost Engineering launches with Space Missions Estimating
Modeling & Analysis Track (MA03)
NASA continually strives to improve cost estimation for the highly advanced technology flown on planetary as well as earth orbiting space missions. Over the years it has been proven that parametric cost models are a desired way to obtain accurate estimates. Still there is still room for improvement. This paper will discuss two of the latest and best methods for obtaining accurate cost estimates using best-of-breed model-based cost engineering techniques.
Keywords: Modeling, Parametrics, Space, NASA, Space Missions, Planetary, Earth Orbiting, electric propulsion, ion thrusters, lasers, parachutes, radar altimeters, thermal protection, case studies, validation
MA03 – Delisle – Model based Cost Engineering
MA03 – Delisle – Paper – Model based Cost Engineering
NRO CAAG Parametric Model for Spacecraft-to-Launch-Vehicle Integration
Modeling & Analysis Track (MA04)
New, innovative launch service providers and spacecraft designs have broadened the tradespace of launch service options for government customers, placing new importance on launch cost, an often understudied component of space programs’ enterprise-level cost. This paper demonstrates a parametric approach to estimating the most mission-dependent component of launch cost, spacecraft-to-launch-vehicle integration engineering. It addresses primary and secondary cost drivers and typical time phasing, and demonstrates the model’s performance compared to NRO and NASA historical costs.
Keywords: Data-Driven, Government, Methods, Modeling, Parametrics, Regression, Space, Launch, Rocket, Integration, ULA, SpaceX, Blue Origin
MA04 – Barkmeyer – NRO CAAG Parametric Model
Integrating Cost into Model-based Systems Engineering Environments
Modeling & Analysis Track (MA05)
Model-based Systems Engineering (MBSE) incorporates digital models to represent system-level physical attributes and operational behavior throughout the system life-cycle. To date, many MBSE efforts have focused on technical requirements with little emphasis on cost. Integrating cost models and MBSE enables rapid exploration of design trades and associated cost impacts, traceability between requirements and cost targets, and earlier identification of affordability issues. We present results from several ongoing efforts as a continuation of our 2019 presentation.
Keywords: Decision Analysis, Manufacturing, Modeling, MBSE, Digital Engineering
MA05 – Kennedy – Integrating Cost into Model-based Systems
NASA Instrument Cost Model (NICM 9) Introductory Training
Modeling & Analysis Track (MA06)
The NICM 9 Tool suite allows for cost and schedule estimation for space flight instruments. This introductory training session will examine a case study utilizing both the System and Subsystem tools, moving from inputs and assumptions to the cost and schedule estimates and interpretations. An overview of the NICM model development methodology will be given. Instructions to obtain follow-on training and to download NICM 9 will conclude the talk.
MA06 – Mrozinski – NASA Instrument Cost Model
Cost and Throughput Analysis for the NASA Ames Arc Jet Modernization Program
Modeling & Analysis Track (MA07)
The facilities of NASA’s Arc Jet Complex are used to simulate the aerothermodynamic heating that a spacecraft endures throughout hypersonic atmospheric entry, and to test candidate Thermal Protection System (TPS) materials and systems. Ms. Scheel and Dr. Smart analyze the cost and throughput impact of several different modernization options. Estimating the cost requires independent research into highly specialized subsystems and customizing inflation application. Throughput analysis of the number of possible test runs per year is conducted via a probabilistic simulation.
MA07 – Scheel – Paper – Cost and Throughput Analysis
MA07 – Scheel – Cost and Throughput Analysis
Is Your Organization Ready for Model-Based Cost Engineering?
Modeling & Analysis Track (MA08)
Model-based design continues to yield unprecedented speeds in technology solution development. Integrated and data-driven Model-Based Cost Engineering (“MBCE”) promises to estimate concurrently at the speed of design. Taking advantage of MBCE requires vision and a level of organizational maturity. The presenters will discuss critical attributes regarding people, processes, data and technology that impact the success of model-based costing.
Keywords: Early Cost, Functional Requirements, Methods, Process Engineering, Model-based design, MBSE, Cost Engineering
MA08 – Ahearn – Is Your Organization Ready
Mission Operations Cost Estimation Tool (MOCET) 2022 Research
Modeling & Analysis Track (MA09)
The Mission Operations Cost Estimation Tool (MOCET) is a model developed by The Aerospace Corporation in partnership with NASA’s Science Office for Mission Assessments (SOMA). MOCET provides the capability to generate cost estimates for the operational, or Phase E, portion of full NASA space science missions. Research topics being studied in 2022 will be presented, including: Level 2 Work Breakdown Structure (WBS) cost modeling, and extended mission cost modeling.
Keywords: Modeling, Tools, & Analysis Case Studies
MA09 – Hayhurst – Mission Operations Cost Estimation Tool
DICEROLLER: Estimating D&D Costs for the NNSA
Modeling & Analysis Track (MA10)
The National Nuclear Security Administration Office of Programming, Analysis, and Evaluation has developed a model for estimating the cost of Decontamination and Disposition (D&D) activities. This effort involved collecting and normalizing cost data from past D&D projects, and then generating a parametric cost estimating relationship to predict future D&D project costs. The resulting model, named DICEROLLER, will be used to make lifecycle cost estimates and one-for-one replacement cost estimates of capital acquisition projects.
Keywords: Early Cost, Government, Infrastructure, Life Cycle, Parametrics, Regression
MA10 – Matheson – DICEROLLER
MA10 – Matheson – Paper – DICEROLLER
Applying System Readiness Levels to Cost Estimates – A Case Study
Modeling & Analysis Track (MA11)
Estimating cost, schedules and expected technical performance of large complex systems pre-development is difficult. When programs are executed, they are plagued with cost growth and schedule delays due to less than required maturity of some elements. We investigate the James Webb Space Telescope program actual cost and schedule history using System Readiness Level (SRL) methods to reveal trouble areas early. The resulting approach will support future estimating accuracy through higher fidelity information for early decision-making.
Keywords: Budgeting, Cost Management, Methods, Program Management, Risk
MA11 – Malone – Applying System Readiness Levels
Processes & Best Practices Track
Cracking Open the ‘Black Box’ of Product Technical Support Contracts
Processes & Best Practices Track (PB02)
Product Technical Support contracts represent a significant annual cost to the DoD. While these contracts serve as useful “catch-alls” for various programmatic requirements, estimating these costs for future systems can be challenging. This paper analyzes historical technical support costs for various Army ground vehicle systems and demonstrates how combining contracts, work directives, and cost data sources along with categorizations and tags for “Service Categories” provides a comprehensive understanding and management of technical support cost requirements.
Keywords: Bayesian, Data Collection, Data-Driven, Government, Methods, Contract Pricing, Predictive Analytics
PB02 – Bonich – Paper – Cracking Open the Black Box
PB02 – Bonich – Cracking Open the Black Box
Visual Exploration of Data – The Missing Element in CER Development
Processes & Best Practices Track (PB03)
Within cost estimating training literature, it is common for the discussion around Cost Estimating Relationship (CER) development to focus primarily on its statistical parameters (e.g. correlation, equation form, etc.). An underemphasized component of CER development is the need to first visualize and explore your data. This presentation will show how integrating these processes into CER development leads to faster and better results.
Keywords: data visualization, cers, space force, space systems command
PB03 – Kwok – Visual Exploration of Data
Data Management for Cost Engineering Projects
Processes & Best Practices Track (PB04)
Arlene F. Minkiewicz
As cost estimation data becomes more plentiful, data management becomes essential for organizing and preparing data for analysis. Even if a formal data management system is unnecessary for a project, concepts from database management can still be incorporated to ensure higher data quality. This paper will contain an overview of relational and non-relational models, along with a case study of a database that contains structured, unstructured, and semi-structured data.
Keywords: Data Collection, IT, Databases
PB04 – Cuiule – Data Management
PB04 – Cuiule – Paper – Data Management
Everything You Always Wanted to Know About Affordability… But Were Afraid to Ask
Processes & Best Practices Track (PB06)
The term “Affordability” means different things to different people, depending upon one’s employer, organization, function, background, etc. The term has also morphed over the years – expanded or narrowed – based on one’s viewpoint. During ICEAA’s OEM-COG discussion on Affordability, diverse opinions were revealed. We will assemble various interpretations & implementations of Affordability from multiple perspectives, provide historical background, and explore how our community can clarify, standardize, and promote the concept of Affordability.
Keywords: Cost/Benefit Analysis, Life Cycle, Methods, Modeling, Affordability Analysis
PB06 – Mourikas – Everything You Always Wanted To Know
Predictive Thresholds for Schedule Execution Metrics
Processes & Best Practices Track (PB07)
We called it the search for a unicorn. After an intense week of schedule analysis, checking the numbers, updating visual basic code, then presenting objective evidence that the contractor is not achieving the baseline plan, we got the action: provide the metrics of a good program. We tried, but after reviewing countless schedules, we realized that a “good” program is as rare as a unicorn, because every program deviates from the baseline plan. Although we never found a unicorn (“good program” with good metrics), we developed data driven thresholds predictive of significant milestone slip.
Keywords: Data-Driven, Decision Analysis, Government, Methods, Scheduling, Space, Statistics
PB07 – JonesMichelle – Predictive Thresholds for Schedule Execution Methods
The BS in BoeS – Oh the Games that are Played
Processes & Best Practices Track (PB08)
A good Basis of Estimate (BoE) uses a historical comparison, and a complexity factor. These two choices are not statistically based, meaning a bias is introduced into most BoEs. This paper explores the bias in BoEs through the framework of Game Theory. A BoE goes through a creation, internal review, internal approval, and external review process, where each player in this process will likely have a different payoff matrix. Each player in this process will want to insert their bias into the BoE, which this paper terms “Bias Selectivity” (BS).
Keywords: Bias, BoE, Game Theory, OEM, SETA, Government
PB08 – Burney – Paper – BS in BoeS
PB08 – Burney – BS in BoeS
No Estimation Without Escalation – The Inflation Revolution
Processes & Best Practices Track (PB09)
I downloaded the latest USG Inflation Indices, so my work here is done, right? OSD guidance encourages the analyst to explore program-specific escalation to maximize estimate accuracy vice defaulting to existing published indices. This briefing will provide a basics refresher background on common appropriation inflation rates, their outlay profiles, and how to interpret them. It will then explore how to improve to account for escalation, commodity-specific demand, and project-appropriate expenditure profiles increasing estimate accuracy.
Keywords: Inflation, Escalation, Expenditure Profile, Research, 3%
PB09 – Cardoza – No Estimating Without Escalation
Transforming Cost Estimation Services
Processes & Best Practices Track (PB10)
In the last 10-15 years the government contracting industry has quickened the pace of technology delivery. This emerging change has forced cost estimators, as well as acquisition professionals of all kinds, to change to their own processes to keep pace. Cost estimation shops can benefit by changing their service to match customer needs and delivery tempo, add software product instruction with analyst training programs, and extend the scope of “cost” to include budget formulation. This presentation touches on transformation such cost organization infrastructure and repeatable processes.
Keywords: Communication, Methods, Modeling, Services Transformation
PB10 – McKeel – Transforming Cost Estimation Services
Software & Agile Track
What Does Agile Software Development Need: Predictable Costs or Predictable Outcomes?
Software & Agile Track (SA03)
Despite ongoing laudable attempts to advance the state-of-the-art in agile software estimating processes and techniques, the software development and cost analysis communities are far from ‘cracking the nut’. There is a dire need for a cost estimating approach that accurately predicts agile project costs with the correct granularity to enable budget and execution planning. This paper will detail the exploration of three approaches that utilize extrapolation from actuals to implement a solution.
Keywords: Agile, Extrapolation, Software, Estimating
SA03 – Kosmakos – What Does Agile Software Development Need
SA03 – Kosmakos – Paper – What Does Agile Software Development Need
Simplifying Software Sizing with Simple Function Points
Software & Agile Track (SA04)
Daniel B. French
Simple Function Points (SFP), modelled on the IFPUG Function Point Analysis method was introduced with the goal of simplifying the method while reducing cost, time, and difficulty estimating software size without sacrificing accuracy. We’ll introduce the IFPUG version of SFP and highlight the challenges/opportunities when using to size software. We’ll explore when to use SPF and key differences between SFP and IFPUG FP while providing guidance on using FP measures in software cost estimates.
Keywords: Function Points, Software Sizing, Software Metrics, Software Estimating, SFP, Simple Function Points, SiFP
SA04 – French – Paper – Simplifying Software Sizing
SA04 – French – Simplifying Software Sizing
Agile Team Performance Measurement as a Basis for Accurate Cost Estimation
Software & Agile Track (SA05)
H.S. van Heeringen
Agile team performance metrics Productivity, Cost Efficiency, Delivery Speed, Sprint Quality, and Product quality can be measured in an objective, repeatable and verifiable way, compared to each other, and benchmarked against industry data. The measurement data is used to recalibrate long-term effort, duration, and cost estimates based on the actual productivity delivered, resulting in increased predictability. I’ll show a recent study of 4 teams of one organization, each in a different European country.
Keywords: agile teams, performance measurement, agile estimation, software cost estimation, benchmarking
SA05 – van Heeringen – Agile Team Performance Measurement
Dynamic Software Effort Estimation: How SWEET It Is!
Software & Agile Track (SA07)
Software estimation is a complex and diverse field that must accommodate a variety of technical inputs across the life cycle. Our team developed an Excel-based effort estimation model designed for both flexibility and transparency. It dynamically builds an effort estimating relationship based on analyst-selected data fields and source data. Our paper explores how we used data from ISBSG, statistical analysis, and practical experience to drive the development of this “clear-box” model.
SA07 – Gellatly – Paper – Dynamic Software Effort Estimation
SA07 – Gellatly – Dynamic Software Effort Estimation
Agile Product Roadmap Estimating and Progress Tracking
Software & Agile Track (SA08)
Agile project managers often struggle developing and maintaining a full project plan, given the churn in requirements and priorities typical of an agile environment. PMs often use a Product Roadmap to communicate high-level work priorities and phasing, but these often lack the quantitative information needed to enable tracking progress against overall project goals. This paper will offer an innovative solution to this problem, which has recently gained traction on an actual government agile project.
Keywords: agile, product roadmap, software
SA08 – Smallwood – Agile Product Roadmap
Let’s Go Agile: Data-Driven Agile Software Cost and Schedule Models Derived from DHS Projects
Software & Agile Track (SA09)
Chad J Lucas
This paper offers effort and schedule estimating models for agile software development projects using contractor backlog, monthly contract reports, and requirements documents. The regression models predict effort or schedule using six different sizing measures: Unadjusted Function Points, Simple Function Points, Functional Stories (Requirements), Issues, Stories, and Story Points. We also offer effort and schedule benchmarks. The analysis is based on data collected from 18 agile projects implemented within DHS and DoD between years 2014 to 2021.
Keywords: Backlog, JIRA, CONOPS, Requirements Document, Agile, Data-Driven, Early Cost, Functional Requirements, Regression
SA09 – Rosa – Paper – Lets Go Agile
SA09 – Rosa – Lets Go Agile
Software Phasing and Schedule Growth Analysis
Software & Agile Track (SA10)
DoD software research lacks an understanding of development phasing, effort allocation, and schedule. We evaluated conventional rules of thumb for effort allocation against projects in the SRDR database. We compared how effort allocation varies between projects with high or low schedule growth. Results showed that increasing effort in early phases consistently decreases the total schedule growth, and led to improved allocation guidelines. These findings were significant across multiple categories such as Service and project size.
Keywords: Agile, Cost Management, Early Cost, Government, Scheduling, Software, Phasing
SA10 – Long – Software Phasing Schedule Growth
Trending Topics Track
Minding Your P’s and Q’s as Prices Rise
Trending Topics Track (TT01)
M. Michael Metcalf
Choices and challenges abound in combining individual prices (P’s) and quantities (Q’s) of labor and material into one single measure of overall escalation for a project. Popular constructs include the Laspeyres, Paasche, and Fisher. This research illuminates the issue of which index to use when in the relentless fight against Money Illusion – the tendency to think in nominal rather than real terms. Methods are offered for forecasting inflation probability distributions up to three decades out.
Keywords: Budgeting, Cost Management, Project Controls, Risk, Uncertainty, Inflation, Escalation, Price Indices, Forecasts
TT01 – Karickhoff – Paper – Minding Your P’s and Q’s as Prices Rise
TT01 – Karickhoff – Minding Your P’s and Q’s as Prices Rise
Impacts of Digital Engineering to the Cost Estimate
Trending Topics Track (TT03)
Defense programs across the services are implementing digital engineering throughout the acquisition lifecycle to develop, manufacture, and sustain their platforms. How should cost estimators account for these new practices? This presentation will focus on the impact of implementing DE to the cost estimate. Based on literature reviews, discussions with SMEs, and review of policy, we will propose investments and cost uncertainties for the estimator to consider throughout the lifecycle of a platform.
Keywords: Digital Engineering, Lifecycle Cost Estimating
TT03 – Clayton – Impacts of Digital Engineering
CE^2 : Communication and Empowerment for Cost Estimators
Trending Topics Track (TT04)
Too often, cost estimator training focuses solely on technical abilities, largely ignoring the “soft skills”. The behaviors of being a good communicator and empowering the team were shown in a 2020 ICEAA community survey to be important to cost leadership efficacy. Knowing that technical skills alone will only take estimators so far, this presentation leverages communication and empowerment training to demonstrate how all cost estimators can use soft skills to exponentially impact their analyses.
Keywords: Communication, soft skills, communication, leadership
TT04 – Snyder – Paper – CE2 Communication
TT04 – Snyder – CE2 Communication
What is the OEM COG?
Trending Topics Track (TT05)
Cost and Affordability communities employed by Original Equipment Manufacturers (OEM) face challenges that differ from those in government or other private sector organizations. To identify and address these challenges, ICEAA formed the OEM Cooperative Opportunities Group (OEM COG) to discuss, collaborate and share best practices on topics such as Affordability, Digital Engineering, Risk and Professional Development – from an OEM perspective. Join us for an interactive session as we share lessons learned with one another.
Keywords: Communication, Manufacturing
TT05 – Kurtz – What is the OEM COG
Solving the Climate Crisis with Satellites, Fighter Aircraft & Nuclear Reactors
Trending Topics Track (TT06)
This paper examines the challenges of factory built, low cost, mass produced nuclear reactors, not by focusing on the technology, but the acquisition process. It will consider lessons learnt from satellite and fighter aircraft programmes which lead to the formation of a consortium of nations that could engage in nuclear power generation. Can we solve the problem of becoming zero emitters of greenhouse gasses while electricity demand is increasing globally?
Keywords: Cost Management, Government, Infrastructure, International, Learning Curves
TT06 – Shermon – Solving the climate crisis
TT06 – Shermon – Paper – Solving the climate crisis
DoD Cost Estimating Guide v2
Trending Topics Track (TT07)
Erin E. Veltman
The DoD Cost Estimating Guide has something to offer every estimator! This session will provide an overview of the guide, with special focus on version 2. New material includes updated statute/policy references, expanded discussion of Middle Tier of Acquisition estimating, a recommended reading list, and a process-focused case study. Follow analyst Ava’s progress as she navigates the challenges of a Milestone C production estimate for a fictional helicopter program.
Keywords: DOD/MOD, Government, cost estimating process, policy, best practices
TT07 – Mertz – Dod Cost Estimating Guide v2
TT07 – Mertz – Paper – Dod Cost Estimating Guide v2
Estimating Costs of Climate Change Impacts to Public Infrastructure (CIPI)
Trending Topics Track (TT08)
What is the long-term budgetary cost that certain climate change hazards could impose on public infrastructure in Ontario, through accelerated deterioration, increased operating expenses or compromised service levels? CIPI combines climate data with mapped infrastructure data, and applies a derived climate “elasticity” to cost the impact of specific climate hazards. While quantifying the budgetary impacts of climate change on public infrastructure is relatively new, the methodology could support capital funding decisions and adaptation actions by governments now and in the future.
TT08 – Crummey – Costing Climate Change Impacts
TT08 – Crummey – Paper – Costing Climate Change Impacts
Superheavy Launch: The Coming Paradigm Shift for the Space Industry
Trending Topics Track (TT09)
A new generation of superheavy launch vehicles will provide mass and volume capabilities not available in the last 50 years. This paper describes how superheavy launch will transform space vehicle (SV) design and the space industry. No longer mass-constrained, SVs will be build with different materials and processes, and will include capabilities previously inconceivable. SV developers will adapt their facilities, logistics, and support equipment. Cost estimators may want to revisit parametric CERs dominated by weight.
TT09 – Timm – Superheavy Launch
Up is Up; Why Expert Knowledge Should Be Favored in a Technologically Driven World
Trending Topics Track (TT10)
Arlene F. Minkiewicz
There have been strong advancements in technology and other tools in the data science community such as big data explosion, machine learning, and neural-network algorithms. Many organizations are losing their appreciation for subject matter expert opinion; instead, they are focusing on technological advancements in tools. We will discuss the importance of subject matter experts, the key role that they play in stitching together data and provide some cautionary tales of leaning too much on technology.
Keywords: Soft Skills, Communication, Visualization
TT10 – Minkiewicz – Up is Up
Cost Estimate Kick-Starters
Trending Topics Track (TT11)
Daniel J. Harper
I’d like to present several “Cost Estimate Kick-Starters” I created to “kick-start” your cost estimate, including: LaRRGE [Labor Rate Reference Guide for Estimators] Labor Rates resources including newly added Cyber Labor Rates; GRIPS [GSA Robust Infrastructure Pricing Solutions] The GSA Enterprise Infrastructure Solutions Pricer, or “Turbotax” for Enterprise Infrastructure Solutions; SLiCE [Software License Cost Estimator] lookup tool containing over 4,000 prices (and growing) for software licenses, training, etc.
TT11 – Harper – Cost Estimate Kick-Starters
Volunteering: Maximizing the ROI of your ICEAA Membership
Trending Topics Track (TT12)
Christina N. Snyder
Joining ICEAA is as easy as clicking a few buttons, but to truly benefit from the value of your membership, you need to get involved. Volunteering is the first step to making your membership work harder for you. This session shows how to maximize the return on investment of ICEAA membership through volunteer opportunities with commitments ranging from short-term to long-term at the local, national, and international levels.
TT12 – Snyder – ICEAA ROI
CEBoK® Training Track
Basics & Techniques
CEBoK® Training Track (CEB01)
Jennifer Kirchhoffer, CCEA®
The Basics & Techniques session introduces an overview of cost estimating and analysis and the reasons for doing cost estimates, as well as four essential cost estimating techniques most often used to develop realistic and credible estimates. Additionally, we will review cost estimating products and related topics such as schedule and operations and support estimating, providing the background information and fundamental knowledge from CEBoK® Modules 1-3.
Data Collection & Normalization
CEBoK® Training Track (CEB04)
Justin Knowles, CCEA®
This session covers the Core Knowledge section of CEBoK® Module 4: Data Collection. All estimating techniques and cost estimating models require credible data before they can be used effectively. In this module we will discuss the various types of data, processes needed to collect and analyze the data used in parametric applications, as well as data types, sources, and adjustment techniques.
CEBoK® Training Track (CEB05)
Katiana (Kat) Lemmons
This session covers the Core Knowledge section of Inflation and Index Numbers (CEBoK® Module 5). Proper inflation analysis is essential to the success of any cost estimate or economic analysis. Calculating inflation correctly and understanding the fundamental concepts will enable you to produce cost estimates that are timely, accurate, and credible to support your program’s lifecycle needs. It will also empower you to communicate with key stakeholders on the need to adjust your financial estimates based on changes in the economy.
Analysis, Probability, & Statistics
CEBoK® Training Track (CEB06)
Mike Allen, CCEA®
This session discusses the analytical steps to take after obtaining a set of cost data and covers techniques for displaying and analyzing data graphically and statistical and graphical analysis of univariate and bivariate data sets (CEBoK® Modules 6 & 10). Other topics include measures of central tendency and dispersion and important probability distributions. We also introduce the concept of a random variable; Monte Carlo simulation; and the differences between the normal and lognormal distributions. Finally, we discuss hypothesis testing.
CEBoK® Training Track (CEB07)
Sriram Krishnan, CCEA®
This is a training track presentation of the CEBoK® Module 7 (Learning Curves) will cover the key ideas, analytical constructs, and applications of the module. Beyond the theoretical information, we will present the study questions for Module 7 with steps required to solve the problems using only a calculator as is required on the certification exam.
CEBoK® Training Track (CEB08)
Bill Barfield, CCEA®
This course introduces the basic concepts of regression and provides a demonstration of a simple linear ordinary least squares model (CEBoK® Module 8). This session focuses on the basics required to build and evaluate a simple linear model such as a Cost Estimating Relationship (CER). Key concepts include correlation, minimizing error, homoscedasticity, statistical significance, goodness of fit, confidence intervals, uncertainty, and analysis of variance. The better you understand these concepts, the better you will be able to make inferences about cost data and employ more complicated regression techniques.
CEBoK® Training Track (CEB09)
Sam Easterly, CCEA®
This session will provide motivation for the need for risk analysis and introduce the basic types and uses of risk (CEBoK® Module 9). It will focus on the practical execution of the general risk analysis process: develop a point estimate; identify the risk areas in the point estimate; determine uncertainty around the point estimate; apply correlation between uncertainty distributions; run the Monte Carlo simulation; assess the reasonableness of results; calculate, allocate, and phase risk dollars; and show the results.
Manufacturing Cost Estimating
CEBoK® Training Track (CEB11)
Corey Boone, CCEA®
The goal of the Manufacturing Cost Estimating module (CEBoK® Module 11) is to arm the student with a set of techniques used to address issues unique to estimating in the manufacturing environment. It will be our objective in this module to raise a few of the most common general issues, considerations and concerns the estimator must be aware of in a typical major manufacturing environment and to provide techniques for addressing them. Depending on time and interest of attendees, example problems can be worked as exam preparation.
Software Cost Estimating Using CEBoK-S
CEBoK® Training Track (CEB12)
David Brown, CCEA®
Kevin Cincotta, CCEA®
This session covers the core knowledge of Software Cost Estimating using CEBoK-S (all PCEA/CCEA testable topics are included). It will be of particular interest to anyone studying for the ICEAA certification exam. The session provides an introduction to the basics of the software development and maintenance processes and how to estimate the related effort. The key ideas of Software Cost Estimating include the cost drivers of size, complexity, and capability. In the sizing area, we’ll focus on understanding the physical size, functional size, relative effort measures (agile software development), and RICE(FW)1 objects. We’ll also discuss the primary software development paradigms – Predictive (waterfall), Predictive with modification (incremental, evolutionary, and spiral methods), Agile (iterative, scrum, SAFe), and Hybrid – and how to model them from a cost estimating perspective.
CEBoK® Training Track (CEB13)
Richard Shea, CCEA®
This session covers the Core Knowledge section of Module 13 Economic Analysis of CEBoK®. It will be of particular interest to anyone studying for the ICEAA certification exam. The session provides a practitioner’s perspective for conducting an economic analysis (EA) by reviewing EA concepts, terminology, variables and measures-of-merit. By accounting for monetized costs, monetized benefits, opportunity costs and time-value-of-money (“discounting”), an EA enables one to calculate economic measures-of-merit.
CEBoK® Training Track (CEB14)
Paul Neubert, CCEA®
This session explores the basics of contract pricing (CEBoK® Module 14). We explore various contract types and the factors and considerations related to choosing a contract type. We also explore fee, shared risk, cost-price proposal preparation, the makeup of a good Basis of Estimate (BOE), and evaluation efforts. This session also provides an introduction to cost management. Some methods discussed include Total Ownership Cost (TOC), Cost As an Independent Variable (CAIV), Target Costing, and Activity Based Costing (ABC).
Cost Management and EVM
CEBoK® Training Track (CEB15)
Earl J. Gee II
This session will provide an introduction to the basic concepts of earned value management (CEBoK® Modules 15 & 16), with a focus on implementation, governance, and practical application in support of a project or program. Specific topics will include basic EVM components and data elements, as well as standard earned value analysis techniques. We will use practice problems throughout the presentation to demonstrate and reinforce the basic principles of EVM.