Agile Track
AAE2A in C2ISR: Right Metrics for the Right Audience
Agile Track (AG01)
Elizabeth Ashwood
Kyle Davis
Matthew Hoffman
Jennifer Lee
The PEO Digital C2ISR Division cost team has been utilizing JIRA to develop agile software development metrics for 4 years. This presentation will discuss how we have integrated ourselves with our software development teams, the complexity involved in extracting meaningful information from JIRA, and how we generate useful metrics and trend analyses for a number of different audiences. We will discuss the metrics we develop for each audience – software development teams, system users, senior leadership – and why they are critical.
Keywords: Agile, Decision Analysis, Program Management, Software, Story Points
Air Force Mission Planning Systems: Standing Up Agile Under a Waterfall
Agile Track (AG02)
Trevor Michelson
Lorraine Rappaport
In November 2018, the “Agile Development Manifesto” was released to provide summary guidance to rapidly deliver software products under SAFe Agile. The Mission Planning Systems (MPS) cost team has worked closely throughout the MPS division over the past year to incorporate Agile practices and useful metrics for software development through the release increments. This presentation will provide a journey through our current Agile metrics, how they could be improved, and lessons learned throughout MPS.
Keywords: Agile, Communication, Data-Driven, Early Cost, Performance Management, Program Management, Software, Story Points
The Fact That Your Project is Agile is Not (Necessarily) a Cost Driver
Agile Track (AG03)
Arlene F. Minkiewicz
All true agile projects follow the same philosophy, they do not all apply the same set of practices, tools or processes. Agile projects are value driven, thus subject to change. There are, however, business and contractual requirements for up front estimates creating a conundrum. This paper discusses a methodology and rules of thumb for estimating agile projects, based analysis of publicly available datasets, that provides value to stakeholders and aligns with Earned Value Management requirements.
Keywords: Agile, Early Cost, Software
Empirical Effort and Schedule Models for Agile Processes
Agile Track (AG04)
Wilson Rosa
Barry W. Boehm
Bradford K. Clark
Raymond Madachy
This paper presents a set of practical effort and schedule estimation models for agile software development projects useful during early lifecycle phases. The statistical models were developed using actual data from 36 agile projects implemented within the US DoD, during the time period from 2008 to 2019. The analysis will also demonstrate whether Agile provides better results than traditional software development in the areas of productivity gains, minimizing cost growth and mitigating schedule delays.
Keywords: Agile, Data-Driven, Early Cost, Functional Requirements, Regression, Software
Agile Product Roadmap Estimating and Progress Tracking
Agile Track (AG05)
Blaze Smallwood
Ryan Blackburn
Agile project managers often struggle developing and maintaining a full project plan, given the churn in requirements and priorities typical of an agile environment. PMs often use a Product Roadmap to communicate high-level work priorities and phasing, but these often lack the quantitative information needed to enable tracking progress against overall project goals. This paper will offer an innovative solution to this problem, which has recently gained traction on an actual government agile project.
Keywords: Agile, Methods, Software, Story Points, Roadmap
Cyber Mission Platform (CMP) Program: Analyzing the Full Suite of Agile Metrics
Agile Track (AG06)
Alex Smith
William Lane
Brian Opaska
Guidance, policies and task force recommendations related to agile software development best practices for DoD seem to surface on a daily basis. The DSB, DIB, GAO and OUSD A&S have all weighed in during the last two years. The OUSD A&S “Agile Metrics Guide” released in 2019 provides the most concise discussion of metrics that cost analysts and PMs supporting agile programs should consider. This presentation will discuss the AF Cyber Mission Platform program’s recent agile transformation and how it is collecting and analyzing the key metrics discussed in the OUSD A&S guide.
Automated Data Analysis and Forecasting for Agile Deliveries
Agile Track (AG07)
Eric van der Vliet
Jurjen Bloo
Agile projects deliver results in short iterations. The initiative starts with a budget and there seems to be less control than in a traditional lifecycle while there is much more information. Agile projects use tools to manage their delivery, containing information about tickets, defects, effort, etc. This presentation shows an example of a BI solution that extracts the data out of Agile tooling and reports real-time the actual status as well as an predictive analysis.
Keywords: Agile, Cost Management, Data Collection, Data-Driven, Decision Analysis, Story Points, Metrics, Predictive Analysis
Agile Team Estimation, Performance Measurement and Benchmarking
Agile Track (AG08)
H.S. van Heeringen
Nowadays, more and more organizations switch to agile ways of software development. However, senior management is often not satisfied, as the productivity of the agile teams is often not measured. To improve and to demonstrate success to stakeholders, they need to know which teams are high performers and which are low performers and to see the trends in performance. Standardized metrics for performance measurement, benchmarking, estimation and budgeting are essential for management decision making.
Keywords: Agile, Budgeting, International, Parametrics, Performance Management, Project Controls, Story Points, Agile Team Performance Measurement, Agile Team Cost Estimation, Benchmarking
Wranglin a Meandering Program
Agile Track (AG09)
Leah Walker
Bryan K. Anderson
Melanie Melancon
Cost and schedule estimation for new investments is difficult but more challenging when the program was executed prematurely, leading to cost and schedule overruns. These off-course programs need a holistic remedy to get them back to health. This business case explores the impacts of program ownership transition, conversion of software development methodology, and modified scope and requirements management. Innovative approaches were used to achieve programmatic goals and put the project back on track.
Keywords: Agile, Data Collection, Data-Driven, Functional Requirements, Government, Learning Curves, Life Cycle, Methods, Modeling, Performance Management, Scheduling, Software, Uncertainty, Variables, Scaled Agile, Function Point Analysis, SEER-SEM, Machine Learning, NLP
Analytical Methods Track
Developing CERs to Estimate Commercial IaaS Costs for Federal IT Systems
Analytical Methods Track (AM02)
Cara Cuiule
Amanda Ferraro
Daniel Harper
Richard Mabe
Establishing federal budgets for cloud infrastructure costs prior to selecting a cloud provider requires vendor agnostic cost estimating methods. These methods need to reflect the correlation between rates for a variety of infrastructure instances across all viable cloud service providers. This paper describes research and validation leading to CERs based on over 28,000 virtual machine and storage instances. The predictive analytic approaches presented in this paper can provide valid and verifiable vendor agnostic estimates.
Keywords: Data-Driven, Government, IT, Modeling, Cloud
The Rise of O&S Cost – Data Drive Exponential CERs for Maintenance Cost
Analytical Methods Track (AM03)
Jeremy Goucher
There is no question that O&S cost are a very large part of the lifecycle cost of a system. One common method to estimate O&S cost is annually by analogy to a similar program. Further, anecdotal evidence suggests these costs “lay flat;” that is to say they are roughly the same from one year to the next. However recent analysis of VAMOSC data suggests that not only do these costs grow from year to year, but that they grow exponentially! This study will describe the data analyzed, the equations that were tested, recommended CERs, and engineering explanations of the results.
Keywords: Data-Driven, Life Cycle, Methods, Modeling, Parametrics, Statistics, O&S Cost Estimating, Maintenance Cost, Aircraft Cost
The Median Is Not the Message: The Case for Estimating at the Mean
Analytical Methods Track (AM04)
Dustin Hilton
Kimberly Roye
Christian B. Smart
The use of log-transformed ordinary least squares (LOLS) pervades the development of cost estimating relationships (CERs). This technique estimates at the median. Because of skewness, this means LOLS is typically biased low. The negative impact of this bias is discussed and the need to estimate at the mean is stressed. Modern alternatives to LOLS are discussed. Implementation of these methods in R is provided, including algorithms, goodness of fit, and statistical significance.
In Search of the Production Steady State: Mission Impossible?
Analytical Methods Track (AM05)
Patrick McCarthy
Learning Curves are a vital tool for cost estimators when predicting the number of direct labor hours required for a production run. One challenge of utilizing learning curves is predicting when no additional improvement can be expected, otherwise known as the steady state of the production run. This paper addresses different variables to consider when analyzing data to determine when improvement is likely to cease and the steady state of the production run will commence.
Portfolio Risk Analysis Supporting Agency Budget Closure and Architecture Trades
Analytical Methods Track (AM07)
Stephen R. Parker
Christine L. Anderson
William Laing
Nicole M. White
The NRO CAAG has developed a Portfolio Risk Analysis capability supporting NRO budget closure and architecture trades. A data-driven input approach is employed, integrated with a simulation model for budget analysis. This approach provides outputs that improve decision-makers ability to perform prioritized budget closure analysis. Enhanced dashboard outputs help support these informed portfolio decisions. This presentation provides an overview of this capability, with focus on the development of the simulation inputs and analysis of outputs.
Keywords: Budgeting, Cost Management, Data Collection, Data-Driven, Decision Analysis, Government, Modeling, Monte Carlo, Risk
Diversity in Software Estimation Approaches – Perceptions to Preferences
Analytical Methods Track (AM08)
Shashank Patil
Ria Bakhtiani
For strong foundation of Software Estimates, diversity in analytical abilities & preferences of winning teams are the key factors those must be effectively addressed. In this paper we talk about various preferred approaches adopted by winning teams and their usefulness on critical parameters of Software Estimations; viz: Turnaround Time, Accuracy, Repeatability and Reproducibility. The views in this paper are based on our internal surveys carried out with SMEs and Engagement Managers at various hierarchical levels.
Keywords: Agile, Budgeting, Decision Analysis, Uncertainty, Analytical Skills
The Road Back to EVM
Analytical Methods Track (AM09)
Teresa Price
Tia M. Barnes
Andrew Drennon
Wilfred Tagud
EVM is a management methodology for integrating the oversight of scope, schedule, and cost of projects, and for objectively measuring project prog¬ress and performance. This study provides an insight to EVM and explores current EVM practices at the Federal Aviation Administration. The study assesses emerging performance management techniques and provides recommendations to improve current EVM practices for government programs and projects. This initiative is aimed to encourage industry to review, rethink and advance the application of EVM.
Keywords: EVM, Function Points
The Devil is in the Detail…ed Technical Baseline
Analytical Methods Track (AM10)
Benjamin Truskin
Summary Not Approved for Public Release
Identifying Bias and Influence in Expert Estimates
Analytical Methods Track (AM11)
Caleb Williams
Stephanie DelPozzo
Summary Not Approved for Public Release
Data Management Track
Finding and Analyzing Data for Failure Analysis and Cost Estimates
Data Management Track (DM01)
George Bayer
Bryan K. Anderson
Cost estimators are often presented with incomplete data sets from which they must develop business case solutions. Understanding, interpreting, and improving the data integrity are critical factors for cost estimate accuracy. In this use case, the cost estimators analyze and interpret incomplete and subjective data sets, forecast spares depletion, and estimate obsolescence. Using innovative data mining and machine learning techniques, the estimators demonstrate how improved data can result in better cost estimates and business cases.
Keywords: Cost Management, Data Collection, Data-Driven, Methods, Modeling, Uncertainty, Obsolescence Forecasting
The FlexFile Cookbook: Powerful Analysis via R
Data Management Track (DM02)
Benjamin Berkman
Justin Cooper
The Cost and Hour Report (“FlexFile”) is a new reporting format that promises to change the world of DoD cost analysis by delivering significantly more granular cost and hour data than its predecessor, the well-known 1921 Contractor Cost Data Reporting (CCDR) series. The volume of FlexFile data requires a more powerful approach to manipulating, analyzing and visualizing the data than Excel may offer. This paper provides best practices in R that will enable analysts to exploit FlexFile data.
Keywords: Data Collection, Data-Driven, Government
Data Visualization: A Tableau Story
Data Management Track (DM03)
Joyce Fang
Katie Stacy
Cost estimators perform dual roles. They need to provide analyses on various data to capture the scope of a project, and they also need to effectively communicate their findings to decision makers. Data visualization is the solution to this dichotomy. Effective data visualizations allow viewers to quickly grasp patterns, trends and outliers. Discover the potential and benefits of data visualization through the lens of Tableau and develop strategies of seeing and showing.
Keywords: Communication, Data-Driven, Data Visualization, Analytics, Tableau, Charts, Plots, Graphs
Advanced Data Analytics for Maintenance & Repair Reporting
Data Management Track (DM04)
Paul Hardin
Alexander LoRusso
Tyler Staffin
The 1921-M/R (Maintenance & Repair Parts Data Report) is the DoD system for collecting actual maintenance event and repair part data in the Cost and Software Data Reporting (CSDR) system. This paper will employ the R Shiny package, which is used for the construction of interactive web applications, to demonstrate the analytical value of -M/R data. Additionally, this paper will explore the mechanics of the R Shiny framework within the environment of advanced data analytics.
Keywords: Cost Management, Data Collection, Data-Driven, Government, Life Cycle, Analytics, R, Shiny
ICER-P: Automating the Development of Cost Estimating Relationships
Data Management Track (DM05)
Brian Wells
Sophie Behrend
Clare Egan
The NRO CAAG uses Cost Estimating Relationships (CERs) as a basis for its estimates. Historically, updating CERs has involved a manual Excel process that was very time consuming. The Integrated CER Platform (ICER-P) is the CAAG’s new Python-based tool that serves as a replacement to the legacy Excel process. This presentation gives firsthand accounts of the ICER-P development and deployment process.
Keywords: Data-Driven, Government, Modeling
Machine Learning Track
Remember the Algorithms: A Case Study on Machine Learning in Software Estimation
Machine Learning Track (ML01)
Bryan K. Anderson
Morgan Heimbrock
Machine learning opens the door for software cost and schedule estimation for previously unviable scenarios in traditional frameworks. A reliable software estimation requires several intricate factors and often the data is unattainable or incomplete. By training several machine learning algorithms on actual data from an issue tracking system this case study achieved a more reliable model. Factors like issue description, individuals assigned, and issue events are shown to be important features in modeling development work.
Keywords: Agile, Data Collection, Scheduling, Software, Statistics
Machine Learning Methods to Calculate Cost Estimating Relationships and Effort Adjustment Factors
Machine Learning Track (ML02)
Jonathan Brown
Elizabeth Morris
In today’s data driven world, machine learning is increasingly utilized to gain insights from large amounts of data. In cost estimation, machine learning can enable us to make use of large sets of historical data to improve future estimates. In this paper, we apply machine learning and traditional methods to data from the Software Resources Data Report (SRDR) Database to calculate Cost Estimating Relationships (CERs) and Effort Adjustment Factors (EAFs) used in code size calculation for software cost estimates.
Using Machine Learning to Determine Cyber Security Costs
Machine Learning Track (ML03)
Daniel Harper
Richard Mabe
Summary Not Approved for Public Release
Applying Machine Learning to My Integrated Master Schedule
Machine Learning Track (ML04)
Joe Hirsch
Summary Not Approved for Public Release
Intersections of AI and Cost Estimating: Explainability
Machine Learning Track (ML05)
Jon Kilgore
The emerging field of Explainable Artificial Intelligence (XAI) is focused on helping decision makers to understand and trust underlying machine learning logic and algorithms. Inherent in many AI algorithms is a tradeoff between explainability and potential predictive power. This paper will explore explainability in artificial intelligence and machine learning, and will bridge these challenges to those that the cost estimating community has encountered historically and will increasing encounter as we also adopt machine learning techniques.
Keywords: Communication, Data-Driven, Decision Analysis, Methods, Modeling, Artificial Intelligence, Machine Learning
Machine Learning for Numerical Prediction at Subsystem Level
Machine Learning Track (ML06)
Conner Lawston
Machine Learning has been a widely popular topic in data analytics for the past several decades, from face and speech recognition to robotics, spam filtering and automated assistants. Additionally, new modeling Methods (Decision Trees/Random Forests, Ridge/Lasso, and Principal Component Analysis) can replace the traditional Regression methods with much better results. This presentation will dive into how Machine Learning can accurately and efficiently predict costs based off of dozens, and even hundreds of low level variables.
Keywords: Bayesian, Data-Driven, Modeling, Parametrics, Regression, Statistics, Machine Learning
Cost and Technical Data Convergence in Machine Learning
Machine Learning Track (ML07)
Jeff McDowell
Our community does very well collecting cost data in a structured and repeatable manner. Unfortunately, we do less well collecting cost-driver data yet cost analysis involves relating cost to its causation – relating cost data to non-cost data. Machine learning in cost analysis requires non-cost data to be as well formed as the cost data. This paper converges data science, machine learning, and cost analysis to address this convergence of cost and technical data.
Keywords: Data Collection, Data-Driven, Modeling, Parametrics, Machine Learning
Summarizing Schedules Using Hidden Markov Models and Natural Language Processing
Machine Learning Track (ML09)
Daniel Newkirk
Independent Schedule Risk Assessments (iSRA) and Joint Confidence Level (JCL) are common approaches for evaluating schedule overruns and determining likelihood of completion. The starting point for these analyses is the creation of a summarized schedule, which is difficult and time consuming to create. We have developed an automated schedule summarization method that utilizes Hidden Markov Models and Natural Language Processing (NLP) to address this need.
Keywords: Modeling, Scheduling, Machine Learning, Automation, Python, Hidden Markov Models, Natural Language Processing
Leveraging the Wisdom of Crowds in Estimating Army SW Sustainment
Machine Learning Track (ML11)
Christian B. Smart
Cheryl Jones
Kimberly Roye
The use of modern regression and machine learning techniques can improve predictive accuracy compared to traditional log-transformed ordinary least squares, as well as resolving issues with bias and transformation. The combination of multiple models in an ensemble and cross-validation can further increase accuracy. These techniques are discussed in detail and are applied to an extensive set of software sustainment data for 192 Army systems. Results include models based on release type, software changes, and categories.
Keywords: DOD/MOD, Parametrics, Regression, Statistics, Machine Learning
iAssistant – Machine Assisted Risk Based Cost Estimation
Machine Learning Track (ML12)
David L. Wang
Austin Lee
Rachel Macaulay
Summary Not Approved for Public Release
Modeling Track
Advanced Estimating Methodologies for Conceptual Stage Development
Modeling Track (MM01)
Chuck Alexander
This research paper presents statistical techniques and cost analysis that significantly enhance legacy technology development estimating methodologies. Techniques leveraging independent variables that reflect a comprehensive set of cost drivers relating to technology scale, complexity, type, maturity, and development difficulty are presented. Highly tailored solutions including uncertainty are produced that vastly expand and refine earlier development estimating models. General R&D framework relating key milestones, TRLs and cost benchmarks is constructed and woven into an integrated solution.
Keywords: Bayesian, Data-Driven, Early Cost, Methods, Modeling, Parametrics, Statistics, Uncertainty, Variables, Conceptual Stage, Technology Development
Continuous Enhancements: An Alternative to Maintenance
Modeling Track (MM02)
Daniel Bowers
Emily Freed
With a rapidly evolving threat and the enemy’s ability to adapt and counter our weapons and technology, gone are the days of designing, producing, and maintaining a system for 20-30 years. Modern systems need the ability to adapt and update rapidly to include the latest technology. This new paradigm changes the traditional O&S methodologies and estimating techniques employed by the cost analyst. We will look at the impacts to maintenance costs and the traditional funding profile.
Keywords: Budgeting, Life Cycle
Future Years Cost Tool
Modeling Track (MM03)
Zachary Donchez
Collin Donner
With ever-evolving missions and threats, today’s Department of the Navy (DON) decision makers are challenged to understand the impacts of force structure trade-offs. The DON’s defense strategy shapes requirements for ship and aircraft portfolios, but in the absence of associated cost. This cost tool leverages historical execution and cost data to forecast the full DON Total Obligation Authority across every appropriation. The tool supports force structure drills and a future fleet architecture project.
Keywords: Budgeting, Data-Driven, Modeling, Regression, Statistics
Writing Code for Cost Estimating – How to and Why You Should
Modeling Track (MM04)
Collin Donner
Zachary Donchez
Today’s Department of the Navy (DON) decision makers are challenged to understand the impacts of force structure trade-offs in terms of capability and cost. For a future years cost tool, aircraft procurement costs were analyzed to develop a learning/rate curve estimate for all different types of aircrafts in the DON. This paper uses this example to present how utilizing a programming language can improve reproducibility, reduce time, and enhance traceability in data analysis and reporting.
Keywords: Cost/Benefit Analysis, Data Collection, Data-Driven, Government, Manufacturing, Microsoft Excel, Modeling, Operations, Parametrics, Process Engineering, Regression, Software, Statistics, Learning Curves, Rate Curves, R, Programming, Artificial Intelligence, Historical Trends
Mission Operations Cost Estimation Tool (MOCET) 2020
Modeling Track (MM05)
Marc Hayhurst
Cindy Daniels
Lissa Jordin
Brian Wood
The Mission Operations Cost Estimation Tool (MOCET) is a model developed by The Aerospace Corporation in partnership with NASA’s Science Office for Mission Assessments (SOMA). MOCET provides the capability to generate cost estimates for the operational, or Phase E, portion of full NASA space science missions. Updates made to version 1.4 in 2019 and forthcoming updates in 2020 are discussed.
Keywords: Cost Management, Early Cost, Government, Modeling, Space
Augustine’s Law: Are We Really Headed for the $800 Billion-Dollar Fighter?
Modeling Track (MM06)
Brent M. Johnstone
Augustine’s Law famously proposed fighter aircraft costs are growing so rapidly that by 2054 buying a single tactical aircraft will consume the entire defense budget. Is the situation really so dire? This paper examines the trend in U.S. fighter costs and relates them to generational changes in aircraft design and manufacture. It also examines the new jet fighters of the 2000s to see if Augustine’s Law is really unfolding as its author originally thought.
Keywords: Budgeting, Cost/Benefit Analysis, Data-Driven, Early Cost, Government, Life Cycle, Modeling, Parametrics, Augustine’s Law, Cost Analysis, Fighter Aircraft, Government Procurement, Cost Effectiveness, Cost Estimates, Mathematical Models, Planning
Cost Credibility Enhancements with SRL and MBSE Advanced Tools
Modeling Track (MM07)
Patrick K. Malone
When space based system development programs are less than 15% spent of estimated costs, traditionally almost 80% of design costs are committed! Furthermore, programs with less than the recommended GAO knowledge at program start will likely have higher risk and unexpected cost/schedule growth. Using advanced tools like System Readiness Level (SRL) and Model Based System Engineering (MBSE) methods we show how to identify vulnerabilities early in the development life cycle to mitigate these risks.
Keywords: Cost Management, Government, Methods, Space
Cost, the Forgotten Element in MBSE
Modeling Track (MM08)
Karen Mourikas
Dan Cota
Digital Thread. Model-based Systems Engineering (MBSE). Integrated processes. Life-Cycle Cost analysis. Concepts such as these enable better decision-making and cross-functional collaboration throughout the life cycle by connecting often-siloed elements of design, manufacturing, operations and support, resulting in optimized solutions. However, cost is often absent. We demonstrate case studies incorporating cost analysis for rapid trades and strategic decision-making into an MBSE environment, at different phases of the life cycle, enabling more affordable optimized solutions.
Keywords: Cost/Benefit Analysis, Decision Analysis, Life Cycle, Modeling, MBSE
Cost Estimation and Prediction from Causal Structure
Modeling Track (MM09)
William R. Nichols
Michael D. Konrad
Does your cost estimation method account for the number of difficult requirements, team experience, code quality? For the past three years, the SEI has investigated the causal structure underlying software project data to better understand which factors actually drive software cost. Some factors thought to drive software costs do not actually do so, and other factors, often unmeasured, may be more important. We will share our insights and offer recommendations for improved cost estimation.
Keywords: Bayesian, Cost Management, Early Cost, Life Cycle, Modeling, Statistics, Uncertainty, Causal Discovery, Causal Modeling, Software Cost, Empirical Research
Foundation of Structured Architecture, System & Cost Modeling
Modeling Track (MM10)
Danny Polidi
Mike Crist
Modern software packages can perform complex physics based simulations and alone typically do not consider cost as an input variable. Other packages can aid in determining the cost to a chosen design solution. This paper begins the development of a structured System Engineering approach to System design and defines a standardized modular diagram for a RADAR System applied to military applications in the aerospace industry. It will be demonstrated that a system can be defined in standard terms, can be modeled with modular blocks and can simultaneously be costed by those same modular blocks.
Schedule Milestone Metrics for Government Space Systems
Modeling Track (MM11)
Michael Schiavoni
Come see real data-driven insights you can use immediately, and an analysis framework that can be applied to other data sets. This paper investigates when major milestones tend to occur relative to the overall project schedule. By expressing event dates of historical programs in terms of schedule percent complete, density estimates are generated for PDR, CDR, and BIST. The result is a collection of metrics that can be used to crosscheck the feasibility of a summary schedule, or even to predict the launch date of an in-progress program.
Commercial Cloud Cost Model (C3M)
Modeling Track (MM12)
Ryan Timm
James Smirnoff
Summary Not Approved for Public Release
Processes & Best Practices Track
Assuring Credibility in the Cost Estimate; Part II
Processes & Best Practices Track (PB01)
Henry Apgar
This paper updates the original, first presented at the 2016 ICEAA International Workshop in Bristol, which traced the evolution of cost estimating attributes, settling on cost credibility. Evidence was provided in the words of senior government executives, industry leaders, estimating and engineering handbooks, professional journals, and government auditing manuals. This update incorporates the impact of cost drivers such as program maturity levels and system readiness levels, and newer processes such as model-based system engineering. The paper assesses the impact on the estimating professional.
Keywords: Government, Modeling, Parametrics, Program Management
13 Reasons a Cost Estimate During a Concurrent Engineering Study Could Go Wrong
Processes & Best Practices Track (PB02)
Andy Braukhane
During early phase spacecraft design, the concurrent engineering (CE) approach is proven to be very efficient. But the condensed and iterative nature of CE sessions can also make life hard for a cost estimator. This work discusses 13 problem areas experienced or observed mainly during one-week, inter-disciplinary space system design studies and provides practical examples on how to tackle them, e.g. how to handle rapid data changes, wrong expectations and a diverse engineering team.
Keywords: Communication, Early Cost, International, Space, Concurrent Engineering
Estimation Capability Analysis (ECA) Survey Results – A&D Industry
Processes & Best Practices Track (PB03)
Bruce Fad
Strength of estimating capability is a factor in winning competitive proposals and achieving organizational growth. Answers to clear and pointed questions about estimation capability help identify process shortcomings that may be inhibiting growth. Aggregated results from question surveys, completed by industry participants will be presented along with results interpretation to paint a picture of Estimation Capability Analysis (ECA).
Keywords: Bias, Budgeting, Cost Management, Data Collection, Data-Driven, Decision Analysis, DOD/MOD, Modeling, Parametrics, Process Engineering, Risk, Scheduling, Uncertainty
Cost Estimating Tug-of-War: Strategies for Playing the Game
Processes & Best Practices Track (PB04)
Kellie Scarbrough
Cost estimates face pressures from stakeholders, project managers, estimators, and contractors. Desirable projects recover financial outlays quickly but also deliver on social impact, pressures which drive the creation of overly optimistic estimates that underestimate the cost and risk involved. This is an interactive session to explore this game of scope and cost manipulation and discuss strategies to counter the influences on cost estimates that make them more political posturing than a factual assessment of project price.
Keywords: Cost Management, Data-Driven, Government, Program Management, Project Controls
Storytelling for Cost Estimators
Processes & Best Practices Track (PB05)
Christina N. Snyder
As estimators, we advocate the importance of good data; but without context, estimates and analyses are just numbers. To give power to our work, we need to effectively pair good estimating with good communication. There is no existing best practice guidance for estimators on how to create a compelling narrative to accompany analysis. By leveraging a storytelling structure, we can inspire action, communicate our findings in a way that resonates, and ultimately become more effective.
Keywords: Decision Analysis, Program Management, Storytelling
Lessons Learned Implementing EVM on Government-led Delivery Efforts
Processes & Best Practices Track (PB06)
Joshua Teitelbaum
Summary Not Approved for Public Release
Software Track
Navigating the Minefield – Software Cost Estimation before Requirements
Software Track (SW01)
Carol Dekkers
Daniel French
Estimating the cost and schedule for software development and enhancement projects without solid requirements can be especially challenging when few requirements are known. This presentation presents shortcuts to estimating project scope (size) including t-shirt sizing, patterns, simple FP, early and quick and the one file model. We will also address the fundamentals of software cost estimation and where size fits into the process.
Keywords: Agile, Budgeting, Communication, Data-Driven, Early Cost, Functional Requirements, Government, International, IT, Life Cycle, Methods, Performance Management, Process Engineering, Software
Measure it and They Will Come: Keys to Developing a Successful Software Measurement Program
Software Track (SW02)
Daniel French
Carol Dekkers
Despite the pervasiveness of software development in all areas of industry today, few organizations develop and implement effective software measurement programs to manage their software development projects and leverage information necessary to make business decisions. This presentation discusses reasons why this is, addresses myths and misconceptions regarding software metrics programs, and how to develop an effective program, as well as which software metrics to develop to provide the greatest value.
Keywords: Cost Management, Data-Driven, EVM, Function Points, IT, Methods, Performance Management, Project Controls, Software, Software Measurement, Software Metrics, Software Estimation, Software Quality, Project Management
Software Evaluations: The Increasing Growth of Software for Military Application
Software Track (SW03)
William Gbelee
Justin Cerrato
The DoD has experienced an exponential growth of Software for military systems – Every aspect of the DoD enterprise now includes a software dependent capability: weapons, communications, and resourcing. The vast majority of program and system managers still rely on counting Software Lines of Code (SLOC) to measure software application size. This paper will discuss the growth of software for military applications and to discuss alternative sizing methodologies to measure and manage a program.
Keywords: DOD/MOD, Government, Software, Software Growth, Software Lines of Code, Data
Software Made Simple: A Redux
Software Track (SW04)
Shaun Irvin
William King
At SMC, estimating software requires ESLOC as an input. However, software is generally reported by the contractor as New, Modified, Reused, and Autogenerated. Converting to ESLOC requires estimating factors for each code type, adding additional uncertainty to your final estimate. We propose removing ESLOC conversion and regressing software effort with each code types. This paper expands on Jeremy Goucher’s 2018 paper, by exploring non-linear regressions and accounting for code growth, agile, and software processes simultaneously.
Keywords: Agile, Regression, Software
Improving Estimating Relationships for Army Software Sustainment
Software Track (SW05)
Cheryl Jones
Bradford K. Clark
James Doswell
New approaches were employed to improve Army software sustainment cost estimation: causal analysis and annualization of release data. Causal analysis examines the cause/effect relationships between factors that indicate which CERs should be derived. Converting multi-year data to annualized values has improved CERs. This presentation shows what was discovered using causal analysis and the resulting improved CERs.
Keywords: Decision Analysis, DOD/MOD, Modeling, Parametrics, Software, Sustainment
But Wait, There’s More! Using SFPA for Your Cost, Schedule & Performance Needs
Software Track (SW07)
Katharine Mann
Ryan Hoang
Do you need to estimate software size? Do you want to add value to your program beyond the LCCE? Simple Function Point Analysis (SFPA) can help! We discuss how analysts can engage Program Managers to use SFPA not just for cost estimating, but for scheduling and progress tracking of software development programs. Real DHS Programs and Policies are used to illustrate the benefits of Simple Function Points to the entire organization.
Keywords: Agile, Early Cost, EVMFunction Points, Functional Requirements, IT, Performance Management, Program Management, Scheduling, Software, Simple Function Points
Costing Out an Air Force Software Factory
Software Track (SW08)
Stephanie Quintal
Caitlin Burke
Kristen Marquette
The novel concept of standing up a software factory has left even the most seasoned cost estimator scrambling for guidance. Presented by the Kessel Run cost team, this presentation will provide insights on staffing, physical locale and other hidden stand-up costs. We will discuss real world actuals on team sizing, skill mix, and phasing methods as well as labor rate analysis and acquisition support. Lastly, a template for a generic software factory will be provided.
Keywords: Agile, Data Collection, Modeling, Software
Software Estimating: Is the Problem Solved? Some Myths and Facts
Software Track (SW09)
Sanathanan Rajagopal
Software Estimating is often seen as one of the most difficult domain and an extremely difficult to estimate. However every project (S) now have software and dependency and complexity of which is growing exponentially. In this paper author will explore why people thinks why software estimating is a big issue and will try to break some myths and explore some facts about software estimating.
Keywords: Software, Software Estimating
Is ESLOC Dead?
Software Track (SW10)
Richard J. Shea
The Cost Assessment and Performance Evaluation (CAPE) recommends “Identify and eliminate remaining uses of ESLOC as metric for cost and schedule estimation of software/software-intensive programs” by fiscal year 2022. The Defense Innovation Board (DIB) is proposing a change of how software is procured and developed within the Department of Defense (DoD).
Strategy Track
Strategic Cost Estimation: China’s Military Modernization and Cost Constraints
Strategy Track (SY01)
Jack Bianchi
China’s military modernization efforts appear unrelenting but China’s defense-related cost constraints remain underappreciated by Western strategists and defense planners. Understanding Chinese defense budget limitations and resource tradeoffs is necessary for accurately estimating China’s future force structure and for aiding Allied policymakers in developing competitive strategies that exploit Chinese resource scarcity. This presentation addresses methodologies for estimating China’s platform costs and future force structure, and offers preliminary strategic-level insights and conclusions.
Keywords: Budgeting, International, Statistics
Breached: Why a Cybersecurity Estimate should Start with a Cyber Attack Response
Strategy Track (SY02)
Jeremy Eden
ICEAA’s encouragement of the community to expand into the areas of cyber and data sciences has been popular. However, full understanding the challenges of these fields to better serve clients is a Texas size problem. Well prepare to form a posse with me and ride through the stages, and costs, of cyber-attack.
Keywords: Government, IT, Cyber, Cybersecurity, Cost Estimate
A 3 Market, 10 Dimension Trade
Strategy Track (SY03)
Doug K. Howarth
Any person, company, or government working across three or more related markets decides how to divide the costs between them. Often decision makers give little thought as to how those resource splits need to work when working in conjunction to a common goal. Using the example of the Prompt Global Strike (PGS) initiative, this paper studies ways to optimize costs in three connected markets (air-to-surface missiles, bombers, tanker aircraft) across ten dimensions.
Are you Smarter than an Algorithm?
Strategy Track (SY04)
Andy Prince
Cost analysts rely on mathematical algorithms, experience, and subjective assessments to develop cost estimates. However, these analysts often disagree over what is more important: statistically derived algorithms; or experience and judgment. To try to answer this question cost estimating professionals were surveyed for their expert judgment on the complexity and new design values for 15 NASA science missions. The results may or may not be surprising, but will surely be interesting.
Keywords: Methods, Modeling, Parametrics, Variables
Too Good to be Forgotten: Strategies For Protecting Institutional Knowledge
Strategy Track (SY05)
Nicole Robertson
In today’s competitive market finding ways to sustain and preserve organizations ability to outperform competitors is paramount to success. The institutional knowledge (the business and operational knowledge as well as organizational and political contacts) lost due to restructuring, employee turnover, and retirement is detrimental to the success of our client relationships. This paper explores strategies to preserve institutional knowledge so that we may serve our clients more efficiently and effectively in the future.
Keywords: Communication, Performance Management, Program Management, Institutional Knowledge, Knowledge Management, Client Relationships, Cross-training, Customer Service
How to Win Friends & Influence Programs
Strategy Track (SY06)
Britt Staley
Summary Not Approved for Public Release