Analytical Methods Track
Design to Cost: What it is and Why You Should Care
Analytical Methods Track (AM01)
Karen Mourikas
Henry Apgar
Lisa Colabella
Design to Cost (DTC) has been around since the 1970s, falling in and out of favor. But what exactly is DTC? And will it help control program or product costs? Googling it yields various definitions, sometimes contradictory. Examples fluctuate from large defense programs to mass-produced commercial components. Implementation differs depending upon company, life-cycle phase, and objectives. This presentation is the culmination of an effort by the SoCal ICEAA Chapter to educate our community on DTC.
Keywords: Cost/Benefit Analysis, Decision Analysis, Methods, Parametrics, Project Controls, DTC
AM01 – Mourikas – DtC What it is and Why – ppt
Accuracy, Precision and Uncertainty in Cost Estimates
Analytical Methods Track (AM02)
Timothy P. Anderson
Cost proposals for large government contracts are necessarily very detailed and typically highly precise, expressed down to the nearest dollar. This presentation explores the differences between precision and accuracy in cost proposals in the context of uncertainty and cost growth. Moreover, the presentation is interactive, with thought-provoking questions peppered throughout, providing numerous opportunities for audience participation for improved knowledge and retention.
Keywords: Precision, Accuracy, Uncertainty
AM02 – Anderson – Accuracy, Precision and Uncertainty – ppt
Innovative Risk-Driven Contract Pricing Strategy
Analytical Methods Track (AM03)
Brian Flynn
Robert Nehring
Andi Hylton
*2023 Best Paper: Analytical Methods Category -and- Best Paper Overall
This paper presents a framework or scoring matrix that quantifies contract risk using a numerical evaluation of the many factors that make or break a program, such as: experience of the contractor, stretch in technology, solidity of requirements, and degree of competitive procurement. The framework, in turn, is made operational by leveraging benchmarks from over 60 U.S. contracts, enabling data-driven selection of contract type, incentives, and share lines for use in evaluating future contract prices.
Keywords: Contracts, Strategy, Pricing Approach, Risk
AM03 – Flynn – Innovative Risk-Driven Contract Pricing Strategy – ppt
AM03 – Flynn – Innovative Risk-Driven Contract Pricing Strategy – paper
One Number to Correlate Them All – Efficient Implementation of Correlation
Analytical Methods Track (AM04)
Anh (Anne) Harris
Karen McRitchie
Christian Smart
Correlation is a key consideration in cost and schedule risk analysis, as its exclusion causes significant underestimation of uncertainty. When assigning values in the absence of functional correlation, this can be accomplished by considering every WBS estimate. However, this can be time consuming for a detailed estimate. In this presentation, we discuss an alternative method that uses a single value, which offers significant time savings and discuss its implementation in the SEER model suite.
Keywords: Monte Carlo, Parametrics, Risk, Uncertainty
AM04 – Harris – One Number to Correlate Them All – ppt
AM04 – Harris – One Number to Correlate Them All – paper
Incorporating Risk into Analysis of Alternatives Results: A Novel Methodology
Analytical Methods Track (AM05)
Brittany Clayton
Lauren Mayer
While Analysis of Alternatives (AoAs) commonly calculate the expected cost and effectiveness of alternatives, risks to cost overruns and degraded mission effectiveness are analyzed separately. RAND developed and employed a novel methodology for incorporating these risks into cost and effectiveness results. We collaborated with each AoA working group to identify, quantify and map risks to existing cost and effectiveness model variables, allowing for results to be displayed as a comparison of the alternatives’ risk-adjusted cost-effectiveness.
AM05 – Mayer – AoA Risk Methodology – ppt
Using Bayes’ Theorem to Develop CERs – Extending the Gaussian Model
Analytical Methods Track (AM06)
Christian Smart
David Jo
Bayes’ Theorem is a mathematical method to combine prior experience with new data and is extremely important in leveraging limited information, which is often the case in cost estimating. This paper is an extension of a previous ICEAA paper that dealt with the application of Bayes’ Theorem to cost estimating. In this update we show how the Bayesian approach for linear models can be extended for different, more realistic assumptions.
Keywords: Bayesian, Bias, Methods, Parametrics, Regression, Statistic
AM06 – Smart – Using Bayes Theorem to Develop CERs – ppt
AM06 – Smart – Using Bayes Theorem to Develop CERs – paper
Alternative Risk Measures for Determining Program Reserves
Analytical Methods Track (AM07)
Louis Fussell
NASA project managers hold cost and schedule reserves equal to a 50% joint confidence level and the managing directorates hold reserves to a 70% confidence level. These joint confidence levels are quantile risk measures. This paper discusses the drawbacks of quantile risk measures and proposes the use of super quantiles for determining project reserves. Several projects are analyzed and a comparison of risk measures is presented.
Keywords: Monte Carlo, Program Management, Risk Management, EVM & Scheduling, Risk, Planning
AM07 – Fussell – Alternative Risk Measures – ppt
AM07 – Fussell – Alternative Risk Measures – paper
The Space Between Us: A Novel Collaborative Spacecraft Estimating Framework
Analytical Methods Track (AM08)
Alex Wekluk
Benjamin Truskin
Government space acquisitions are renowned for their complexity and can suffer from notable cost and schedule overruns. The space cost community is composed of disparate organizations employing different cost estimating methods, even when the same industry partners are building the spacecraft. This paper leverages a Technomics internally-developed space estimating framework, industry-released Cost Estimating Relationships from multiple agencies, and NRO CAAG external collaboration experience to examine and contrast previously incompatible methods, and improve synchronization.
AM08 – Wekluk – The Space Between Us – ppt
AM08 – Wekluk – The Space Between Us – paper
Measuring Schedule Uncertainty
Analytical Methods Track (AM09)
Tommie (Troy) Miller
The growing popularity of Joint Cost & Schedule Analysis has highlighted the need for quality SRAs. SRAs require that uncertainty distributions be assigned to all schedule activities. The most popular technique for making this assignment employs a binning structure that assigns each task to a particular bin. Most binning structures have little theoretical support. This paper attempts to improve on this technique by discussing how uncertainty determination and allocation is conducted within the Sentinel program.
AM09 – Miller – Measuring Schedule Uncertainty – ppt
Spread Thin: Managing Coefficient of Variation in Monte-Carlo Based Cost Models
Analytical Methods Track (AM10)
Stephen Koellner
Coefficient of Variation (CV) can be utilized to determine whether sufficient uncertainty is captured in Monte-Carlo based estimates. This topic explores common barriers to capturing program level risk using the interpretation of a WBS as a linear combination of distributions. A WBS CV equation is provided to model perturbations of a baseline case and then randomized WBSs are generated to analyze CV at scale. Estimators can apply these insights to improve program estimate risk calculations.
AM10 – Koellner – Spread Thin Managing CV – ppt
Data Science Track
Follow the Money in Government Estimates
Data Science Track (DS01)
Geoffrey P. Boals
Struggle with finding the correct source for actuals? Contracts for RDTE items are only bundles with no breakouts for the data you need? This session we will assist estimators in how to use government contract, Military Interdepartmental Purchase Requests (MIPRs), and other sources to find the relevant data needs for a strong basis for estimating. The session will detail a proposed data structure to store relevant data from the separate data sources to help power your estimates.
Keywords: Budgeting, Data Collection, DOD/MOD, Government, Variables
DS01 – Boals – Follow the Money in Government Estimates – ppt
DS01 – Boals – Follow the Money in Government Estimates – paper
“Cracking the Code” : Demystifying Programming Languages for Data Analytics
Data Science Track (DS02)
Kyle Ferris
John Maddrey
In recent years, buzzwords such as “Machine Learning” and “Artificial Intelligence” have made their way into the cost community’s vocabulary. While data science concepts such as these certainly advance the cost estimating industry, their significance is lost when analysts find themselves unfamiliar with fundamental programming principles. This presentation aims to demystify and compare commonly used programming languages such as R and Python, with the purpose of helping analysts who wish to develop coding expertise but don’t know where to begin.
Keywords: Data-Driven, IT, Methods, Modeling, Software, Statistics, Programming, Coding, Data Science, Data Analytics, Software Development
DS02 – Ferris – Cracking the Code – ppt
Level up your Cost Analysis with Data Science
Data Science Track (DS03)
Sarah Green
How can developments in the field of data science help us transform many of our analytics capabilities in the Cost Community? AFCAA is leveraging the AF’s cloud-based platform VAULT and state-of-the-art toolsets to in order to increase process automation, data curation and predictive models, and therefore significantly increase both quality and efficiency of our data analytics. This presentation will include a live demonstration including a demo of custom cost applications, tools and dashboards to help you in your everyday estimating- come see how the cost community is being revolutionized!
DS03 – Green – Level up your Cost Analysis – ppt
Web Applications for Cost Analysis Use: R Shiny vs Python Plotly Dash
Data Science Track (DS04)
Eric J. Hagee
Christine K. Clay
Web applications are a new tool for the cost community. Here, we compare and contrast two common web application tools, R Shiny and Python Dash. We develop a dashboard in each tool to display various USA government inflation measures and compare/contrast the use of either tool. Our findings are summarized, future thought points identified, and relevant background on the tools, inflation measures, use cases, and tool implementation provided.
Keywords: Communication, Data-Driven, R, Python, Shiny, Dash, Data Visualization, Web Application, Tools Comparisons, Dashboards
DS04 – Hagee – Web Applications for Cost Analysis Use – ppt
Using Machine Learning to Improve Cost Data Normalization
Data Science Track (DS06)
Bryan Mariscal
Cost data normalization is time intensive, but remains critical to building a defensible cost model. SSC’s Unmanned Space Vehicle Cost Model (USCM) database contains over 120 normalized datasets that span nearly a 50-year history. Developing machine learning models from the older data that generalize well to newer data would be difficult due to data obsolescence. However, we have identified an example use case with promising initial results that will be examined in detail.
Keywords: Data-Driven, Methods, Modeling, Machine Learning, Algorithms, Data Normalization, Process Improvement
DS06 – Mariscal – Using Machine Learning to Improve Cost Data – PPT
Automating the Data Preparation Process using R Programming
Data Science Track (DS08)
Zachary West
One of the most challenging aspects of the cost estimating process is the collection and consolidation of data prior to carrying out any analysis. Initial data collection and preparation steps are often manual, messy, and time consuming. I plan to present an automated solution using R Programming that efficiently collects and merges structured data reports, organizes and prepares the relevant data, and exports the results to support cost estimating efforts.
DS08 – West – Automating the Data Preparation Process using R – ppt
IT& Cloud Computing Track
Interesting Results from EVAMOSC or “Wow There is a Lot of O&S Data”
IT & Cloud Computing Track (IT01)
Daniel Germony
EVAMOSC is an new OSD-CAPE developed database which ingests and normalizes Operations & Sustainment (O&S) data on over a thousand weapon systems for the DoD. In this presentation, the EVAMOSC team will review interesting results it has found related to O&S data using the hundreds of millions of datapoints it has ingested and normalized. Pareto rule? Try hyper-Pareto rule! How did COVID impact vehicle maintenance? Not how you would have expected. Are field units experiencing inflation/escalation? All this and more.
IT01 – Germony – Interesting Results from EVAMOSC – ppt
Applying Simple Function Point Analysis to an 804 Rapid Acquisition Program Cost
IT & Cloud Computing Track (IT02)
Bob Hunt
Rainey Southworth
Chad Lucas
When software is developed under the Middle Tier of Acquisition (Section 804), Adaptive Acquisition Framework (AAF), these programs tend to be actual agile software development programs. These programs focus on delivering capability in a period of 2-5 years with rapid prototypes and rapid fielding with proven technology. Sizing a Software development program remains a major challenge. This paper outlines the advantages and issues associates with using Simple Function Point analyses to size and estimating the cost for a Major Rapid Acquisition Software Program.
Keywords: Agile, Acquisition, Software, Cost Estimation
IT02 – Hunt – Applying Simple Function Point Analysis to an 804 – paper
IT02 – Hunt – Applying Simple Function Point Analysis to an 804 – ppt
Cloud-Based Machine Learning in the DoD Environment
IT & Cloud Computing Track (IT03)
Conner Lawston
Bryan Eckle
Follow up to the 2022 Presentation “Supercharging Machine Learning with Cloud Computing.” This project has made huge strides in the past year- having made over 5,000 predictions since February, all 100% autonomously and within the DoD ADVANA environment. There have been numerous improvements (like increasing ML accuracy via PCA, & speeding up calculation time with SPARK) and lessons learned as we expanded into production for the Joint Chiefs, MDA, Navy, Marine Corps, and more!
Keywords: Modeling, Machine Learning, Cloud, Robotic Process Automation
IT03 – Lawston – Cloud Based Machine Learning – ppt
Addressing CI/CD with Automated Testing in Your Software Estimate
IT & Cloud Computing Track (IT04)
Arlene F. Minkiewicz
Agile development and DevOps/DevSecOps practices are becoming increasingly common in software development projects in commercial, government and government contractor organizations. Three practices arising are Continuous Integration, Continuous Delivery and Continuous Deployment. Automated testing is clearly an investment, but one that promises to lead to productivity and quality improvements. This paper presents the general concept of automated testing focused on practices highlighted above. The cost impacts of employing automated testing practices will be demonstrated and discussed.
Keywords: Software, Agile Software Development, DevSecOps
IT04 – Minkiewicz – Automated Testing – ppt
IT04 – Minkiewicz – Automated Testing – paper
Comparing the Cost of Cloud Services: AWS vs Azure
IT & Cloud Computing Track (IT05)
Gunnar Nichols
Patrick Casey
Sergey Kozin
Cloud Services continue to play a growing role in how the internet is used to provide capabilities to users. Amazon Web Services and Azure are the two biggest Cloud Service Providers on the market today. This research investigates cost differences for cloud services comprised of comparable virtual machines for workloads with defined computational requirements. This analysis is utilized to characterize scenarios where cost savings should be considered when supporting a Cloud Service Provider decision.
Keywords: Data-Driven, IT, Software
IT05 – Nichols – Comparing the Cost of Cloud Services – ppt
Shining Rays of Light & Savings on Cloud Portfolios: An Important Advance!
IT & Cloud Computing Track (IT06)
Kenneth Rhodes
Alex Wekluk
RJ Krempasky
Existing cloud cost estimating and pricing tools have at least two significant drawbacks: 1) they require many assumptions, therefore a detailed understanding of current/future architectures and 2) they do not provide insights required to manage costs. Technomics developed a groundbreaking alternative that will revolutionize how cloud costs are estimated and managed. This paper describes our parametric-based toolset, which enables cloud lifecycle costs estimation based on known programmatic design requirements and cost reduction and efficiency analysis.
IT06 – Rhodes – Shining Rays of Light – ppt
Modeling & Case Studies Track
Budget Execution and Margin Simulation (BEAMS)
Modeling & Case Studies Track (MC01)
Erik Burgess
ODNI works with organizations in the Intelligence Community to submit multibillion-dollar budget requests to Congress, based on cost estimates that are not exact. For an agency portfolio, this study quantifies the impact of budgeting to higher or lower confidence levels. By modeling the interactions among programs and the mechanics of the annual budget closure process, we show that estimates somewhat higher than 50 %tile are the “sweet spot” between minimizing growth and maximizing missions delivered.
Keywords: Budgeting, Space
MC01 – Burgess – Budget Execution Margin SImulation – ppt
Foundational Cost Models
Modeling & Case Studies Track (MC02)
Paul Broaddus Franklin Jr.
This study will unveil more than forty-five previously unpublished simple and learning-augmented cost estimating relationships for common construction materials and equipment. These relationships and other data allow us to investigate the hypothesis that the error distribution of conventional building components follow a skew exponential power distribution rather than the lognormal one usually applied to defense estimates. We also examine whether backcasting to the intercept can isolate approximate labor cost underlying composite expense data. This paper shares those relationships in the hope of expediting cost estimating from the ground-up.
Keywords: Parametric Estimation, Machine Learning, Construction, Infrastructure, Program Management, Operations Research, Defense
MC02 – Franklin – Foundational Cost Models – ppt
CSI EU (Cost Scene Investigation – European Union)
Modeling & Case Studies Track (MC03)
Doug Howarth
*2023 Best Paper: Modeling & Case Studies Category
In 2000, several Airbus units from European Union member countries launched the Airbus A380. The program was projected to sell 1250 units with a development cost of €9.5 billion but shut down after selling 251 copies, as costs more than doubled. This paper examines its causes and how the entire debacle might have been prevented, as all the information needed to assess its viability existed before it started.
Keywords: Development Costs, Sales, Demand
MC03 – Howarth – CSI EU – Paper
Trouble With the Curve: Engineering Changes and Manufacturing Learning Curves
Modeling & Case Studies Track (MC04)
Brent M. Johnstone
Engineering changes pose a dilemma for estimators: If learning curves assume cost improvement due to repetitive build, what happens when that repetition is interrupted by a change of task? Design changes are common occurrences, but rarely addressed in learning curve literature. This paper addresses how to analyze an engineering change by breaking it into its pieces and outlines techniques to calculate the reversionary impact on the learning curve to derive the estimated cost of change.
Keywords: Labor, Learning Curves, Methods, Modeling, Operations, Manufacturing, Engineering Changes
MC04 – Johnstone – Trouble With the Curve – Paper
MC04 – Johnstone – Trouble With the Curve
Developing a Schedule Model From a Cost Modeler’s Perspective
Modeling & Case Studies Track (MC05)
Benjamin Kwok
Daniel Newkirk
Cost estimators need to utilize and develop estimating methods for project components beyond cost. Schedule is one of those components. The team that brought you USCM is now applying their model building experience to the world of schedules. This journey has yielded some exciting new methods, products, and processes. It has also brought challenges as we could not address schedule exactly as we would with cost.
Keywords: Data Collection, Data-Driven, Modeling, Schedule, Databases, Metrics, Django, Python
MC05 – Kwok – Developing a Schedule Model – PPT
MC05 – Kwok – Developing a Schedule Model – paper
All in the Hierarchy: Meta-Estimators to Standardize Work Breakdown Structures
Modeling & Case Studies Track (MC06)
Maura Anne Lapoff
Patrick Shelton
The National Nuclear Security Administration (NNSA) has demonstrated how to use Machine Learning and Natural Language Processing to map disparate cost data to a standard, high-level Work Breakdown Structure (WBS). However, mapping data to deeper WBS levels becomes increasingly complex due to the hierarchical relationship between levels, rendering common machine learning models inadequate. Here we demonstrate how to implement a Hierarchical Classification Machine Learning scheme to map multi-level, hierarchical cost data to a common WBS.
Keywords: Data-Driven, Government, Modeling, Hierarchical, Construction, Classification
MC06 – Lapoff – All in the Hierarchy – paper
MC06 – Lapoff – All in the Hierarchy – ppt
The Cost of Bureaucracy
Modeling & Case Studies Track (MC07)
Zachary Matheson
Jenna Vandervort
Daniel Mulrow
The National Nuclear Security Administration (NNSA) has separate management methods for capital construction projects, with enhanced requirements for larger projects. Naively, one might expect that enhanced requirements might increase program management costs and possibly impact schedule. However, as the enhanced management requirements are tied to a dollar threshold, comparing costs is not straightforward. In this paper, we share results of a regression analysis that developed separate cost and schedule regressions for each management method and a comparison of the costs by management type.
Keywords: Decision Analysis, Early Cost, Government, Parametrics, Regression, Statistics
MC07 – Matheson – Cost of Bureaucracy – ppt
MC07 – Matheson – Cost of Bureaucracy – paper
Unprecedented – Accurate Estimating in the Hypersonics Era
Modeling & Case Studies Track (MC08)
Eric Sick
Christian Smart
Many cutting-edge programs and emergent domains, such as hypersonics, have few direct historical precedents. This presents challenges in estimating the cost of such programs at the system level. A key insight is that many of the components on these systems have been implemented on numerous other historical programs, so one way to successfully tackle this challenge is by estimating at a component-level Work Breakdown Structure. This paper provides comparative examples of accurate estimates for aerospace advanced technology programs, including commercial space and government hypersonics programs
Keywords: Cost/Benefit Analysis, Early Cost, Methods
MC08 – Sick – Unprecedented Accurate Estimating Hypersonics – ppt MC08 – Sick – Unprecedented Accurate Estimating Hypersonics – paper
Systems Engineering Cost Estimation Aligned with MIL-STD-881F
Modeling & Case Studies Track (MC09)
F. Gurney Thompson III
Vivian Tang
This presentation reviews advancements in estimating top-level Systems Engineering efforts. We will propose a new model structure for alignment with MIL-STD-881F definitions, and improved scope control and output usefulness. We will discuss the drivers of effort distribution within systems engineering activities and resources, including system criticality level, safety risks, logistics scenarios, human factors considerations, technology readiness level and more. This presentation also discusses ongoing data analysis efforts to support model validation and prototype demonstration.
MC09 – Thompson – Practical Estimating of Hardware Lifecycle – ppt
Cost Estimating: The Estimator’s Guide to a Parametric Universe
Modeling & Case Studies Track (MC10)
Taryn Anne Reilly
Imagining tomorrow’s cost estimating using parametric methods. Experience the advantages of parametric estimating throughout the opportunity and program lifecycle. Learn ways to significantly improve your estimating efficiency, accuracy and traceability through historical data calibration. Develop cost early so it influences the design process. Understand the importance of a standard WBS and curating completed program data to build calibrated models. Augment information in the Cost Community through data sharing, improving publicly available sources, methods, models.
MC10 – Reilly – Estimators Guide to a Parametric Universe – ppt
Processes & Best Practices Track
Shortfall Analysis – Creative Approaches to Problem Quantification
Processes & Best Practices Track (PB01)
George Bayer
Brian Carroll
Government acquisitions start with problem identification as a means of solving public sector problems or “shortfalls” and inefficiencies with technical hardware and software solutions. To justify government business cases, analysts are tasked to (1) define the problem, (2) identify impacted stakeholders, and (3) creatively quantify and monetize impacts to operations and services. Using a proven approach, the team demonstrates how to effectively perform shortfall analyses and monetize the largest underlying value to justify business cases.
Keywords: Cost/Benefit Analysis, Decision Analysis, Process Engineering, Regression, Statistics, Business Case, Shortfall Analysis, Critical Thinking, Root Cause Analysis, Cost Avoidance, Monetization
PB01 – Bayer – Shortfall Analysis – ppt
PB01 – Bayer – Shortfall Analysis – paper
You’ve Invested, Now Sustain it: Insights into the Sustainment Review Process
Processes & Best Practices Track (PB02)
Adam Kidwell
John Liss
Alex Bonich
Congress, in NDAA 2017, established Sustainment Reviews to assess the planning and execution of Major Defense Acquisition Program (MDAP) Product Support strategies. Readiness is a critical aspect of Defense Operations, and rising supply chain costs highlight the fact that effective planning, budgeting, and execution of a program’s Sustainment Strategy is crucial to its long-term success. This paper provides an objective, experience-based perspective of Sustainment Reviews and best practices to ensure successful outcomes.
Keywords: DOD/MOD, Life Cycle, Performance Management, Sustainment, Congress, Policy, Readiness
PB02 – Kidwell – Youve Invested Now Sustain – ppt
PB02 – Kidwell – Youve Invested Now Sustain – paper
Educating Future Cost Engineers in the Space Station Design Workshop (SSDW)
Processes & Best Practices Track (PB03)
Fabian Eilingsfeld
Nicolaus Millin
The space sector seeks young cost engineers who can work in a concurrent engineering environment. One means for scouting young talent is the annual Space Station Design Workshop (SSDW) at the University of Stuttgart, Germany. Two competing teams (size: 20) each must design a space station within one week. SSDW’s concurrent design facility (CDF) format offers students a unique opportunity to learn in a hands-on, international, and interdisciplinary environment. The use of cost estimating tools for teaching at SSDW is described. Lessons learned from the 2022 post-workshop survey are presented.
Keywords: Space, Education, Young Talent
PB03 – Eilingsfeld – Educating Future Cost Engineers – ppt
From Software CONOPS to ROM Estimate in Six Easy Steps
Processes & Best Practices Track (PB04)
Carol Dekkers
Dan French
Creating software estimates using a Concept of Operations (ConOPS) document is a challenge for software estimators due to the low fidelity of the requirements. This presentation will illustrate how to create a defensible ROM (Rough Order of Magnitude) Estimate using the new IFPUG software functional size measure. Carol and Dan will also explain how to use this size metric to develop a ROM Cost and Schedule by leveraging various software estimating techniques.
PB04 – Dekkers – From Software CONOPS to ROM Estimate – ppt
PB04 – Dekkers – From Software CONOPS to ROM Estimate – paper
Interviewing Subject Matter Experts
Processes & Best Practices Track (PB05)
Melissa Stone
As estimators, we often must obtain technical information from Subject Matter Experts (SMEs) in order to provide thorough and accurate analysis. But sometimes getting the information you need is easier said than done. This presentation discusses the steps for interviewing SMEs in order to make the most of their time and your time. It explores the benefits of building a cooperative relationship of mutual respect and the “dos” and “don’ts” of SME consultations.
PB05 – Stone – Interviewing Subject Matter Experts – ppt
NATO Transformation Command: Launching a Cost Estimating Capability
Processes & Best Practices Track (PB06)
Candace Mahala
Ryan Feeks
Michael Thompson
After over 70 years of establishing NATO, the Transformation Command recognized a cost estimating gap in its capabilities. They launched the development of a new cost estimating capability, for the pre-concept and concept stages remaining solution-agnostic at programmatic level. This is the story of how to grow and cultivate cost estimating from program inception, establishing the fundamentals, integrating with other NATO organizations’ cost communities and developing a closed-loop cost track for robust data.
PB06 – Mahala – NATO Transformation Command – ppt
Managing Schedule Risk Expectations During Program Execution
Processes & Best Practices Track (PB07)
Patrick Malone
Program managers use Integrated Master Schedules to complete projects. It is a time phased logical view of the work and activity network. Organizations identify risks using rule-based risk taxonomies; or selecting risk exposures and impacts from tables grading them low to high. These assessments are carried forward but not always implemented in the schedule. We investigate schedule risk methods to provide an approach to more accurate forecasting resulting in a process to implement across programs.
Keywords: Schedule, Risk, Program Management, Analysis
PB07 – Malone – Managing Schedule Risk Expectations – paper
PB07 – Malone – Managing Schedule Risk Expectations – ppt
Take the Blue Pill!
Processes & Best Practices Track (PB08)
Javier Provencio
Chris Hutchings
Cassidy Shevlin
Digital ecosystems are distributed technical systems that are scalable, sustainable, and often self-organizing; much like ecosystems found in the natural world. We will offer insight into what a digital ecosystem encompasses, along with relatable examples, and then explicate the virtues and value for our community in areas such as on-demand design to cost, MBSE, in-service support, and simulation. Finally, we will examine emergent relevant and significant use cases.
Keywords: Cost Management, Cost/Benefit Analysis, Data-Driven, Decision Analysis, Infrastructure, Methods, Operations
PB08 – Provencio – Take the Blue Pill – paper
Accidental Proof of Calculus for Cost Analysts
Processes & Best Practices Track (PB09)
Jack Snyder
Brittany Smith
Ever get derailed chasing your tail checking calculations, only to discover the cross check initiating the train wreck was flawed to begin with? Join us on an unflattering journey that began with a quick integral cross check that snowballed into the perfect proof for why calculus is a requirement for cost analysts. MATLAB, calculator, or by hand were the academic tools of the calculus trade, we’ll also share how we use Excel for integral calculus.
PB09 – Snyder – Accidental Proof of Calculus – ppt
Methodology for Assessing Reasonableness of Large Scientific Facilities’ Costs
Processes & Best Practices Track (PB10)
Raymond D. Woods
Valerie Rockwell
Aerospace analysts developed a methodology to address the questions and challenges that arise when looking at reasonableness of large scientific facilities’ costs. This methodology is an iterative process of developing tailored reasonableness ranges and systematically addressing lower WBS levels to refine estimates. Aerospace will share their experience in using this method, when it is applicable, the benefits of implementing this, and associated products – including a specialized ‘Scorecard’ – to provide insights to program management.
PB10 – Woods – Assessing Reasonableness of Large Scientific Facilities – ppt
Software & Agile Track
NLP: A New Approach to SFP Estimation
Software & Agile Track (SA01)
David H. Brown
The need for data-driven methods for predicting software size has never been greater. This is especially true for agencies that rely on SME opinion or T-shirt based sizing as the primary method for estimating the size and cost of agile-developed software. This paper offers an alternative method — use of Natural Language Processing to automate the estimation of Simple Function Point counts – that offers two important advantages: 1) consistency across multiple counts and 2) speed.
Keywords: Data-Driven, Early Cost, Software, Agile, SFP, NLP
SA01 – Brown – NLP A New Approach – ppt
SA01 – Brown – NLP A New Approach – paper
The Symbiotic Nature of Requirements Quality and Software Cost Estimation
Software & Agile Track (SA02)
Colin Hammond
*2023 Best Paper: Software & Agile/IT & Cloud Computing Category
It is widely recognised that high quality software requirements lead to reliable functional size estimates, what is less apparent is the potential contribution of functional sizing to help assess and improve the quality of requirements. This paper looks at the observations from assessments of software user stories using our NLP (Natural Language Processing) requirements analysis tool, and how this symbiotic relationship is revealed and enhanced with automated analysis tooling.
Keywords: Cost/Benefit Analysis, Data-Driven, NLP, Quality, Sizing, Function Points, Cost Reduction, Reduce Risk, Reduce Waste
SA02 – Hammond -Symbiotic Relationship between Requirements Quality – Paper
SA02 – Hammond -Symbiotic Relationship between Requirements Quality – ppt
A Novel Approach for Early Phase Agile Software Estimating and Sizing
Software & Agile Track (SA03)
Wilson Rosa
Sara Jardine
This study investigates how well three high-level size measures (Capability Gaps, Capabilities, and Epics) accurately predict total agile development effort and schedule at the earliest acquisition phase. These measures were obtained from early program documents including Mission Needs Statement, Concept of Operations, and Release Roadmap. Analysis of Variance and Goodness-of-Fit were used to introduce six new estimation models based on data from 20 agile projects implemented between 2014 to 2022 in the DHS and DoD.
Keywords: Data-Driven, IT, Parametrics, Regression, Software, Agile Software Development, Cost Estimation, Mission Needs Statement, Analysis of Alternatives, Rough Order Magnitude, Capability Gaps, Capabilities, Epics, Software Acquisition, Software process, Time Estimation
SA03 – Rosa – Novel Approach for Early Phase Agile – ppt
SA03 – Rosa – Novel Approach for Early Phase Agile – paper
Scheduling Agile-Fall – Best Practices
Software & Agile Track (SA04)
Sam Kitchin
Ryan Webster
A high-quality schedule can improve a program’s ability to clarify requirements, predict cost, and communicate effectively by creating a common denominator that links complex efforts across a project. Many Agile programs operate using a hybrid waterfall approach making a properly built schedule critical for understanding the relationship between agile development and government requirements. This presentation will demonstrate best practices for schedule construction in an Agile environment and its implications on cost, delivery, and performance.
SA04 – Kitchin – Scheduling AgileFall – ppt
Tales from the Trenches – Challenges of Data Analytics in a DevSecOps Program
Software & Agile Track (SA05)
Alex Smith
The large databases of information captured in software planning tools within DevSecOps environments contain a wealth of information. However, this data is user-input and not created with data analysis in mind, which results in several challenges when attempting to analyze. This presentation walks through the real-world process that resulted from collecting, cleaning, reviewing, and analyzing over 20,000 observations of agile software data to facilitate DevSecOps planning and road-mapping activities for a program entering the Software Acquisition Pathway.
Keywords: Government, IT, Story Points, DevSecOps, Data Analysis
SA05 – Smith – Tales from the Trenches – ppt
How Efficient are Your Efficiency Levers?
Software & Agile Track (SA07)
Eric van der Vliet
For application service projects an important aspect is making services more efficient by means of efficiency levers like juniorization, test automation, robotics, etc. The challenge is to combine these efficiency improvements and determine the effort and cost impact during the maintenance period. This presentation presents lessons learned with respect to applying efficiency levers in large application service projects and shows an estimation tool that supports a combination of multiple efficiency levers with different impacts and durations and calculates the monthly impact.
Keywords: Budgeting, Data-Driven, Parametrics, Software
SA07 – vander Vliet – How Efficient are Your Efficiency Levers – ppt
Technical Innovations Track
Ticking Time Bomb of Telework: Communication and Time Management
Technical Innovations Track (TI01)
Jennifer Aguirre
Shannon Cardoza
Annie Bagay
Today’s evolving work environment has demonstrated the importance of emphasizing soft skills in the workplace. Employers must seek candidates with high interpersonal abilities in conjunction with technical expertise. Hard skills often dominate in job-related training, while critical foundations like the ability to communicate and manage workload effectively are often overlooked. This presentation will focus on expanding a person’s marketability by discussing techniques to open these communication pipelines and increase productivity despite an ever-growing workload.
Keywords: Communication, Program Management
TI01- Aguirre – Ticking Time Bomb of Telework – ppt
AI and Cost Estimation: Data Science’s Expanding Role in Cost Estimating
Technical Innovations Track (TI03)
Kevin McKeel
Daniel Harper
Greg Forman
Data Science has taken on an expanded presence in Cost Analysis. E.g., NLP is used to automate functional software sizing in commercial models. Data Science methods and tools such as NLP can be also used to extract data to identify spending trends and agency funding. We will present an overview of modern usages of data science, to include Machine Learning, AI and data visualization. We will also survey cost and budget data sources which are publicly available and ripe for analysis using data science methods.
Keywords: Artificial Intelligence, Data Science, Machine Learning, NLP
TI03 – McKeel – AI and Cost Estimation – ppt
Software Estimation Using Functional Size Derived From User Stories
Technical Innovations Track (TI04)
Karen McRitchie
Esteban Sanchez
Alton Ng
Natural language processing and ISO standard functional size metrics offer an opportunity for an improved approach to sizing and estimation directly from user stories. This presentation will offer an overview of how functional size generated by ScopeMaster can be used for directly for estimation in SEER-SEM. This approach uses either Cosmic or Simple Function Points and considers the uncertainty around sized, ambiguous and potentially missing requirements.
Keywords: Agile, Functional Requirements, Story Points, Sizing
TI04 – McRitchie – Software Estimation Using Functional Size – ppt
Software Maintenance and Software Obsolescence: A Tale of Two Cost Drivers
Technical Innovations Track (TI06)
Sanathanan Rajagopal
Software Obsolescence is one of the key issues in the modern warfare technologies especially in the cyber space where vulnerabilities of obsolete software are exploited. Understanding these vulnerabilities, predicting, managing them cost effectively are now key priorities of any key defence contracts. In this original research paper the author looks at the relationship of software maintenance and its key cost drivers with software obsolescence. Author will further explore the appropriate mitigation strategy and resolution approach for managing software obsolescence while providing an insight o
Keywords: Cost Management, Parametrics, Software, Software Maintenance, Software Obsolescence
TI06 – Rajagopal – Software Maintenance and Obsolescence – ppt
Wouldn’t It Be Nice to Use All of Your Data Points: An Introduction to NICE
Technical Innovations Track (TI07)
Steven Read
*2023 Best Paper: Processes & Best Practices/Technical Innovations Category
The difficulty of having non-homogeneous data when fitting a CER can be addressed using data imputation. A Non-linear Iterative Constrained Estimator (NICE) is an approach to both data imputation and CER fitting that considers the whole model and uses the WBS relationships to recalibrate systems of equations for a better fit. This presentation will explain what NICE is, how it works, and begin to explore what problems this method can solve.
Keywords: Data-Driven, Methods, Modeling
TI07 – Read – Wouldn’t It Be Nice – ppt
TI07 – Read – Wouldn’t It Be Nice – Paper
VEGA: Shining Light on the Battles of Data Collection and Management
Technical Innovations Track (TI08)
Abby Schendt
Omar Akbik
Raymond Vera
Every cost organization battles both the collection and management of cost, schedule, technical, and programmatic data. This paper describes how the Office of Cost Estimating and Program Evaluation (CEPE) at NNSA has built and matured this capability since the office’s inception in 2015. This paper will present seven years of lessons-learned that will pay dividends for any cost organization looking to optimize how it collects and manages mission critical data.
Keywords: Data Collection, Data-Driven, Government, Database, Data Storage, Policy, Data Structure, Data
Trending Topics Track
Hilbert’s Problems III: The Post-Pandemic Cost Community
Trending Topics Track (TT01)
Peter Braxton
*2023 Best Paper: Data Science/Trending Topics Category
Upended by the pandemic, the cost community stands at an inflection point. Revolutions in telework and data science open doors to collaboration and innovation. After a decade, we revisit the seminal Hilbert’s Problems paper and reassess Professional Identity, Cost Estimating Techniques, Cost Estimating Implementation, Cost and Schedule Risk Analysis, and Integration with Other Disciplines. We summarize recent progress and issue challenges for future research. We recommend taking advantage of trends in Digital Engineering and Agile and providing meaningful dashboards instead of static briefs.
Keywords: Cost Management, Methods, Program Management
TT01 – Braxton – Hilberts Problems III – paper
TT01 – Braxton – Hilberts Problems III – ppt
Placing a Value on Employee Well-Being
Trending Topics Track (TT02)
Kevin Buck
Asma Abuzaakouk
MITRE applies Social Return on Investment (SROI) principles to inform government workplace Diversity, Equity, Inclusion, and Accessibility (DEIA) decisions. An Investment Value Management Framework (IVMF) compares DEIA solutions and identifies optimal solutions. The framework includes process guidelines and an SROI model to estimate cost, benefit, uncertainty, and risk. The model translates metrics to benefits, and an early warning system manages lifecycle DEIA investment performance, comparing projected to actual SROI.
Keywords: Inclusion, Equity, SROI, Performance, Investment, Optimization
TT02 – Buck – Placing a Value on Employee Wellbeing – ppt
TT02 – Buck – Placing a Value on Employee Wellbeing – paper
Standardising and Governing Cost Estimation and More in the Australian Defence
Trending Topics Track (TT03)
Kimberley Rowlands
In August 2020 the Australian Defence Department established a central governing branch to standardise costing practices and professionals within Defence. This presentation will walk you through the challenges we faced, our approach to overcoming them and the resulting successes. It will outline what we’re focussing on now, understanding the environment of the cost professionals we work with and how we might leverage from each other. As the branch head, I am interested in maturing the profession of cost estimation in the Australian Defence Department.
TT03 – Rowlands – Standardising Cost in AUS Gov
Modeling Battery Manufacturing Complexity
Trending Topics Track (TT04)
Sara McNeal
Fossil fuels are the primary energy source used to drive our economies, but with limited supplies, the reliance on batteries has increased. Batteries are energy storage devices, and the chemical molecules react and produce voltage differences between positive and negative electrodes. These differences produce current flow in the energy forms to power products. In this study, the emphasis is researching cost and specifications to identify variables that have strong relationships with manufacturing complexity for structure.
TT04 – McNeal – Modeling Battery Manufacturing Complexity – PPT
TT04 – McNeal – Modeling Battery Manufacturing Complexity – Paper
Sustainment Contracts Cost Growth Analysis
Trending Topics Track (TT06)
Candice Schultheis
Contract type varies based on a program’s position in the acquisition lifecycle, the type of items/skills procured, and associated risks. Awarded fixed-price contracts are cited as having “locked in” program costs or result in savings claims at initial award, but these claims may not reflect the actual contract cost performance that occurs over time. This analysis quantified sustainment contract cost changes over time and measured cumulative growth. Cost growth comparisons between fixed-price and cost-type contract groups are presented. Root causes are identified, and work-level characteristics are analyzed.
TT06 – Schultheis – Sustainment Contracts Cost Growth Analysis – ppt
Cost Engineering a MVDC Power & Energy Design for Navy Surface Combatants
Trending Topics Track (TT07)
Richard Shea
Henry Jones
Ann Hawpe
Victor Sorrentino
Since maritime MVDC systems do not exist, a statistically significant cost estimating relationship is not easily determined. The components required for a MVDC system do exist at some scale and technical level of maturity. However, there is nothing in production meeting all the same requirements of a Navy surface combatant ship installation. Therefore, this paper follows an approach to compare the theoretical costs of a MVDC to a MVAC architecture in a US Navy Ship.
Keywords: DOD/MOD, Manufacturing, Operations, Uncertainty
TT07 – Shea – Cost Engineering a MVDC Power & Energy Design – ppt
TT07 – Shea – Cost Engineering a MVDC Power & Energy Design – paper
The Economics of Rocket Reusability
Trending Topics Track (TT08)
Ryan Timm
Reusability of space launch vehicles promises some obvious recurring cost savings; but we delve deeper to enumerate and evaluate the effects of reusability on non-recurring costs of plant and equipment and rate and learning impacts. We model and explore the interplay of refurbishment and launch cadence and the maximum number of launch vehicle reuses. Finally, we compare hypothetical costs for the launch of expendable, partially reusable, and fully reusable space launch systems.
Keywords: Learning Curves, Methods, Modeling, Space
CEBoK-S Training Track
CEBOK, CEBoK-S and Certification Exams Overview
CEBoK-S Training Track (CEBoKIntro)
An introductory session to outline ICEAA’s our knowledge base and product line. Our instructors will provide an overview of the ICEAA certification exams, how to prepare, and answer your questions.
CEBoK-S Lesson 0: Overview & Curriculum
CEBoK-S Training Track (CEBoK00)
Kevin Cincotta
Jennifer Kirchhoffer
This short lesson outlines the goals and objectives for CEBoK-S and where it fits in the context of ICEAA’s product line. The lesson provides a high-level roadmap for the CEBoK-S curriculum and gives acknowledgements to the contributing subject matter experts and industry leaders who were involved in CEBoK-S v1.0 development.
CEBoK-S Lesson 1: Introduction to Software Cost Estimating
CEBoK-S Training Track (CEBoK01)
Bob Hunt
This lesson explains the importance of software cost and schedule estimating for CEBoK-S audiences, and includes historical contexts, examples of software program failures and success, and highlights what students can gain from CEBoK-S. Note that CEBoK-S is intended to supplement the audience’s understanding of the cost estimating tools and techniques covered in CEBoK®, introducing the differences between software and hardware estimating and what is included in systems vs. software cost estimating.
CEBoK-S Lesson 2: Software Paradigms
CEBoK-S Training Track (CEBoK02)
Eric van der Vliet
This lesson introduces the two major paradigms involved in custom software development, predictive and Agile, and presents a variety of software development methods and where they differ from perspective of software cost estimation of custom software development efforts. Included are highlights of waterfall, iterative, spiral, Agile, Scrum, SAFe, DAD, and other approaches to give the software cost estimator a basic understanding of the activities and cost drivers involved.
CEBoK-S Lesson 3: CEBoK-S 5-Step Estimating Process
CEBoK-S Training Track (CEBoK03)
Jennifer Kirchhoffer
This lesson outlines the major considerations of our CEBoK-S Five-step Estimating Process for custom software development including the pre-requisite data, cost drivers, and considerations involved in the software development process. We will identify and discuss types of estimates for custom software development efforts and present the CEBoK-S 5-step software estimating process.
CEBoK-S Lesson 4: Estimating Custom Software Development
CEBoK-S Training Track (CEBoK04)
David H. Brown
Kevin Cincotta
Lesson 4 outlines where we use relevant data & custom-built software Cost & Schedule Estimating Relationships (CER & SER) to estimate software development effort, cost and schedule duration. We will present three ways to create the software estimate using estimating techniques: an overview of estimating approaches, how to create a software development estimate based on CEBoK-S Five-step Estimating Process and four estimating approaches, developing a schedule estimate, time-phasing the estimate, cross-checking the estimate, and reviewing estimates prepared by others.
CEBoK-S Lesson 5: Software Sustainment
CEBoK-S Training Track (CEBoK05)
Carol Dekkers
Software sustainment is a definable, robust term, used to describe the second stage of the software life cycle, in place of the term Software Operations and Support (O&S). The first stage, called the investment stage, has been covered in other lessons. Software sustainment is a parent term for many types of software-related work as we’ll outline in this lesson. Software maintenance, in the context of CEBoK-S, is defined to be a child component within software sustainment.
CEBoK-S Lesson 6: Estimating Procured Solutions
CEBoK-S Training Track (CEBoK06)
Arlene F. Minkiewicz
Identify key characteristics of cost estimating considerations for, and differences between Commercial Off-the-Shelf (COTS) software and Enterprise Resource Planning (ERP) systems among other procured software solutions. This lesson will provide an introduction to procured software solutions, describe COTS software and the major types of procured software solutions, and cost drivers and estimating approach(es) for each. We will consider cost growth and sustainment in procured software solutions.
CEBoK-S Lesson X: Software Size
CEBoK-S Training Track (CEBoKX)
Carol Dekkers
This lesson focuses on the various units of measure for software size, options and methods for their measurement, and how to select an appropriate unit of measure suited to your estimate. We also discuss conversions and rules of thumb, and then finally how to ensure whatever size estimate you are given as the software cost estimator can be validated with a checklist of questions.
CEBoK-S Lesson Y: Productivity
CEBoK-S Training Track (CEBoKY)
David H. Brown
Kevin Cincotta
This lesson outlines the complex topic of productivity, which is often used interchangeably with several other terms including throughput, delivery rate and even velocity (in agile software development.) We discuss productivity as a concept and in general, before delving into what constitutes productivity, how to measure it, and the relationship between size and productivity.
CEBoK-S Lesson Z: Commercial Estimating Models
CEBoK-S Training Track (CEBoKZ)
Karen McRitchie
This lesson highlights how various commercial estimating models provide support to our CEBoK-S Five-Step Estimating Process, and how they support creating the custom software development (Lesson 4), procured software solutions (Lesson 6), and software sustainment (Lesson 5) estimates. We will use these models to cross-check out our Midnight and FFS examples from Lesson 4, using the COCOMO II Web Tool, Galorath Inc;s SEER-SEM®, Unison’s TruePlanning® for Software, Quantitative Software Management (QSM)’s Slim Estimate®and additional models including Namcook Consulting LLC’s Software Risk Master (SRM) and COCOMO II-based automated estimating tools.