What’s the Big Deal? Is Agile Software Development Really Different in the DoD Acquisition Environment?
Agile Track (AG01)
Increasingly more projects are leaning towards the use of Agile as a way of mitigating the cost, schedule, and performance of complicated development efforts. Due to complicated acquisition requirements and environment of the DoD, the advantages of Agile versus non-Agile projects are blurred. This research will compare and contrast cost and technical data from multiple DoD acquisition programs to better understand the impact of Agile software development in the DoD.
Keywords: Agile, DOD/MOD, Government,Software
AG01 – PPT – Barbre – What’s the Big Deal?
Scrum Agile Software Metric Analysis for AF Information Systems
Agile Track (AG02)
Programs transitioning from waterfall to agile software development are driving change to the cost analysis process and the way we develop metrics to predict future performance. This case study discusses AF C2ISR portfolio research in a portfolio of information system projects following the Scaled Agile Framework (SAFe). Continuous development and requirements flow introduces additional challenges. This research will discuss the team’s evolving approach, study of productivity metrics over time and lessons learned generated from the initial research.
AG02 – PPT – Davis – Scrum Agile Software
Agile Management for Rapid Acquisition
Agile Track (AG03)
According to PARCA in the April 2018 Agile Program Manager’s Guidebook, “Agile philosophies promote rapid incremental product deliveries, provide flexibility to respond to changing requirements, and advocate close customer collaboration.” While Rapid Acquisition programs may receive reprieve from the procedural requirements defined by the DOD 5000.01, program managers and the cost estimating community maintain their responsibility to manage programs effectively. This paper presents how Agile program management best practices can be applied to Rapid Acquisition.
Keywords: Agile, Cost Management Government, Program Management, Rapid Acquisition
AG03 – PPT – Deane – Agile Management for Rapid Acquisition
AG03 – Paper – Deane – Agile Management for Rapid Acquisition
Understanding Federal Sector Agile Productivity: A Benchmark Study
Agile Track (AG04)
Software estimation in the federal sector is challenging, relying heavily on team productivity. We will present the results of an Agile productivity bench marking study (2017-2018) that was performed across a variety of diverse federal projects from multiple agencies. In addition to quantitative metrics, we captured the effects of federal acquisition constraints on how Agile is implemented, drivers of successful projects, Agile estimation best practices, and other key findings that were part of the analyses and the results.
AG04 – PPT – McKeel – Understanding Federal Sector
Pitfalls to Avoid in Agile Fsm Productivity Measurements
Agile Track (AG05)
Using Functional Sizing on Agile projects requires integration of delivery process with measurement process. The gaps, if any, can take away the whole purpose of measurement. Few key things should be kept in mind when doing functional point counting (FP) for sprints or release which are described as part of this paper. It would also provide some recommendations on how to do productivity benchmarking for same.
Keywords: Agile, EVMFunction Points, Functional Requirements
AG05 – PPT – Thapar – Pitfalls to Avoid in Agile Fsm
Agile Estimation Challenges When Starting a New Team for a New Product
Agile Track (AG06)
Eric van der Vliet
This presentation tells the story of an experienced Cost Estimator with an SPC4 certification that starts with a new team without sufficient Agile experience the development of a complete new product. The budget of the program is fixed and has a duration of two years. Because of the innovative character of the product is it impossible to start with a complete backlog although the program must be managed and controlled and of course estimated.
Keywords: Agile, Communication, Cost Management, Early Cost, Functional Requirements, Project Controls, Software,Story Points, Uncertainty, Cost Estimation
Analysis & Modeling Track
Don’t Just Use Your Data… Exploit It!
Analysis & Modeling Track (AM01)
The Data Age is here. Data are being collected at an exponential rate and cost analysts are struggling to exploit it. Limited structured contextual information and clunky formatting are significant barriers to efficient use. This paper demonstrates the importance of establishing a modern data strategy that considers how analysts identify, access and use data in the context of an example exploitation of a large CSDR data set for the Army’s Stryker Family of Vehicles.
Keywords: Data Collection, Data-Driven, Methods, Modeling, Statistics
AM01 – PPT – JamesA – Don’t Just Use Your Data… Exploit It!
AM01 – Paper – JamesA – Don’t Just Use Your Data… Exploit It!
Production System Cost Modeling Within an MBSE Environment
Analysis & Modeling Track (AM02)
Model-based Systems Engineering (MBSE) incorporates digital models to represent system-level physical attributes and operational behavior throughout the system life-cycle to support product development. To date, many MBSE efforts have focused on technical requirements with little emphasis on cost. Integrating cost models into MBSE provides visibility into cost impacts of design decisions. This presentation explores optimizing production-system design, manufacturing processes, and operations, by integrating various internal and industry production-system cost models into an MBSE environment.
AM02 – PPT – KennedyD – Production System Cost Modeling
An Upgrade to Anderlohr’s Retrograde Method for Broken Learning
Analysis & Modeling Track (AM03)
Breaks in production cause havoc for managing and estimating program costs. Estimators must consider the loss of efficiencies that will result from the break as well as the expected rate of learning after the break. Anderlohr’s Retrograde Method is a popular technique for estimating the cost of broken learning, but it suffers from a weakness that arise when the learning slope changes at the break. This paper describes this weakness and suggests a robust solution.
Keywords: Learning Curves, Monte Carlo
AM03 – PPT – Miller – An Upgrade to Anderlohr’s
AM03 – Paper – Miller – An Upgrade to Anderlohr’s
The Beginning of the End of Traditional Analogous ‘Bottom-up’ Estimating
Analysis & Modeling Track (AM04)
Traditional bottom-up estimating methodologies are under fire due to inconsistency and inaccuracy. Leveraging advanced data-driven techniques is key for accurate, rapid cost estimating. Based on a recent endorsement provided by the DCAA in their 2018 Audit Manual, acceptance of alternative estimating methodologies other than ‘bottom-up’ are now acceptable. We will present a disruptive process for transitioning from a traditional costly, time-consuming analogous ‘bottom-up’ Basis of Estimate (BOE) bidding process to a new, more cost-effective process
Keywords: Cost Management, Data Collection, Data-Driven, Methods, Parametrics, Process Engineering
AM04 – PPT – PriceChris – The Beginning of the End
Adaptive Curve Fitting: An Algorithm in a Sea of Models
Analysis & Modeling Track (AM05)
Adaptive Curve Fitting (ACF) is a novel technique that analyzes finite time series data such as monthly expenditures for government acquisition contracts. By intelligently fitting known resource phasing curve forms to an existing sequence of data, it generates a custom model that extrapolates the remaining values. This enables a variety of joint cost, schedule, and phasing analyses. ACF is based in theory and empirical research, shares similarities with existing techniques, and introduces several key innovations.
Keywords: Smoothing, Burn Rate Trend, Forecasting, Multi-Modal, Crosscheck, EAC/ETC/EVM, Automated, Program Management, Budgeting, Portfolio Analysis
AM05 – PPT – Schiavoni – Adaptive Curve Fitting
AM05 – Paper – Schiavoni – Adaptive Curve Fitting
Hybrid Cost Estimating: The Union of Macro and Micro-Parametrics
Analysis & Modeling Track (AM06)
At the pre-concept phase of a project the nature of the solution is varied and the number of options numerous. As the project moves into the later phases the feasibility studies begin to produce more information and the number of options starts to reduce. In a hybrid cost estimating framework solution it is possible to migrate from a macro to a micro parametric cost model without need for training or skills.
Keywords: Budgeting, Cost/Benefit Analysis, Decision Analysis, DOD/MOD, Life Cycle, Parametrics
AM06 – PPT – Shermon – Hybrid Cost Estimating
AM06 – Paper – Shermon – Hybrid Cost Estimating
Schedule Estimating Relationship Development Using Missile & Radar Datasets
Analysis & Modeling Track (AM07)
This paper addresses the challenges of collecting useful historical schedule data for missiles and radars. The objective was to develop schedule estimating relationships and descriptive statistics for predicting the time between major acquisition milestones based on technical parameters. Major milestones are not always clearly defined and consistent. The difficulties of collecting and analyzing schedule data are highlighted. Various schedule drivers are discussed. The results provide a cross-check on the reasonableness of projected schedules for new programs.
Keywords: Data Collection, Data-Driven, Regression, Scheduling
AM07 – PPT – Jardine – Schedule Estimating Relationship
AM07 – Paper – Jardine – Schedule Estimating Relationship
Estimating China’s Defense Expenditures
Analysis & Modeling Track (AM08)
While DoD releases detailed budget data, China releases virtually none. The Center for Strategic and Budgetary Assessments developed a straightforward, adaptable, and strategy-driven program, the Strategic Choices Tool (SCT), which allows users to modify planned U.S. military force structure and modernization spending over the next decade. We are developing an SCT for Chinese defense spending. This presentation addresses cost estimation methodologies for China’s procurement, personnel, and O&M costs, and preliminary data collection and modeling efforts.
No Keywords Provided
AM08 – PPT – Bianchi – Estimating China’s Defense Expenditures
Communication & Visualization Track
The Legacy of Parametric Estimating
Communication & Visualization Track (CV01)
This paper chronicles the people, methods, and achievements over past centuries leading to today’s acceptance of proven parametric methods for credibly predicting future costs of the world’s most significant tools, software, weapons, structures, and processes. One detailed example is how parametric methods were used to effectively estimate the construction cost of medieval European Castles. A final salute recalls the International Society of Parametric Analysts (ISPA) which is now integral to ICEAA.
Keywords: Methods, Modeling, Parametrics
CV01 – PPT – Apgar – The Legacy of Parametric Estimating
CV01 – Paper – Apgar – The Legacy of Parametric Estimating
Simplifying the Estimate without Sacrificing Quality
Communication & Visualization Track (CV02)
A typical Plan of Actions and Milestones for an ACAT 1 cost estimate is six months. This includes time to define the scope, define the WBS, research and analyze data, develop the estimate, iterate, brief, iterate again, brief again, and so on. Simplifying the estimate will result in shorter research and analysis periods, easier model revisions, easier quality assurance, and easier to understand presentations. This presentation will discuss various methods and processes for simplifying the estimate without sacrificing quality.
Keywords: Data Collection, Data-Driven, Early Cost, Methods, Modeling, Parametrics, Risk, Statistics, Uncertainty, Variables
CV02 – PPT – Goucher – Simplifying the Estimate
Data Visualization – a Product of Human Design
Communication & Visualization Track (CV03)
Cost estimates are based on data analysis and statistical methodologies that capture complex behavior in mathematical terms. One of the biggest challenges estimators face is describing their analyses in a way that a decision maker can quickly grasp and apply. Data visualization is an important skill set that analysts can use to present complex results in a clear and compelling manner. This presentation will provide an overview and show the benefits of data visualization.
Keywords: Data Visualization, Charts, Plots, Communication, Analytics
CV03 – PPT – Kwok – Data Visualization
The Point Estimate Is Not the Point
Communication & Visualization Track (CV04)
The purpose of cost estimating and analysis is to inform planning decisions and secure sufficient / timely funding to facilitate program execution. Or is it? The authors will explore how tangible and intangible roles of cost estimators can better inform design, acquisition strategy, and planning decisions early. Traceable, repeatable, credible and flexible analytical models help to paint the quantitative picture, enabling decision makers to accurately plan a successful program.
Keywords: Communication, Data-Driven, Decision Analysis, Government, Microsoft Excel, Modeling,Uncertainty
CV04 – PPT – SnyderJ – The Point Estimate Is Not the Point
Clearly Communicating Your IGCE to Decision Makers
Communication & Visualization Track (CV05)
All that work gathering data, crunching numbers and running sensitivity tests for Independent Government Cost Estimates. Not done yet–now comes communicating them. It’s more art than science. Leaders need the key details and facts. But they can get hazy amidst probability distributions, parametrics, discounting, inflation and other elements. This presentation uses a real (sanitized) cost estimate, providing tried-and-true examples of getting the message across—understandably, concisely, and decision-ready.
Keywords: Communication, Cost Management, Cost/Benefit Analysis, Government
Comparing Cloud Costs Equitably: J-Funded Capability Development
Computing Track (CC01)
Government agencies often base cloud investment decisions on prices quoted in published cloud calculators and rate cards that invariably apply different service offering assumptions and cost drivers. The likelihood of making decisions based on incomplete or misunderstood data is high, and misinformed investment decisions can unfortunately be quite costly. MITRE evaluated several predominant commercial and government vendor solutions and developed a suite of vignettes that highlight challenges in comparing price quotes on an apples-to-apples basis.
Keywords: Cost/Benefit Analysis, Government, IT, Variables, Cloud Costing
CC01 – PPT – Buck – Comparing Cloud Costs
The 11th Commandment: Thou Shalt Migrate to the Cloud
Computing Track (CC02)
And God gave unto Moses on Mt. Sanai and to General Mattis in the Pentagon the 11th Commandant: Migrate to the Cloud. This paper presents research that supports mission owners, program offices, and cost components in their quest for the Holy Grail – cost effective and cyber-security compliant migration of legacy systems and data to the cloud. It offers innovative artifacts to support up-front requirements and trade-space analyses and back-end cloud design, build, testing and deployment.
Keywords: Cost/Benefit Analysis, Decision Analysis, IT, Life Cycle, Software, Cloud
CC02 – PPT – Hagerty – The 11th Commandment
CC02 – Paper – Hagerty – The 11th Commandment
Predictive Analytic Estimates of Cloud Costs for Government IT Systems
Computing Track (CC03)
The White House’s “Cloud Smart” strategy encourages agencies to consider life cycle costs focused on three main “pillars”: workforce, procurement, security. IT funding requires two years to approve and allocate a budget; so commercial Cloud rates will not be known as the budget is approved. Cost estimates must therefore capture historical workforce, procurement and security cost trends to estimate future solutions. Predictive analytic estimating approaches can provide effective estimates using calibrated historical data.
Keywords: Agile, Life Cycle, Methods, Modeling, Parametrics
CC03 – PPT – Mabe – Predictive Analytic Estimates
Measurements in Cyberspace
Computing Track (CC04)
Addressing cybersecurity and information security is becoming increasingly important within the DoD and beyond. With software systems constantly talking amongst themselves and across various networks, concern is growing about the protection of our sensitive data and applications. But one must also consider how much security is too much and when the ‘right’ degree of cybersecurity has been achieved. This paper discusses cyber and information security and presents measurements and metrics suitable for accessing the practicality and efficacy of mitigation strategies.
Keywords: Software, Cybersecurity, Measurement
CC04 – PPT – Minkiewicz – Measurements in Cyberspace
CC04 – Paper – Minkiewicz – Measurements in Cyberspace
Cost Analysis Needed for Blockchain Efforts
Computing Track (CC05)
U.S. government offices are exploring blockchain to address key challenges in their missions. The motivation stems from blockchain enabling peer-to-peer information sharing, without a centralized authority, which in turn supports execution of processes which span organizations. Currently, government lacks tools to describe blockchain projects with sufficient consistency to support acquisition and cost analysis. This presentation proposes a blockchain descriptive framework as a first step, with a focus on cost elements and drivers for blockchain projects.
Keywords: Cost/Benefit Analysis
CC05 – PPT – Reed – Cost Analysis Needed for Blockchain Efforts
Forecasting Future Amazon Web Services Pricing
Computing Track (CC06)
The National Reconnaissance Office (NRO) Cost and Acquisition Assessment Group (CAAG) produces independent cost estimates to support decision making, budgeting and trade studies. Cloud service costs procured from Amazon Web Services (AWS) are becoming increasingly scrutinized. A thorough analysis was conducted to collect historical AWS prices and model the downward trend. Autoregressive time series models were fit to storage and compute service prices, resulting in annual price reduction rates to be applied to future estimates.
Keywords Not Provided
CC06 – PPT – Souiri – Forecasting Future Amazon Web
CC06 – Paper – Souiri – Forecasting Future Amazon Web
Data Collection & Management Track
“Big Data” Analytics in Operations Research
Data Collection & Management Track (DM01)
As the world becomes more connected and data-driven, accumulating masses of data is becoming a more common corporate strategy. This paper discusses the current PRICE® Systems data collection methodology within the company that utilizes web scraping/crawling. The team will present both lessons learned and potential shortcomings in the current method of approach, along with plans for future endeavors.
Keywords: Data Collection, Methods, Web Scraping
DM01 – PPT – Cuiule – “Big Data” Analytics
DM01 – Paper – Cuiule – “Big Data” Analytics
A Discussion on Data Reliability: Evaluating Qualitative and Quantitative Data
Data Collection & Management Track (DM02)
Obtaining data is an integral part of any analysis and collecting good data that is accurate and robust can be challenging. The quality and validity of data can strongly influence the final cost position recommendation that is presented to accurately and effectively inform decision makers. This paper will discuss data reliability, how to evaluate qualitative and quantitative data, and how to create and evaluate metrics to track the quality of data to ensure accurate cost estimates.
Keywords Not Provided
DM02 – PPT – DeStefano – A Discussion on Data Reliability
DM02 – Paper – DeStefano – A Discussion on Data Reliability
How to Create a Cost Estimate Using Data Science and RRRrrr Studio!
Data Collection & Management Track (DM03)
ICEAA’s effort to expose the community to data science concepts and advantages to cost estimators has been popular. However, you want to know how to apply tools and techniques for creating an estimate NOW. Well prepare to create your very first data science cost estimate. There will be code, there will be data, and they will be for ye to plunder and take as your own because “not all treasure is silver and gold mate.”
Keywords: Data Collection, Data-Driven, DOD/MOD, Government, Data Science, R Studio, Data Capture, How To
DM03 – PPT – Eden – How to Create a Cost Estimate
DM03 – Paper – Eden – How to Create a Cost Estimate
When Data isn’t Enough
Data Collection & Management Track (DM05)
How can an analyst transform messy data into valuable insights, and why should they bother? This paper argues the importance of expanding the analyst skill set into the broader realm of data science and outlines several techniques to clean and process large volumes of data. Using R, we demonstrate how to look for important relationships among our data and explore some text mining techniques to utilize an otherwise unhelpful text field. Finally, we explore best practices in visualizing and communicating our findings.
Keywords: Cost/Benefit Analysis, Data-Driven, Methods, Statistics, R, Tableau, Text Mining
DM05 – PPT – Scarbrough – When Data isn’t Enough
DM05 – Paper – Scarbrough – When Data isn’t Enough
CDMS: Developing a Database Solution for Data Management
Data Collection & Management Track (DM06)
The NRO CAAG has a data issue…too much for spreadsheets to handle! The detail and variety of data the CAAG has collected for 40 years has forced a revamp of their data management and storage systems. The CAAG Data Management System (CDMS) will be the next generation database solution to store/normalize CAAG data. This presentation gives firsthand accounts of the challenges encountered and how the CAAG overcame and implemented our solution.
Keywords: Data Collection, Government, IT, Methods, Operations, Space
Machine Learning Track
Growing Maintenance Costs: Understanding How Weather Impacts Maintenance
Machine Learning Track (ML01)
The in-service maintenance is often a major cost driver. This paper explores an innovative approach to identifying maintenance trends in the Federal Aviation Administration (FAA). Using NLP and other Machine Learning techniques, the FAA’s RMLS maintenance text data is correlated with the ITWS weather data. The session goal is to cover one way on how to think about NLP in a cost setting. Followed by implementation with Python and other data science technologies.
Keywords: Data-Driven, Government, Methods, Modeling, Software, Statistics
ML01 – PPT – AndersonBryan – Growing Maintenance Costs
Machine Learning Assisted Data Extraction and Normalization
Machine Learning Track (ML02)
Data collection and normalization is a key component of analysis. However, it often requires significant time and effort to properly extract quality normalized data from the raw datasets. Given the recent proliferation of larger datasets a more automated approach to data extraction and normalization is required. This research applies machine learning text classification algorithms and methods to automate historically manual data normalization tasks.
Keywords: Data Collection, Methods, Dara Science, Machine Learning
ML02 – PPT – BrownJonathan – Machine Learning Assisted Data
The Robot Forecaster–A Continued Study of Artificial Neural Network Application
Machine Learning Track (ML03)
If your job includes predicting the future then you understand the difficulties of the “unknown”, of all the little gremlins (sometimes called risks) that can mess up the most elegant and well thought-out plan. There are data models and statistical methods that can help a great deal, but if the conditions are right, a relatively new tool that is as accessible as it is misunderstood might just be what you need. We will cover the basics of Artificial Neural Networks and I’ll share the progress I’ve made since ICEAA 2018 in putting these digital brains to work.
Keywords: Agile, Bayesian, Budgeting, Communication, Cost/Benefit Analysis, Data Collection, Data-Driven, Decision Analysis, Government, IPM, IT, Learning Curves, Manufacturing, Methods, Microsoft Excel, Modeling, Monte Carlo, Operations, Parametrics, Performance Management, Process Engineering, Program Management, Project Controls, Regression, Risk, Scheduling, Software, Space, Statistics, Uncertainty, Variables, Machine Learning, Artificial Intelligence, Artificial Neural Networks
ML03 – PPT – Eskue – The Robot Forecaster
Don’t be Scared, Machine Learning is Easy
Machine Learning Track (ML04)
Thus far the field of cost estimating has only explored the pedagogical utilization of machine learning within cost estimating. The novelty of machine learning makes it difficult to find real use cases demonstrating the efficient application of machine learning to the cost estimating process. This paper and accompanying presentation will document a case study exploring the data cleaning and modeling process for a cost estimate done on DoD installations.
Keywords: Government, Modeling, Machine Learning
ML04 – PPT – JohnsonMary – Don’t be Scared, Machine Learning is Easy
Machine Learning and Natural Language Processing Applications for Cost Analysis
Machine Learning Track (ML05)
How can machine learning and natural language processing enhance the traditional methods of cost analysis? By making it easier for humans to analyze large datasets, and by automating many of the manual, time-consuming tasks needed to cleanse and prepare the data for analysis. This presentation examines several applications in which we integrated Natural Language Processing using python libraries, with various Machine Learning methods, in particular trees, randomization, and boosting, to improve prediction accuracy.
Keywords: Data-Driven, Methods, Modeling, Machine Learning, Natural Language Processing, Data Science, Analytics
ML05 – PPT – Mourikas – Machine Learning and Natural Language
Beyond Regression: Applying Machine Learning to Parametrics
Machine Learning Track (ML06)
Cost estimating has relied primarily upon regression analysis for parametric estimating. However, regression analysis is only one of many tools in data science and machine learning, and is a small subset of supervised machine learning methods. In this paper, we look at a variety of methods for predictive analysis for cost estimating, including other supervised methods such as neural networks, deep learning, and regression trees, as well as unsupervised methods and reinforcement learning.
Keywords: Data-Driven, Methods, Regression, Statistics
ML06 – PPT – Roye – Beyond Regression
ML06 – Paper – Roye – Beyond Regression
Data Impacts on System Readiness and Cost
Methods Track (MM01)
Operations and support costs are becoming increasingly in focus as system readiness is addressed. This paper will discuss the impact data has on forecasting readiness and the corresponding effects on cost. Identifying “correct” data and sources, such as CSDRs and OEM data reporting systems, plays an important role in modeling both cost and readiness. Different approaches to improve readiness and their impacts on cost and data needs will be discussed.
Keywords: Data Collection, Decision Analysis, Government, Operations, Performance Management
MM01 – PPT – BrownPaul – Data Impacts
A New Approach When Cost/Capability Trades Matter Most
Methods Track (MM02)
Consideration of life cycle costs (LCC) early in the acquisition process, specifically prior to Milestone A, is paramount to credible requirements generation and improves the probability of program success. This paper describes a parametric universal wheeled ground vehicle LCC model based on CSDR data and designed for integration into a modeling and simulation environment that produces alternative vehicle designs. The LCC model uses outputs from a requirements-driven preliminary design model to generate informed cost vs. capability/requirements trades.
Keywords: Data-Driven, Early Cost, Life Cycle, Parametrics, Tactical Wheeled Vehicles
MM02 – PPT – Cherwonik – A New Approach
MM02 – Paper – Cherwonik – A New Approach
Five Steps for Improving the Accuracy of Rough Order of Magnitude Estimates – Bell V-280 Wing Predictive Cost Analytics Case Study
Methods Track (MM03)
According to the GAO Cost Estimating and Assessment Guide, ROMS are “developed when a quick estimate is needed, and few details are available”. This leads to challenges with ROMs especially when estimates become “locked in” early, based on the least accurate data. We will discuss five enabling steps that estimators can take to improve the accuracy of ROM estimating. This includes using a predictive analytic data driven methodology for leveraging and extrapolating limited data sets.
Keywords: Data-Driven, Methods, Modeling, Parametrics
MM03 – PPT – Jasnoff – Five Steps for Improving the Accuracy
Intellectual Property Valuation: An Overview
Methods Track (MM05)
Intellectual property (IP) refers to inventions, literary and artistic works, designs, and symbols, names and images used in commerce. IP is protected by patents, copyright and trademarks, which enable earning of financial benefit from these inventions or creations. While most cost estimators already know that, what do we know about IP valuation? If asked to derive a cost estimate for IP valuation, what approaches could be taken? This overview will answer that question and more.
Keywords: Budgeting, Life Cycle, Methods, Program Management, Software
MM05 – PPT – PrinceC – Intellectual Property Valuation
PAC Template – Program Acquisition Cost (PAC) Template
Methods Track (MM06)
One of the goals of our young and upcoming cost organization was to create a universal cost model template. Fast forward 6 years and the template we developed introduced standardization, a starting point for new analysts, minimized tedious work, and reduced common errors. We would like to share the success story of the cost model template and the lessons learned through the 6-year development process.
MM06 – PPT – White – PAC Template
MM06 – Paper – White – PAC Template
Planning & Strategy Track
Sinking, Fast and Slow
Planning & Strategy Track (PS01)
How do sunk costs impact a cost estimate? Your response might depend on whether you approach the issue from the perspective of an accountant or a behavioral economist. This presentation explores the literature on sunk costs and proposes strategies to improve cost estimating practice by improving our understanding and categorizing of sunk costs.
Keywords: Data Collection, Methods, Sunk Costs, Cognitive Bias
Self-Organizing Markets And Time
Planning & Strategy Track (PS02)
Markets demonstrate statistically significant self-organization concerning how they respond to changes in prices and the product features offered to them. The nature of these self-organizing activities changes over time. What works for a market now may not work a few years from now. Being able to characterize market self-organization now and in the future is key to optimizing financial success, which this paper examines.
Keywords: Decision Analysis, Variables, Multidimensional Economics, ME, 4D, Economic Tug-of-war, Profit Optimizing
PS02 – PPT – Howarth – Self-Organizing Markets And Time
PS02 – Paper – Howarth – Self-Organizing Markets And Time
Contracting for Agile Projects
Planning & Strategy Track (PS04)
The nature of agile, with its flexible approach to requirements and stress on incorporating feedback from frequent stakeholder engagements, makes contracting for agile projects particularly difficult. This paper proposes a new idea that uses a capability-based incentive structure, which is currently being considered on upcoming Department of Defense contracts. The paper will present the mechanics of implementing this approach, its pros and cons, and a comparison to other approaches recently used on similar projects.
Keywords: Agile, Software, Contracts, Incentive
PS04 – PPT – Smallwood – Contracting for Agile Projects
Enhance Estimator Success & Organization Competitiveness in Supplier Assessment
Planning & Strategy Track (PS05)
This presentation reviews the use of an effective methodology to perform Supplier Assessment. Estimator success and organizational competitiveness are enhanced when supplier quotations can be assessed and validated, leading to stronger negotiation positions as well as basis for preferred suppliers. This paper offers a quantitative methodology to benchmark, monitor and evaluate potential suppliers of a customer-specified technology. Once established, a Supplier Assessment capability creates the outcome of timely consistent bid evaluations as well as ceiling prices for negotiation.
Keywords: Cost Management, Methods, Parametrics, Supplier Assessment, Affordability, Negotiation, Preferred Vendors
PS05 – PPT – Swaren – Enhance Estimator Success
Application of Conjoint Regression to Requirement Analysis
Planning & Strategy Track (PS06)
Conjoint analysis provides a simple method of measuring how the tradeoff between different capabilities can affect the perceived value of a product. Normally associated with marketing research, it can also be utilized to help estimate the cost of developmental items based on their expected requirements. This presentation explores the best practices, common pitfalls, and validation techniques associated with conjoint regression, and includes a notional example derived from an application of this technique in a USMC tradeoff analysis study.
Keywords: Cost/Benefit Analysis, Data-Driven, Decision Analysis, Methods, Modeling, Statistics
Processes & Best Practices Track
Measuring Portfolio Value for Government Programs and Initiatives
Processes & Best Practices Track (BP01)
George Bayer Jr.
Transformational government initiatives which require major capital investments or acquisitions are complex, difficult to measure, and challenging to articulate to decision-makers. Considering the complex dependencies and implementation risk of individual programs, risk-adjusted cost estimates for portfolios are often overstated. This paper examines how cost estimators and analysts measure portfolio value and generate stakeholder advocacy for major government initiatives, which require multiple acquisitions for implementation and agency policy changes to change behaviors and realize value.
Keywords: Cost/Benefit Analysis, Decision Analysis, Government, Risk, Portfolio Analysis
BP01 – PPT – Bayer – Measuring Portfolio Value
BP01 – Paper – Bayer – Measuring Portfolio Value
Flexfiles: The Next Generation in Contractor Cost Reporting
Processes & Best Practices Track (BP02)
The Department of Defense (DoD) has collected contractor cost data in largely the same manner since the 1970s. The FlexFile initiative represents the next generation in contractor cost data collection. The FlexFile builds upon its predecessor, the DD 1921 reports, with time-phased actuals and additional detail that lend insight into contractor’s native cost elements. This paper offers a brief history of DoD cost data collection, an overview of the DD 1921 reports as well as the FlexFile, and examples of the greater detail provided by the FlexFile relative to the DD 1921 reports.
Keywords: Data Collection, DOD/MOD, Government, FlexFiles, CSDR
BP02 – Website – Janicki – Flexfiles
2020 Census Life Cycle Cost Estimate (LCCE): Using the GAO Cost Estimation and Analysis Guide to Drive Enhancement
Processes & Best Practices Track (BP03)
The 2020 Census LCCE is a large, complex estimate with a projected $15 billion total cost. In 2016, the GAO assessed the estimate as only partially meeting any of the GAO’s four characteristics of a quality cost estimate. Using the GAO’s Cost Estimating and Assessment Guide, the 2020 Census LCCE team used a data-driven analysis to improve the LCCE and its supporting documentation. The result was a drastic improvement in the 2017 GAO assessment.
Keywords: Communication, Data-Driven, Government, Life Cycle, Methods,Modeling
BP03 – PPT – Kobilarcik – 2020 Census Life Cycle
GAO Updated, New, and Coming Best Practice Guides
Processes & Best Practices Track (BP04)
This session will go through GAO’s updated, new, and forthcoming best practice guides. This includes a review of the best practices described in the cost guide, schedule guide, technology readiness assessment guide, and Agile assessment guide.
Keywords Not Provided
BP04 – PPT – Leotta – GAO Updated
BP04 – Paper – Leotta – GAO Updated
Contractually Speaking: The Story of DoD Contract Vehicles
Processes & Best Practices Track (BP05)
A contract is the legal document holding government and contractor accountable for their responsibilities and can play a significant role in cost and schedule issues throughout its lifetime. This paper utilizes the Contracts Database and CADE to investigate whether or not contract vehicles themselves contribute to these issues. We also dive into the acquisition process to determine the efficiency of different contract vehicles and how profit may impact a contract’s success or failure.
Keywords: Data Collection,Data-Driven,DOD/MOD,Government
BP05 – PPT – Olbum – Contractually Speaking
BP05 – Paper – Olbum – Contractually Speaking
Business Case on the Cost/Benefit of U.S. Government Support of Contractors in order to Maintain Competition
Processes & Best Practices Track (BP06)
The DoD is facing shrinking industrial bases in its acquisitions due to contractor mergers, specific military applications, and scarcity of materials. National security, strategic posturing, and Congressional interest make it vital to keep these industries viable and competitive. Mitigating the risk of “shrinking industrial bases” offers unique opportunities in structuring these acquisitions. This study looks at the costs and benefits associated with strategies for acquisition of defense systems within the constraints of shrinking industrial bases.
Keywords: Cost/Benefit Analysis, Decision Analysis, Government, Infrastructure, Learning Curves, Life Cycle, Risk
BP06 – PPT – Pardoe – Business Case
Engineering the Acquisition Process: Better Value Through Mechanism Design
Processes & Best Practices Track (BP07)
As both the regulator and the only buyer in a market that measures over $500 billion a year in acquisitions, the Defense Department has tremendous leverage but does not make full use of it. Limited competition in the prime contractor market results in higher prices and lower quantities purchased. There have been some recent strides made to achieve better value, but much more can be done if the government will pursue strategic approaches. In this paper, we consider the use of mechanism design to achieve better value.
Keywords: DOD/MOD, Government, Modeling, Economic Analysis
BP07 – PPT – Smart – Engineering the Acquisition Process
BP07 – Paper – Smart – Engineering the Acquisition Process
Creating a Cost Driver S-Curve
Risk Track (RR01)
S-Curve analysis is not just for Risk Analysis. It can be used for both Contractor and Government Cost Driver analysis. The presentation will: 1) describe the similarities and differences between Cost Driver and Risk Analysis; 2) provide a rational for doing an S-curve on Cost Drivers early in the procurement lifecycle; 3) provide detailed steps to developing a Cost Driver S-Curve; and 4) display examples of Cost Driver S-Curves and additional outputs that result from the analysis.
Keywords: Cost Management, Modeling, Monte Carlo, Cost Driver
RR01 – PPT – Burney – Creating a Cost Driver S-Curve
The 3-Point Method Redux: Estimating Cost Uncertainty Given Only a Baseline Cost
Risk Track (RR02)
Leveraging a 2007 paper entitled “Estimating Cost Uncertainty when only Baseline Cost is Available,” this presentation revisits the 3-point method: a scenario-based application of the Analytical Hierarchy Process (AHP). This presentation steps through an overview and notional example on applying the 3-point method to demonstrate how pessimistic and optimistic costs are derived from a given baseline cost. These three costs, in turn, serve as the requisite three parameters of the triangular distribution.
Keywords: Early Cost, Methods, Microsoft Excel, Modeling, Risk, Scheduling, Uncertainty, AHP, Baseline, Beta, Beta-PERT, Confidence Level, Cost, Coefficient of Variation, CV, Distribution, Expert Judgment, Intensity Level, Lognormal, Maxwell Risk Criteria Matrix, Mode, Most Likely, MRCM, Optimistic, Pairwise Comparison, Pessimistic, Point Estimate, Probabilistic, Probability, Project, Reserves, Ratio, Relative Risk, Risk, S-curve, Scenario, Schedule, Scores, Simulation, SME, Standard Deviation, Subject Matter Expert, Three-Point, Triangular, Uncertainty, Variance, Weighting, Weights
RR02 – PPT – Greenberg – The 3-Point Method Redux
When is Less More? Level of Detail in Cost and Schedule Risk Assessment
Risk Track (RR03)
When performing cost and schedule risk assessment (C)SRA, the level of detail in the analysis schedule dictates the time to build, validate, and review the model. In this brief, we will discuss the pros and cons of using a more or less detailed analysis schedule as the basis for these models, whether 10s or 10,000s of lines. We will explore the level of effort, accuracy, and value (stakeholder understanding) of each approach through the use of real-life scenarios, based upon the authors’ experience developing models for NASA and other federal agencies. An overview of the basic requirements for any SRA or JCSA, including detailed schedule for execution, monitoring, and control of the schedule to validate model critical path will also be provided.
Keywords Not Provided
RR03 – PPT – Krepel – When is Less More?
Risk: Intentional Interaction With Uncertainty
Risk Track (RR04)
In this session we review how Risk Management is too often perceived as a compliance exercise rather than a management tool. Because of this outlook, the vast majority of risk registers record too few risks. The limitations and pitfalls of restricting the amount of risks recorded leads to a reliance on point estimates, an incomplete risk mitigation plan, and an absence of tangible impacts and analysis. It’s not the big risk that lead your project to failure, it’s the 30 little risks that occur in a unique sequence of events that spirals into a perfect storm.
RR04 – PPT – Lemay – Risk: Intentional Interaction With Uncertainty
Scheduling & Programming Track
Impact of Scope Changes on Schedule Growth
Scheduling & Programming Track (SP01)
Can cost analysts predict and thus estimate schedule growth based on scope changes? Is there a relationship between the two? Leveraging from the same research and data as the 2018 ICEAA Workshop paper “Impact of Scope Changes on Software Growth” by Jonathan Brown and Gail Flynn, this research focuses on the possible schedule impacts due to pure growth and scope changes. Analysis was performed from a scheduling perspective to show the impact scope changes can have on the program schedule.
Keywords: Scheduling, Government, Scope growth, SRDRs
SP01 – PPT – Bowers – Impact of Scope Changes
The Programmatic Estimating Tool (PET)
Scheduling & Programming Track (SP02)
The Programmatic Estimating Tool (PET) provides a method for adjusting cost estimates in scenarios where programs face rigid schedule and/or budget phasing constraints. PET integrates program cost, schedule, and budget phasing into a single user-friendly tool. Using historical correlation between cost, schedule, and phasing model residuals to generate a tri-variate conditional distribution, PET can be used to estimate the impact of: Schedule and/or phasing constraints on cost, Cost and/or phasing constraints on schedule.
SP02 – PPT – Laing – The Programmatic Estimating Tool (PET)
SP02 – Paper – Laing – The Programmatic Estimating Tool (PET)
A Program Managers Guide to Reliable Subcontractor Reporting
Scheduling & Programming Track (SP03)
Prime Government contracts are executed with significant subcontractor content. Objective measurement of their performance can be difficult leading to methods such as percent spent or level of effort. These methods can result in erroneous progress masking true performance. Using discrete earned value best practices provides prime contractors and Government agencies realistic subcontractor performance for early corrective action. This paper investigates how to implement low risk objective EV techniques to promote reliable and effective subcontractor reporting.
Keywords: Budgeting, Cost Management, Government, Program Management, Project Controls, Risk, Subcontractor Performance
SP03 – PPT – Malone – A Program Managers Guide
SP03 – Paper – Malone – A Program Managers Guide
Blending Contract and Project Management to Achieve Cost Savings
Scheduling & Programming Track (SP04)
Tia M Barnes
This study examines the waterfall that occurs between segregating projects and contract management, and how interdependencies drive project cost savings. This interpretive study will visit the discussion around the pitfalls of segregating these interrelated activities, and how the process can impact your agency bottom line. It will exam how cost avoidance can be achieved and provide recommendations to assist in the minimization of costs, schedule, and risk.
Keywords: Cost Management, Cost/Benefit Analysis, Methods, Program Management, Risk, Scheduling
SP04 – PPT – PriceTeresa – Blending Contract and Project Management
SP04 – Paper – PriceTeresa – Blending Contract and Project Management
EVM Visualization: The Radar Tool
Scheduling & Programming Track (SP05)
This research presents an innovation in EVM data analytics and visualization that gives acquisition managers the potential to better understand system level risk and to laser-focus their attention on critical elements of cost. The tool transforms a massive set of disaggregated data into a manageable form, and extracts “critical” elements by filtering on a set of user-defined, rules-based criteria. The tool holistically displays absolute growth, percent growth, and percent complete in one radar graph.
Keywords: Cost Management, Microsoft Excel, EVM
SP05 – PPT – Staffin – EVM Visualization: The Radar Tool
SP05 – Paper – Staffin – EVM Visualization: The Radar Tool
Earned Scope Management: Scope Performance Evaluation for Software Projects
Scheduling & Programming Track (SP06)
Three of the major constraints on projects are scope (size), cost (money) and time (schedule) and while the Earned Value Management (EVM) and Earned Schedule (ES) techniques manage the cost and time constraints, they do not explicitly tackle the scope constraint. Even though scope management is recognized as a key success factor in software projects, there is a lack of formal techniques in the literature to manage scope. This session presents the Earned Scope Management (ESM) technique developed since 2015 and currently tested in Mexico.
Keywords: Decision Analysis, EVMFunction Points, Performance Management, Project Controls, Earned Scope Management
Examining the Effects of Recourse on Estimates
Scheduling & Programming Track (SP07)
We construct a set of simple project planning scenarios and apply multi-stage stochastic programming to demonstrate the effect of different recourse options on optimized resource allocation policies.
Keywords Not Provided
The 3C’s of Measurement and Cost Estimating Success: Create, Confirm, Convince
Software Track (SW01)
To many, it may be obvious that good cost estimating is fundamental to software development–. However, most technology organizations remain unaware of the benefits of formal cost estimating and measurement – even when they are technology-centric and use defined project management methods. This presentation promotes a 3C approach to increasing awareness, gaining acceptance and understanding why formal cost estimating is critical to organizational success.
Keywords Not Provided
SW01 – PPT – Dekkers – The 3C’s of Measurement
From Point A to Point Estimate: How Requirements Become Function Points
Software Track (SW02)
IFPUG function point analysis has been successfully used to size software by organizations for 30+ years. However, there remains confusion as to how the process works. This presentation demystifies the Function Point Analysis process and educates those interested in how function points are counted and/or are considering the use of FP in their organizations. The presenters will discuss functional/non-functional requirements, good/poor requirements and how an FP count is developed based on the functional requirements.
Keywords: Early Cost, Functional Requirements, IT, Software
SW02 – PPT – French – From Point A to Point Estimate
Estimating and Managing Modern Software Development Programs
Software Track (SW03)
The recent final report of the Defense Science Board (DSB) Task Force on Acquisition of Software for Defense System (February 2018) outlines key challenges facing Defense with resect to software acquisition. The summary report states, “Software is a crucial and growing part of weapons systems and the Department needs to be able to sustain immortal software indefinitely. “ Identified issue range from poor initial estimates to the Federal Government’s inability to utilize modern developmental tools, such as the “Software Factory”.
Keywords: Agile, EVMFunction Points, Software
SW03 – PPT – Hunt – Estimating and Managing
SW03 – Paper – Hunt – Estimating and Managing
Army Software Sustainment Cost Estimating Results
Software Track (SW04)
The Army has conducted a study over the past six years to improve the estimation accuracy of software sustainment systems cost. Based on an extensive data call of 192 Army systems, data analysis revealed several types of cost estimating relationships based on release type, release rhythm, and three categories of data. Analysis of a sustainment cost risk model was also conducted. This presentation will show the study results including what worked and did not work.
Keywords: Data-Driven, DOD/MOD, Parametrics, Regression, Risk, Software, Statistics
SW04 – PPT – JonesCheryl – Army Software Sustainment
SW04 – Paper – JonesCheryl – Army Software Sustainment
The Journey to Better ERP Estimation
Software Track (SW05)
Software estimation is never easy, especially when implementing large complicated Enterprise Resource Planning (ERP) Systems comprised of significant off-the-shelf (OTS) components combined with glue code interfaces to numerous legacy systems. Data to support such an estimate is sparse, incomplete, and often inconsistent. This paper presents the first phase of a journey toward improvements to this situation, utilizing analysis of historical data to inform future estimates, focusing on process and lessons learned.
SW05 – PPT – Kilgore – The Journey to Better ERP Estimation
Software Cost Estimation – Why Is It Different?
Software Track (SW06)
Harold van Heeringen
Software Cost Estimation is often considered the most difficult part of an integral cost estimate. Unrealistic software estimates often result in disasters. Examples are new tunnels, metro lines or factories that can’t start their operation because everything is ready, except for the software part. One of the main reasons for this is often an inaccurate and optimistic software cost (and schedule) estimate. Why is estimating software differently and how is ICEAA dealing with this?
Keywords: Software, Software Cost Estimation, SCEBOK, Functional Size, Agile Team Estimation
Space & Missiles Track
Adapting Existing Cost Model to Estimate Section 804 Rapid Prototyping and Rapid Fielding Activities
Space & Missiles Track (SM01)
Analogous prototype program cost data doesn’t readily exist and the development of prototype specific cost databases is a lengthy process. Our immediate Cost Estimating needs must address estimating prototype satellites using existing cost models. Our evolving process elicits and applies explicit cost model adjustments to existing cost models to directly address the explicit differences between prototypes and operational systems, specifically in the areas of CERs to Model Commercial-Like Programs and New Design and Heritage.
Keywords: Data Collection, Methods, Data Science, Machine Learning, Function Points, Software Sizing, IFPUG, Software Metrics, Estimation
SM01 – PPT – Garcia – Adapting Existing Cost Model
The Efficacy of NASA’s Joint Confidence Level Policy
Space & Missiles Track (SM03)
In January 2009 NASA policy established the performance of a Joint Cost Schedule Confidence Level (JCL) analysis as the basis for project funding and external commitments. This paper examines the efficacy of that policy relative to reducing cost overruns. Projects executed before the policy implementation are compared to projects approved post-policy implementation. Various statistical techniques are used to perform comparative analyses and alternative explanations are explored.
Keywords: Budgeting, Cost Management, Government, Modeling, Program Management, Risk, Space
SM03 – PPT – PrinceAndy – The Efficacy of Nasa’s
“Phase 0” Space Mission Estimates
Space & Missiles Track (SM04)
Michel van Pelt
How to prepare very early estimates for future space missions, within little time and using even less inputs? Based on experience with preparing estimates for the very first Principal Investigator proposals for new ESA Science missions, this paper provides rules-of-thumb, observed high-level cost drivers and trends, checklists, potential pitfalls and general lessons learned.
Keywords: Early Cost, Government, Life Cycle, Methods, Modeling, Parametrics, Space
SM04 – PPT – van Pelt – “Phase 0” Space Mission Estimates
SM04 – Paper – van Pelt – “Phase 0” Space Mission Estimates
An Analytical Explanation for Vertical Integration Behavior in the Marketplace
Space & Missiles Track (SM05)
Vertical integration is a costly and difficult-to-reverse strategy with significant risks, yet it has experienced a sharp rise in popularity within the satellite sector over the last decade. Building on previous research, this session demonstrates how commercial parametric tools can be used in combination with business-case analysis to understand why aerospace companies are increasingly pursuing vertical integration strategies. Additionally, it discusses how the framework for this analysis can be applied across other industries.
Keywords: Cost/Benefit Analysis, Decision Analysis, Learning Curves, Life Cycle, Modeling, Parametrics
SM05 – PPT – WilliamsC – An Analytical Explanation
SM05 – Paper – WilliamsC – An Analytical Explanation
Estimating Missile G&C Development Cost: An Important Advance
Space & Missiles Track (SM06)
Paul Hardin III
This paper describes improved cost estimating methodologies for tactical missile Guidance and Control (G&C) Development Engineering (DE), generally the largest contributor to missile development cost and typically one of the most challenging cost elements to estimate. These methodologies represent an alternative approach to commonly used cost factors (e.g., factor of prototype production cost) which have error metrics that normally exceed 50%. The research developed CERs with error metrics below 50% and as low as 20%.
Keywords: Data-Driven, Early Cost, Methods, Modeling, Parametrics, Regression, Statistics, Variables
SM06 – PPT – York – Estimating Missile G&C
SM06 – Paper – York – Estimating Missile G&C
A Comparative Analysis of Nuclear Security Enterprise Estimates
Space & Missiles Track (SM07)
Various government publications recommended adapting multiple methods and approaches to enhance the quality, validity and reliability of a program cost estimate. This study utilizes a COTS software tool and two approaches based on different system architectures, for a Nuclear Security Enterprise estimating effort. The cost estimate results exhibit a phenomenon of near-homogeneity. Model techniques, critical parameters, and cost drivers are analyzed and evaluated to help comprehend the model behaviors.
Keywords Not Provided