Agile Track
Agile Software Development Cost Factors: A Case Study
Agile Track (AG01)
Blaze Smallwood
The lack of data on government agile software development programs has made estimating costs for new agile development programs challenging. This paper seeks to address this challenge through a case study of several completed DoD agile projects with cost, schedule, and performance data. Several relevant metrics will be examined, including cost per story point, cost per requirement, scope growth rates, impacts of team size changes on velocity and productivity, and various others.
Keywords: Agile, software, story points
AG01 – PPT – Smallwood – Agile Software Development
Using Function Points to Manage Agile Product Backlog: Fact vs. Fiction
Agile Track (AG03)
Daniel French
Carol Dekkers
The Agile framework continues to grow in popularity as IT organizations struggle to deliver projects on time and budget. Agile is not a silver bullet that many think it is and the primary reason for this is that Agile estimation methods are completely subjective. This presentation details how IFPUG function points and other rule based size metrics can be effectively used to help Agile teams estimate sprint sizes, calculate velocity and manage the product backlog.
Keywords: Agile, budgeting, early cost, functional requirements, IT, program management, software, function points, software metrics, software sizing
AG03 – PPT – French – Using Function Points
Agile Software Development Cost Risk for Information Technology Programs
Agile Track (AG04)
Adam Letcher
This paper addresses the issue of how to analyze and quantify cost risk within an Agile software development environment. Because of the unique nature of Agile software development, some of the traditional cost estimating and risk analysis practices should be altered. This paper proposes a process for assessing Agile development risk using a capability sizing metric and presents the governing equations for doing so.
Keywords: Agile, EVM, function points, IT, methods, risk, software, uncertainty
AG04 – PPT – Letcher – Agile Software Development Cost Risk
AG04 – Paper – Letcher – Agile Software Development Cost Risk
Accurate Agile Estimation
Agile Track (AG05)
Kevin McKeel
Estimating software development costs in an Agile development environment is challenging due to undefined scope, lack of planning beyond the next few sprint cycles, and distrust of traditional software estimation techniques within the Agile world. This presentation will detail an IRS case study, and describe how these challenges were overcome.
Keywords: no keywords provided
AG05 – PPT – McKeel – Accurate Agile Estimation
A Perfect Marriage: Mr. and Mrs. Agile-Cloud?
Agile Track (AG06)
Teresa Price
Wilfred Tagud
John Sullivan
Andrew Drennon
What mandate brought together no one should tear apart. This interpretive study analyzes the relationship between Cloud computing services and Agile Methodology to determine if it’s a right fit for achieving cost avoidance. In addition, the study includes experience-based input from the FAA’s Chief Information Technology officers as well as other agencies. The research will conclude with the challenges and identified lesson learned when both are implemented together. “I can’t say, ‘I Do’ without you…”
Keywords: Agile, cost/benefit analysis, decision analysis, infrastructure, methods, program management, software, cloud services
AG06 – PPT – Price – Mr. and Mrs. Agile-Cloud
An Implementation of Automated Structural Design-To-Cost in a Model Based Engineering Environment
Agile Track (AG07)
Christopher Price
Leonor Hagberg
Brig Bjorn
Apinut “Nate” Sirirojvisuth
Daphne Biddle
The integration of Computer-Aided Design (CAD) tools with Cost Analytic tools in COTS Integration Environments allows designers to quickly and easily estimate the cost of structural designs as they are designing, thereby directly supporting Design-To-Cost, in a Model Based Engineering (MBE) environment. This paper will demonstrate how quick and easy it is to estimate modifications to a structural assembly using such an integration environment.
Keywords: Agile, cost/benefit analysis, data-driven, decision analysis, early cost, functional requirements, methods, modeling, parametrics, process engineering, variables
AG07 – PPT – Price – An Implementation of Automated Structural Design-To-Cost
AG07 – Paper – Price – An Implementation of Automated Structural Design-To-Cost
Scaled Agile Deliveries: Do We Still Need Estimates?
Agile Track (AG08)
Eric van der Vliet
Organizations prefer agile software development methodologies because of the high level of agility of requirements and the focus on delivering business value. Self-controlling teams deliver as much as possible value within the available budget. Attention for estimation is reducing what has a direct impact on the control of agile deliveries. This presentation explains what amount of estimation is required in what stage of a scaled agile delivery to provide the right level of delivery control.
Keywords: Agile, budgeting, software, story points, epics, features, estimation, control, scaled
AG08 – PPT – van der Vliet – Scaled Agile Deliveries
The Concepts of Size and Productivity in an Agile World
Agile Track (AG09)
Harold van Heeringen
Many organizations have moved to agile development software development. One of the cornerstones of one of the main Agile methods, Scrum, is to de-professionalize estimation and to use an arbitrary effort unit of measurement, the story point, to estimate projects. Because of a lack of understanding of the matter, management in these organizations believe that this is the way to go, but now they are facing huge challenges in estimation and budgeting. it is explained why agile has resulted in lower estimation maturity in the industry, and the ways this can change again using industry standards.
Keywords: no keywords provided
AG09 – PPT – van Heeringen – The Concepts of Size and Productivity in an Agile World
Acquisition & Operations Track
Potential Impacts of Non-Major Program Data Collection on Cost Estimating
Acquisition & Operations Track (AO01)
Brandon S. Bryant
Stephanie Myrick
The FY 2017 NDAA requires the armed services to collect cost data on all acquisition programs over $ 100 M. The requirement creates a challenge of balancing data collection standardization and flexibility for programs that have not been required to follow traditional data collection strategies. However, this new requirement is an opportunity to improve DOD cost analysis. This paper provides implementation strategies, how to leverage current data collection tools, and potential impacts on cost estimating.
Keywords: Data collection, DOD/MOD, government
AO01 – Bryant – PPT – Potential Impacts of Non-Major Program Data
AO01 – Paper – Bryant – Potential Impacts of Non-Major Program Data
Using Sustainment CSDR Data
Acquisition & Operations Track (AO03)
Sandra B. Enser
Operations and support costs can exceed 60% of the total Life Cycle Cost of aerospace and defense systems. This work is increasingly performed by contractors rather than by organic facilities. This paper will demonstrate how the Sustainment Cost and Software Data Report (CSDR) data collected by DCARC can be used to estimate future sustainment costs, support Performance-Based Logistics business case analyses, identify maintenance and repair problems, streamline review of hours and costs in negotiations, and evaluate contractor profits.
Keywords: Cost management, data collection, data-driven, life cycle, operations, PBL, CSDR
AO03 – PPT – Enser – Using Sustainment CSDR Data
AO03 – Paper – Enser – Using Sustainment CSDR Data
Using Army Software Sustainment Cost Estimating Results
Acquisition & Operations Track (AO04)
Cheryl Jones
James Doswell
Bradford Clark
The Army has completed an initial analysis of software sustainment cost and performance data collected from ~250 Weapons, C4ISR, and ERP systems. The analysis addresses primary resource distributions and cost estimating relationships across multiple functional domains, and establishes a foundation for efficient resource allocation decisions across the Army systems portfolio, and projected policy and process changes. The results, including the detailed statistical analysis, will be made available for use by participants.
Keywords: Cost management, data-driven, DOD/MOD, parametrics, regression, software, sustainment, maintenance
AO04 – PPT – Jones – Using Army Software
Cost and Competition in U.S. Defense Acquisition
Acquisition & Operations Track (AO05)
Eric Lofgren
The cost estimator has a major role in determining the price, and therefore value, of major systems acquisition in the Department of Defense. Two primary costing methodologies include “should cost” and “will cost” analysis, and are affected by “must cost” realities. This paper explores the history of these costing methods and places them in a theoretical context, first with respect to the meaning of competition, and second with respect to the nature of cost.
Keywords: Government, methods, parametrics, performance management, process engineering
AO05 – PPT – Lofgren – Cost and Competition
AO05 – Paper – Lofgren – Cost and Competition
Maturing Cost Estimation in a Rapid Acquisition Environment
Acquisition & Operations Track (AO06)
Jennifer Manring
Natalie Faucher
DoD is moving toward rapid acquisition approaches to deploy critical capability to the warfighter quickly. Rapid acquisitions have shortened timelines and less definition than traditional acquisitions. These constraints challenge traditional cost estimation practices. Multiple case studies were conducted to understand the challenges to cost estimating processes in a rapid acquisition environment. Easily implementable recommendations are made to help ensure credible and confident cost estimates are developed, even within the constraints of these challenging environments.
Keywords: Cost/benefit analysis, rapid acquisition, cost estimation process
AO06 – PPT – Manring – Maturing Cost Estimation
Exploring How Systems Age (and Fail) and the Impact on O&S Cost Estimates
Acquisition & Operations Track (AO07)
Patrick McCarthy
Alex Krencicki
Significant time is spent estimating the development and production costs of weapon systems. However, estimating how systems age and fail during the O&S phase, as well as trying to identify optimal sustainment and maintenance strategies can be every bit as challenging and have a significant impact on the estimated lifecycle cost. This paper will address obstacles associated with predicting fleet aging profiles, including dynamic failure distributions and the integration of right-censored data into the estimate.
Keywords: Life cycle, methods, modeling, operations, statistics
AO07 – PPT – McCarthy – Exploring How Systems Age (and Fail)
AO07 – Paper – McCarthy – Exploring How Systems Age (and Fail)
Tabular CARDs: Orderly Data for the Cost Community
Acquisition & Operations Track (AO08)
Jeff McDowell
Recent CAPE guidance has brought positive exciting change to the Cost Analysis Requirements Description (CARD). This change is a new focus on standardized program data content via workbook templates balanced with reduced-scope narrative content. This paper demystifies the new CARD templates by describing each of the tables emphasizing how they were designed to satisfy enduring and recurring cost estimator needs. Also their relevance to MIL-STD-881D Product Extensions and CSDR Plan Standards will be presented.
Keywords: Data collection, government, life cycle, CARD
AO08 – PPT – McDowell – Tabular CARDs
Estimating Software Sustainment Costs
Acquisition & Operations Track (AO09)
Arlene Minkiewicz
Software sustainment costs can make up as much of 90% of the total ownership cost of a program, yet the software industry continues to struggle with the best way to predict these costs. Traditional cost drivers used for acquisition estimates don’t necessarily apply for sustainment. This paper discusses an on-going research project collecting cost and technical data from evolving software systems in an effort to determine the best sustainment cost drivers and cost estimating relationships.
Keywords: Data collection, decision analysis, life cycle, regression, software, software sustainment, software maintenance
AO09 – PPT – Minkiewicz – Estimating Software Sustainment Costs
AO09 – Paper – Minkiewicz – Estimating Software Sustainment Costs
Portfolio Analysis: Estimating the UK Defence Budget
Acquisition & Operations Track (AO10)
Dale Shermon
This paper will consider how to build a portfolio analysis of a government budget and, once established, what questions can be considered through this analysis. I will examine the capability gaps and the cost of replacement for obsolete capabilities. As an example the paper will consider the UK MOD budget for current and future expenditures with lessons learnt trying to conduct this ambitious analysis.
Keywords: Budgeting, cost management, DOD/MOD, government, life cycle, methods, modeling, parametrics, project controls
AO10 – PPT – Shermon – Portfolio Analysis
AO10 – Paper – Shermon – Portfolio Analysis
Diversity of Maintenance Logs and Delay
Acquisition & Operations Track (AO11)
Bryan Kenneth Anderson
Maintenance activities play a leading role during the In-Service Management phase in the FAA’s AMS. This paper explores the potential relationships between FAA maintenance activity and cost incurred by the public. The paper objective is to determine if there is a correlation between facility type, maintenance activity incurred by the FAA, and flight delays impacting the public that could drive reclassification of facility type.
Keywords: Data-driven, methods, uncertainty
AO11 – PPT – Anderson -Diversity of Maintenance Logs and Delay
AO11 – Paper – Anderson -Diversity of Maintenance Logs and Delay
Developing an Independent Government Cost Estimate
Acquisition & Operations Track (AO12)
Richard Shea
Shavaiz Saood
The Independent Government Cost Estimate (IGCE) is an unbiased cost estimate based upon Government requirements and inputs without input from potential contractors. The IGCE is a tool used by the Government during source selection as the basis for reserving funds for the contract, comparing costs propose by submitting contractors, and used as a guideline to determine contract proposal price reasonableness. This presentation discusses the steps building the IGCE and the next steps in the process.
Keywords: Cost/benefit analysis, government, life cycle, uncertainty
AO12 – PPT – Shea – Developing an Independent Government Cost Estimate
Quantifying Annual Affordability Risk of Major Defense Programs
Acquisition & Operations Track (AO13)
Thomas J. Coonce
David Tate
Using DoD Selected Acquisition Reports (SARs), the authors use a variety of statistical techniques to estimate the distribution of possible funding profiles that might result from a new program, given its initial estimates, program characteristics, and funding climate. The authors then extend this method to address the remaining annual cost risk of partially-completed programs. The results allow analysts to assess the annual confidence levels of a given program.
Keywords: Bayesian, cost management
Comprehensive Perspectives Track
An Empirical and Visual Tale of a Cross-Country Bicycle Adventure
Comprehensive Perspectives Track (CP01)
Rick Collins
Maggie Dozier
Orly S. Olbum
Paul Lanier Hardin, III
The ‘road’ from Anacortes, Washington to Bar Harbor, Maine is paved with amazing landscapes, small towns, interesting people and cold beer. This 74 day cross-country cycling trip was an ideal opportunity to collect data that might explain daily riding speed. This presentation (and companion paper) paper describes the journey and post-ride analysis of the data using influence diagrams and constrained optimization (via Excel Solver) and will hopefully inspire others to get on a bike and experience the ‘power’ of cycling.
Keywords: Data collection, data-driven, influence diagrams, modeling, optimization, parametrics, optimization, statistics
CP01 – PPT – Collins – An Empirical and Visual Tale
CP01 – Paper – Collins – An Empirical and Visual Tale
Have All the Cost Estimates Already Been Done? Data Science in Cost Analysis
Comprehensive Perspectives Track (CP03)
Jeremy Eden
Data Science and “big data” or the analysis of large quantities of consolidated and searchable information are having great impacts on operations, industries, scientific fields, and analytical disciplines, including cost estimating. This paper will provide a basic understanding of data science, discuss some current issues and solutions for cost estimators using data science, and suggest some steps ICEAA can take to incorporate data science with current cost estimating training and certifications to capture and shape the benefits of the data science revolution in cost analysis.
Keywords: Agile, data collection, data-driven, government, methods, microsoft excel, modeling, parametrics, data science, big data, analogous
CP03 – PPT – Eden – Have All the Cost Estimates Already Been Done?
Introduction to the Organizational Cost Community Framework: Update
Comprehensive Perspectives Track (CP04)
Terry Josserand
Edwin P. Chamberlin
Leone Z. Young
One specific area of emergent research and assessment, with respect to organizational role and responsibility overlaps and gaps, exists within an organization’s ability to generate sustainable and defensible cost estimates. This paper will assist organizations facing the challenge to improve their cost estimation and analysis capabilities through the general characterization of the four interdependent cost functions, their processes, and the participants all within the systematic Organizational Cost Community Framework.
Keywords: Infrastructure, management, lifecycle, framework, process improvement, roles, responsibilities, capabilities, government, R&D
CP04 – PPT – Josserand – Introduction to The Organizational Cost Community
CP04 – Paper – Josserand – Introduction to The Organizational Cost Community
WBS vs CES: Navigating Different Structures for Software Systems
Comprehensive Perspectives Track (CP05)
Brad Dahlin
Bakari Dale
Army estimates are created using either a Work Breakdown Structure (WBS) and the Cost Element Structure (CES). Determining which structure to use for your cost estimate can be a challenge. Through this presentation, the authors will present when to use which structure, and what information and level is required for each and provide information as to when a WBS/CBS is required.
Keywords: Cost management, cost/benefit analysis, government, cost element structure
CP05 – PPT – Dahlin – WBS vs CES
Being Certain About Uncertainty: Part 2
Comprehensive Perspectives Track (CP07)
Andy Prince
Christian B. Smart
This paper addresses the difficult and pervasive challenge of identifying extreme cost growth early in a project’s life cycle and preventing it before it happens. The paper examines how DoD and NASA have implemented policies and practices to minimized or eliminate extreme cost growth and why those policies can sometimes fail. Finally, we propose some remedies that could help and identify some warning signs that a project may be headed for trouble.
Keywords: Process engineering, cost growth
CP07 – PPT – Prince – Being Certain About Uncertainty: Part 2
CP07 – Paper – Prince – Being Certain About Uncertainty: Part 2
Where Have All the Estimators Gone?
Comprehensive Perspectives Track (CP08)
Cris Shaw
Tom Dauber
Andrea West
Bryan Kenneth Anderson
The declining budgets, expanding oversight, sequestration challenges, and acquisition changes faced by Government programs has resulted in the increased importance of cost estimating and analysis, and higher value of qualified cost estimators. However, like many associations and organizations, ICEAA has seen a decline in membership despite expanding the overall focus of the organization into new areas. We will identify key influencing events, analyze recent ICEAA membership trends and recommend potential areas for membership growth.
Keywords: Government, cost estimating, cost analysis, certification, membership, data analytics, clearance
CP08 – PPT – Shaw – Where Have All the Estimators Gone?
CP08 – Paper – Shaw – Where Have All the Estimators Gone?
Investigation of the Disconnect Between the Use of CSDRs Cost and Contracting Communities
Comprehensive Perspectives Track (CP09)
Marc Stephenson
Brian Davis
Today, the organization of project offices separates the program manager (PM) from the contracting officer. It might seem that for this reason, the PM’s cost team often fails to adequately support contract negotiations. However, this paper will demonstrate that a more fundamental problem is the disconnected structure and flow of information to each of the functional teams. It will investigate the ties and prospective improvements to bring the cost estimator back into the contracting cycle.
Keywords: no keywords provided
CP09 – PPT – Stephenson – Investigation of the Disconnect
Processes of Weapon Systems Acquisition
Comprehensive Perspectives Track (CP10)
Eric Lofgren
One major outcome of recent reforms in defense acquisition has been the organizational separation of research and development from production and sustainment. But is the reform simply a change in organization charts or does it have real implications for the cost estimator? This companion to a 2017 paper will build a historical understanding of the defense innovation and procurement processes and suggest a proper role for the cost estimator under the new organization.
Keywords: government, uncertainty, decision analysis, acquisition process, management
CP10 – PPT – Lofgren – Processes of Weapon Systems Acquisition
CP10 – Paper – Lofgren – Processes of Weapon Systems Acquisition
Economic/Data Analysis Track
Applying Economic Theory to Cost Recovery in DoD Working Capital Funds
Economic/Data Analysis Track (EA01)
Kathryn Connor
Michael Vasseur
A DoD working capital fund must recover its costs, but some customers have questioned the cost of these services, which are perceived to be too high and include readiness and other costs not directly tied to customer demand. This study identifies and tailors best commercial practices in order to better align customer incentives with DoD goals and readiness needs. We recommend that this DoD working capital fund implement preferential pricing for those activities that contribute to readiness.
Keywords: Cost management, DOD/MOD
EA01 – PPT – Connor – Applying Economic Theory to Cost Revovery
Diamonds in the Rough: How to Normalize Cost Accounting Data
Economic/Data Analysis Track (EA02)
Niatika Griffin
Jason DeLorenzo
Every government agency has a cost accounting system. For cost estimation purposes, this is a wealth of actual cost data that can be used as a reference for estimates. Unfortunately, the needs of financial analysts differ from cost analysts. This paper explores how to normalize cost accounting system data using FAA’s Delphi system as a case study. We conclude given certain assumptions the data can be normalized into usable source data and customized reports.
Keywords: no keywords provided
EA02 – PPT – Griffin – Diamonds in the Rough
EA02 – Paper – Griffin – Diamonds in the Rough
Projecting Future Costs with Improvement Curves: Perils and Pitfalls
Economic/Data Analysis Track (EA04)
Brent M. Johnstone
Improvement curves are one of the most common projection tools used by cost estimators. Their use is surrounded however by perils and pitfalls. Common errors include: the fallacy of “straight edge and graph paper” projection, the dangers of recovery slopes, failure to understand how development and production environments differ, and the dangers of using learning curve slopes to measure production line efficiency. This paper examines these potential pitfalls and proposes ways to avoid them.
Keywords: Labor, learning curves, manufacturing, methods, modeling, parametrics
EA04 – PPT – Johnstone – Projecting Future Costs with Improvement Curves
EA04 – Paper – Johnstone – Projecting Future Costs with Improvement Curves
A Case Study for Future Munition: An Analysis of Alternatives
Economic/Data Analysis Track (EA05)
Faye Kim
Meagan Gadreault
An Analysis of Alternatives (AoA) is essential in making a rational funding decision particularly before MS A decisions. An AoA was conducted to determine the Total Ownership Costs (TOC) for 21 possible Courses of Actions (COAs) for the design of a future munition. Multiple options within the future munition’s three main hardware sections and software section create the COAs, and differences in these options within the COAs drive the TOC that impact the overall AoA decision.
Keywords: Cost/benefit analysis, government, life cycle, Microsoft Excel, modeling
EA05 – PPT – Kim – A Case Study for Future Munition
Improved Cost and Technical Data Collection for Contractors
Economic/Data Analysis Track (EA06)
Greg Kiviat
John Swaren
As a follow on to the 2017 ICEAA Workshop presentation “Lessons Learned in Leveraging Historical Cost, Schedule and Technical Data” the authors present results of a Part II post-study enhancement applying standardized CARD (Cost Analysis Requirements Description) to existing data collection process, per audience feedback. The team has applied this enhanced approach and is exploring how contractors can leverage government standard process to improve the existing approach and gain insight beyond model calibration (e.g. CER development, analogous programs, etc).
Keywords: Cost management, data collection, data-driven, DOD/MOD, government, modeling, parametrics, lessons learned, estimating
EA06 – PPT – Kiviat – Improved Cost and Technical Data
EA06 – PPT – Kiviat – Improved Cost and Technical Data
Calibrating Use Case Points Using Bayesian Analysis
Economic/Data Analysis Track (EA07)
Kan Qi
Anandi Hira
Elaine Venson
Barry Boehm
Use Case Points (UCPs) has been widely used to estimate software size for object-oriented projects. Yet, many research papers criticize the UCPs methodology for not being verified and validated with data, leading to inaccurate size estimates. This paper explores the use of Bayesian Analysis to calibrate the use case complexity weights of the UCPs method to improve size and effort estimates. Bayesian Analysis integrates prior information (in this study, we use the weights defined by the UCPs method and the weights suggested by other research papers) with parameter values suggested by data. The effectiveness of the calibration approach has been evaluated by our empirical study of 34 use case driven projects. We found that the Bayesian estimates of the use case complexity weights consistently provide better estimation accuracy compared to the weights proposed by the original UCPs method, the empirically calibrated weights, and the expert-based weights.
Keywords: Bayesian, cost management, cost/benefit analysis, data collection, early cost, methods, parametrics, regression, statistics, uncertainty, variables, effort estimation, use case analysis, use case driven process, model calibration, data normalization, software size metrics, model-based analysis, incremental estimation model, use case points, object-oriented modeling, project management, Bayesian analysis, local calibration
EA07 – PPT – Qi – Calibrating Use Case Points
Robust Non-Design, Code, Test, and Integration Cost Estimating Relationships
Economic/Data Analysis Track (EA08)
Britt Staley
Nicole Robertson
Computer program development (CPDEV) models are founded in accurately estimating the design, code, test, and integration (DCTI) of the computer program. Given that non-DCTI support costs can account for 50 percent or more of the estimate total, it is also critical to accurately capture these indirect costs. This paper analyses almost two decades of historic data from nineteen CPDEV efforts to derived timely non-DCTI cost estimating relationships for use in these types of estimates.
Keywords: Data-Driven, government, methods, modeling, parametrics, regression, software, statistics
EA08 – PPT – Staley – Robust Non-Design, Code, Test and Integration
The Art of Employing Data Science to Improve Cost Data Analysis
Economic/Data Analysis Track (EA10)
Richard Shea
Shavaiz Saood
While large data sets are highly desired and used frequently, there is not a universally accepted standard format for large data extractions and transfers. This presentation focuses on gaining insights into large amounts of data, spotting inconsistencies, and getting data into a usable format. This presentation examines the possibility of putting data into a standard database format so it can be easily manipulated and cross compared against other datasets that may have comparable elements.
Keywords: Data collection
EA10 – PPT – Shea – The Art of Employing Data Science
EA10 – Paper – Shea – The Art of Employing Data Science
Demand, Recurring Costs, And Profitability
Economic/Data Analysis Track (EA12)
Douglas K. Howarth
Customers in all markets collectively abide by their self-imposed demand curves, which dictate their responsiveness to changes in price and the maximum quantities of products they can absorb. Concurrently, producers in all markets face recurring costs, which typically fall over time due to a variety of factors. Producers can effectively model demand and recurring costs before product launch. Understanding how demand curves relate to recurring costs is key to enhancing profitability, which this paper examines.
Keywords: Profitability, profits, demand frontier, aggregate market demand, end of program
EA12 – PPT – Howarth – Demand, Recurring Costs, and Profitability
EA12 – Paper – Howarth – Demand, Recurring Costs, and Profitability
A Robot Brain Might Be the Best Forecasting Tool Possible
Economic/Data Analysis Track (EA13)
Nathan Eskue
Do you have to predict the future as part of your job—revenue, headcount, demand, production, etc.? Is it frustrating that something always seems to mess up your estimates? The robot brain—Artificial Neural Network (ANN)—can predict patterns in the midst of chaos. I’ll share its mechanics/purpose, my experience using ANN’s to self-learn (what could go wrong?), and how that same technology can produce forecast accuracy better than any other method.
Keywords: Bayesian, bias, cost management, cost/benefit analysis, data collection, data-driven, decision analysis, learning curves, manufacturing, methods, modeling, Monte Carlo, parametrics, performance management, regression, risk, statistics, uncertainty, variables
EA13 – PPT – Eskue – A Robot Brain
Data Science Cost Estimating Challenges: Cut Your Time in Hack!
Economic/Data Analysis Track (EA14)
Joe Rohner
Lorraine Feury
The increasing velocity and variety of cost analysis data demands holistic data science approaches performed in near-real time. How can you discover multiple innovative, unbiased, cutting-edge solutions in just three days? Host your own Data Science Hackathon cost estimating challenge! Hackathons are environments that incorporate external open-source tools coupled with super-smart data scientists to solve challenges impeding success. Learn Hackathon best practices that serve as innovative catalysts and transform how your organization solves complex problems.
Keywords: Bayesian, cost/benefit analysis, data collection, data-driven, decision analysis, methods, modeling, parametrics, performance management, risk, data science, hackathon, innovation
EA14 – PPT – Rohner – Data Science Cost Estimating Challenges
Methods & Models Track
Learning Rate Sensitivity Model
Methods & Models Track (MM01)
Timothy P. Anderson
Nichols F. Brown
In space cost estimates, learning curves are used to estimate cost of small quantity acquisitions. Recently, spacecraft providers have started proposing unprecedentedly large constellations. The authors have developed a methodology to test assumptions about learning rates versus proposed cost estimates, providing a data-driven assessment of whether a proposed learning rate/cost combination is feasible or even likely, and further describes the learning rate would be necessary in order to meet a proposed cost estimate.
Keywords: Budgeting, data-driven, learning curves, manufacturing, modeling, space
MM01 – PPT – Anderson – Learning Rate Sensitivity Model
MM01 – Paper – Anderson – Learning Rate Sensitivity Model
An Approach Towards Determining Value Through the Application of Machine Learning
Methods & Models Track (MM03)
Christopher Hutchings
Within the field of data analytics, machine learning is a method used to develop algorithms that lend themselves to prediction; often known as predictive analytics. These analytical models allow ‘interrogators’ to expedite the production of reliable and repeatable results and uncover hidden insights through learning from legacy relationships and trends in the available data. The intention of this paper is to demonstrate a value based engineering application of this embryonic field of analysis.
Keywords: Bias, data-driven, decision analysis, functional requirements
MM03 – PPT – Hutchings – An Approach Towards Determining Value
MM03 – Paper – Hutchings – An Approach Towards Determining Value
Building Dynamic Cost Estimating Models
Methods & Models Track (MM04)
Miranda Jones
Organizations are driving requirements for both cost estimators and analysts to be agile and deft in the application of dynamic, heterogeneous data efficiently with process repeatability. Developing versatile estimating models saves time, reduces errors, and drives consistency. This presentation will provide an overview of the “what” and “why” of dynamic estimating models and the techniques for building adaptable models that can be applied to any type of cost estimating discipline.
Keywords: Data collection, data-driven, methods, Microsoft Excel, modeling
MM04 – PPT – Jones – Building Dynamic Cost Estimating Models
Unmanned Space Vehicle Cost Model: Past to Present
Methods & Models Track (MM05)
Ben Kwok
Chinson Yew
Space, the final frontier. These are the cost estimates that employ the Unmanned Space Vehicle Cost Model (USCM). USCM’s 45+ year mission: to assist in the development of cost estimates, to seek out new data sources and new methodologies, to boldly evolve to meet the ever changing landscape of the cost world…Take a journey into USCM’s development, a model traveling at warp speeds since the 60’s that has undergone major evolutions over the past decades.
Keywords: Data collection, data-driven, government, methods, parametrics, space, statistics, automation, data visualization
MM05 – PPT – Kwok – Unmanned Space Vehicle Cost Model
Just-In-Time Cost Estimate in a Multidisciplinary Design Environment
Methods & Models Track (MM07)
Apinut “Nate” Sirirojvisuth
F. Gurney Thompson, III
Poor cost estimates cause inefficiency in the program and run the risk of increased bottom-line, program delay and cancellation. One remedy is to have an effective cost management framework that can evolve and adjust as the program matures. In this research case study, we present the formulation of cost-capability trade space that automates multidisciplinary collaboration resulting in a just-in-time cost estimate from any design change or technology infusion. An aircraft redesign example will be presented.
Keywords: Cost management, cost/benefit analysis, decision analysis, modeling, performance management
MM07 – PPT – Sirirojvisuth – Just-In-Time Cost Estimate
Estimating Future Air Dominance
Methods & Models Track (MM08)
David Stem
Estimating the cost of aircraft programs early in development presents special challenges. The process requires a consideration of the content of the program and various methods to be employed. The Air Force Cost Analysis Agency recently provided cost advice to the Scientific Advisory Board on how aircraft programs are estimated, recent historical experience/lessons learned, and approaches to reduce life cycle costs while the program is in the conceptual design stage.
Keywords: Decision analysis, DOD/MOD, early cost, government, methods, risk, aircraft development estimating methods
MM08 – PPT – Stem – Estimating Future Air Dominance
MM08 – Paper – Stem – Estimating Future Air Dominance
Commercial Applications of Predictive Analytics: Requirements-Driven Forecasting
Methods & Models Track (MM09)
John Swaren
Our job is often to fine-tune a predictive capability, process or system by best applying knowledge to justify key drivers for forecasting. Regardless of methodology, internally-developed or licensed, the common objective is creating valid/defensible estimates based on actual historical data as well as performance parameters. This paper will examine four commercial case studies where unique Predictive Analytics methods were implemented to both leverage proprietary knowledgebases as well as reflect requirements metrics to create justify forecasts.
Keywords: Data-driven, decision analysis, functional requirements, methods, modeling, parametrics, predictive analytics, estimating
MM09 – PPT – Swaren – Commercial Applications of Predictive Analystics
Design Life Study
Methods & Models Track (MM10)
Alex Wekluk
Colleen Adamson
Matt Reiley
The ODNI/SRA/CA division studied Design Life (DL) impact on the cost of traditional satellite acquisition strategies and associated launch costs. If launch costs continue to decrease it may be prudent to move away from constellations of extremely reliable vehicles to constellations of less reliable vehicles at an affordable price. A constellation of shorter DL vehicles tolerates a higher risk at the unit level and increases flexibility to evolve ahead of a changing technology landscape.
Keywords: Cost/benefit analysis, data-driven, decision analysis, government, modeling, parametrics, risk, space
MM10 – PPT – Wekluk – Design Life Study
Proactive Estimating: The Analysis of Sixth Generation Aircraft
Methods & Models Track (MM11)
Dale Shermon
This paper will explore some of the options and alternatives which as a cost community we should be pursuing for new projects. It will examine the big, first order assumptions which we should be considering to ensure that we have a voice and that the cost is considered at the forefront of the decision process. As an example the paper will consider the options for a sixth generation fighter capability.
Keywords: No keywords provided.
Management, EVM & Scheduling Track
Modern Methods for Budget-Constrained Schedule Analysis
Management, EVM & Scheduling Track (MS01)
Nick DeTore
There’s a need across the community for methods to optimize schedules based upon constraints imposed by the available budget. Work cannot be performed until funding is available, yet current SRA/JCL tools don’t reconcile this discrepancy. The cost/schedule analysis communities understand the prevalent and vital concept of a budget-constrained schedule and yet research has been inconclusive. This paper introduces a robust method of adjusting the schedule within complex program plans in order to satisfy a budget.
Keywords: Budgeting, cost management, data-driven, government, methods, modeling, monte carlo, risk, scheduling, statistics, uncertainty, joint confidence analysis (JCL), schedule risk analysis (SRA)
MS01 – PPT – DeTore – Modern Methods
MS01 – Paper – DeTore – Modern Methods
Integrating Excel Cost Models and MS Project Schedules
Management, EVM & Scheduling Track (MS02)
Melvin R. Etheridge, Jr.
The FAA requires non-risk-adjusted “resource loaded” schedules as part of its Milestone Decision process. Integrating cost and schedule risk in MS Project schedules and Excel cost models is challenging. These challenges stem from the fundamental differences between a cost estimate, which is a tree, and a schedule, which is a network and the dependency of some cost elements to duration. This paper investigates techniques for linking risk-adjusted schedules and cost estimates. It highlights the need for close collaboration between cost estimators and schedulers.
Keywords: Cost/benefit analysis, government, IPM, methods, microsoft excel, Monte Carlo, program management, risk, scheduling, uncertainty
MS02 – PPT – Etheridge – Integrating Excel Cost Models
The Living Estimate: Leveraging LCCEs Throughout the Program Lifecycle
Management, EVM & Scheduling Track (MS04)
Sriram Krishnan
Kevin Schutt
While cost estimates are crucial for program planning and approval – after program approval, where does the estimate go? This paper examines how the LCCE can support and inform a program office beyond the planning stages. We will also look at the cost analyst’s role in the Program Control and Business Management functions, and how s/he can add value to all aspects of the Program Control team.
Keywords: Cost management, program management
MS04 – PPT – Krishnan – The Living Estimate
Your Schedule is in Shambles and This is Why: A Systematic Approach to Why So Many Programs Fail
Management, EVM & Scheduling Track (MS05)
Kai Lemay
Patrick Myers
In this presentation, we explore the root causes for why programs fail to achieve schedule targets, either set by them or set by external stakeholders. In this presentation we review how requirements, contracts and contract requirements, program management approaches (or lack thereof), scope creep, unrealistic estimates, and not taking a data driven decision approach to decision making all come together to form a perfect storm that prevents programs from achieving the schedule targets.
Keywords: Project controls, scheduling
MS05 -PPT – Lemay – Your Schedule is in Shambles and This is Why
MS05 -Paper – Lemay – Your Schedule is in Shambles and This is Why
Adventures in Using Contractor Cost Data Reports for Wheeled and Tracked Vehicles Analysis
Management, EVM & Scheduling Track (MS06)
Kimberly Roye
Jennifer Scheel
The Cost Assessment Data Enterprise (CADE) contains multiple types of contractor cost data reports (e.g. CDSRs, FCHRs, CPRs) that cost analysts may find useful in developing cost estimating relationships. Understanding the best report(s) to use depends on the analysis being performed and the optimal choice is critical to meaningful analysis. This paper will provide lessons learned to help analysts successfully navigate available cost reports and avoid common pitfalls.
Keywords: Data-driven
MS06 – PPT – Roye – Adventures in Using Contractor Cost Data Reports
Risk & Uncertainty Track
To Monte Carlo Or Not To Monte Carlo…That Is The Question
Risk & Uncertainty Track (RU01)
Joe Bauer
In cost estimating, the two most often used risk / uncertainty analysis methodologies are Monte Carlo simulation and Method of Moments. Have you ever stopped to wonder about the difference between the two? Fear not! In this presentation, the authors will compare and contrast the two risk / uncertainty analysis methodologies through several case studies. The authors will also share several unique methods for allocating risk dollars across the program phases.
Keywords: Risk, monte carlo, method of moments, risk phasing
RU01 – PPT – Bauer – To Mote Carlo Or Not To Monte Carlo
RU01 – Paper – Bauer – To Mote Carlo Or Not To Monte Carlo
Risk-Adjusted Contract Price Methodology
Risk & Uncertainty Track (RU02)
Peter J. Braxton
Keith S. Hetrick
Orly S. Olbum
Traditional risk approaches rely on program-level risk and uncertainty benchmarks from SARs, but governments and other buyers need to assess and manage risk at the contract level. For the first time, the Risk-Adjusted Contract Price Methodology explicitly models both “off-the-shareline” risk and “on-the-shareline” risk to present a complete and accurate distribution of final contract price. Drawing from a robust CLIN-level database of cost, fee, and price changes over time, it incorporates historical benchmarks for contract changes and other growth.
Keywords: Cost management, data-driven, government, modeling, risk, uncertainty
RU02 – PPT – Braxton – Risk-Adjusted Contract Price Methodology
RU02 – Paper – Braxton – Risk-Adjusted Contract Price Methodology
Estimating the Cost of Pharmaceuticals: Managing Cost and Expectations
Risk & Uncertainty Track (RU03)
Thurman D. Gardner
Robert Hunt
Recently headlines have been addressing the high price of pharmaceuticals. Contrary to what it may seem, greed does not drive pricing, it is the unknown and recovery of costs. Science provides a big piece of this, but the regulatory pathway is almost as big a factor, and in some cases bigger. Estimating new products requires a solid methodology and forward-thinking approaches to adequately bound projected costs and expectations.
Keywords: Cost management, program management, risk, uncertainty, biotech
RU03 – PPT – Gardner – Estimating the Cost of Pharmaceuticals
RU03 – Paper – Gardner – Estimating the Cost of Pharmaceuticals
Calculating a Project’s Reserve Dollars from its S-Curve
Risk & Uncertainty Track (RU04)
Marc Greenberg
A probabilistic method was developed to calculate funds a project would need if it exceeded the project’s point estimate. This conditional cost reserve is depicted as the amount of funds held in reserve “above the project of program.” The method requires three data inputs to calculate cost reserve: (1) the project cost dispersion (measured by its coefficient of variation), (2) the project point estimate, and (3) the confidence level of the project point estimate.
Keywords: Budgeting, cost management, early cost, government, life cycle, methods, Microsoft Excel, modeling, risk, statistics, uncertainty, budget, confidence level, distribution, variance, margin, point estimate, portfolio, probability, probabilistic, project, reserves, s-curve, simulation, UFE, value
RU04 – PPT – Greenberg – Calculating a Project’s Reserve Dollars
Enhancing Risk Calibration Methods
Risk & Uncertainty Track (RU06)
Christian B. Smart
Calibration methods such as the Enhanced Scenario-Based Method allow analysts to establish cost risk analyses that are based on objective data. Some methods currently in use rely on the normal and two-parameter lognormal. Empirical data, however, indicates that a three-parameter lognormal is more appropriate for modeling cost risk. We discuss three-parameter lognormals and how to calibrate cost risk using this distribution. We compare the results with traditional calibration to two-parameter normal and lognormal distributions.
Keywords: Data-driven, DOD/MOD, early cost, government, modeling, risk, statistics, uncertainty
RU06 – PPT – Smart – Enhancing Risk Calibration Methods
RU06 – Paper – Smart – Enhancing Risk Calibration Methods
RU06 – Excel Spreadsheet – Smart – Enhancing Risk Calibration Methods
Software Estimating Track
Impact of Scope Changes on Software Growth
Software Estimating Track (SW01)
Jonathan Brown
Gail Flynn
The SEI DoD Software Factbook summarizes MDAP/MAIS SRDR data for DoD programs. The mean value reported for ESLOC growth is 106%. While accurate, the SEI’s and other similar analyses capture total software growth, including the impact of scope changes. This paper introduces “Pure Software Growth” which differentiates planned scope changes from traditional software growth. Several programs are analyzed from this perspective to show the difference between pure and total growth and the unexpected impact this could have on estimates.
Keywords: Government, software, uncertainty, software growth
SW01 – PPT – Brown – Impact of Scope Changes on Software Growth
SW01 – Paper – Brown – Impact of Scope Changes on Software Growth
Predicting Maintainability for Software Applications Early in the Life Cycle
Software Estimating Track (SW02)
Cara Cuiule
Maintainability is defined as the difficulty of altering a software system’s source code, thus it is tied very closely to the concept of software maintenance. The following research is an investigation of the methods used for measuring this characteristic (including the Maintainability Index). Guidance on how maintainability affects maintenance effort will be proposed. This will be followed by a discussion of which metrics could possibly predict maintainability early in the life cycle.
Keywords: Cost management, early cost, methods, software, software cost estimation, software maintenance, maintainability, maintainability index
SW02 – PPT – Cuiule – Predicting Maintainability for Software Applications
SW02 – Paper – Cuiule – Predicting Maintainability for Software Applications
Software Made Simple: Effort Adjustment Factors and the Accuracy of the Estimate
Software Estimating Track (SW04)
Jeremy Goucher
This research investigates the sensitivity of estimated hours for software development in relation to effort adjustment factors (EAFs). The analysis highlights the importance of performing original data analysis for new estimates, rather than relying on rules of thumb or industry standards. Accuracy and precision metrics are used to demonstrate the applicability of the ESLOC method to a variety of software projects, including agile projects. Finally, the analysis is supported with data from over thirty historic programs.
Keywords: Agile, data collection, data-driven, government, methods, software
SW04 – PPT – Goucher – Software Made Simple
SW04 – Paper – Goucher – Software Made Simple
Cost of Software Obsolescence Resolutions
Software Estimating Track (SW05)
Sanathanan Rajagopal
Software plays an important role in defence. Almost every project in defence has software elements with various degrees of complexity and dependencies. This has brought its own challenges to the availability-based contracts. The challenges to both the contractors and the suppliers is that they have to have a good understanding of the whole life cost of the product and have confidence in the whole life cost model at the time of negotiation and contract signing. This paper will look into the ways to estimate the cost of Software Obsolescence resolutions of real-time defence software.
Keywords: Cost management, DOD/MOD, IT, modeling, software, software obsolescence
SW05 – PPT – Rajagopal – Cost of Software Obsolescence Resolutions
Software Data Collection and Analysis for Proposal Evaluation
Software Estimating Track (SW06)
Ken Rhodes
Eric Hippensteel
Software data collection forms provide necessary information for the Government to assess the validity of proposed software development effort, productivity, sizing, and schedule. This paper discusses use of the forms with an RFP to validate contractor proposals and compare across multiple bids. This data collection and evaluation approach helps ensure selection of the best vendor and defend against protests, and was recognized by Defense Procurement and Acquisition Policy (DPAP) as a DoD acquisition best practice.
Keywords: Data collection, software
SW06 – PPT – Rhodes – Software Data Collection and Analysis
SW06 – Paper – Rhodes – Software Data Collection and Analysis
A Probabilistic Method for Predicting Software Code Growth: 2018 Update
Software Estimating Track (SW09)
Eric M. Sommer
Bopha Seng
David LaPorte
Michael Ross
Software estimating is challenging. SMC’s approach has evolved over time to tackle this challenge. Originally based on Mike Ross’s 2011 DSLOC Estimate Growth Model, we’ve updated our model to include more recent SRDR data and an improved methodology (Orthogonal Distance Regression). Discussions will focus on non-linear relationships between size and growth, unique growth for new, modified, and unmodified DSLOC, as well as correlation between DSLOC types and future efforts to include space flight software data.
Keywords: Methods, parametrics, software
SW09 – PPT – Sommer – A Probabilistic Method
SW09 – Paper – Sommer – A Probabilistic Method
Establishing Standards as the Basis for Effective Measurement and Affordability
Software Estimating Track (SW10)
Pete Pizzutillo
Measurement of application development output has long been a controversial topic. Yet as contracting relationships within industry and public sector become more strategic, buyers and sellers of software development and sustainment services require consistent and effective measures of application development output to provide: objective visibility into application development output; a rational basis for Application Development and Maintenance (ADM) investment decisions; and vendor and buyer accountability supported by data, not subjective judgments.
Keywords: Agile, budgeting, cost/benefit analysis, data-driven, decision analysis, IT
SW10 – PPT – Pizzutillo – Establishing Standards
Why Does Software Cost So Much? Toward a Causal Model
Software Estimating Track (SW11)
Michael D. Konrad
Robert Stoddard
How can we control the cost of software-intensive systems? To contain costs we need to better understand which factors truly drive costs versus those merely correlated with cost. In this talk, we will share results from application of a newly developed causal discovery and modeling tools to evaluate these potential causes of code quality and effort: number of requirements, software size, schedule, team experience; programmer productivity and error-proneness; and architecture pattern violations.
Keywords: Bayesian, cost management, early cost, functional requirements, life cycle, modeling, project controls, software, statistic, causal learning, causal discovery, causal modeling, software cost, software estimation, empirical research
SW11 – PPT – Konrad – Why Does Software Cost So Much?
SW11 – Paper – Konrad – Why Does Software Cost So Much?
Technology & Innovation Track
Social Media and Submarines: How Machine Learning and Unconventional Methods Can Change Cost Estimating
Technology & Innovation Track (TI01)
Omar Akbik
Jeffrey Pincus
As technology advances and analytical techniques evolve, machine learning and big data systems stand to dramatically change the cost industry- these changes are coming sooner than we may think. Industry requires timely, efficient and defensible analysis, and new data sources are allowing previously unexplored opportunities to meet these goals. We explore how automated algorithms can identify market trends to forecast program cost with greater accuracy, and in less time than historically required by an analyst.
Keywords: Data collection, data-driven, decision analysis, early cost, government, infrastructure, IT, manufacturing, methods, modeling, Monte Carlo, operations, parametrics, process engineering, program management, regression, risk, software, statistics, variables, big data, analytics, machine learning, artificial intelligence, algorithms
TI01 – PPT – Akbik – Social Media and Submarines
Estimating the ‘Total Cost of Ownership’ of Cybersecurity in an Increasingly XaaS world.
Technology & Innovation Track (TI02)
Zachary Jasnoff
David Cass
Richard Mabe
Converging technology trends in XaaS have profound effects on how organizations are evaluating decisions regarding XaaS outsourcing and hybrid deployments as more business functions move to the cloud. Most organizations have a security skills gap that XaaS and moving to the cloud can help solve giving choices on insourcing or outsourcing cybersecurity. This paper explores how XaaS impacts the TCO of cybersecurity and also deliver guidance on the estimating the cost of the DFARS cyber policy to defense programs.
Keywords: Cost/benefit analysis, infrastructure, IT, performance management, program management
TI02 – PPT – Jasnoff – Estimating the Total Cost of Ownership
Costs Considerations in Refreshing Vulnerable IT Networks
Technology & Innovation Track (TI03)
John Leahy
Virtually all IT networks must deal with the growing threat of cybersecurity intrusion and yet retain sufficient features to meet mission needs. Many IT network components are close to, if not past, their end of life. This presentation will provide a high level view of alternatives analysis focused on the cost of a network refresh with secure, state of the art components. Topics include different approaches to network refresh and assessment of stakeholder interests.
Key words: IT Networks, End-of-Life, Cybersecurity
TI03 – PPT – Leahy – Costs Considerations
Machine Learning & Non-parametric methods for Cost Analysis
Technology & Innovation Track (TI04)
Karen Mourikas
Nile Hanov
Joseph King
Denise Nelson
The world of big data opens up new opportunities for ICEAA, such as machine learning and non-parametric methods. These methods are more flexible since they do not require explicit assumptions about the structure of the model. However, a large number of observations is needed in order to obtain accurate results. Hence, the use of big data. This presentation examines several non-parametric methods, with examples related to our community, and discusses opportunities and limitations going forward.
Keywords: Data-driven, methods, modeling, non-parametric, machine learning, big data
TI04 – PPT – Mourikas – Machine Learning
Application of K-Means Clustering Methodology to Cost Estimating
Technology & Innovation Track (TI06)
Jacob Walzer
Cluster Analysis provides an efficient method of analyzing large datasets and grouping them based on similar characteristics. While not suitable for all datasets, clustering allows analysts to easily classify data while considering all relevant variables. This presentation explores the best practices, common pitfalls, and validation techniques associated with clustering, and includes a notional example derived from an approved application of this technique utilized in a USMC estimate.
Keywords: Data-driven, decision analysis, methods, modeling, regression, statistics
TI06 – PPT – Walzer – Application of K-Means Clustering
Modeling Technology and System Readiness Level Impacts on LCC
Technology & Innovation Track (TI07)
Apinut (Nate) Sirirojvisuth
Bakari Dale
Technology Readiness Levels (TRLs) is an important construct used to assess maturity of a technology and set criteria for inclusion in a program with an implied level of risk. In this research, we attempt to quantify the risk implications in terms of cost and schedule impacts. A new cost model will be introduced that utilizes technology maturity assessment of constituent components of a system as a metric to determine cost, schedule, and uncertainty response.
Keywords: Cost management, cost/benefit analysis, DOD/MOD, government, parametrics
TI07 – PPT – Sirirojvisuth – Modeling Technology and System Readiness