# 2021 Breakout Session Summaries

#### Analytical Methods Track

Investigating Causal Effects of Software and Systems Engineering Effort
Analytical Methods (ANA01)
James Alstad
Mike Konrad
Anandi Hira

Causal discovery can help identify factors that drive project costs. We applied causal discovery to software and systems engineering cost estimation model calibration datasets to determine the causes of effort. Due to few variables resulting from the standard causal discovery algorithms, we came up with a technique called Weak Signal Analysis to better tease out weaker causes. Finally, we estimate the numerical impact of each such causal relationship, resulting in cost estimating equations.

Keywords: Methods, Modeling, Parametrics, Software, Statistics, Causal Inference, Causal Discovery, Systems Engineering

ANA01 – ppt – Alstad – Investigating Causal Effects of Software

Assessing Regression Methods via Monte Carlo Simulations
Analytical Methods (ANA02)
Michael Schiavoni
Richard Bearce

This presentation provides new insights while building on past research into multiplicative-error model regression methods. Prior evaluations have primarily been limited to mathematical arguments or comparisons of sample statistics using a few datasets. Conversely, this effort executes Monte Carlo simulations with thousands of iterations, thus enabling the direct estimation of measures such as true population bias. Furthermore, the capabilities/limitations of each method are explored by simulating different model forms, sample sizes, error distributions, and variances.

Keywords: R, Python, code snippets, nonlinear least squares, constrained optimization, maximum likelihood estimation, parametrics, accuracy, precision, convergence, correlation, LOLS, PING, MRLN, GRMLN, MLE, ZMPE, MPE-ZPB, MUPE, IRLS, Levenberg Marquardt, COBYLA, GRG Nonlinear

ANA02 – ppt – Schiavoni – Assessing Regression Methods

The Missing Link: An Evolution of Portfolio Natural Selection
Analytical Methods (ANA03)
Catherine Dodsworth
Remmie Arnold
Stella O’Sickey

“It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is the most adaptable to change.” –Charles Darwin. This research presents a conceptual framework and methodology for solving the missing link between learning curve estimates, prediction intervals, and S-curves and generating analytically-based affordable cost constraints to naturally select trade space in a portfolio. This holistic approach expands the evolution of the S-curve.

Keywords: Cost Management, Data-Driven, Learning Curves, Methods, Statistics

Reducing Lifecycle Cost Through Aircraft Modernization
Analytical Methods (ANA04)
Jeremy Goucher

The Department of the Navy is planning to upgrade one of the oldest fighter jets in the inventory. To support a milestone decision, an independent Government cost estimate was developed from an engineering build up including over 1,500 individual material procurement line items. Additionally, dozens of past programs were analyzed to provide parametric inputs to the cost model. The detailed data, methodology, and risk adjusted phased results will be presented. Additionally, certain metrics will be compared to past upgrade programs to determine if there are commonalities across platforms.

Keywords: Data Collection, Data-Driven, Government, Methods, Modeling, Risk

ANA04 – ppt – Goucher – Reducing Lifecycle Cost

Adding Cost Credibility with SRL and MBSE Advanced Tools
Analytical Methods (ANA05)
Patrick Malone

When space based system development programs are less than 15% spent of estimated costs, traditionally almost 80% of design costs are committed! Furthermore, programs with less than the recommended GAO knowledge at program start will likely have higher risk and unexpected cost/schedule growth. Using advanced tools like System Readiness Level (SRL) and Model Based System Engineering (MBSE) methods we show how to identify vulnerabilities early in the development life cycle to mitigate these risks.

Keywords: Budgeting, Cost Management, Cost/Benefit Analysis, Decision Analysis, Early Cost, Methods, Software, Space

ANA05 – ppt – Malone – Adding Cost Credibility

SatSim: Estimating Satellite Costs via Simulation
Analytical Methods (ANA06)
Daniel Newkirk
Ben Kwok

Cost Estimating Relationships (CERs) are the standard for traditional cost estimators in estimating satellite designs. As new satellites and architectures are proposed, traditional CERs become less defensible as the design uniqueness increases. In order to better estimate future systems, the Space and Missile Systems Center (SMC) cost research team developed a simulation-based approach to estimate satellite costs with equivalent or better accuracy than traditional CERs.

Keywords: Data-Driven, Modeling, Monte Carlo, Space, Cost Estimating, Python, Cost Model

ANA06 – ppt – Newkirk – SatSim Estimating Satellite Costs

Foundation of Structured Architecture, System & Cost Modeling
Analytical Methods (ANA07)
Danny Polidi
Mike Crist

Modern software packages can perform complex physics based simulations and alone typically do not consider cost as an input variable while other packages specialize in determining cost. This paper begins the development of a structured System Engineering approach to System design and defines a standardized modular diagram for a RADAR System applied to military applications in the aerospace industry. It will be demonstrated that a system can be defined in standard terms, modeled with modular blocks and be costed by those same modular blocks.

Keywords: Keywords not provided.

Beyond the Matrix – The Quantitative Cost and Schedule Risk Imperative
Analytical Methods (ANA08)
Christian Smart

The first step in risk management is gaining an appreciation for risk and uncertainty. However, despite a long history of cost and schedule growth in all types of projects, we tend to focus on averages rather than recognize extremes. Project management is blind to risk. The fixation on averages leads to an underestimation of risk. The use of qualitative methods, including risk matrices, is prevalent, but should be analyzed with rigorous quantitative methods.

Keywords: Risk

ANA08 – ppt – Smart – Beyond the Matrix

#### Cyber Security & Cloud Computing Track

Improved CERs to Estimate Commercial IaaS Costs for Federal IT Systems
Cyber Security & Cloud Computing (CYC01)
William Gbelee
Cara Cuiule

Previous research established vendor agnostic estimating methods for the following Infrastructure as a Service (IaaS) costs: virtual machine and storage data. New data has led to price updates to the dataset resulting in over 27,000+ world-wide data points. This includes publicly available government data. The statistical analytic approach provides validated vendor agnostic models. The analysis was conducted and verified by randomly separating the samples of data into training and test sets.

Keywords: Data-Driven, Government, IT, Cloud

CYC01 – ppt – Gbelee – Developing CERs to Estimate Commercial

Planning a Cloud Migration Effort: Cost Estimating Considerations
Cyber Security & Cloud Computing (CYC02)
Kyle Ferris

Cloud First vs. Cloud Smart? Private Cloud vs. Public Cloud? Multi-Cloud vs. Hybrid Cloud? With the growing prevalence of cloud solutions and strategies across the Federal Government, developing a defensible cost estimate for the migration of legacy data to cloud environments becomes increasingly complex. Responding to these challenges, this presentation will demystify the fundamentals of cloud computing, and establish important cost estimating considerations when defining the purpose and scope of a cloud migration effort.

Keywords: Cost/Benefit Analysis, Functional Requirements, Government, Infrastructure, IT, Life Cycle, Program Management, Software, Cloud Computing, Cloud Migration, Data Analytics

CYC02 – ppt – Ferris – Planning a Cloud Migration Effort

Cybersecurity Cost Issues Facing Today’s Cost Analyst
Cyber Security & Cloud Computing (CYC03)
Bob Hunt

There are three general approaches to coat analysis: Botton-up (engineering build up), Economic/Cost Benefit Analysis, and Top-down (parametric). Today Cyber cost analysis is principally done by Botton-up (engineering build up) and/or Economic/Cost Benefit Analysis. This paper will address the advantages and disadvantages of these two approaches. In addition, this oaper will propose a new parametric approach to cyber security cost analysis based on cyber “actors” and business sector.

Keywords: Life Cycle, Modeling, Program Management, Cybersecurity

CYC03 – ppt – Hunt – Cybersecurity Cost Issues Facing Today’s

Estimating Cloud Infrastructure: Requirements, Methodologies, and Uncertainty
Cyber Security & Cloud Computing (CYC04)
Olivia Lindsey
Alex Smith

The DoD is fully committed to the cloud – but where do we start? We’ll explore multiple existing models available for estimating cloud infrastructure including those created by NRO CAAG and PRICE Systems, and data sheets available directly through Amazon. We’ll discuss technical baseline depth needed and uncertainty considerations for each scenario while exploring economies of scale and pricing differences between security classifications for Government systems and providing ready-to-use factors and reference points for multiple cloud scenarios.

Keywords: Cloud

CYC04 – ppt – Lindsey – Estimating Cloud Infrastructure

#### Machine Learning & Data Science Track

Comprehending Chaos: Leveraging Text to Improve Analysis
Machine Learning & Data Science (MLD01)
Omar Akbik
Cadence Doyle

Data has become the world’s most valuable industrial commodity. The defense industry is no stranger to this reality. The quantitative application of unconventional data sources, such as text, can fill gaps that exist in traditional analysis, and reduce reliance on opinion based inputs that often require significant uncertainty adjustments. This paper will explore natural language processing (NLP) for classification and clustering in cost analysis and how the authors have applied it in practice.

Keywords: Data-Driven, Decision Analysis, Methods, Modeling, Statistics, Machine Learning, Natural Language Processing

The Algorithm of 10,000 Variables
Machine Learning & Data Science (MLD02)
Bryan Anderson
Morgan Heimbrock

Machine learning opens the door for software cost and schedule estimation for previously unviable scenarios in traditional frameworks. A reliable software estimation requires several intricate factors and often the data is unattainable or incomplete. By training several machine learning algorithms on actual data from an issue tracking system this case study achieved a more reliable model. Factors like issue description, individuals assigned, and issue events are shown to be important features in modeling development work.

Keywords: Bayesian, Data Collection, Data-Driven, Modeling, Software, Statistics, Variables

MLD02 – ppt – Anderson – The Algorithm of 10,000 Variables

Innovative Techniques for Analyzing Incomplete Data to Improve Cost Estimates
Machine Learning & Data Science (MLD03)
George Bayer
Bryan Anderson

Cost estimators are often presented with incomplete data sets from which they must develop business case solutions. Understanding, interpreting, and improving the data integrity are critical factors for cost estimate accuracy. In this use case, the cost estimators analyze and interpret incomplete and subjective data sets, forecast spares depletion, and estimate obsolescence. Using innovative data mining and text analysis techniques, the estimators demonstrate how improved data can result in better cost estimates and business cases.

Keywords: Data Collection, Data-Driven, Life Cycle, Obsolescence, Forecasting, Data analysis

Applying Natural Language Processing Techniques to Categorize Cost Estimation Data
Machine Learning & Data Science (MLD04)
Cara Cuiule

In cost estimation, datasets used for analysis can include text data that needs to be normalized to common terminologies, such as component categorizations. Natural language processing (NLP) and machine learning (ML) could potentially be used to solve this problem. This paper will define terminology and common techniques for both NLP and ML. It will also include a case study using a hardware dataset and NLP techniques using Python.

Keywords: Data Collection, Data-Driven, Methods, Machine Learning

Machine Learning and Parametrics for Software Joint Confidence Level Analysis
Machine Learning & Data Science (MLD05)
Sara Jardine
Kimberly Roye
Christian B. Smart

Joint Confidence Level analysis has proven to be very successful for NASA. It is typically conducted using bottom-up resource-loaded schedules. However, the use of high-level parametrics and machine learning has been successfully used by one of the authors. This approach has some advantages over the more detailed method. In this paper, we discuss the approach and provide an example of the application of machine learning and parametric analysis to software programs.

Keywords: IT, Parametrics, Risk, Scheduling, Statistics, Uncertainty

Data With A Purpose: Technical Data Initiative
Machine Learning & Data Science (MLD06)
Jeff McDowell
Courtney Clark

Technical characteristics are known to influence cost and as such are integral to cost estimating methods. Cost-driver data is an enduring cost community need yet is often an afterthought to cost data collection. This presentation is an overview of CADE’s Technical Data Report (TDR) to systemically capture this must-needed information as part of the CSDR process. Also a sample Power BI case will illustrate the powerful analysis empowered by integrating TDR data with FlexFile cost data.

Keywords: Data Collection, Data-Driven, Variables

MLD06 – ppt – McDowell – Data With A Purpose

USCM11 – an Evolution of Techniques Used to Build Cost Models
Machine Learning & Data Science (MLD07)
Benjamin Kwok
Daniel Newkirk

The development of the 11th version of USCM incorporates many innovative processes that paves the way for how future cost estimating models could be developed. This presentation will showcase analysis, processes, and techniques such as automating the generation of CERs, using machine learning techniques to classify CERs as either primary or secondary methods, using an analytics approach to evaluate your model, and more.

Keywords: Data-Driven, Decision Analysis, Methods, Modeling, Regression, Space, Statistics, CERs, automation, machine learning, visualization, analytics, USCM, SMC

MLD07 – ppt – Kwok – USCM11 – an Evolution of Techniques

Dealing with Missing Data – The Art and Science of Imputation
Machine Learning & Data Science (MLD08)
Kimberly Roye
Christian B. Smart
Dustin Hilton

Missing data is a common occurrence when collecting and analyzing data. Even when a data set includes many data points, many variables of interest will have omitted values. The most common way to deal with this situation is to exclude the data points from analysis. However, this is not ideal. We discuss a better way to deal with this issue, which is the use of imputation, a statistically rigorous method for filling in the holes.

Keywords: Data-Driven, Modeling

#### Operations & Support Analysis Track

Continuous Enhancements: An Alternative to Maintenance
Operations & Support Analysis (OSA01)
Daniel Bowers
Emily Freed
Christine Branham

With a rapidly evolving threat and the enemy’s ability to adapt and counter our weapons and technology, gone are the days of designing, producing, and maintaining a system for 20-30 years. Modern systems need the ability to adapt and update rapidly to include the latest technology. This new paradigm changes the traditional Operations & Support methodologies and estimating techniques employed by the cost analyst. We will look at the impacts to maintenance costs and the traditional funding profile.

Keywords: Budgeting, Cost/Benefit Analysis, Modeling, Statistics

OSA01 – ppt – Bowers – Continuous Enhancements

Using Analytics to Aid in Performance Based Logistics Decision Making Processes
Operations & Support Analysis (OSA02)
Paul Brown

As OEMs push towards high dollar value, long-term Performance Based Logistics (PBL) contracts, decisions surrounding these types of contracts should be driven by an analytical process. PBL contracts can be worth billions of dollars and are sold as ways to improve system performance while reducing costs. Modeling and simulation can be used to evaluate the performance and cost impacts of the PBL-suggested improvements and assist decisions makers select an appropriate way forward.

Keywords: Budgeting, Cost/Benefit Analysis, Data-Driven, Decision Analysis, DOD/MOD, Government, Modeling, Performance Management

OSA02 – ppt – Brown – Using Analytics to Aid in Performance

Army Software Sustainment: Righting Our Cost Estimating Assumptions
Operations & Support Analysis (OSA03)
Cheryl Jones
Bradford Clark
Robert Charette
Paul Janusz

A challenge in estimating software Total Ownership Costs at the beginning of a software acquisition is estimating the cost of software after development. Rules-of-thumb are often used to estimate the annual change in the software during the sustainment lifecycle. This session will present data analysis on software sustainment profiles that show a rise and fall in the number of changes over long periods of time contrary to the practice of using a constant annual change.

Keywords: Data-Driven, Life Cycle, Modeling, Software, Sustainment

Practical Estimating of Hardware Lifecycle Maintenance and Obsolescence Mitigation Strategies
Operations & Support Analysis (OSA04)
Gurney Thompson
Nate Sirirojvisuth

There are many strategies that can be employed to maintain a hardware system, and many phenomena that have been observed in systems with long lifespans which can greatly affect cost and reliability. This paper will present our research approach and will examine case studies to compare various maintenance strategies, approaches to mitigating technology obsolescence issues, modeling of non-constant failure rates, effects of mid-life upgrades, and more.

Keywords: Maintenance, Technology Obsolescence, Modeling, Tools, Analysis Case Studies

OSA04 – ppt – Thompson – Practical Estimating

#### Processes & Best Practices Track

Business Transformation of Life Cycle Cost Estimating at PEO STRI / Processes
Processes & Best Practices (PBP01)
James Golden

The U.S. Army’s PEO STRI used the tenets of Better Buying Power and Continuous Process Improvement (CPI) to improve life cycle cost estimating, focusing on the processes and applications that produce credible data-driven, comprehensive, and reliable cost estimates. Implementation resulted in earlier and faster production of estimates; increased confidence in program estimates; and improved organizational ability to defend estimates. This presentation describes the challenges, processes, and internal controls associated with success of PEO STRI initiatives.

Keywords: Keywords not provided.

PBP01 – ppt – Golden – Business Transformation of Life Cycle

Addressing Challenges in Costing Unique Large Scientific Facilities
Processes & Best Practices (PBP02)
Marc Hayhurst
Matthew Marshall
Vera Scheidlinger
Valerie Rockwell

The costing of large scientific facilities poses many challenges to cost estimators as these facilities are typically uniquely designed for specific types of research. However, when the facilities and hardware are examined at a lower level, there may be similar components which can be consulted to evaluate a new proposed system. Aerospace will share their experience in costing unique facilities including the importance of historical data, cost scaling, sensitivity analysis, and other evaluation techniques.

Keywords: Cost Management, Government, Methods

PBP02 – ppt – Hayhurst – Addressing Challenges in Costing

Effective Affordability Engineering Teams – Top 5 Things You Need to Know
Processes & Best Practices (PBP03)
Zach Jasnoff

Much is written on Affordability Analysis, but what does it take to implement an effective Affordability Engineering team within a DOD contractor? This presentation explores challenges / pitfalls facing establishing an effective team and the top five things needed to overcome them. Also discussed are the methodologies, technologies and best practices required to fully explore the cost / performance trade space. A Model Based Cost Engineering case study demonstrating Affordability Analysis principles will also be presented.

Keywords: Cost Management, Methods, Program Management

PBP03 – ppt – Jasnoff – Effective Affordability Engineering Teams

Cybersecurity Cost Estimating Factors for Business IT Systems
Processes & Best Practices (PBP04)
Richard Mabe

PRICE® Systems and MITRE analysts developed an initial set of cybersecurity cost estimating factors for the 2020 ICEAA Workshop. Peer feedback on the factors identified several issues requiring additional research to resolve. This a paper presents the follow-on data updates and analysis to address these issues, and to update the factors accordingly. The results can be applied to estimate costs for cybersecurity specific activities supporting development and operations of IT systems for the federal government.

Keywords: Cost Management, Data-Driven, Variables, Cybersecurity

PBP04 – ppt – Mabe – Cybersecurity Cost Estimating Factors

Where To Miss, What To Give Up
Processes & Best Practices (PBP05)
Doug Howarth

Many projects miss some or all of their targets and specifications. Unrealistic requirements force designs made with skyrocketing costs. To produce a successful product, one must know which conditions to keep and which to drop. Producers should discover what the market wants, doesn’t have, and can afford. In-depth market analysis reveals configurations more likely to fall within cost limits while satisfying customer requirements.

Keywords: Optimization, Hypernomics

PBP05 – ppt – Howarth – Where To Miss, What To Give Up

The Art of the Interview
Processes & Best Practices (PBP06)
Joe Bauer

Cost estimation is often called an art and a science. From an art perspective, the estimator must be able to effectively communicate with people to elicit information and deliver actionable intelligence. In this presentation, we will discuss interview techniques and other “soft skills” that enable the estimator to effectively obtain useful information, build rapport and collaborate with a number of different audiences, to include subject matter experts, engineers, program managers, customers, and stakeholders.

PBP06 – ppt – Bauer – The Art of the Interview

Using Integrated Program Management (IPM) Principles Effectively
Processes & Best Practices (PBP07)
Mike Thompson

Integrated Program Management(IPM) is not just managing the Technical aspects for the program and looking to achieve the program goals. Besides the technical piece, there is an entire team effort, which includes Cost Estimates and Analysis, Scheduling, and Earned Value Management, which work together to present a total picture of the program, based on separate data. In the past there has been a tendency to “silo” these disciplines and not identify or use the them in an integrated way. The purpose of this paper is to illustrate how the use of IPM Principles.

Keywords: Performance Management, Program Management, Project Controls

Forecasting of Agile deliveries
Processes & Best Practices (PBP08)
Eric van der Vliet
Jurjen Bloo

Agile is the most common delivery approach for software projects. Single teams are able to control their delivery, more challenging is the delivery of multiple teams or multiple release trains. How can management keep control in such a situation? How can management determine what value has been delivered? This presentation is about a Power BI based solution that consolidates the information from different teams and release trains and consolidates this information in trends for management to take decisions with respect to performance, quality and value.

Keywords: Agile, Data Collection, Data-Driven, Decision Analysis, Performance Management, Program Management, Project Controls, Software, Statistics

PBP08 – ppt – van der Vliet – Forecasting of Agile deliveries

#### Soft Skills & Communication Track

Does Cost Team Leadership Matter
Soft Skills & Communication (SSC01)
Christina Snyder

An anonymous survey of 150+ cost analysts unanimously reported that a cost-team lead’s effectiveness ultimately impacts the team’s products. However, there has been minimal guidance as to what defines good leadership. Using the ten behaviors identified by Google’s Project Oxygen, this paper seeks to understand what skills are necessary for successful cost leadership. The findings lead to a simple conclusion that mirrors that of Project Oxygen: improving our soft skills will improve cost leader efficacy.

Keywords: Communication, Program Management, soft skills

Faster…Better…Cheaper…Improving Counting Productivity and Delivery
Soft Skills & Communication (SSC02)
Sheila Dennis

Do you want to get the best ROI from your function point team? From a business perspective, there are five core goals for any program: 1-Effectively manage workflow; 2-Proactively manage end user expectations; 3-Accurately plan, budget and forecast deliveries; 4-Accurately estimate deliverables; and 5-Show value to the organization. A function point team should also strive for these goals. Come learn best practices, tips and techniques for building a valuable counting support unit for your organization.

Keywords: EVMFunction Points, Process Engineering

SSC02 – ppt – Dennis – Faster…Better…Cheaper…

Estimator development and process automation
Soft Skills & Communication (SSC03)
Steven Glogoza

As technology, software, and tools continue to automate, it becomes increasingly challenging to develop personnel. The design and modeling of automated processes may leverage artificial intelligence, but the cost estimator will have to continue to navigate a murky gray environment that requires critical thinking, intellectual curiosity, and analytical aptitude. As we automate estimating activities and continue to develop future estimators, how do we design to capture process velocity without sacrificing insight and credibility?

Keywords: People Development

SSC03 – ppt – Glogoza – Estimator development and process automation

Finding the Story in Your Data
Soft Skills & Communication (SSC04)
Karen Richey Mislick

This presentation will cover data visualization and how to find the story within your data. It discusses how people process information and offers tips for creating effective graphics using data visualization principles and techniques to inform decision-making. Different approaches to visualizing data will be discussed including decluttering your graphics, choosing informative visuals, focusing the audience’s attention using pre-attentive attributes, thinking like a designer, and implementing effective storytelling techniques.

Keywords: Communication, Data-Driven, Decision Analysis, Performance Management, Program Management, Project Controls, data visualization, storytelling with data, pre-attentive attributes, Gestalt principles

SSC04 – ppt – Mislick – Finding the Story in Your Data

#### Software & Agile Track

Icebergs or Shifting Sands – What’s the Key to Software Estimation?
Software & Agile (SWA01)
Carol Dekkers

ICEAA is a world leader when it comes to estimating tangible programs. With CEBoK, successful estimation of large and small scale programs involving hardware, satellites, buildings, equipment and networks is not only possible, it’s become a science. Now, what happens when you bring software-intensive systems into the mix? It’s said that estimating software development is like estimating an iceberg, and with the new agile development, maybe even like shifting desert sands. What do you think? Can we create successful software estimates with so much uncertainty? Let’s explore the topic.

Keywords: Agile, Software, Story Points, SCEBoK

SWA01 – ppt – Dekkers – Icebergs or Shifting Sands

Build It and They Will Come: Keys to Developing a Successful Software Metrics Program
Software & Agile (SWA02)
Daniel French
Carol Dekkers

Despite the pervasiveness of software development in all areas of industry today, few organizations develop and implement effective software measurement programs to manage their software development projects and leverage information necessary to make business decisions. This presentation discusses reasons why this is, addresses myths and misconceptions regarding software metrics programs, and how to develop an effective program, as well as which software metrics to develop to provide the greatest value.

Keywords: Cost Management, Data Collection, Program Management, Software, Software Sizing. Software Metrics, GQM

SWA02 – ppt – French – Measure It and They Will Come

Get to the Point. What’s the Deal with Different Function Points Methodologies?
Software & Agile (SWA03)
Anandi Hira
Katharine Mann
Paul Cymerman
Ian Brown

Since the development of IFPUG Function Points (FPs), many variants have emerged to simplify software sizing and improve cost estimation. Which of these sizing techniques should one use? To understand differences among methods, we compare IFPUG FPs, COSMIC FPs, and Simple FPs for effort estimation on University of Southern California (USC)’s dataset of Unified Code Count (UCC) enhancements. Additionally, we investigate automated tools to understand the correlation between Objective FPs and the other FPs methods.

Keywords: Agile, Cost Management, Cost/Benefit Analysis, Early Cost, Software, Function Points, Software Sizing, Software Cost Estimation, Software Effort Estimation

Air Force: Standing Up Agile Under A Waterfall
Software & Agile (SWA04)
Trevor Michelson

On November 2018 the “Agile Development Manifesto” was released to parts of the Air Force to provide summary guidance to rapidly deliver software products under Agile Development. Multiple cost teams have worked closely throughout the divisions the past few years to better transition how divisions will work to incorporate Agile practices and useful metrics for software development through the release increments. This presentation will provide a journey through the usefulness of our current Agile metrics, how they will be improved and how we have made use of them throughout the DoD.

Keywords: Agile, Data-Driven

SWA04 – ppt – Michelson – AF Standing Up Agile

Cyber Mission Platform (CMP) Program: Analyzing the Full Suite of Agile Metrics
Software & Agile (SWA05)
Alex Smith
Brian Opaska

Guidance, policies and task force recommendations related to agile software development best practices for DoD continue to surface on a daily basis. The DSB, DIB, GAO and OUSD A&S have all weighed in during the last two years. The OUSD A&S “Agile Metrics Guide” provides a concise discussion of metrics that cost analysts and PMs supporting agile programs should consider. This presentation will discuss the AF Cyber Mission Platform program’s agile transformation, approach to estimation, and provide techniques and visualizations for analyzing the key metrics discussed in the OUSD A&S guide.

Keywords: Agile, IT, Program Management, Software, DevSecOps

SWA05 – ppt – Smith – CMP – Analyzing the Full Suite of Agile Metrics

Lessons Learned from Software Maintenance Datasets
Software & Agile (SWA06)
Arlene Minkiewicz

Effort applied to the maintenance of software applications is thought to be between 65% to 85% of the total ownership costs of the application. Yet there are significantly fewer studies focused on maintenance as compared to development, especially based on public domain datasets. This paper presents results from a study focused on maintenance effort data from the ISBSG’s Maintenance and Support (ISBSG M&S) database along with data from other sources.

Keywords: Data-Driven, Life Cycle, Software

Secure Software Development Levels and Costs
Software & Agile (SWA07)
Elaine Venson
Bradford Clark
Barry William Boehm

While the growing field of Software Security provides solutions to address security vulnerabilities, financial issues are still a barrier to their introduction in projects. Evaluating the cost-effectiveness of security practices requires understanding the impact of increasing degrees of security on the development effort. This session will present a rating scale for establishing levels of secure software development, along with effort estimates provided by experts, and the steps envisioned to propose a security cost model.

Keywords: Cost/Benefit Analysis, Early Cost, Modeling, Software, Software Security, Cybersecurity

SWA07 – ppt – Venson – Secure Software Development Levels and Costs

Grandma’s Secret Hotdish Recipe for SW Planning: SAFe and Analytics
Software & Agile (SWA08)
Leah Walker
Bryan Anderson
Melanie Melancon

In today’s environment, programs want to develop and deliver quickly, but struggle to manage to cost and schedule baselines. Changes in scope, staffing and process require non-standard analysis of data amidst constant change. The authors employed innovative approaches, leveraging function point analysis and machine learning tools, to generate estimates and metrics to drive business outcomes. This business case explores the impacts of program ownership transition, conversion of software development methodology, and modified scope and requirements management.

Keywords: Keywords not provided.

SWA08 – ppt – Walker – Grandma’s Secret Hotdish Recipe for SW Planning

#### Technical Management Track

How Green Was My Labor: The Cost Impacts of Manufacturing Personnel Changes
Technical Management (TCM01)
Brent Johnstone

Estimators are frequently confronted with manpower increases or decreases and asked to calculate shop performance impacts. However, existing learning curve literature offers little guidance how to do so. This paper identifies issues associated with both new hires (so-called “green labor) and workforce reductions and offers an analytical format. Based on a study of a large workforce expansion on a mature aircraft program, a model to analyze future manpower changes is presented as well as several example cases.

Keywords: Data-Driven, Labor, Learning Curves, Manufacturing, Methods, Modeling

A Perspective on Cost and Schedule Impacts of Non-Technical Project Management
Technical Management (TCM02)
Terry Josserand
Alexander Lee
Ricardo Valerdi
Leone Z. Young

Project and program management have distinct definitions, but the interplay between technical and non-technical roles within these fields has limited research. This observational study evaluates non-technical project management and its impact on the cost and schedule of major nuclear weapon modernization programs by utilizing historic data to comprehend these emerging phenomena. The study results yield multiple observations and recommendations that will assist senior leadership in their review and decision-making regarding future acquisitions.

Keywords: Budgeting, Data Collection, Data-Driven, Decision Analysis, DOD/MOD, Early Cost, Government, IPM, Life Cycle, Methods, Program Management, Project Controls, Technical, Non-Technical, Project Management

TCM02 – ppt – Josserand – A Perspective on Cost and Schedule

Lessons Learned from Implementing Global Project Control
Technical Management (TCM04)
Dale Shermon

This paper examines the challenges for organisations to initiate and maintain excellence in Project Controls working with the multi-disciplinary fields of scheduling, cost estimating, risk management, EVM, reporting and monitoring. With more than fifty global stakeholder and 150+ deliverables we will explore the highs and lows of a Project Controls change programme and the lessons learned.

Keywords: Cost Management, International, Methods, Performance Management, Project Controls, Risk, Scheduling