2015 Workshop Proceedings

Below are copies of all of the presentations and handouts from this year’s Workshop. We encourage you to share them with colleagues or others who may find them of use, but ask that the files not be distributed to large groups or used for demonstration without the author’s permission. If you are unsure whether or not an intended use of these files is appropriate, please email us at iceaa@iceaaonline.org

Authors and presenters: many presentations were changed just before the Workshop or other versions of the presentations were used. If the file here is not the latest version, please email the new version to us and we will post the new file here.

Use the links in the table below to easily navigate to your desired track or presentation.

Government Processes TrackIt Ain't Easy Being Green... Sustainable Manufacturing with an eye on Cost Avoidance and StewardshipKammy Mann, Jessica Boatwright
Government Processes TrackHyperbolic Discounting and Defense AcquisitionsEric Lofgren
Government Processes TrackMasters in Cost Estimating and AnalysisDan Nussbaum, Greg Mislick
Government Processes TrackATC Zero: Estimating the True Impact of the Chicago Control Center FireKellie Scarbrough
Government Processes TrackWhere Does Your Cost Estimate Go?Emily Stafford, Christopher Metz
Government Processes TrackHealth CERs – Using an interdisciplinary approach to estimating the cost of caring for Canada’s veteransErin Barkel, Carleigh Malanik
Government Processes TrackManufacturing Assembly Plan (MAP) Tool: Bridging the Gap between Performance and the Construction ProcessRichard Lee, Haitham Ghannam, Edward Walsh
Government Processes TrackThe Navy Modernization Program: Estimating the Cost of Upgrading AEGIS Guided Missile CruisersJeremy Goucher
Government Processes TrackFederal Shared Services: The future Shared-First and challenges facing the Federal Shared Services Providers (FSSPs) and their customersRuth Dorr, Marilyn Fleming, Sarah Pope
Government Processes TrackMining for Cost Estimating Relations from Limited Complex DataMark Jacobs, Shawn Hayes
Government Processes TrackBusiness Case Analysis WizardLauren Nolte, Cassie Capots, Marcie Pfeuffer
Government Processes TrackMethods Used in Pricing and Conveying NASA Assets for Use by Commercial ProgramsJames Roberts, Torrance Lambing
Government Processes TrackFacilitating Predictive Cost Analytics via Modelling V&VJohn Swaren
Government Processes TrackAnalysis of Alternatives from a Cost Estimating PerspectiveKirby Hom, Kenny Ragland
Government Processes TrackLessons learned from applying cost capability curves on an Air Force AoARobert Georgi
Government Processes TrackNASA Commercial Crew Cost Estimating - A Look at Estimating Processes, Challenges and Lessons LearnedLance Cole, Rick Battle
Methods & Models TrackClass List of Characteristics: Changing the Landscape of Estimating by Removing BiasPaul Breon
Methods & Models TrackAchieving Affordability through high fidelity ROM cost estimatesEric Buller
Methods & Models TrackThe Peculiar World of Sole Source Contracting; Maybe There is a Better WaySandy Burney, Shawn Larson
Methods & Models TrackThe Impacts of Design Change on Reliability, Maintainability, and Life Cycle CostAndreas Viberg, Oskar Tengo
Methods & Models TrackCost Estimating Challenges in Additive ManufacturingJoe Bauer, Patrick Malone
Methods & Models TrackGeneralized Degrees of Freedom (GDF)Shu-Ping Hu
Methods & Models TrackLong Term Impact of Ship Concepts on Operating and Support Cost AffordabilityEric Buller, Stuart Merrill, Bryan Miller
Methods & Models TrackBottom Up Methods of Estimating Depot Level RepairablesTim Lawless, James Black, Dave Goldberg, Qing Qing Wu
Methods & Models TrackA Prime Contractor's View Using JCL Tools and TechniquesChip Woods
Methods & Models TrackInterviewing Subject Matter ExpertsMelissa Teicher
Methods & Models TrackMigration of Microsoft Excel Tools to Next Generation Platforms: “Can You Hear the Footsteps?”Jeremy Eden
Methods & Models TrackSeven Degrees of Separation: The Importance of High-Quality Contractor Data in Cost EstimatingCrickett Petty
Methods & Models TrackBiometric Analytics Cost EstimatingJoseph Sarage
Parametrics TrackA Regression Method for Sparsely-Populated DatasetsDaniel Barkmeyer
Parametrics TrackA Parameter Driven WorldJeffrey Platten
Parametrics TrackThe NRO CER Analysis ToolDonald MacKenzie
Parametrics TrackGenerating a Semi-Automated “"Similar To"” Basis Of Estimate from a Complex Parametric Hardware Cost Model for AntennasDanny Polidi, David Bloom
Parametrics TrackUsing Historical Cost Reports to Derive System Specific FactorsJocelyn Tague, William Banks
Parametrics TrackA CER Development Process for Spares EstimatingCheryl Wilson
Parametrics TrackImprovement Curves: An Early Production MethodologyBrent Johnstone
Parametrics TrackBuilding an Integrated Aerospace Electronics Development Parametric Model from the Ground UpDavid Bloom
Parametrics TrackMacro-parametrics and the applications of multi-colinearity and Bayesian to enhance early cost modellingDale Shermon
Program Management TrackEssential Views of the Integrated Program Management Reports (IPMRs)Thomas Coonce, Glen Alleman
Program Management TrackThe graph is always greener on the other side: examples of how to interpret and fix bad visual representations of dataStephen Ketcham
Program Management TrackCost ManagementGeorges Teologlou
Program Management TrackMeeting Fiscal Constraints: the Evolving Role of Performance MeasurementSteve Green, Kevin Davis, Kurt Heppard
Program Management TrackDifferent Incentives in Government ContractingJennifer Leotta
Program Management TrackA Maturity Analysis of the American and European Cost Community Mark Gilmour
Program Management TrackThe Psychology of Cost EstimatingAndy Prince
Program Management TrackAgile Project Management Controls: Issues with EVM and a Viable AlternativeOmar Mahmoud, Blaze Smallwood
Program Management TrackPresenting EstimatesMelissa Teicher
Program Management TrackUsing Total Lifecycle Cost as the Basis for Proposal Cost EvaluationRyan Timm
Program Management TrackSeeing What We Need to See vs. What We Hope to See: Visualizing Integrated Cost and Schedule Data for Earned Value AnalysisBrian Leach
Risk TrackThe Impact of Selected Assumptions and Core Tenets on Schedule Risk Assessment Results (A Progressive Model Comparison)James Quilliam
Risk TrackA Mathematical Approach for Cost and Schedule Risk AttributionFred Kuo
Risk TrackData-Driven Guidelines for Correlation of Cost and Schedule GrowthSidi Huang, Mark Pedigo, Chris Shaw
Risk TrackWhy Are Estimates Always Wrong: Estimation Bias and Strategic MisestimationDan Galorath
Risk TrackRisk Mitigation: Some Good News after the Cost / Schedule Risk Analysis ResultsDavid Hulett
Risk TrackTriangular Distributions and CorrelationsJennifer Lampe
Risk TrackBeyond Correlation: Don't Use the Formula That Killed Wall StreetChristian Smart
Risk TrackA “Common Risk Factor” Method to Estimate Correlations between DistributionsMarc Greenberg
Risk TrackUsing Stochastic Optimization to Improve Risk PrioritizationEric Druker
Risk TrackMethodology for Integrating Risk mitigation Activities into JCL analysisJames Taylor, Justin Hornback
Risk TrackConnecting the Dots: Integrating the risk cube with the POEChristina Snyder
Risk TrackWine as a Financial InvestementMichael Shortell, Adiam Woldetensae
Software Estimating TrackHow Much Does Software Maintenance Cost?Cheryl Jones, James Judy, John McGarry
Software Estimating TrackSoftware size measures and their use in software project estimationHarold van Heeringen
Software Estimating TrackEarly Phase Software Cost and Schedule Estimation ModelsWilson Rosa
Software Estimating TrackAt the Intersection of Technical Debt and Software Maintenance CostsArlene Minkiewicz
Software Estimating TrackAgile Estimation Using Functional MetricsThomas Cagley
Software Estimating TrackExploring DoD Software Growth: A Better Way to Model Future Software UncertaintyNicholas Lanham
Software Estimating TrackConsiderations in Data Center Migration Life Cycle Cost Estimating: A Joint Information Environment (JIE) PerspectiveDennis McKeon
Software Estimating TrackNASA Software Cost Estimation Model: An Analogy Based Estimation MethodJairus Hihn
Software Estimating TrackEffective use of Function Points for Analogous Software EstimationDaniel French
Software Estimating TrackThe Continual Pursuit of the One True Software Resource Data Reporting (SRDR) DatabaseRemmie Arnold, Peter Braxton
Software Estimating TrackSoftware Size GrowthCorinne Wallshein
Software Estimating TrackForecasting Software MaintenanceAlea DeSantis
Software Estimating TrackPut some SNAP in your estimating modelDavid Lambert
Software Estimating TrackEstimating Agile Software DevelopmentBob Hunt
Parametrics TrackImproving Parametric Software Cost Estimation Through Better Data CollectionBob Koury
Cost Estimating Advanced TrackProb/Stat AdvancedMarc Greenberg, Christian Smart
Cost Estimating Basic TrackEconomic Analysis BasicKammy Mann, Joseph Sarage
Cost Estimating Basic TrackContract PricingPeter Braxton, Rick Collins
Cost Estimating Basic TrackEVM BasicDavid Bach, Thomas Harless
Cost Estimating Advanced TrackEconomic Analysis AdvancedJoseph Sarage
Parametric Training TrackComplex Models HardwareDavid Bloom
Cost Estimating Advanced TrackSoftware Cost Estimating AdvancedHarold van Heeringen, Bob Hunt
Integration TrackData Science and Cost EstimatingEric Druker
Integration TrackTarget Costing and TradesMarc Greenberg, John Deem
Integration TrackCost Implications of WSARABob Hunt
Cost Estimating Advanced TrackMonte Carlo SimulationsJohn Swaren, Blake Boswell
Cost Estimating Advanced TrackManufacturing Cost EstimatingJohn Deem, Charles Kurtz, Kammy Mann
Cost Estimating Basic TrackLearning Curve AnalysisGurney Thompson, Michael Mahoney
Cost Estimating Advanced TrackProbability Distributions for RiskPeter Braxton
Cost Estimating Basic TrackProb/Stat BasicTim Anderson, Stacy Dean
Cost Estimating Basic TrackSoftware Cost EstimatingBlaze Smallwood, Arlene Minkiewicz
Parametric Training TrackRegression Nonlinear/MultivariateDavid Harris
Parametric Training TrackOther Parametric ApplicationsJohn Swaren
Cost Estimating Advanced TrackRisk Analysis AdvancedChristian Smart, Amanda Wilson
Parametric Training TrackComplex Models Company-DevelopedChris Price
Parametric Training TrackCER Risk and S-CurvesChristian Smart, Chris Price
Parametric Training TrackRegression MultiplicativeHerve Joumier
Integration TrackJoint Cost and Schedule RiskDavid Hulett, Eric Druker
Integration TrackContracts Risk/RCAPeter Braxton
Integration TrackCost ManagementTom Dauber, Zachary Jasnoff, John Deem
Cost Estimating Basic TrackRisk Analysis BasicEric Druker, Sarah Green
Cost Estimating Advanced TrackGAO Cost Assessment GuideBruce Fad, Neil Chakrabarti
Cost Estimating Basic TrackCost Estimating BasicsTed Mills, Chris Dalton
Cost Estimating Basic TrackCost Estimating TechniquesPatrick K. Malone, Matthew McGovern
Integration TrackBasic Schedule AnalysisJustin Hornback
Cost Estimating Advanced TrackCost and Software Data ReportsEmily Beltramo, Brandon Bryant
Integration TrackCost Estimating in PMBoK vs. CEBoKCassie Capots, Lauren Nolte
Integration TrackAdvanced Schedule AnalysisJustin Hornback
Cost Estimating Basic TrackInflationSam Toas, Bill Seeman
Cost Estimating Basic TrackData CollectionEric Plumer, Christina Snyder
Cost Estimating Basic TrackData AnalysisWalter Bednarski, Alex Wekluk
Integration TrackSchedule Risk AnalysisDavid Hulett
Cost Estimating Advanced TrackTechnical BaselinesJason Dechoretz, Paul Tetrault, Greg Hogan, Sam Toas
Cost Estimating Advanced TrackWork Breakdown Structure (WBS)Chris Dalton
Parametric Training TrackParametric Analysis OverviewAnthony Demarco, Steve Glogoza
Parametric Training TrackCost Estimating Relationships (CERs)Anthony Demarco
Parametric Training TrackRegression LinearPeter Braxton, Remmie Arnold

Professional Papers & Presentations

Government Processes Track

Health CERs – Using an interdisciplinary approach to estimating the cost of caring for Canada’s veterans
Erin Barkel
Carleigh Malanik
Government Processes Track

GP01 Presentation Barkel Health CERs

Reports from the Veterans and the National Defence and Canadian Forces Ombudsmen have brought attention to the prevalence of mental illness among veterans. However, there are no published estimates for the cost of treating this growing population. A key obstacle to this research is that the relevant data is dispersed across several government organizations. We fill this research gap by leveraging actuarial and epidemiological studies of the veteran population to create new cost estimating relationships.

NASA Commercial Crew Cost Estimating – A Look at Estimating Processes, Challenges and Lessons Learned
Lance Cole
Rick Battle
Government Processes Track

GP03 Presentation Cole NASA Commercial Crew

To support annual PPBE budgets and NASA HQ requests for cost information for commercial crew transportation to the International Space Station (ISS), the NASA ISS ACES team developed system development and per flight cost estimates for the potential providers for each annual PPBE submit from 2009-2014. This paper describes the cost estimating processes used, challenges and lessons learned to develop estimates for this key NASA project that diverted from the traditional procurement approach and used a new way of doing business.

Federal Shared Services: The Future for Administrative Shared Services and Challenges Facing Shared Service Providers (SSPs) and Their Customers
Ruth Dorr
Marilyn Fleming
Sarah Pope
Government Processes Track

GP04 Presentation Dorr Federal Share Services

This paper will discuss the progress being made in the “Shared-First” approach for sharing of administrative services. Federal Agencies are being asked to execute their missions to ever increasing standards in a resource constrained environment. Administrative shared services have been identified as a way to reduce redundancy and improve efficiency in Federal business solutions. The identification of shared services as a Presidential cross-agency priority along with the designation of several providers for administrative shared services are recent developments in the movement toward shared services within the Federal government.

Lessons learned from applying cost capability curves on an Air Force AoA
Robert Georgi
Government Processes Track

GP05 Presentation Georgi Lessons Learned AoA

New direction from Air Force leadership established a requirement for the presentation of life cycle cost versus capability tradeoff analysis for all AoA final reports. This presentation examines the approach that was used for a recent AoA that was a pilot program for the Cost Capability Curve process. When appropriately implemented cost capability analysis can provide significant advantages. There are lessons learned from attempting to implement this approach. We will also discuss examples where this approach does not apply.

2015 Best Paper: Government Processes Track

The Navy Modernization Program: Estimating the Cost of Upgrading AEGIS Guided Missile Cruisers
Jeremy Goucher
Krystian Jenkins
Government Processes Track

GP06 Presentation Goucher Navy Modernization

GP06 Paper Goucher Navy Modernization

The Navy is well into its 20 year, $16B (CY10$) plan to modernize 84 AEGIS warships. This cost estimate covers eleven of the planned ships and accounts for maintenance and upgrades to the ship and the combat systems. The estimate leverages recent contract and shipyard performance data and interviews with engineers, resulting in a detailed study and recommended cost savings initiatives. The methods and data in this estimate will assist ship modernization cost efforts across the fleet for the foreseeable future.

Analysis of Alternatives from a Cost Estimating Perspective
Kirby Hom
Kenny Ragland
Government Processes Track

GP07 Presentation Hom Analysis Alternatives

GP07 Paper Hom Analysis Alternatives

Subsequent to the Weapon Systems Acquisition Reform Act of 2009, there have been other significant DoD policy changes that require Service sponsors to conduct an Analysis of Alternatives (AoA) prior to Milestone A. “About three-quarters of a program’s total life cycle cost is influenced by decisions made before it is approved to start development“ [GAO-09-665]. From a cost estimating perspective, a successful AoA rests with the study approach, overall costing methodologies, key insights from trades analyses, and lessons learned.

Mining for Cost Estimating Relations from Limited Complex Data
Mark Jacobs
Shawn Hayes
Government Processes Track

GP08 Presentation Jacobs Mining For Relations

NASA’s robotic Earth and space science missions cover a diverse group of projects addressing complex science objectives that include challenging implementation approaches. Progress applying Principle Component Analysis techniques covering project management, systems engineering, mission assurance, integration & test, and spacecraft subsystems is described. Supporting data analysis efforts include a large detailed set of technical and programmatic input candidates that are analyzed to identify the primary spacecraft cost drivers.

Manufacturing Assembly Plan (MAP) Tool: Bridging the Gap between Performance and the Construction Processes
Richard Lee
Haitham Ghannam
Edward Walsh
Government Processes Track

GP09 Presentation Manufacturing Assembly Plan

This presentation explains the four module build plan for the VIRGINIA Class Submarine (VCS) program and provides an in-depth look into labor hours by Section 1/2A, Section 2B/5, Section 6/7, Section 8/9, Final Assembly and Test (FAT), and Post Shakedown Availability (PSA). Further understanding these concepts will not only allow cost estimators and program managers to better understand the current construction build strategy, but it will also aid in better aligning cost estimates with reality at the deck plate.

It Ain’t Easy Being Green… Sustainable Manufacturing with an eye on Cost Avoidance and Stewardship
Kammy Mann
Jessica Boatwright
Government Processes Track

GP10 Presentation Mann Easy Being Green

GP10 Paper Mann Easy Being Green

The DoD, one of the largest energy consumers in the world, has committed to sustainability. An important element of sustainable weapon systems acquisition currently not under the purview of the DoD’s strategic plan is sustainable manufacturing. This paper will leverage current research and industry best practices to identify the impact of sustainable manufacturing on cost avoidances relating to human health and environmental benefits. Recommendations will also be provided for complying with the FAR sustainability requirements.

Business Case Analysis Wizard
Lauren Nolte
Cassie Capots
Marcie Pfeuffer
Government Processes Track

GP11 Presentation Nolte Business Case Analysis Wizard

This paper describes the purpose and design of a “Business Case Analysis (BCA) Wizard” that was created for the Air Mobility Command (AMC) Enterprise Learning Office (ELO). As the value of BCAs within the learning transformation at AMC has grown, ELO identified the need to develop a tool that allows “non-cost analysts” to develop high-quality BCAs. This “BCA Wizard” guides users through a standard process of conducting a BCA via a user-friendly interface.

Masters in Cost Estimating and Analysis
Dan Nussbaum
Greg Mislick
Government Processes Track

GP12 Presentation Nussbaum Masters Cost Estimating

This presentation provides an update on the all-Distance Learning Masters Degree and/or Certificate Program in Cost Estimating and Analysis (MCEA / CCEA) offered at the Naval Postgraduate School (NPS).

Methods Used in Pricing and Conveying NASA Assets for Use by Commercial Programs
James Roberts
Torrance Lambing
Government Processes Track

GP13 Presentation Roberts Methods Used NASA Assets

Different agreement types including EUL’s, Use Permits, Land Leases and others are used to convey NASA assets no longer needed by programs such as Space Shuttle and construction of the Space Station. By law, some situations prohibit NASA from pricing facilities in competition with the local market. Costs are estimated either via market-based or actual cost methods, according to circumstances. This paper will explain and clarify how these costs are estimated and pricing is established.

ATC Zero: Estimating the True Impact of the Chicago Control Center Fire
Kellie Scarbrough
Government Processes Track

GP14 Presentation Scarbrough ATC Zero

GP14 Paper Scarbrough ATC Zero

For seventeen days, the Chicago Air Route Traffic Control Center went offline after a contractor set fire to the intricate communications network that controls some of the busiest airspace in the country. Workers scrambled around the clock for seventeen days to restore functionality to the center. This paper seeks to determine the true economic impact of the event, including the costs of time, operations, and environmental impacts.

Where Does Your Cost Estimate Go?
Emily Stafford
Christopher Metz
Government Processes Track

GP15 Presentation Stafford Where Cost Estimate Go

You have spent several months developing your cost estimate and presenting it to the customer. Now what? This paper will track what happens to your cost estimate after it has been delivered to the customer. It will follow the estimate through the budget formulation process, ultimately becoming a part of the President’s Budget. It will also discuss the cost estimator’s role during the budget process, and the influence of the budget process on future estimates.

Facilitating Predictive Cost Analytics via Modelling V&V
John Swaren
Government Processes Track

GP17 Presentation Swaren Predictive Modeling VV

We are often asked to “fine tune” estimating software to unique situations. That “calibration” effort is key to Verification & Validation of parametric models. In this presentation, we show an example of using Predictive Analytics to develop a method to predict a Software-estimating model’s inputs as a function of Key Performance Parameters (KPPs). We demonstrate how a calibrated knowledgebase can best be used to V&V an estimating approach reflective of past and future requirement metrics.

Hyperbolic Discounting and Defense Acquisitions
Eric Lofgren
Government Processes Track

GP18 Presentation Lofgren Hyperbolic Discounting

GP18 Paper Lofgren Hyperbolic Discounting

Cost estimation has largely been approached as an engineering problem which can be solved using data and techniques. While these are undoubtedly invaluable considerations, the broader production process is an inherently open-ended system subject to human interactions which are ill-defined and non-deterministic. This paper explores how behavior and institutions affect cost outcomes through the avenue of hyperbolic discounting, a time-inconsistent model of personal valuations. It will be argued that evidence for hyperbolic discounting in both government agents and contractor management have negative effects on cost estimates and will proffer some solutions based on commitment schemes.

Methods & Models Track

Cost Estimating Challenges in Additive Manufacturing
Joe Bauer
Patrick Malone
Methods & Models Track

MM01 Paper Bauer Additive Manufacturing

MM01 Presentation Bauer Additive Manufacturing

Additive manufacturing possibilities are virtually endless: from rapid prototyping, to printing food, to recreating human tissue. This presentation highlights the challenges of estimating 3D printing processes and offers some solutions to the estimator. Challenges include: amortizing non-recurring tooling across multiple programs, capturing software-centric nature of 3D printing, printing speed, and printing media limitations. This presentation highlights the use of existing cost estimating tools and areas for future development.

Long Term Impact of Ship Concepts on Operating and Support Cost Affordability
Eric Buller
Stuart Merrill
Bryan Miller
Methods & Models Track

MM03 Presentation Buller Long Term Impact Ship Concepts

Achieving affordability can be challenging especially once a program enters full rate production. Developing accurate Operating and Support (O&S) cost estimates that reveal tradeoff impacts and sustainment risks provides decision makers with affordability analysis during key acquisition events. We intend to demonstrate through analysis the impact of legacy and new ship concepts on O&S costs and program affordability. We will examine the cost tradeoffs for each ship concept in each O&S cost element.

The Peculiar World of Sole Source Contracting; Maybe There is a Better Way
Sandy Burney
Shawn Larson
Methods & Models Track

MM04 Presentation Burney Peculiar World of Sole Source

In 2014, 21% of U.S. Government contracts were sole source, accounting for 31% of total contracted value. They typically require negotiation and certification of all cost estimates, which can take up to 18 months. This light-hearted presentation will explore the peculiar world of developing and negotiating sole source estimates, including differences from budgetary and competitive estimating, as well as offer suggestions for achieving fair and reasonable sole source prices for less time and money.

Migration of Microsoft Excel Tools to Next Generation Platforms: “Can You Hear the Footsteps?”
Jeremy Eden
Methods & Models Track

MM05 Paper Eden Footsteps

MM05 Presentation Eden Footsteps

Analysis tools are popular solutions for extended deployments and at the same time an acceleration of platform migrations is occurring to keep up with rapidly changing technology and end user demands. This paper provides specific examples of platform migration challenges, resulting failures, and how they can be best addressed. It will also provide specific best practices for ICEAA member take-away that can be followed to help ensure smoother transitions across platforms.

2015 Best Paper: Methods & Models Track

Generalized Degrees of Freedom (GDF)
Shu-Ping Hu
Methods & Models Track

MM06 Paper Hu Generalized Degrees

MM06 Presentation Hu Generalized Degrees

Minimum-Percentage Error/Zero-Percentage Bias (ZMPE) method is commonly used for multiplicative-error models. But ZMPE users do not adjust degrees of freedom (DF) for constraints included in the regression process. This generates misleading ZMPE CER fit statistics and underestimates the CER error distribution variance. Hence, ZMPE results are incomparable with those derived without constraints. This paper details why DF should be adjusted and proposes a Generalized Degrees of Freedom measure to compute fit statistics for constraint-driven CERs.

Bottom Up Methods of Estimating Depot Level Repairables
Tim Lawless
James Black
Dave Goldberg
Qing Qing Wu
Methods & Models Track

MM07 Presentation Lawless Bottom Up

Depot Level Repairables (DLR) costs comprise a significant share of total Operations and Support costs as they capture the scope of repairing and/or replacing failed hardware items. This paper will provide detailed descriptions of bottom up methods to estimate DLR costs using programmatic technical baseline documentation and acquisition costs as well as supply system requisitions and repair price data sources. Emphasis will be placed on producing data-driven analysis to inform cost estimates and estimate customers.

Seven Degrees of Separation: The Importance of High-Quality Contractor Data in Cost Estimating
Crickett Petty
Methods & Models Track

MM08 Presentation Petty Seven Degrees

MM08 Paper Petty Seven Degrees.pdf

The popular notion that any two people are linked to one another on average via a chain with “six degrees of separation” is based on a relatively small sample. Using massive data sets, researchers have since discovered that the average degrees of separation is closer to seven. This highlights the need for complete and thorough data in any analysis, and cost estimating in particular. Despite its importance the collection of high-quality contractor cost data is challenging. Processes for meeting these challenges are presented.

Biometric Analytics Cost Estimating
Joseph Sarage
Methods & Models Track

MM09 Paper Sarage Biometric Analytics.pptx

MM09 Presentation Sarage Biometric Analytics

The purpose of this paper is to present cost estimating methods as it pertains to the Biometric Field by which statistical analysis techniques can be applied to better predict the costs and relationships with Biometric data. The paper will explore analytical methods in which Biometric data can be used for trend analysis, forecasting, and strategic decision making within various organizations and applications.

Interviewing Subject Matter Experts
Melissa Teicher
Methods & Models Track

MM10 Presentation Teicher Interviewing SME

MM10 Paper Teicher Interviewing SME.pptx

As estimators, we often have to obtain technical information from Subject Matter Experts (SMEs) in order to provide thorough and accurate analysis. But sometimes, getting the information you need is easier said than done. This paper discusses the steps for interviewing SMEs in order to make the most of their time and your time. It explores the benefits of building a cooperative relationship of mutual respect and the “dos” and “don’ts” of SME consultations.

The Impacts of Design Change on Reliability, Maintainability, and Life Cycle Cost
Andreas Viberg
Oskar Tengo
Methods & Models Track

MM11 Tengo Presentation Impacts Design Change

When making decisions about a design change to a technical system, it is essential to understand how it affects reliability and maintainability, as that directly impacts both overall performance and cost of operations and maintenance throughout the life cycle. LCC analysis based on modeling and simulation of relevant scenarios is an effective way to gain that understanding. This paper presents an approach for such analysis and illustrates its successful application in a recent project.

A Prime Contractor’s View Using JCL Tools and Techniques
Chip Woods
Methods & Models Track

MM12 Presentation Woods Prime Contractor View

For several years now, Lockheed Martin has been integrating cost and schedule data into a single model to perform integrated analysis. This paper will discuss Lockheed Martin’s JCL process, the tools used to develop JCL curves, and how this analysis can be used to shape and steer the program – including modeling funding constraints.

Class List of Characteristics
Paul Breon
Methods & Models Track

MM13 Presentation Breon Class Characteristics

MM13 Paper Breon Class List Characteristics

CLOC (Class List of Characteristics) is a coding system wherein each digit of the code indicates one of the cost drivers for the product/activity. CLOC gives the ability to “Call a thing by what it really is” and creating a universal language that can be utilized for any purpose. CLOC does NOT include Bias like quantity, cost, etc. These things are used in conjunction with the code to do any kind of analysis needed.

Achieving Affordability through high fidelity ROM cost estimates
Eric Buller
Methods & Models Track

MM14 Presentation Buller Achieving Affordability ROM

Achieving affordability can be challenging especially if early decisions are made without proper cost estimates. Developing high fidelity ROM cost estimates and identifying capability tradeoffs during the requirements generation phase can be critical to the success of achieving long term affordability. We intend to demonstrate through analysis that preforming high fidelity ROM cost estimates with capability tradeoffs allow decisions makers to informed choices on mission requirements that achieve affordability for future programs.

Parametrics Track

A Regression Method for Sparsely-Populated Datasets
Daniel Barkmeyer
Parametrics Track

PA01 Presentation Barkmeyer Regression Method

A method is presented for including sparsely-populated independent variables in a regression analysis without introducing subjectivity or statistical bias. The method is shown to reproduce the result of a traditional regression method for a fully populated dataset, and to converges to the same result on a sparsely-populated dataset as that dataset decreases in sparseness. Application of the method to develop a new, empirically-derived model for spacecraft testing schedule duration will also be presented.

2015 Best Paper: Parametrics Track

Improvement Curves: An Early Production Methodology
Brent Johnstone
Parametrics Track

PA03 Paper Johnstone Improvment Curves

PA03 Presentation Johnstone Improvment Curves

Learning slope selection is a critical parameter in manufacturing labor estimates. Incorrect ex ante predictions lead to over- or understatements of projected hours. Existing literature provides little guidance on ex ante selection, particularly when some actual cost data exists but the program is well short of maturity. A methodology is offered using engineered labor standards and legacy performance to establish a basic learning curve slope to which early performance asymptotically recovers over time.

The NRO CER Analysis Tool
Donald MacKenzie
Parametrics Track

PA05 Presentation MacKenzie NRO CER Analysis

PA05 Paper MacKenzie NRO CER Analysis.pdf

The NRO Cost and Acquisition Assessment Group (CAAG) uses its CER Analysis Tool, CERAT, to inform CER development. CERAT analyzes CERs for sensitivity to influential data points (IDPs). The impacts on estimated costs for a target data point when data points are removed are the primary outputs. The changes in recorded CER constants when data points are removed also help in assessing overall CER “stability.” CERAT also provides several other aids to CER developers.

A Parameter Driven World
Jeffrey Platten
Parametrics Track

PA06 Presentation Platten Parameter Driven World

PA06 Paper Platten Parameter Driven World

Parametric estimating is gaining acceptance. DFARS requires DOD contractors to maintain an estimating system. Parameters are the characteristics, features, or measurable factors that define a product or program. What makes a good parameter? What are the pros and cons of parametric cost estimating? How do you develop and maintain a parametric estimating database/model? What does the future hold, beyond just estimating? How do you gain parametric control over touch labor, support labor, and parts costs to improve affordability?

Generating a Semi-Automated “Similar To” Basis Of Estimate from a Complex Parametric Hardware Cost Model for Antennas
Danny Polidi
David Bloom
Parametrics Track

PA07 Presentation Polidi Similar To Antennas

This paper discusses the development of a “Similar-To” Basis Of Estimate (BOE) generation tool used in conjunction with a Complex Parametric Antenna Cost Model. Starting with the parametric model based on quantifiable sizing mechanisms which are designed to quickly and accurately calculate the “top-down” cost for all engineering and operations disciplines and functions required for antenna development and test, cost is generated, “similar to” programs are identified, and a BOE can be auto-populated.

Macro-parametrics and the applications of multi-colinearity and Bayesian to enhance early cost modelling
Dale Shermon
Parametrics Track

PA08 Presentation Shermon Macro Parametrics

PA08 Paper Shermon Macro Parametrics

This paper will consider the spectrum of parametric cost models from cost estimating relationships (CER) to micro and finally macro-parametric models. This will lead to a description of the Family of Advanced Cost Estimating Tools (FACET) parametric suite of models and their top-down capability to estimate costs incorporating the application of multi-colinearity and bayesian techniques.

Using Historical Cost Reports to Derive System Specific Factors
Jocelyn Tague
William Banks
Parametrics Track

PA09 Paper Tague Using Historical Reports

PA09 Presentation Tague Using Historical Reports

Non‐Recurring Engineering (NRE) is often a cost driver for new, complex systems and must be given high attention during Milestone B cost estimating. Our team developed an approach for developing system specific Cost Estimating Relationship (CER) using actual costs for from combat weapon systems with shared capabilities, resulting in a credible defensible, yet flexible approach that can be replicated for a wide range of estimates.

A CER Development Process for Spares Estimating
Cheryl Wilson
Parametrics Track

PA10 Presentation Wilson CER Development

This presentation shows the direction of a group of Spare Parts Estimators and how we are advancing our statistical skills and CER development processes. It shares the progress in our CER development journey attained through education, research, analysis and teamwork.

Improving Parametric Software Cost Estimation Through Better Data Collection
Bob Koury
Parametrics Track

PA11 Presentation Koury Improving Parametric Better Data

In 2010 PRICE Systems LLC in conjunction with PEO STRI agreed to work together to improve the data being collected to support software development cost estimation in the command. The paper will compare and contrast the experiences of two other collection efforts with those of original effort. The paper will identify core findings which may be used universally to improve the quality, quantity, and applicability of data being collected.

Building an Integrated Aerospace Electronics Development Parametric Model from the Ground Up
David Bloom
Parametrics Track

PA12 Presentation Bloom Integrated Aerospace Model

This paper presents the results of a multiple year effort to create a business specific set of integrated parametric cost models for complex electronics hardware development in such a way that all disciplines and activities that participate in that development process have a piece of the model and a stake in the data collection that drives the model. This “grounds up” approach provides that added benefit of cost allocation guidance once the development is approved.

Program Management Track

Essential Views of the Integrated Program Management Reports (IPMRs)
Thomas Coonce
Glen Alleman
Program Management Track

PM01 Presentation Coonce Essential Views IPM

During this session, the authors will describe key views of the IPMRs and other data that will allow a government stakeholder to a) understand the cost, schedule and technical status on applicable contracts, b) investigate the nexus of performance problems, and c) observe likely future problems. The presentation represents a “synthesis” of existing traditional earned value management metrics and other metrics designed to provide an integrative picture to assist in keeping the program “green”.

Meeting Fiscal Constraints: the Evolving Role of Performance Measurement
Steve Green
Kevin Davis
Kurt Heppard
Program Management Track

PM02 Presentation Meeting Fiscal Constraints

Performance measurement tools can reflect the strategic intent of management and help them prioritize initiatives to meet strategic objectives. In fiscally austere times, the Balanced Scorecard can help organizations address these tough decisions. Applying concepts from the Balanced Scorecard to cost analysis and budgeting, this paper develops an integrated aligned system of strategic goals, performance metrics, and cost parameters at a university facing pressure to manage costs. There may be lessons learned for the DoD.

The graph is always greener on the other side: examples of how to interpret and fix bad visual representations of data
Stephen Ketcham
Program Management Track

PM03 Presentation Ketcham Graph Greener Handout

In my role performing independent reviews of estimates, I’ve seen a quality spectrum of different types of graphs, charts, and forecasts. These visuals summarize key points and remain in decision-makers minds, often providing the convincing point to win funding approval for the project. Unfortunately they can also be poorly crafted, employ scare tactics or even frame the data in a misleading way. This presentation provides bad examples of visuals, and discusses possible improvements.

Seeing What We Need to See vs. What We Hope to See: Visualizing Integrated Cost and Schedule Data for Earned Value Analysis
Brian Leach
Program Management Track

PM04 Presentation Leach Seeing What We Need

When applied properly, data visualization can serve analysis by showing us what is otherwise difficult to see (and not always what we hope to see). With the emergence of the UN/CEFACT XML standard, integrated cost and schedule data can be shown in exciting new ways. This paper will examine and explore innovative new ways to show integrated cost and schedule data which have practical applications in earned value management system analysis.

Different Incentives in Government Contracting
Jennifer Leotta
Program Management Track

PM05 Presentation Leotta Different Incentives Contracting

PM05 Paper Leotta Different Incentives Contracting

This paper shows stock prices are used as the primary incentive for executives of publicly traded government contractors rather than profit sharing contracts. Looking at the reaction of companies’ stock prices to public and private sector news, this paper shows stock prices do not capture “bad news” for both the public and private sector customers compared to overall market indicators; and would make poor contract incentives. Finally, alternate methods of contract incentives will be explored.

Agile Project Management Controls: Issues with EVM and a Viable Alternative
Omar Mahmoud
Blaze Smallwood
Program Management Track

PM06 Presentation Mahmoud Aglie Project Management

As more government software programs adopt agile practices, program managers are trying to find effective ways to monitor and control cost and schedule. The challenge is that traditional project control techniques, like EVM, are not well suited to deal with the requirements fluidity and new terms/metrics that come with agile projects. This paper will describe the inherent issues with using EVM on agile projects and offer a recommendation for a more effective solution.

2015 Best Paper: Program Management Track

2015 Best Paper Overall

The Psychology of Cost Estimating
Andy Prince
Program Management Track

PM07 Presentation Prince Psychology Cost Estimating

PM07 Paper Prince Psychology Cost Estimating

Research into human psychology has yielded amazing findings into how we process information and how we use information to make decisions. Cost estimators can use psychology and behavioral economics to improve not only our cost estimates but how we communicate and work with our customers. By understanding how our customer’s think, and more importantly, why they think the way they do, we can have more productive relationships and greater influence. The end result will be better decisions by the decision makers.

Cost Management
Anthony DeMarco
Program Management Track

PM08 Presentation DeMarco Cost Management

Cost Management has a key role in project management and is the management trend due to the actual financial and economic worldwide constraints. Cost Management solution was built to be in line with the recommendations of the GAO (US Government Accountability Office) that stated that a large number of Defense projects are at risk of experiencing cost overruns and schedule delays. GAO found that the two major causes of project failure were first an inaccurate initial estimate and secondly poor subsequent cost control.

Using Total Lifecycle Cost as the Basis for Proposal Cost Evaluation
Ryan Timm
Program Management Track

PM09 Presentation Timm Total Lifecycle Cost

This paper discusses benefits and obstacles of using the TLCC of an entire system instead of the bid price for the purpose of proposal cost evaluation. An example acquisition of a space vehicle (SV) ground system explains the impact to TLCC of portions of the ground system developed by the SV manufacturer, SV operations, external software interfaces, government furnished IT infrastructure, and legacy sustainment until transition. Using TLCC allows higher-level requirements and enables innovative architectures.

A Maturity Analysis of the American and European Cost Community
Mark Gilmour
Program Management Track

PM10 Presentation Gilmour Maturity Analysis

PM10 Paper Gilmour Maturity Analysis

The Cost Engineering Health Check (CEHC) is a cost capability benchmarking tool used to identify strengths and weaknesses within organizations costing functions. In 2014 the authors of this paper conducted a CEHC at the ICEAA conference in Denver. That workshop used e-voting to anonymously gather the views of the audience in attendance. This paper presents a summary of these results and contrasts them against data gathered from European workshops, providing a snapshot of the international costing capability.

Presenting Estimates
Melissa Teicher
Program Management Track

PM11 Tiecher Presenting Estimates

Decision‐makers rely on cost estimators to guide them in making the most educated judgments on what is best for the program or organization. A thorough and well‐organized presentation can make the difference between an easy decision and a difficult one. This paper discusses how public speaking skills contribute to an effective estimate presentation. It also delivers tips and tricks on how to make an estimate presentation more visually pleasing and easier to understand.

Risk Track

Using Stochastic Optimization to Improve Risk Prioritization
Eric Druker
Risk Track

RI02 Presentation Druker Stochastic Optimization

RI02 Paper Druker Stochastic Optimization

Minimizing the cost of complex programs is critical for government agencies trying to meet their missions in today’s fiscal. Today, identifying cost-savings measures is a manual procedure where an analyst must make an educated guess as to what risks to mitigate. This process is manual and slow. This paper will outline a methodology for using the emerging field of stochastic optimization to automate the identification of cost-savings measures on complex programs.

Why Are Estimates Always Wrong: Estimation Bias and Strategic Misestimation
Dan Galorath
Risk Track

RI03 Presentation Galorath Estimates Always Wrong

RI03 Paper Galorath Estimates Always Wrong

The biggest source of estimation error usually comes from people, either by accident or strategically. This is a disaster because viable estimates are core to project success as well as ROI determination and other decision making. Most people don’t know how to estimate. Those who estimate are often optimistic, full of unintentional bias and sometimes strategically mis-estimating.
This paper discusses the issues of estimation bias and strategic mis-estimation as well as how to mitigate these issues.

A “Common Risk Factor” Method to Estimate Correlations between Distributions
Marc Greenberg
Risk Track

RI04 Presentation Greenberg Common Risk Factor

A “common risk factor” method uses expert-derived risk factors to estimate correlation between two distributions. The premise is to estimate mutual information among risk factors that underlie each distribution. Approximation of such mutual information leverages properties of the joint probability distribution of a unit square. Geometric outputs of each pair of common random variables are compared leading to an estimated ‘intersection’ of pair of common risk factors. This intersection is then assumed to be a proxy of correlation.

Data-Driven Guidelines for Correlation of Cost and Schedule Growth
Sidi Huang
Mark Pedigo, PhD
Chris Shaw
Risk Track

RI05 Paper Huang Data Driven Guidelines

RI05 Presentation Huang Data Driven Guidelines

When running Monte Carlo simulations, one of the most difficult aspects to account for is correlation between the independent variables. Ignoring correlation generally reports overly optimistic analysis results; industry standard suggests that when no additional insight is available, a default value of 0.2 can be used. This study analyzes historical NASA program data to develop data-driven guidelines for correlation. Moving forward, this study can be replicated to other industries and organizations with critical schedules.

Risk Mitigation: Some Good News after the Cost / Schedule Risk Analysis Results
David Hulett
Risk Track

RI06 Presentation Hulett Risk Mitigation News

The results of your cost‐schedule risk analysis say you are likely to be overrun on both schedule and budget. Are you are out‐of‐luck? No. Do not forget the risk mitigation phase. Risk mitigation typically proceeds from a prioritized risk list developed from analysis of the risk and uncertainty‐loaded schedule. Complete mitigation of risks is not usually feasible, and uncertainty is usually deemed to be irreducible. Some good news comes from the risk mitigation actions.

A Mathematical Approach for Cost and Schedule Risk Attribution
Fred Kuo
Risk Track

RI07 Presentation Kuo Mathematical Approach

RI07 Paper Kuo Mathematical Approach

The level of sophistication in cost estimate has progressed substantially in the last decade. This is especially true at NASA where, spearheaded by the Cost Analysis Division (CAD) of NASA Headquarters, cost and schedule risks in terms of confidence level have been codified. In this paper, the author attempts to employ the portfolio approach in calculating risk attribution; first applying to a portfolio of cost risks and extend the same concept to schedule risks analysis.

Triangular Distributions and Correlations
Jennifer Lampe
Risk Track

RI08 Presentation Lampe Triangular Distributions

Triangular distributions are often used in estimating cost risk because the math is relatively simple. The concept of an estimate having a lower limit, an upper limit, and a mode is easy to understand.
We show the math behind triangular distributions and correlations.
What does it mean to say that two risks are 30% correlated?
How do risk analysis programs handle triangular distributions and correlations?
How can you use random numbers in Excel to generate a Monte Carlo simulation for any defined triangular distribution?

The Impact of Selected Assumptions and Core Tenets on Schedule Risk Assessment Results (A Progressive Model Comparison)
James Quilliam
Risk Track

RI09 Paper Quilliam Impact Selected Assumptions

RI09 Presentation Quilliam Impact Selected Assumptions

In the quest to ensure the sound representation of Schedule risk assessment (SRA) simulations this case study will provide a progressive model comparison of schedule risk assessment assumptions and core tenets. The elements of this approach will focus on the: methodology and tools; the progressive assumptions and core tenets applied; conclusions and lessons learned for practitioners. This will greatly enhance the understanding and confidence that leadership and project teams have in the schedule risk assessment results.

Wine as a Financial Investement
Michael Shortell
Adiam Woldetensae
Risk Track

RI10 Presentation Shortell Wine as Financial Investment

The purpose of this presentation is to look into wine as an investment and analyze its characteristics as a financial asset. The goal here is to look into vintage wines through time, analyze its profitability and predict how investing in wine could be a good source of diversification. Our objective is to detect the types of wines that are a good source of investment and their associated risk. We aim to identify a good investment portfolio for a wine collector in order to maximize returns and minimize the associated risk.

2015 Best Paper: Risk Track

Beyond Correlation: Don’t Use the Formula That Killed Wall Street
Christian Smart
Risk Track

RI11 Presentation Smart Beyond Correlation

RI11 Paper Smart Beyond Correlation

Risk models in cost estimating almost exclusively rely on correlation as a measure of dependency. Correlation is only one measure (among many) of stochastic dependency. It ignores the effect of tail dependency, when one extreme event affects others. This leads to potential outcomes that do not make sense such as a program with a large schedule overrun but no cost overrun. This method has been widely blamed for the financial crisis a few years ago. The copula method is introduced as a way to overcome these deficiencies.

Connecting the Dots: Integrating the risk cube with the POE
Christina Snyder
Risk Track

RI12 Paper Snyder Connecting the Dots

RI12 Presentation Snyder Connecting the Dots

DoD guidance instructs program managers to include risk and uncertainty analysis in their cost estimates. Program managers also maintain a risk cube that tracks known program risks and assigns a likelihood and consequence. Risks, regardless of if they are programmatic, technical, or schedule defined, will have an effect on the overall program cost. This paper investigates the linkage between identified risks and POE cost and shows how to translate the risk cube into dollars.

Methodology for Integrating Risk mitigation Activities into JCL analysis
James Taylor
Justin Hornback
Risk Track

RI13 Presentation Taylor Methodology Risk Mitigation

RI13 Paper Taylor Methodology Risk Mitigation

The intent of this whitepaper is to outline a an improved methodology for capturing risk mitigation activities for both cost and schedule into a Joint Confidence Level (JCL) analysis and assessing mitigation effectiveness.

Software Estimating Track

The Continual Pursuit of the One True Software Resource Data Reporting (SRDR) Database
Remmie Arnold
Peter Braxton
Software Estimating Track

SW01 Presentation Arnold Continual Pursuit

This paper will demonstrate the ability for one database to store Software Resource Data Report (SRDR) source data in its original form, as submitted by WBS element and reporting event. This database allows evaluations, interpretations, and annotations of the data, including appropriate pairing of Initial and Final reports; mapping of SLOC to standard categories for the purposes of determining ESLOC; normalization of software activities to a standard set of activities; and storage of previous assessments.

Agile Estimation Using Functional Metrics
Thomas Cagley
Software Estimating Track

SW02 Presentation Cagley Agile Estimation

Agile methods have matured and are now being integrated into many different approaches to the development of software. Estimation has been problematic for all methods, agile to plan based and therefore it tends to be a lightning rod for experimentation and synthesis such as is being described in this paper. Agile Estimation Using Functional Metrics has presented a path for integrating the discipline found in functional metrics with the collaborative approaches found in agile parametric estimation.

Put some SNAP in your estimating model
David Lambert
Software Estimating Track

SW03 Presentation SNAP Model

An effective estimating model is dependent upon having the capacity to accurately size the problem domain for the solution being developed. Function Point Analysis is a widely accepted industry and ISO standard for sizing functional requirements; however, it does not take into account the sizing of non-functional requirements. Learn how the newly released Software Non-Functional Assessment Process (SNAP) can provide a more complete and accurate assessment of project size.

Forecasting Software Maintenance
Alea DeSantis
Software Estimating Track

SW04 Paper DeSantis Forecasting Software Maintenance

SW04 Presentation DeSantis Forecasting Software Maintenance

Providing a defensible estimate for software maintenance can be complicated. By overlaying historical data with lifecycle events one can derive event driven factors and use trend analysis to forecast future Software Problem Reports. Topics addressed are: common-cause variation, special-cause variation, severity level, complexity level, and decay rate. This method moves away from using a general defect rate calculation to a more complete product-specific calculation.

Effective use of Function Points for Analogous Software Estimation
Daniel French
Software Estimating Track

SW05 Presentation French Effective Function Points

Analogous estimation is often used to estimate software projects, especially in the early phases, because it is believed to be quicker and easier than other techniques. However, many organizations do not employ this technique effectively and correctly. One pitfall is improperly applying the technique & basing these estimates on Software Lines of Code (SLOC). This presentation will discuss common mistakes made when using analogous estimates as well as how the use of IFPUG function points can greatly improve this estimation technique

NASA Software Cost Estimation Model: An Analogy Based Estimation Method
Jairus Hihn
Software Estimating Track

SW06 Presentation Hihn NASA Software Model

Software development activities are notorious for their cost growth. While there has been extensive work on improving parametric methods there is very little focus on the use of models based on analogy and clustering algorithms. In this paper we will describe the methodology being used in the development of a NASA Software Cost Model using data mining clustering algorithms and evaluate its performance by comparing it to estimates from COCOMO II, a calibrated COCOMO II, liner regression, and K-nearest neighbor models.

Estimating Agile Software Development
Bob Hunt
Software Estimating Track

SW07 Presentation Hunt Estimating Agile

The use of Agile Software Development continues to grow in large and small Federal programs. These programs often utilize Story Points as a key size metric. This presentation will roovideman update on Agile Cost Estimating techniques and emphasize how to,utilize Story Points as a key size metric.

How Much Does Software Maintenance Cost?
Cheryl Jones
James Judy
John McGarry
Software Estimating Track

SW08 Presentation Jones Software Maintenance

To accurately estimate, budget, allocate, and justify the resources required to meet evolving mission and service affordability requirements across the system life-cycle we must be able to answer the fundamental question: How much does software maintenance cost? This presentation provides a summary of the findings from an ongoing U.S. Army study that is focused on developing improved software maintenance estimation processes, cost relationships, and associated models.

Exploring DoD Software Growth: A Better Way to Model Future Software Uncertainty
Nicholas Lanham
Software Estimating Track

SW09 Paper Lanham DoD Software Growth

SW09 Presentation Lanham DoD Software Growth

This presentation highlights trends and estimating relationships derived from the April 2014 SRDR data. This analysis provides the estimating community with several new variables to better model future software growth uncertainty. In addition, the analysis includes growth by contract type, domain, and program type as well as introduces a multivariate model to predict final hours. All of the relationships included within this slide set have been derived from the best 400+ data points, were approved by DCARC, and are DoD-specific.

Considerations in Data Center Migration Life Cycle Cost Estimating: A Joint Information Environment (JIE) Perspective
Dennis McKeon
Software Estimating Track

SW10 Presentation McKeon JIE Migration

The potential benefits of consolidating data centers has led to the creation of initiatives such as the DoD Joint Information Environment (JIE). The goal is to reduce IT infrastructure costs by consolidating system support functions, improve hardware utilization, and lower energy and facility costs. This paper is from the perspective of an organization migrating to a consolidated data center. It will describe the migration steps, cost estimating approach, benefits, risks, and organizational challenges from their perspective.

At the Intersection of Technical Debt and Software Maintenance Costs
Arlene Minkiewicz
Software Estimating Track

SW11 Paper Minkiewicz Technical Debt

SW11 Presentation Minkiewicz Technical Debt

Ward Cunningham introduced the technical debt metaphor in 1992 to ease business leaders understanding of the cost of making poor decisions during software development. While this author has seen many different definitions of technical debt, it is basically an indication of the amount of ‘should fix’ issues that remain in production code. This paper reports on an on-going research effort to understand technical debt and its implications on software maintenance cost and effort estimation.

2015 Best Paper: Software Estimating Track

Early Phase Software Cost and Schedule Estimation Models
Wilson Rosa
Software Estimating Track

SW12 Paper Rosa Early Phase Software

SW12 Presentation Rosa Early Phase Software

Software cost estimates are more useful at early elaboration phase, when source lines of code and Function Points are not yet available. This study introduces effort estimation models using functional requirements as a size predictor and evaluates development schedule using estimated effort based on data from 40 military and IT programs delivered from 2006 to 2014. Statistical results show that estimated functional requirements and peak staff are significant contributors and that estimated or actual effort is a valid predictor of development duration.

Software size measures and their use in software project estimation
Harold van Heeringen
Software Estimating Track

SW14 Paper VanHeeringen Software Size Measures

SW14 Presentation VanHeeringen Software Size Measures

Using the right size measure is extremely important for software cost engineering purposes. However, the number of software size measures is growing rapidly and there is a lot of confusion in the industry with regard to the methods that should be used. The advantages and disadvantages of the major software size measures are discussed and recommendations are given on what size measures should be used to estimate the different types of software systems.

Software Size Growth
Corinne Wallshein
Software Estimating Track

SW15 Presentation Wallshein Software Size Growth

NCCA will present their findings including cost estimating relationships (CERs) of software size growth using selected subsets. CERs will be compared using initially estimated parameter distributions against CERs using software size. Other parameters include requirements, staff size, hours, and duration. As the 2014 update of the Joint Agency Cost and Schedule Risk and Uncertainty Handbook advises use of standard error for uncertainty bounds, NCCA will compare bounds for recommended CERs, by selected data subsets.

Training Tracks & Sessions

Cost Estimating Basic Training Track

Cost Estimating Basics
Chris Dalton
Ted Mills

CEB 01 – Cost Estimating Basics

The Cost Estimating Overview session introduces the basics of cost estimating and analysis and the reasons for doing cost estimates. Additionally, the motivation for cost estimating, why and how it is used, and an overview of the challenges and issues will be presented. Cost estimating products will be reviewed and a general overview of the process and methods used to develop cost estimates will be discussed, providing the background information and fundamental knowledge needed for the other CEB sessions. This session will provide an overview of the ICEAA certification process and give trainees an understanding of what to expect in the subsequent CEB sessions.

Cost Estimating Techniques
Patrick Malone
Matthew McGovern

CEB 02 – Cost Estimating Techniques

The Cost Estimating Overview session introduces the basics of cost estimating and analysis and the reasons for doing cost estimates. Additionally, the motivation for cost estimating, why and how it is used, and an overview of the challenges and issues will be presented. Cost estimating products will be reviewed and a general overview of the process and methods used to develop cost estimates will be discussed, providing the background information and fundamental knowledge needed for the other CEB sessions. This session will provide an overview of the ICEAA certification process and give trainees an understanding of what to expect in the subsequent CEB sessions.

Data Collection
Eric Plumer
Christina Snyder

CEB 03 – Data Collection

This session covers the Core Knowledge section of Module 04 Data Collection of CEBoK. All parametric estimating techniques and cost estimating models require credible data before they can be used effectively. Training in Data Collection and Analysis discusses the processes needed to collect and analyze the data used in parametric applications, as well as data types, sources, and adjustment techniques.

Sam Toas
Bill Seeman

CEB 04 – Inflation

This session covers the Core Knowledge section of Module 05 Inflation and Index Numbers of CEBoK. Proper inflation analysis is essential to the success of any cost estimate or economic analysis. Calculating inflation correctly and understanding the fundamental concepts will enable you to produce cost estimates that are timely, accurate, and credible to support your program’s lifecycle needs. It will also empower you to communicate with key stakeholders on the need to adjust your financial estimates based on changes in the economy.

Data Analysis
Walter Bednarski
Alex Wekluk

CEB 05 – Data Analysis

This session discusses the first analytical steps to take after obtaining a set of cost data, with a particular emphasis on techniques for displaying and analyzing data graphically. It addresses univariate, bivariate, and multivariate data sets, with a focus on statistical analysis of univariate data sets and graphical analysis of bivariate data sets via scatter plots. It introduces measures of central tendency, such as mean, median, and mode; measures of variability, such as standard deviation, variance, and coefficient of variation (CV); and measures of uncertainty, such as confidence and prediction intervals. A couple of vivid examples illustrate the perils of drawing conclusions from your numerical data without looking at a graph.

Learning Curve Analysis
Michael Mahoney
Gurney Thompson

CEB 06 – Learning Curve Analysis

This is a training track presentation of the CEBoK Module 7 (Learning Curves). The presentation will cover the key ideas, analytical constructs and applications of the learning curve module. The target audience are those preparing to take the ICEAA certification exam. Beyond the theoretical information, we will present the study questions for Module 7 with steps required to solve the problems using only a calculator as is required on the certification exam.

Risk Analysis Basic
Eric Druker
Sarah Green

CEB 07 – Risk Analysis Basic

This session will provide motivation for the need for risk analysis and introduce the basic types and uses of risk. It will focus on the practical execution of the general risk analysis process: (1) Develop a point estimate; (2) Identify the risk areas in the point estimate; (3) Determine uncertainty around the point estimate; (4) Apply correlation between uncertainty distributions; (5) Run the Monte Carlo simulation; (6) Assess the reasonableness of results; (7) Calculate, allocate, and phase risk dollars; and (8) Show the results.

Probability and Statistics Basic
Tim Anderson
Stacy Dean

CEB 08 – ProbStat Basic

This will be a review of statistical and probability concepts relevant to cost estimation and uncertainty. Topics will include statistics; measures of central tendency; measures of dispersion; and important probability distributions and their linkage to cost estimates. We also introduce the concept of a random variable; Monte Carlo simulation; the Inverse CDF method; and thoughts on the differences between the normal and lognormal distributions. Finally, we discuss hypothesis testing and its relationship to cost estimating.

Software Cost Estimating
Arlene Minkiewicz
Blaze Smallwood

CEB 09 – Software Cost Estimating

This session will convey the principles of this critical area of specialized cost estimating. It addresses the basics of software development and some of the things to look out for in software estimating, and includes specific examples of applying software estimating techniques. The presentation will walk through examples of how to approach a software cost estimate using a number of estimating relationships and ideas, commercial models that can be used to construct estimates, and various methods of software development.

Economic Analysis Basic
Kammy Mann
Joseph Sarage

CEB 10 – Economic Analysis Basic

This session covers the Core Knowledge section of Module 13 Economic Analysis of CEBoK. It will be of particular interest to anyone studying for the ICEAA certification exam. The session provides a practitioner’s perspective for conducting an economic analysis (EA) by reviewing EA concepts, terminology, variables and measures-of-merit. By accounting for monetized costs, monetized benefits, opportunity costs and time-value-of-money (“discounting”), an EA enables one to calculate economic measures-of-merit.

Contract Pricing
Peter Braxton
Rick Collins

CEB 11 – Contract Pricing

This session takes a holistic view of the contracting and contract pricing from a cost estimator’s viewpoint, with emphasis on acquisition planning, contract pricing and cost/price analysis. The discussion of acquisition planning will focus on contract types, specifically how cost risk is shared between parties and the profit/fee potentials commensurate with that risk. The use of cost estimating techniques to establish a basis of estimate (BOE) as the foundation of the cost/price proposal (i.e. cost pricing) will be explained in detail.

EVM Basic
David Bach
Thomas Harless

CEB 12 – EVM Basic

This session will provide an introduction to the basic concepts of earned value management (EVM), with a focus on practical application in support of a project or program. Specific topics will include basic EVM components and data elements, as well as standard earned value analysis techniques. A sample case study will be introduced and built upon throughout the session to demonstrate and reinforce these basic principles of EVM.

Cost Estimating Advanced Training Track

GAO Cost Assessment Guide
Neil Chakrabarti
Bruce Fad

CEA 01 – GAO Cost Assessment Guide

In March 2009, the United States Government Accountability Office (GAO) published, GAO Cost Estimating and Assessment Guide, Best Practices for Developing and Managing Capital Program Costs. The guide was developed to establish a consistent methodology that is based on best practices and that can be used across the federal government for developing, managing, and evaluating capital program cost estimates. This training will introduce the twelve-step process, describe the characteristics and contents of a high quality estimate, and illustrate the application of the guide to GAO assessments.

Technical Baselines
Jason Dechoretz
Paul Tetrault
Sam Toas

CEA 02 – Technical Baselines

Understanding the technical and programmatic issues of the program/product you are estimating is key to developing a good cost estimate. This session discusses what information you need to baseline the technical and programmatic aspects of the program, the questions that need to be asked to ensure a clear understanding and how these issues translate into a cost perspective.

Work Breakdown Structure (WBS)
Chris Dalton

CEA 03 – Work Breakdown Structures

The Work Breakdown Structure (WBS) is a critical part of managing and planning any project. The WBS is the best tool for ensuring consistency and focus on the project. This presentation will discuss the key attributes of a good WBS in scoping and planning of projects. The presentation will provide the generally accepted concepts of the use and application of the WBS as well as why it is often misunderstood and not used as it should for maximum effectiveness.

Cost and Software Data Reports
Emily Beltramo
Brandon Bryant

CEA 04 – Cost and Software Data Reports

This training session will give an overview of Cost and Software Data Reports (CSDRs) and how they can be employed to facilitate the development of better and more credible cost estimates. This training session will also demo and discuss the Cost Assessment Data Enterprise (CADE) tool. CADE collects, organizes, and displays data, including EVM, CSDRs, Selected Acquisition Reports (SARs), budget data, and technical data all in one centralized location.

Risk Analysis Advanced
Christian Smart
Amanda Wilson

CEA 05 – Risk Analysis Advanced

This session will delve into advanced topics in risk, including a short and entertaining refresher on basic cost risk analysis. We will perform an analysis of historical risk data by studying the effect of program size on cost / schedule growth and how we can use this information in our risk analysis. This session covers the Related and Advanced Topics section of Module 09 Cost and Schedule Risk Analysis of CEBoK.

Monte Carlo Simulations
Blake Boswell
John Swaren

CEA 06 – Monte Carlo Simulations

Monte Carlo simulations are the most commonly used tool for performing risk analysis. This training will walk through a brief history of Monte Carlo simulations, from their use in evaluating Pi, through their application on the Manhattan project. The bulk of the training will be an interactive demonstration where the attendees create their own Monte Carlo simulation, alongside the instructor, in Excel (using a few lines of Visual Basic coding). The simulation will be run and the results analyzed in real-time to provide attendees a deeper understanding of where S-Curves come from and how to interpret them.

Probability Distributions for Risk
Peter Braxton

CEA 07 – Probability Distributions for Risk

Cost Estimating Advanced builds on the topics covered in Cost Estimating Basic. Topics addressed include: understanding the shape of a distribution and how it is related to that distribution’s parameters; alternative specification of distributions, such as by any two of a central point (mean, median, and/or mode), a coefficient of variation (CV), and a percentile; correction of distributions for understatement of both mean and CV; and useful rules of thumb for characterizing distributions.

Probability and Statistics Advanced
Marc Greenberg
Christian Smart

CEA 08 – ProbStat Advanced

This training session builds upon the Probability / Statistics Basic concepts relevant to cost estimation and uncertainty analysis. Topics to be presented include: ordinary least squares, outlier analysis, maximum likelihood estimation, and method of moments. Instructors will provide you with useful approaches to apply advanced statistical concepts to data sets used in your cost estimate generation efforts and approaches for the techniques to ensure you can apply the techniques appropriately in their real-world estimating scenarios.

Manufacturing Cost Estimating
John Deem
Charles Kurtz
Kammy Mann

CEA 09 – Manufacturing Cost Estimating

The goal of the Manufacturing Cost Estimating module is to arm the student with a set of techniques used to address issues unique to estimating in the manufacturing environment. It will be our objective in this module to raise a few of the most common general issues, considerations and concerns the estimator must be aware of in a typical major manufacturing environment and to provide techniques for addressing them.

Software Cost Estimating Advanced
Bob Hunt
Harold van Heeringen

CEA 10 – Software Cost Estimating Advanced

This session: 1) explores functional size measurement techniques with parametric models and historical data; 2) explains functional size and the estimation laws that should be understood before estimating a software project; 3) presents a case is given where a reality check based on functional size and historical data prevented a project disaster; 4) explores current practices in agile software development and present alternative approaches to estimating the cost and schedule for an agile program.

Integration Training Track

Basic Schedule Analysis
Justin Hornback

INT 01 – Basic Schedule Analysis

Examination of Program performance continues to identify the development and monitoring of program schedules as a weakness in programs that do not meet technical, cost, and schedule goals. This workshop presentation will cover scheduling basics, desired attributes of a valid project schedules, and basic schedule analysis techniques important to programmatic and cost estimator analysts. Schedule analysis tools and techniques will include, but are not limited to, three point estimates and schedule risk analysis.

Advanced Schedule Analysis
Justin Hornback

INT 02 – Advanced Schedule Analysis

This workshop will build on topics covered by Basic Schedule Analysis, expanding into the detailed process to conduct schedule analysis. Topics include common schedule analysis metrics and how to use them to analyze a schedule. Using an Integrated Master Schedule (IMS) to create an analysis schedule and employing schedule risk analysis with resource/cost loading to build the framework for Joint (Cost and Schedule) Confidence Level (JCL) creation.

Schedule Risk Analysis
David Hulett

INT 03 – Schedule Risk Analysis

Project scheduling is only the start of understanding when your project will finish. This workshop shows how simulating uncertain durations allow the user to determine how much overrun there might be at different target levels of certainty. We introduce the Risk Driver method of driving the Monte Carlo simulations using the risks usually found starting from the Risk Register. We then prioritize the risks for effective risk mitigation.

Joint Cost and Schedule Risk
Eric Druker
David Hulett

INT 04 – Integrated Cost Schedule Risk Analysis

Recognizing the need to ensure results from cost and schedule risk analyses are compatible, agencies, such as NASA, have recently implemented policy requiring the integration of these two, formerly separate, practices. This training course will provide analysts an overview on how they can integrate their cost estimates, schedules, and risk register into a single cohesive model that can then be analyzed to predict future cost and schedule growth and identify their sources so that mitigative actions can be taken. A live demonstration of an integrated cost/schedule risk analysis (ICSRA) tool, Polaris, will be conducted in class so that analysts can familiarize themselves with how an ICSRA model works.

Contracts Risk
Peter Braxton

INT 05 – Contracts-Risk

This session will present cost estimating and risk analysis as tools to aid in facing contract management challenges. This session is adapted from “Cost Estimating and Risk Analysis as Tools for Contract Management,” twice presented at the Government Contract Management Conference (GCMC), sponsored by sister society National Contract Management Association (NCMA). It also draws from “Analysis of Large O&S Proposals: Lessons Learned,” presented at ICEAA 2013.

Cost Management
Tom Dauber
John Deem
Zachary Jasnoff

INT 06 – Cost Management

This session provides an introduction to cost management, discussing tools and techniques currently in use by leading-edge organizations, the role of cost estimating/analysis, and the related discipline of Cost As an Independent Variable (CAIV). Cost management takes a holistic approach, incorporating cost considerations into the overall management approach, and together with other measures of performance, help lead to decisions that provide optimal value to organizations and customers, be they shareholders, consumers, or warfighters.

Cost Implications of WSARA
Bob Hunt

INT 07 – Cost Implications of WSARA

The Cost Implications of WSARA presentation places the legislation in the context of an ever-evolving acquisition framework. Further, it covers the role played by acquisition in management of the defense enterprise, especially its connection with the two other major decision-support systems employed in the Department: the Joint Capabilities Integration and Development System, and the Planning, Programming, Budgeting, and Execution System. These other two systems, requirements and resourcing, can and do influence acquisition outcomes, sometimes significantly.

Cost Estimating in PMBoK vs. CEBoK
Cassie Capots
Lauren Nolte

INT 08 – Cost Estimating in PMBoK vs CEBoK

The Project Management Body of Knowledge and Cost Estimating Body of Knowledge each give guidance on effective cost estimating for successful project execution. There is a good deal of overlap between the two curricula, but also significant differences. More importantly, in many instances, the same terms are used differently. Mastering cost estimating as it exists in both requires you to remember everything you learn as you study these bodies of knowledge. And then forget it.

Target Costing and Trades
John Deem
Marc Greenberg

INT 09 – Target Cost Management Implementation

This session provides a practical introduction to tools and techniques used by government and industry to incorporate cost considerations into enterprise cost management efforts by supporting decisions that provide optimal value to the enterprise. The presentation is based on Module 16 Cost Management of CEBoK© still covering the essence initiatives to reduce total ownership costs, CAIV, Target Costing and Activity-Based Costing while incorporating additional references useful to estimators and cost analysts preparing for assignments to support the affordability mission.

Data Science and Cost Estimating
Eric Druker

Slides available upon request: iceaa@iceaaonline.org

For decades, cost analysts have relied on the same statistical approaches to building estimates. These approaches face numerous limitations stressing their utility for estimating cost. Data Science is the extraction of knowledge from data. With the recent rise in computing power and democratization of complex methodologies we now have access to an array of approaches previously unthinkable when it came to cost estimating. This exploration will introduce data science to the cost estimating community.

Parametrics Training Track

Parametric Analysis Overview
Anthony Demarco
Steve Glogoza

PAR 01 – Parametric Analysis Overview

This introductory session provides an overview of the parametric estimating technique. Parametric estimating develops cost estimates based upon the examination and validation of the relationships which exist between a project’s technical, programmatic, and cost characteristics as well as the resources consumed during its development, manufacture, maintenance, and/or modification. Attendees will learn the key ideas, primary analytical constructs, practical applications, and applicable commercial-off-the-shelf tools.

Cost Estimating Relationships (CERs)
Anthony DeMarco

PAR 02 – Cost Estimating Relationships

Cost Estimating Relationships (CERs) are critical to cost analysis. At the most basic level, CERs are mathematical expressions developed using historical cost and technical data to relate cost with other independent variables. This module provides key definitions, describes the CER development process and significance tests. A walkthrough of notional CERs will be provided, including an example of basic CER creation. The module ends with a discussion of validation and advanced topics, including calibration.

Regression Linear
Remmie Arnold
Peter Braxton

PAR 03 – Regression Linear

This session covers the linear regression subset of Module 08 Regression Analysis of CEBoK. Upon completion of this session, the student will be able to: 1) Determine the linear regression equation for a bivariate data set using ordinary least squares (OLS); 2) Calculate appropriate statistics related to the linear regression and use them to determine goodness of fit and statistical significance of the model: and 3) Use these mathematical modeling techniques to create Cost Estimating Relationships (CERs) from provided normalized data.

Regression Nonlinear and Multivariate
David Harris

PAR 04 – Regression Nonlinear and Multivariate

Upon completion of this session, the student will be able to: 1)Apply logarithmic transformations to enable the determination of best fit using OLS for data exhibiting power, exponential, or logarithmic functional forms; 2) Determine the linear regression equation for a multivariate data set using ordinary least squares (OLS); 3) Calculate appropriate statistics use them to determine goodness of fit and statistical significance of the model; and 4) Use these mathematical modeling techniques to create Cost Estimating Relationships (CERs) from provided normalized data.

Regression Multiplicative
Herve Joumier

PAR 05 – Regression Multiplicative

Development of CERs has, in the past, been based on explicit solutions of the OLS Y = a+bX+E, where E is a Gaussian additive error term or the logarithmic transform of Y=aX^b*E. Applying “general-error” regression allows to determine the optimal coefficients for any curve shape and to choose the error model. This tutorial focuses on development of CERs having multiplicative (i.e., percentage) errors.

CER Risk and S-Curves
Chris Price
Christian Smart

PAR 06 – CER Risk and S-Curves

This training session compares S-Curves generated using standard errors of the estimate (SEE’s), confidence intervals (CI’s), and prediction invervals (PI’s), arguing for PI’s as the appropriate technique. It is shown how and why SEE-based techniques, while commonly used, almost always underestimate both risk and uncertainty. Finally, we provide a step-by-step guide on how to generate PI-based S-curves, with the mathematical intuition behind each step.

Complex Models Company-Developed
Chris Price

PAR 07 – Complex Models Company-Developed

“Company Developed Complex Models” covers the material in Chapter 4 of the Parametric Estimating Handbook (PEH), which describes a proven ten-step approach for developing a multi-CER cost prediction model from historic data. The handbook defines a “company-developed” model as a special-purpose model, typically developed by an organization for a specific platform or product. A brief comparison of the GAO Cost Estimating Process and the PEH approach is also provided.

Complex Models Hardware
David Bloom

The Complex Hardware Models session (PT08) of the ICEAA Parametric training track provides a short history of complex model parametric estimating and continues with an overview, examples and recommendations to develop Basis of Estimates (BOE for Rough Order Magnitude (ROMs), Business Case Analysis, and Proposals. Best practices and lessons learned from the Parametric Estimating Reinvention Laboratory and personal experience provide insight into practical applications and limitations of the toolsets.

Other Parametric Applications
John Swaren

PAR 09 – Other Parametric Applications

Other Parametric Applications covers the material in Chapter 8 of the Parametric Estimating Handbook (PEH) This training will address what to consider when developing and implementing parametric estimating tools. We will also give descriptions of general and specialized applications where parametric tools can be used. We will show examples of where and how these techniques have been implemented.