Identifying enterprise leverage points in Defense Acquisition Program performance

Full text

(1)Identifying Enterprise Leverage Points in Defense Acquisition Program Performance by Joseph Robert Wirthlin B.S. Engineering Sciences United States Air Force Academy, 1994 S.M. Engineering and Management Massachusetts Institute of Technology, 2000 Submitted to the Engineering Systems Division in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Engineering Systems at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY September 2009 © 2009 Joseph Robert Wirthlin. All rights reserved. The author hereby grants to MIT permission to reproduce and to distribute publicly paper and electronic copies of this thesis document in whole or in part in any medium now known or hereafter created. Signature of Author…………………………………………………………………………………………………………………….. Engineering Systems Division September 05, 2009 Certified by…………………………………………………………………………………………………………………………………. Warren P. Seering Weber-Shaughness Professor of Mechanical Engineering and Engineering Systems Thesis Supervisor Certified by…………………………………………………………………………………………………………………………………. Sheila E. Widnall Institute Professor, Aeronautics and Astronautics and Engineering Systems Certified by…………………………………………………………………………………………………………………………………. Donald R. Lessard Epoch Foundation Professor of International Management, Sloan School of Management Accepted by………………………………………………………………………………………………………………………………… Nancy Leveson Professor of Aeronautics and Astronautics and Engineering Systems Chair, ESD Education Committee. 1.

(2) Disclaimer The views expressed in this work are those of the author and do not reflect the official policy or position of the United States Air Force, Department of Defense, or the U.S. Government.. 2.

(3) Identifying Enterprise Leverage Points in Defense Acquisition Program Performance by Joseph Robert Wirthlin Submitted to the Engineering Systems Division on September 05, 2009 in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Engineering Systems. ABSTRACT Large, complex systems development programs in the Department of Defense are finding it more difficult to deliver desired capabilities to the end user on time and on budget than ever before. Evidence exists that almost all developmental programs on record are over cost and schedule, costing the Department and ultimately the U.S. taxpayer billions of dollars more than anticipated. Numerous studies over many decades have addressed various aspects of the problems plaguing these efforts with many recommendations. Unfortunately, most of these recommendations have been ignored or poorly implemented with limited success. This work embodies an exploratory systems approach to characterize the system of acquiring large, complex, socio-technological systems for the Department of Defense. Through a series of qualitative studies and in-depth interviews with individuals working in the Joint Capabilities Integration Development System (JCIDS), the Planning, Programming, Budgeting, and Execution (PPBE) process, and the Acquisition system, a model of the larger “enterprise of acquisition” or Acquisition System was developed. The model has a scope ranging from the very early beginnings of any program through the conclusion of developmental activities. The methodology used consisted of stringing together the individual pieces of the system defined by probabilistic distributions of time and corresponding probabilistic decision points into a model ideal for discrete-event simulation. An extensive program of verification and validation of the model was carried out to increase confidence in the model and its simulation outcomes. Experimental system interventions, designed to mimic potential policy interventions and/or system changes, were introduced into the model and the corresponding outcomes analyzed. Results show several interventions have varying degrees of influence and suggest no single antidote exists for solving the problems related to Acquisition. Furthermore, many of the outcomes of the system can be described as emergent behaviors versus problems stemming from poor program management, program risk management, or requirements management. Thesis Supervisor: Warren P. Seering Title: Weber-Shaughness Professor of Mechanical Engineering and Engineering Systems. 3.

(4) [blank]. 4.

(5) Biographical Summary of Author Robb Wirthlin is a Lieutenant Colonel in the United States Air Force. He graduated from the United States Air Force Academy in 1994 with a degree in Engineering Sciences and a Foreign Language Minor. He has served in various acquisition and systems engineering roles within the Air Force ranging from assignments at an Air Logistics Center, a Development Center, and at an operational facility. He was awarded a masters degree in Engineering and Management from the Massachusetts Institute of Technology in 2000. He loves his wife and children very much.. 5.

(6) [blank]. 6.

(7) Acknowledgements I am profoundly grateful for the support, encouragement and guidance of my committee, Dr. Warren Seering, Dr. Sheila Widnall, and Dr. Don Lessard. I can not express enough of my thanks for their insight and wisdom, especially as they helped me to see things that were not always clear to me and prodded me in areas where I needed encouragement. For Warren, in particular, I want to thank you for the countless hours spent teaching, guiding, and talking with me. Your enthusiasm and positive attitude carried me along when I felt especially discouraged. Without your confidence and resolve, I would not have finished this dissertation. I want to express my heartfelt thanks to Dr. Eric Rebentisch. As my perennial sounding board and a good friend, I am forever grateful for your timeless hours of advice and assistance. I never could have imagined the twists and turns this effort has taken and I’m grateful for the breakthroughs you handed to me in my most frustrating moments. I owe the clarity of this dissertation to Claire Betar, PhD, a dear friend and mentor. Without your patient, kind critiques of my writing style and voice, this work would fall short in many areas. I’m grateful you’ve helped me to clearly communicate and express the concepts in my dissertation. I’ve learned to love the power of the written word, the beauty of the English language, and the importance of proper grammar. I regret that I’ve never had the privilege of being a student in one of your classrooms. I also want to thank Capt Paul Conner for the hours of dedicated research done for me, especially after just finishing your own challenging degree program. You did some real grunt work that I didn’t have the time or patience to do. Without your help, I am sure I would still be wallowing in the midst of multiple databases trying to sort out data. To the professionals working in the DOD Acquisition system, I thank you for your dedicated service, candid insights, feedback, and information. This dissertation would not have been possible without your efforts. I hope this work will someday make your work better. To my friends and colleagues at MIT in the Engineering Systems Division and especially the Lean Advancement Initiative: Thank You! I will miss the daily interaction, the helpful feedback, and the stimulating discussions. To those of you in the Silo – Dan, Dave, Joao, Sid, Pedzi, Dan, Claudia, Sebastian, and Damien – I can not thank you enough. We worked, laughed, and played hard together. I look forward to strengthening our friendships and collaborating professionally in the future. I would like to thank my parents for helping me learn to work, encouraging me to reach for the stars, and teaching me to trust in God. I am grateful for their examples and love. My wife and children have supported me and encouraged me, made me laugh and motivated me to do my best. I want to thank them and especially thank my wife for encouraging me to go to MIT and believe in myself that I could go back again to MIT and earn a doctorate. I am grateful for the heavy load she carried that enabled me focus on the work needed to complete this degree. I believe God is the source of all knowledge and He has inspired me and blessed me to complete this challenging degree. I have learned a lot and I am looking forward to learning even more. Furthermore, our family has learned from, grown, and overcome great obstacles with the support, blessings and divine intervention of Our Heavenly Father, for which I will always be thankful and grateful. 7.

(8) [blank]. 8.

(9) Table of Contents List of Figures .............................................................................................................................. 12 List of Tables ............................................................................................................................... 15 List of Acronyms ......................................................................................................................... 17 CHAPTER 1 -- INTRODUCTION ........................................................................................... 23 Research questions, approach, and methods ............................................................................. 25 Research Limitations ................................................................................................................ 25 Dissertation Outline .................................................................................................................. 26 CHAPTER 2 – LITERATURE REVIEW ................................................................................ 28 Product development processes ................................................................................................ 29 Risk ....................................................................................................................................... 30 Portfolio Management strategies .......................................................................................... 33 Enterprise Risk and Portfolio Execution............................................................................... 37 Enterprise Acquisition System .................................................................................................. 38 JCIDS .................................................................................................................................... 41 PPBE ..................................................................................................................................... 49 Acquisition ............................................................................................................................ 59 Contractors ............................................................................................................................ 64 Modeling and Analysis ............................................................................................................. 66 Summary of Literature Review – Key Take-aways .................................................................. 69 CHAPTER 3 - ACQUISITION ................................................................................................. 70 Initial Observations and Analysis ............................................................................................. 72 Data Coding and Analysis .................................................................................................... 75 Further Discussion ................................................................................................................ 76 Potential measures for further research................................................................................. 80 Conclusions ............................................................................................................................... 81 CHAPTER 4 -- THE OTHER PARTS OF THE ACQUISITION SYSTEM ....................... 83 Results and Analysis ................................................................................................................. 85 The Program Element Monitor ............................................................................................. 85 Other Identified Issues .......................................................................................................... 88 Other issues ........................................................................................................................... 98 Chapter Conclusion: Implications of this study ...................................................................... 102 CHAPTER 5 -- A MODEL OF THE ENTERPRISE ACQUISITION SYSTEM .............. 104 Model design and depiction .................................................................................................... 106 Model Scope ....................................................................................................................... 108 Model Symbology............................................................................................................... 110 Model Assumptions ............................................................................................................ 115 Conclusions ............................................................................................................................. 116 9.

(10) CHAPTER 6 – VERIFICATION AND VALIDATION ....................................................... 118 Verification of free style model .............................................................................................. 118 Hand Modeling ................................................................................................................... 119 PPBE modeling ................................................................................................................... 120 Validation of free style model................................................................................................. 121 Verification of computer simulation model ............................................................................ 124 Validation of computer model ................................................................................................ 125 ACAT I Programs ............................................................................................................... 128 ACAT II/III Programs......................................................................................................... 128 Actual Data Results............................................................................................................. 129 Comparison of model outcomes and actual data – Final Validation ...................................... 137 All ACAT results vs all actual data .................................................................................... 138 ACAT I model results vs ACAT I actual results ................................................................ 140 ACAT II model results vs ACAT II actual results ............................................................. 142 ACAT III model results vs ACAT III actual data results ................................................... 144 Conclusions ............................................................................................................................. 146 CHAPTER 7-- MODEL RESULTS AND REPRESENTATIVE OUTPUT ....................... 148 Model Parameters ................................................................................................................... 148 Model Simulation.................................................................................................................... 149 Typical model output .......................................................................................................... 150 Other analyses done; preliminary results ................................................................................ 154 Program End Points ............................................................................................................ 154 DSM representation and preliminary analysis .................................................................... 159 Model Sensitivities.............................................................................................................. 166 Additional Questions using Sensitivity Test Data .............................................................. 168 Other Questions .................................................................................................................. 169 Further Model Results and Analysis ....................................................................................... 170 Conclusions ............................................................................................................................. 174 CHAPTER 8 -- HYPOTHESIS TESTING AND ANALYSIS .............................................. 176 Hypothesis............................................................................................................................... 177 Key Questions ......................................................................................................................... 178 Experimental Interventions ..................................................................................................... 178 Requirements Swim Lane Interventions ............................................................................. 179 Planning, Programming, Budgeting, Execution System Swim Lane Interventions ........... 182 Acquisition Swim Lane Interventions ................................................................................ 183 Other combinations ............................................................................................................. 191 Greatest Impact Interventions ............................................................................................. 196 Final Analysis and Conclusions .............................................................................................. 198 CHAPTER 9 -- CONCLUSIONS AND SUMMARY ............................................................ 203 Qualitative Observations......................................................................................................... 203 Quantitative Findings .............................................................................................................. 208 Overarching Conclusions ........................................................................................................ 210 Policy Implications and Potential System Improvements ...................................................... 212. 10.

(11) Future work ............................................................................................................................. 214 Recommendations ................................................................................................................... 216 Summary ................................................................................................................................. 216 Bibliography .............................................................................................................................. 218 Appendix A – Other Studies and Recommendations to Improve Acquisition Outcomes .. 226 Key Takeaways ....................................................................................................................... 226 Discussion of Individual Studies ............................................................................................ 227 Summary and Conclusions ..................................................................................................... 239 Appendix B – Sample Questions used in Acquisition Study ................................................. 240 Appendix C – Sample questions used for second study......................................................... 242 Appendix D – Description of Model and Data Documentation ............................................ 244 Introductory description and explanation ............................................................................... 244 Project Attributes ................................................................................................................ 246 ACAT Discussion ............................................................................................................... 248 The Programming, Planning, Budgeting and Execution Process ....................................... 249 Detailed Model Explanation ................................................................................................... 261 The Pre-Milestone A Swim Lanes ...................................................................................... 263 The Pre-Milestone B Swim Lanes ...................................................................................... 299 The Pre-Milestone C Swim Lanes ...................................................................................... 332 Summary ............................................................................................................................. 372 Appendix E – Model Documentation ...................................................................................... 373 SIMAN Code .......................................................................................................................... 373 SIMAN Global Code .............................................................................................................. 536 Enterprise Acquisition Process Model .................................................................................... 584 Submodel for Module Acquisition Planning Activities PreB ............................................. 654 Submodel for Module Acquisition Planning Activities PreC ............................................. 656 Submodel for Module Developmental Test and Evaluation ............................................... 700 Submodel for Module Early Operational Assessment ........................................................ 708 Submodel for Module Independent document preA........................................................... 726 Submodel for Module Independent document preB ........................................................... 733 Submodel for Module Independent document preC ........................................................... 740 Submodel for Module Joint Integration PreA ..................................................................... 749 Submodel for Module Joint Integration PreB ..................................................................... 761 Submodel for Module Joint Integration PreC ..................................................................... 772 Submodel for Module Joint Interest preA .......................................................................... 784 Submodel for Module Joint Interest preB ........................................................................... 797 Submodel for Module Joint Interest preC ........................................................................... 810 Submodel for Module Set ACAT level .............................................................................. 875 Submodel for Module Study for ICD Development ........................................................... 884. 11.

(12) List of Figures Figure 1: An example of a notional product development process .............................................. 29 Figure 2: The Total Acquisition System ....................................................................................... 40 Figure 3: From Chapter 1.3 of the Defense Acquisition Guidebook ............................................ 41 Figure 4: Relationship of Strategic Processes with JCIDS ........................................................... 43 Figure 5: Connections between JCIDS and Acquisition............................................................... 45 Figure 6: PPBE Timeline .............................................................................................................. 50 Figure 7: "On-year" PPBE schedule ............................................................................................. 51 Figure 8: "Off-year" PPBE Schedule ............................................................................................ 51 Figure 9: The US Air Force Corporate Process ............................................................................ 53 Figure 10: One MAJCOM’s corporate structure .......................................................................... 53 Figure 11: Figure from a USAF MAJCOM.................................................................................. 54 Figure 12: Concurrent Program and Budget Review Process ...................................................... 55 Figure 13: PPBE Timing............................................................................................................... 59 Figure 14: An Overview of the Acquisition System ..................................................................... 60 Figure 15: Overlay of Acquisition and Requirements processes .................................................. 61 Figure 16: Ways to Study a System .............................................................................................. 66 Figure 17: A Model Taxonomy .................................................................................................... 67 Figure 18: Portfolio Manager Capability Matrix .......................................................................... 77 Figure 19: Portfolio Domain Space .............................................................................................. 79 Figure 20: A Holistic View of the Acquisition System .............................................................. 105 Figure 21: The “Wall Chart” of the Defense Acquisition System (December 2008 version) .... 108 Figure 22: Model Scope in Relation to the Overall Acquisition System .................................... 109 Figure 23: Conceptual model of the Acquisition Enterprise ...................................................... 110 Figure 24: Close-up of the Requirements portion of the Pre-MS A swim lane.......................... 111 Figure 25: Close-up view of three elements of Pre-MS A Requirements swim lane ................. 112 Figure 26: Final Model Representation ...................................................................................... 114 Figure 27: Example of Scanned image of model feedback form................................................ 122 Figure 28: Image of marked-up paper model ............................................................................. 123 Figure 29: Data Sources for Model Validation ........................................................................... 126 Figure 30: Example Contract Performance Tables as found on Page 2 of each MAR ............... 135 Figure 31: Unequal sample-t test ................................................................................................ 138 Figure 32: Histogram of time elapsed for all ACAT programs between MS B and MS C ........ 139 Figure 33: Histogram of actual program data time elapsed between MS B and MS C .............. 139 Figure 34: Histogram of time elapsed for ACAT I data between MS B and MS C ................... 141 Figure 35: Histogram of time elapsed for ACAT I data between MS B and MS C ................... 141 Figure 36: ACAT II model data between MS B and MS C ........................................................ 143 Figure 37: ACAT II actual time elapsed data between MS B and MS C ................................... 143 Figure 38: ACAT III model histogram of time elapsed between MS B and MS C.................... 145 Figure 39: Histogram of ACAT III actual time elapsed between MS B and MS C ................... 145 Figure 40: Representative and partial output of simulation file ................................................. 151 Figure 41: Histogram of model output at Milestone C ............................................................... 151 Figure 42: Histogram of MS C arrivals after 100000 iterations ................................................. 152 Figure 43: Partitioned DSM of Acquisition System ................................................................... 159 Figure 45: Pre-MS A Partitioned DSM ...................................................................................... 161 Figure 47: Pre-MS B Partitioned DSM....................................................................................... 163 12.

(13) Figure 49: Pre-MS C Partitioned DSM....................................................................................... 165 Figure 50: Comparison of MS C arrivals between forced formal system and any shortcuts ..... 170 Figure 51: Graphical depiction of model outcomes .................................................................... 173 Figure 52: Depiction of categories why schedules slip across development efforts .................. 231 Figure 53: Graphical representation of PPBE Swim Lane ......................................................... 250 Figure 54: First third of PPBE model ......................................................................................... 251 Figure 55: Middle third of PPBE model ..................................................................................... 253 Figure 56: Last third of PPBE model .......................................................................................... 257 Figure 57: Final Model Representation ...................................................................................... 262 Figure 58: Final model with descriptive labels ........................................................................... 263 Figure 59: Pre-MS A portion of model with close up sections marked...................................... 264 Figure 60: Early Pre-MS A close up ........................................................................................... 264 Figure 61: Early Pre-MS A close up in requirements swim lane after initial screening ............ 270 Figure 62: Close up of another portion of the early Pre-MS A requirement swim lane ............. 272 Figure 63: Pre-MS A early PPBE activity .................................................................................. 274 Figure 64: Pre-MS A close up of JCIDS process for ICD Development ................................... 275 Figure 65: Pre MS-A Independent document process ................................................................ 276 Figure 66: Pre-MS A Joint Integration document process, part I ............................................... 279 Figure 67: Pre-MS A Joint Integration document process, part II.............................................. 281 Figure 68: Pre-MS A Joint Interest document process, part I..................................................... 282 Figure 69: Pre-MS A Joint Interest document process, part II ................................................... 284 Figure 70: Pre-MS A PPBE and early Acquisition activities ..................................................... 286 Figure 71: Pre-MS A AOA activities.......................................................................................... 289 Figure 72: Pre-MS A Acquisition activities parallel to AOA ..................................................... 291 Figure 73: Pre-MS A COA approval processes .......................................................................... 293 Figure 74: Pre-MS A Acquisition swim lane after COA selection............................................. 294 Figure 75: Pre-MS A final funding check prior to MS A approval ............................................ 295 Figure 76: Pre-MS A Milestone Decision activity ..................................................................... 297 Figure 77: Pre-MS B Swim Lanes with Reference Figures ........................................................ 299 Figure 78: Pre-MS B Early Requirements Swim Lane............................................................... 300 Figure 79: Pre-MS B Entry into formal requirements process at MAJCOM ............................. 302 Figure 80: Pre-MS B Requirements swim lane MAJCOM process ........................................... 303 Figure 81: Pre-MS B Requirements swim lane JCIDS process .................................................. 304 Figure 82: Pre-MS B Requirements Independent Document process ........................................ 305 Figure 83: Pre-MS B Requirements swim lane Joint Integration process, Part I ....................... 307 Figure 84: Pre-MS B Requirements swim lane Joint Integration process, Part II ...................... 309 Figure 85: Pre-MS B Requirements swim lane Joint Interest process, Part I ............................. 311 Figure 86: Pre-MS B Requirements swim lane Joint Interest process, part II ............................ 313 Figure 87: Pre-MS B early acquisition swim lane activities ...................................................... 315 Figure 88: Pre-MS B Acquisition costing and acquisition planning .......................................... 317 Figure 89: Pre-MS B Contract start-up activities ....................................................................... 318 Figure 90: Pre-MS B Acquisition swim lane Systems Engineering activities............................ 318 Figure 91: Pre-MS B Acquisition swim lane preparations for Acquisition Panels .................... 320 Figure 92: Pre-MS B PPBE Funding check ................................................................................ 322 Figure 93: Pre-MS B Acquisition swim lane Milestone B decision ........................................... 324 Figure 94: Pre-MS B Acquisition swim lane financial uncertainty engine ................................ 325. 13.

(14) Figure 95: Pre-MS B Contractor swim lane uncertainty generator and contract engine ............ 327 Figure 96: Pre-MS B Acquisition swim lane program management and oversight loop ........... 329 Figure 97: Pre-MS C Swim Lanes with Reference Figures ........................................................ 332 Figure 98: Pre-MS C Early Requirements Swim Lane............................................................... 333 Figure 99: Pre-MS C Entry into formal requirements process at MAJCOM ............................. 335 Figure 100: Pre-MS C Requirements swim lane MAJCOM process ......................................... 336 Figure 101: Pre-MS C Requirements swim lane JCIDS process ................................................ 337 Figure 102: Pre-MS C Requirements Independent Document process ...................................... 338 Figure 103: Pre-MS C Requirements swim lane Joint Integration process, Part I ..................... 340 Figure 104: Pre-MS C Requirements swim lane Joint Integration process, Part II .................... 342 Figure 105: Pre-MS C Requirements swim lane Joint Interest process, Part I ........................... 344 Figure 106: Pre-MS C Requirements swim lane Joint Interest process, part II.......................... 347 Figure 107: Pre-MS C PPBE Early funding check ..................................................................... 349 Figure 108: Pre-MS C early acquisition swim lane activities..................................................... 350 Figure 109: Pre-MS C Acquisition costing and acquisition planning ........................................ 351 Figure 110: Pre-MS C Contract start-up activities ..................................................................... 352 Figure 111: Pre-MS C Early Systems Engineering Preliminary Design Review activity .......... 353 Figure 112: Pre-MS C Acquisition swim lane Critical Design Reviews .................................... 354 Figure 113: Pre-MS C Acquisition swim lane fabrication, assembly and testing ...................... 356 Figure 114: Pre-MS C Acquisition swim lane System Verification Review.............................. 359 Figure 115: Pre-MS C Acquisition swim lane preparations for Acquisition Panels .................. 360 Figure 116: Pre-MS C PPBE Funding check .............................................................................. 362 Figure 117: Pre-MS C Acquisition swim lane Milestone B decision ......................................... 363 Figure 118: Pre-MS C Acquisition swim lane financial uncertainty engine .............................. 365 Figure 119: Pre-MS C Contractor swim lane uncertainty generator and contract engine .......... 367 Figure 120: Pre-MS C Acquisition swim lane program management and oversight loop ......... 369. 14.

(15) List of Tables Table 1: Table of Validation and Approval Authority.................................................................. 46 Table 2: Modification Thresholds (Financial Thresholds) ........................................................... 47 Table 3: Document Certification/Validation Authority ................................................................ 48 Table 4: Issues specific for Program Element Monitors (PEMs) ................................................. 86 Table 5: Common issues identified by individuals in JCIDS and PPBE...................................... 89 Table 6: Common issues identified by individuals in JCIDS and the user community ............... 92 Table 7: Issues identified by individuals within JCIDS only ....................................................... 93 Table 8: Issues identified by individuals within PPBE only......................................................... 95 Table 9: Example table of hand modeling trials ......................................................................... 119 Table 10: Explicit PPBE hand-modeling .................................................................................... 120 Table 11: Multi-source Acquisition Program Schedule Data ..................................................... 132 Table 12: Multi-source Acquisition Program Cost Data ............................................................ 133 Table 13: t-Test: Two-Sample test assuming unequal variances between data for all ACATs .. 140 Table 14: ACAT I model and actual data test results ................................................................. 142 Table 15: ACAT II model and actual t-test results ..................................................................... 144 Table 16: t-test results for ACAT III data ................................................................................... 146 Table 17: Basis statistics for Milestone C model output ............................................................ 153 Table 18: Basic statistics for MS C model output with additional iterations ............................. 154 Table 19: Analysis of model terminating points ......................................................................... 155 Table 20: Analysis of model terminating points excluding early rejections .............................. 156 Table 21: Analysis of model terminating points excluding early rejected & diverted programs 157 Table 22: End point summary statistics for sample of 48500 ..................................................... 158 Table 23: End point summary statistics for sample of 100,000 .................................................. 158 Table 24: Original Model data outcomes at MS C ..................................................................... 167 Table 25: Changed model outcomes at MS C ............................................................................ 168 Table 26: Original model outcomes at MS C with no excursions allowed ................................ 168 Table 27: Air staff intervention model outcomes at MS C with no excursions allowed ............ 168 Table 28: Additional Data Analysis ............................................................................................ 172 Table 29: Requirements swim lane analysis ............................................................................... 173 Table 30: Air Staff intervention results ...................................................................................... 180 Table 31: MAJCOM approval bodies result ............................................................................... 181 Table 32: Critical comments results ........................................................................................... 182 Table 33: Funding Stability Results ............................................................................................ 183 Table 34: Acquisition Kill results ............................................................................................... 184 Table 35: Approval bodies results .............................................................................................. 185 Table 36: Technical Interventions Results .................................................................................. 186 Table 37: PDR intervention results ............................................................................................. 187 Table 38: CDR Intervention........................................................................................................ 188 Table 39: DRR Intervention........................................................................................................ 188 Table 40: TRR Intervention Results ........................................................................................... 189 Table 41: Test trades intervention results ................................................................................... 190 Table 42: SVR Intervention ........................................................................................................ 191 Table 43: Systems Enginering Interventions .............................................................................. 192 Table 44: SE and Acquisition Kill Intervention Results............................................................. 193 Table 45: MAJCOM and Acquisition approval bodies intervention results............................... 194 15.

(16) Table 46: Funding and Technical Uncertainty Intervention Results .......................................... 195 Table 47: Random Eight Intervention results ............................................................................. 196 Table 48: Top Three Interventions Results ................................................................................. 197 Table 49: All Interventions combined results ............................................................................. 198 Table 50: Most and Least Significant Categories of Reasons for Schedule Problems ............... 231 Table 51: Description and Decision Authority for ACAT I – III Programs ............................... 249 Table 52: Approval Authority level for JCIDS documents based on ACAT level .................... 276. 16.

(17) List of Acronyms A1 A2 A3 A4 A5 A6 A7 A8 A9 ACAT ACC ACWP ADM AF AFAKSS AFAM AFB AFB AFC AFDO AFG AFI AFIT AFMC AFRL AFROCC AIS ALC AOA APB APOM AQ AQX ASC ASD AT&L. Manpower and Personnel Intelligence Air, Space and Information Operations Logistics Plans and Requirements Communications Installations and Mission Support Strategic Plans and Programs Analyses, Assessments and Lessons Learned Acquisition Category Air Combat Command Actual Cost of Work Performed Acquisition Decision Memorandum Air Force Air Force Acquisition Knowledge Sharing System Air Force Acquisition Model Air Force Board Air Force Base Air Force Council Award Fee Designating Official Air Force Group Air Force Instruction Air Force Institution of Technology Air Force Materiel Command Air Force Research Laboratory Air Force Requirements for Operational Capabilities Council Automated Information System Air Logistics Center Analysis of Alternatives Acquisition Program Baseline Amended Program Objective Memorandum refers to SAF/AQ refers to SAF/AQX Aeronautical Systems Center Assistant Secretary of Defense Acquisition, Technology and Logistics. BAC BCG BCP BCWP BCWS BES BPM BR. Budget at Completion Boston Consulting Group Budget Change Proposal Budgeted Cost for Work Performed Budgeted Cost for Work Scheduled Budget Estimate Submission Business Process Modeling Budget Request. 17.

(18) C3I C4CS C&I CAE CBA CDD CDR CJCSI CJCSM CIO C-level COA COCOMS COMACC CONOPS CP CP CPD CPI CPIF CPR CRS CSAF CS&P CSP CV CV. Command, Control, Communications, and Intelligence Command, Control, Communications, and Computer Systems Communication and Information Component Acquisition Executive Capabilities Based Assessment Capability Development Document Critical Design Review Chairman of the Joint Chiefs of Staff Instruction Chairman of the Joint Chiefs of Staff Manual Chief Information Officer Corporate level Course of Action Combatant Commands Commander ACC Concept of Operation Capability Plan Change Proposal Capability Production Document Cost Efficiency Cost Plus Incentive Fee Cost Performance Report Congressional Research Service Chief of Staff of the Air Force Competitive Sourcing and Privatization Cost, Schedule, and Performance Vice Chairman Cost Variance. D DAB DAES DAMIR DAPA DAU DAWIA DOD DODI DOTMLPF DCR DIA DRR DSM DT&E DTIC. Disconnect Defense Acquisition Board Defense Acquisition Executive System Defense Acquisition Management Information Retrieval Defense Acquisition Performance Assessment Defense Acquisition University Defense Acquisition Workers Improvement Act Department of Defense Department of Defense Instruction Doctrine, Organization, Training, Material, Leadership and Education, Personnel and Facilities DOTMLPF Change Recommendation Defense Intelligence Agency Design Readiness Review Design Structure Matrix Developmental Test and Evaluation Defense Technical Information Center. EAC. Estimate at Completion. 18.

(19) ECP EOA EPP ESC EVMS. Engineering Change Proposal Early Operational Assessment Enhanced Planning Process Electronic Systems Center Earned Value Management Systems. FCB FIFO FMECA FOC FRP FSA FY FYDP. Functional Capabilities Board First In, First Out Failure Mode, Effects, and Criticality Analysis Full Operational Capability Full Rate Production Functional Solutions Analysis Fiscal Year Future Years Defense Program/Plan. GAO CGIC GE. Government Accountability Office Global Cyberspace Integration Center General Electric. HPT HQ. High Performance Team Headquarters. I ICAF ICD ICE IDA IOC IOT&E IPL IPT IRSS ISP IT ITAB ITT. Initiative Industrial College of the Armed Forces Initial Capabilities Document Independent Cost Estimate Institute for Defense Analysis Initial Operating Capability Initial Operational Test & Evaluation Integrated Priority List Integrated Process Team Integrated Requirement Support System Integrated Support Plan Information Technology Information Technology Acquisition Board Integrated Test Team. J1 J2 J3 J4 J5 J6 J7 J8 JCB JCD JCIDS. Manpower and Personnel Joint Staff Intelligence Operations Logistics Strategic Plans and Policy Command, Control, Communications, and Computer Systems Operational Plans & Joint Force Development Force Structure Resources and Assessment Joint Capabilities Board Joint Capability Document Joint Capabilities Integration Development System. 19.

(20) JFCOM JIC JPD JPG JROC JS. Joint Forces Command Joint Integrating Concepts Joint Potential Designator Joint Programming Guidance Joint Requirements Oversight Council Joint Staff. KPP KSA. Key Performance Parameter Key System Attribute. LCMP LRIP. Life Cycle Management Plan Limited Rate Incremental Production. MAIS MAJCOM MAR MAUT MBI MDA MDAP MILCON MIT MR MS. Major Automated Information System Major Command Monthly Acquisition Report Multi-Attribute Utility Theory Major Budget Issue Milestone Decision Authority Major Defense Acquisition Program Military Construction Massachusetts Institute of Technology Management Reserve Milestone. NAS NIP NPD NPS NSS. National Academy of Sciences National Intelligence Program New Product Development Naval Postgraduate School National Security Strategy. O OMB OPR OSD OT&E. Offset Office of Management and Budget Officer Performance Report Office of the Secretary of Defense Operational Test and Evaluation. P&R PB PBD PCP PD PDM PDR PE PEM PEO PHA. Planning and Requirements President’s Budget Program Budget Decision Program Change Proposals Product Development Program Decision Memorandum Preliminary Design Review Program Element Program Element Monitor Program Executive Officer Physical Health Assessment. 20.

(21) PM POM POPS PPBE PPBES PPD PR. Program Manager Program Objective Memorandum Probability of Program Success Planning, Programming, Budgeting, and Execution Planning, Programming, Budgeting, and Execution System (no longer favored) Program Planning Document Program Review. QDR. Quadrennial Defense Review. RAND R&D RCT RFP RMP RDT&E RSR. Research ANd Development Corporation Research and Development Requirements Crosswalk Table Request for Proposal Radar Modernization Program Research, Development, Test and Evaluation Requirements Strategy Review. SACOM SAE SAF SAF/AQ SAF/AQX SAF/FMB SAF/XC SAR SAR SE SES SIMAN SLRG SMART SPG SPI SPO SPOC SPRG SSA SV SVC SVR. Sustainment/Acquisition Composite Model Service Acquisition Executive Secretary of the Air Force Secretary of the Air Force – Acquisition Secretary of the Air Force – Acquisition Integration Secretary of the Air Force – Budget Secretary of the Air Force – Warfighting Integration & Chief Information Officer Special Access Required Selected Acquisition Report Systems Engineering Senior Executive Service SIMulation Analysis Senior Leadership Review Group System Metric and Reporting Tool Strategic Planning Guidance Schedule Efficiency System Program Office Special Access Required (SAR) Programs Oversight Committee Special Program Review Group Source Selection Authority Schedule Variance Service System Verification Review. T&E TDS TEMP TRR. Test and Evaluation Technology Development Strategy Test and Evaluation Master Plan Test Readiness Review. US. United States. 21.

(22) USAF USC USD. United States Air Force United States Code Undersecretary of Defense. VAC VSM. Variance at Completion Value Stream Mapping. 22.

(23) CHAPTER 1 -- INTRODUCTION Throughout the twentieth century, and into the beginnings of the 21st, the United States military has enjoyed unprecedented superiority in the systems and methods used to gain victory on the battlefield. These tangible results are the outcome of thousands of people working to design, develop and acquire complex weapons systems. However, throughout the past four decades, and perhaps even longer, the United States Defense establishment has been fighting another war; one that it appears to be losing badly--that of budgets and schedules out of control in the development of its systems. Furthermore, the trends seem to be getting worse. In the early spring of 2009, the Government Accountability Office (GAO) released scathing reports on the state of defense acquisitions [1]. Nearly all of the complex systems and development examined by these reports were over budget or over schedule or both [2]. These reports come on the heels of and are merely an appendix to the many reports that have been issued since the early 1960s decrying the state of defense acquisition and bemoaning its outcomes. In one of the more recent studies, the Defense Acquisition Performance Assessment (DAPA) examined the history of acquisition reform in the US military and found that most of the substantive reform suggestions and recommended policy changes in those historical studies were either ignored or trivialized [3, 4]. Although the DAPA report's own conclusions have been warmly embraced, their own recommendations have met a similar fate: a tepid response from both the Department of Defense and congressional leadership as noted in the summary of a recently published National Academy of Sciences report about the early phases of Air Force Acquisition [5]. The structure and appearance of the organizations responsible to acquire new systems have only grown more complicated through the years. Between policy choices and statutory requirements, the Department of Defense has developed a number of processes and organizations that help manage systems acquisition. A virtual army of largely unsung skilled professionals toil to deliver these systems. 23.

(24) to the field. Nevertheless, Congressional concern about the acquisition of systems is high. In the House Armed Services Committee’s report on the FY 2007 defense authorization bill it states: Simply put, the Department of Defense (DOD) acquisition process is broken. The ability of the department to conduct the large-scale acquisitions required to ensure our future national security is a concern of the committee. The rising costs and lengthening schedules of major defense acquisition programs lead to more expensive platforms fielded in fewer numbers. The committee's concerns extend to all three key components of the acquisition process including requirements generation, acquisition and contracting, and financial management [6] . The idea that all of the major system components are not functioning properly resonates with many of those working in the acquisition system. Recently, various organizations have suggested product portfolio management and better risk management as the way to address the worsening trends of defense acquisition [7]. The thinking goes that if systems are managed as portfolios, trade-offs could be made across that portfolio, both to manage the throughput and also to optimize resource deployment to get better outcomes. Risk is a natural part of that discussion. The United States Air Force is currently engaged in an effort to adopt these ideas and is, therefore, quite interested in portfolio management and risk. Some might argue, though, that despite the processes, policies and other controls that are in place, and based on historical performance, it appears that the Defense Department is willing to pay any price versus managing to a cost or schedule. Still others despair over the daunting challenges the acquisition system faces. For all of the reasons outlined above, this study was undertaken to better understand the performance of the overall acquisition system, including its major processes and important stakeholders. What follows has become an instructive journey through a process of research that did not have as a foregone conclusion any ideas or recommendations, the use of any modeling or simulation approach, or any other kind of analysis framework. The easy answer would have been to look merely at the outcomes of the acquisition system and conclude that the acquisition process is the broken link in the chain, but rather this journey took a deeper and broader look at all of the components of acquisition. This approach led to a series of insights and discoveries culminating in the current form. 24.

(25) of this research that uses discrete event simulation to verify and validate the insights and contributions documented in this work.. Research questions, approach, and methods The questions that guided this research are neither new nor profound. Simply stated, the main question was, "How does the acquisition system work?" A follow-up question was, "Why does the system behave the way that it does?" And finally, "Are there things that can be done to improve the system?" Initially, a great deal of effort was spent reading as much as possible that was written about the system. The sources for this information included official documentation, books, and journal articles or other materials written about the acquisition system. Over time, this research effort was expanded to include the other portions of the acquisition system, namely the requirements portion and the funding portion of the system. After becoming well-versed in literature, several small studies were undertaken to better understand the acquisition system. The first study was done with acquisition professionals, and the second study looked upon those learnings and interviewed players in the other two systems. Building upon all these efforts, a model was developed to capture the things that were learned as well as to frame the problem in a way that could be studied in depth and in a repetitive manner in order to gain insight and understanding about the behavior of the system.. Research Limitations The research presented here is not intended to be the final word, nor the last study ever conducted about the overall acquisition system. The sheer size and complexity of the system required several assumptions to be made, which will be delineated in later chapters, in order to keep the problem tractable. Furthermore, even though a number of people were interviewed, and a great deal of. 25.

(26) effort was put into the verification and validation of the information received and recorded, these people still represent a small sample of the overall workforce in the Department of Defense. These people undoubtedly carry their own biases and understandings of the system. While a great deal of effort was made to ensure that the responses and their understanding of the system was reasonable, undoubtedly there are a multitude of differing opinions throughout the department. Therefore, there is a possibility that certain things were omitted or misrepresented. Other items may have received disproportionate weight or importance in this discussion. However, it is hoped that the results of this work will provide a broad foundation for future research, and even greater insights into the operation and behavior of the Department of Defense's acquisition system.. Dissertation Outline The following is a brief description of the outline of this dissertation. Chapter 2 contains a review of the literature. Following some initial definitions, discussions about product development processes will take place. Much of the space is devoted to the topics of risk and portfolio management in a product development context, followed by an overview of the extended acquisition system, sometimes called the enterprise of acquisition, including a discussion of the three major sub processes of acquisition management, requirements, and the financial process. Finally, there will be a short discussion about using simulation for modeling and analysis and key conclusions synthesized from all of the literature. Chapter 3 describes the results of the first in-depth study of acquisition done as part of this work. It investigates the use of portfolios and risk in system development and examines the acquisition system in more depth. The examination reviews many of the insights into the system-level process gained by interviewing key players within the acquisition system. Chapter 4 presents the analysis of another study of acquisition, but focusing solely on the requirements and financial processes involved. Together, these two studies help lay the foundation for the modeling of the research that this dissertation describes. Chapter 5 describes the development of the model of the extended acquisition. 26.

(27) system, embodying all of the insights and other things learned in the earlier stages of this research. The basic structure, approach and rationale of the modeling choices will be given in this chapter. Chapter 6 covers the steps taken to verify and validate the model. Chapter 7 explains the operation of the model. It introduces the initial setup and operation of the model, as well as providing a glimpse of the typical output from the model and a representative set of outcomes. A secondary analysis using Design Structure Matrices is presented, showing the insights gained from using this tool and perspective. Chapter 8 introduces the specific hypothesis, key questions and interventions that were implemented by simulating the model under specific conditions. The analysis and interpretation of the interventions and their results comprise the bulk of this chapter. Finally, chapter 9 concludes by outlining the several conclusions that can be drawn from this work with an overall summary of the dissertation. Included in this chapter are recommendations for future work as well as policy recommendations that will positively impact the enterprise of acquisition. Several appendices exist to give better understanding of the model. Appendix A lists a representative sampling of questions used in the initial interviews of the different acquisition subsystems. Appendix B contains a thorough step-by-step explanation of the model details. Appendix C contains a copy of the model source code in the SIMAN simulation language. Appendix D contains an overview of other studies about cost and schedule performance of the acquisition system. Major contributions of this work include the introduction of a qualitative and quantitative approach to studying large complex systems using discrete-event simulation, and, showing that Acquisition System outcomes are influenced by emergent behaviors of the system. The emergent behaviors of the system are those unexpected consequences, system attributes and influences stemming from process design, interactions, and execution of the component processes of the larger system which were neither designed, neither intended nor anticipated.. 27.

(28) CHAPTER 2 – LITERATURE REVIEW This chapter contains the background and the rationale for studying this problem through a close examination of the literature. Since the process of developing large complex systems for the defense establishment is very complicated, various areas of literature will be examined in order to thoroughly evaluate the domain space of the overall process, broken down into four key areas. First, generic product development processes will be reviewed, followed by a more focused discussion of risk. Combined, these two topics lead into a discussion about portfolio management, followed by reviewing the literature about the enterprise of acquisition, consisting of JCIDS, PPBE, and the traditional acquisition system (comprised of government personnel and contractors). Next, a short examination of the relevant literature using modeling and simulation for these kinds of activities will be discussed. Finally, these will all be wrapped up into key conclusions, which set the stage for a thorough understanding of the key processes and issues at work within the acquisition system. A few definitions are in order. First, the United States Air Force processes used in the development of large complex systems will be considered as a surrogate for all the other branches of service. The terms “acquisition, acquisition system, acquisition program” all refer to their application under the auspices of the United States Air Force. Second, the terms “project and project management” are often interchangeably used with the terms “program and program management” in the US Air Force. There are some differences between usages of the terms because a project typically refers to a smaller development effort of a larger program. A program, then, might be the F-16 or the C-17 or another large defense system. A project, on the other hand, might be a sensor that is going to be part of the F-16 or the C-17 or a satellite space system. However, in terms of high-level discussions, the meaning is interchangeable although the word “program” is the preferred vernacular. The literature also, albeit somewhat sloppily, regards and treats both of these terms nearly the same, i.e. projects and programs.. 28.

(29) Product development processes Since the overall purpose of the Defense Department's acquisition system is the development of a solution to a defined material need, it is only natural to first look at product development processes in general. There are many different approaches that one can take in developing a new product. One of the most common forms is that of a stage gate process, where new products are developed over time and slowly make their way through a defined product development process [8]. This process consists of several distinct phases where short-term goals are realized. These phases are called stages. In order to proceed to the next stage, a gate or milestone review must be successfully accomplished. A gate is an opportunity for leadership to review the progress of the development project and determine whether or not it will proceed. During this incubation period, if you will, certain projects are expected to be killed, while others that show promise will be carried forward gaining more and more definition and fidelity until they are delivered [8, 9]. The U.S. Air Force has adopted this approach and manages with a somewhat similarly structured phase gate process [10].. Figure 1: An example of a notional product development process. [8]. In the product development literature, a recent trend has been to focus on some of the underlying mechanics required for product development. More specifically, focus has been on the decisions that are required throughout the lifecycle of the process to bring a product to fruition. By focusing on decisions, this literature tends to be broad, borrowing ideas and building upon them from. 29.

(30) many different academic fields such as engineering, marketing, finance, or operations. One of the more seminal papers in this field, “Product Development Decisions: A Review of the Literature,” written by Krishnan and Ulrich [11], reviews a large number of previously published material that discusses such things as projects, program management, risk, portfolio management, and other areas that are essential for developing new products. Similarly, Kahneman, Tversky and Lovollo [12-16] have made significant contributions to product development by studying the psychology of managerial decision-making. In these papers, a recurring theme is learning to manage the risk and uncertainty that may exist when leaders are presented with a decision about a product in development. Furthermore, decisions in these realms tend to be marked with over optimistic projections and managerial biases that can cloud a decision’s real outcome.. Risk The literature reveals some theoretical work linking risk to product development projects. However, a sampling of the literature shows the definition and meaning of risk in this field is often muddled. Among the general meanings of risk, there are competing definitions depending upon the perspective of the various disciplines [3, 17-22]. However, the common elements of these definitions revolve around probabilistic inputs tending to uncertain outcomes. In product development literature, several kinds of specific risk are enumerated, such as: schedule, performance, development cost, technology, market, and business risk [23]. McManus and Hastings [24] add categories of risk such as: disaster, failure, degradation, market shifts, need shifts, extra capacity, and emergent capabilities. Miller and Lessard [25] enumerate additional kinds of risk, particular to megaprojects, and equally applicable to DOD Acquisition efforts. These are: program stability risk; economic environment risk; and optimism risk. There are even the process-oriented categories of risks of operational, design, manufacturing, and performance according to Chase [26]. Finally, let’s not forget interdependencies which can comprise a distinct category of risk [19]. Lessard. 30.

(31) and Miller [27] further caution that "risks are multidimensional and thus need to be unbundled for clear understanding of causes, outcomes, and drivers.” Keizer, et al [28], recently addressed risks in new product development (NPD) using a multidimensional approach. They sought to demystify the various kinds of NPD risks along the lines of technological, business, and organizational risks. They developed a taxonomy of nearly 142 “risks” clustered into twelve main risk areas. These risks contain three variables of interest: likelihood, impact, and ability of the product development team to influence the risk within their constraints. These twelve categories are: organization and project management risks; commercial viability risks; consumer acceptance and marketing risks; product family and brand positioning risks; manufacturing technology risks; product technology risks; supply chain and sourcing risks; trade customer risks; competitor risks; public acceptance risks; intellectual property risks; and screening and appraisal risks [28]. Keizer’s enumeration of risks corresponds nicely with Williams’ earlier bibliography of research relating to project risk management [29]. Among the risks in product development identified were: time risk, cost risk, performance risk, and the contractual aspects of risk [30]. Notably, Williams [29] also acknowledges the hand of multiple disciplines (Management Sciences, Operations Research, Engineering, and Psychology/Decision analysis) in shaping the concepts of risk important to projects. He further proposes adding another dimension, predictability, to the traditional understanding of risk, impact vs. probability, in order to distinguish between the outcomes of an intrinsically uncertain situation, aleatoric probability, and outcomes relating to a measure in belief of a proposition, epistemic probability. This observation opens the door to understanding risk from a psychological perspective. Kahneman and others [12-14, 16, 31] have identified the notion of “framing” as a way for us to take mental shortcuts in dealing with complex and risky issues which lead decision makers to discount extreme events because the probability is too low to evaluate intuitively.. 31.

(32) In essence, there are nearly as many kinds of risks as there are ways to describe risk, and care must be taken how the word “risk” is defined. There is general agreement in the literature about the kinds of risks common to PD. Effectiveness of risk mitigation activities, however, is difficult to demonstrate because it depends on un-provable counterfactuals [32]. Managing, measuring and mitigating risk is essential to PD, but no clear consensus has yet emerged regarding how to do that. Miller and Lessard [25] nevertheless suggest project outcomes are the most appropriate means to measure risk. Given an understanding of the risks facing PD, several frameworks exist that suggest ways to manage risk for the product development practitioner. Most of them follow a pattern of risk identification, risk analysis, and risk disposition to describe risk management. Examples of these include references by Frame [19], the Risk Management Guide for the DOD [33], and even an entry in Wikipedia [34]. There are also many other frameworks that focus on a particular portion of these generic risk management frameworks and advocate using various tools and processes for that specific area within risk management. Bresnahan [35] and Hastings & McManus [24] for example, each have differing frameworks for approaching risks depending on the task at hand or the phase (initial concept, prototype, final design) of a project in the product development cycle. Frame [19] elaborates on this by saying “the risks a product encounters vary dramatically over its life”. For example, risks encountered in the investment phase are quite different in content and impact from those encountered in the maturity phase. Oehmen [36] made an important observation about risk management and the larger product development enterprise. He extended the common risk management frameworks beyond their traditional boundaries by adding two framework elements that are ignored or otherwise assumed by most other frameworks: the monitoring of risks and the integration of risks. The “integration of risk” element implies methods by which management pulls together the “big picture” regarding overall risk.. 32.

(33) This element should capture the cause and effect network effects among and between multiple projects. “Monitoring of risks” is the framework element describing how management is informed of specific project risks. He documents and describes over fifty-seven different risk management methods and where they are most applicable to be used (for example, FMECA). However, only one method out of the fifty-seven is associated with the integration element. This method is called “scenarios” and is mentioned briefly elsewhere by Miller and Lessard [25]. No explicit method is identified with the monitoring element. Furthermore, he postulates aggregation as a method to use at the enterprise level to manage risk. Shapira [37] agrees with his assertion, but both are devoid of specifics. Given the above, Oheman’s framework seems to imply a link to portfolios of projects and their management, but no further elaboration is given.. Portfolio Management strategies A portfolio, in its most simple definition, is simply a collection of items brought together with a common characteristic. From a product development perspective, portfolios refer to product development projects or programs that have something in common. The common characteristic can be organizationally based (a common reporting chain), resource-based (draw upon the same monies), personality dependent (the same manager), or any other combination. Several authors have suggested managing product portfolios as a way to improve the overall outcomes of product development in terms of the bottom line to a company or meeting the emergent market needs, etc. [38, 39]. However, bringing the concepts of risk and portfolios together may be more difficult than it seems. Managing product portfolios through a conceptual risk measure common across the products in the portfolio is seen as very desirable; however, it is not easily done. Shapira [37] noted that among most executives surveyed, aggregation of risk is very rarely done and although desirable, is usually considered too hard to do. A recent RAND study agreed with both sentiments [40]. However, Aloysius [41], in discussing R&D projects, suggests that firms can consider projects collectively and that risk aggregation helps in. 33.

No results found