2012 ITEA Annual Symposium David G. Smith Sep 2012
The Flight Plan
Why; the Research Problem and questions
Basis and Background; literature review
Clarification; unique terminology
How; the methodology and limitations
What; the findings—what did they say?
Why; implications and future research
Overview of the Research
Question
There is a lack of quantitative evidence either
supporting or refuting the claim that ToC PM improves project cost, schedule, performance, and overall
effectiveness
Theory of Constraints Project Management was brought to the Air Force Flight Test Center at considerable expense, and has never been comparatively studied.
Overview of the Research
Question
Questions were designed to address project:
Cost
Schedule
Performance
Overall Effectiveness
Four research (and associated null) hypotheses were crafted to address each question
Definitions of Uncommon Terms
412th Test Wing
Air Force Flight Test Center
Concerto®
Critical chain project management (CCPM)
Critical path project management (CPPM)
Developmental test and evaluation
Project Management Overview
Projects fail at alarming rates
Costing time, money and resources Failing to deliver the desired end state
Volatile projects have the highest failure rate
Flight test projects are highly volatile
Prior AFFTC project management lacked structure CCPM introduced in 2001
Never quantitatively studied
The Triad-
pick any two?
Overall
Effectiveness
Cost
Performance
Schedule
Critical Path Method (CPM)
Program Evaluation and Review Technique (PERT)
Review of the Literature
Lack of comparative studies noted
No peer-reviewed studies of volatile projects
Only anecdotal evidence of success at the AFFTC
None-the-less, ToC implemented in 2001
Literature strongly supports ToC PM
Reduced cost, ABC automotive-3 more units weekly Increased timely performance-late pharmaceutical
projects reduced by 50%
Delivering project content-777 from late and over
budget to on-time delivery as designed
Improving overall effectiveness-quicker project
ToC PM 5 Step Process:
Identify the system constraint
Exploit the system constraint
Subordinate everything else to the system constraint.
Elevate the system constraint
Lather, rinse, repeat…
Why test military aircraft?
Required by regulation
Enhances safety of flight
Fiscally smart
Testing minimizes risk
ID’s problems early
Provides fix before production
Multiple examples:
Bradley
B-1 Bomber
Description of the Methodology
Retrospective quantitative causal-comparative design
Self-report descriptive research (2 groups)
N=100, n=62 (line managers) N=10, n=10 (senior managers)
Likert-type scale
ANOVA (t-test, Mann-Whitney U-test, modified
Bonferroni correction)
Factor analysis Bayesian analysis
Comparison of actual projects
Bayesian approach
Different approach from frequentist analyses:
Frequentist:
Analysis based on a model: Look at p (getting the data we
observed|null hypothesis value of the parameter)
No prior information
Estimate an unknown constant (parameter)
Hypothesis test, p-value, confidence intervals type I/II errors
Bayesian:
Use hierarchical models
Estimate a random variable, get a density function Incorporate prior information
Clean interpretation: no hypothesis tests, no p-values, no
Limitations of the Research
Lack of randomization, manipulation, and in-equality of groups
Larger sample size?
Ethics and perception?
No comparison of identical projects executed differently
Discussion of Findings
62 line manager responses (Edwards AFB population of 100)
36 experiment (uses ToC)
26 control
10 senior responses (entire population)
5 experiment 5 control
ANOVA (t-test and Mann-Whitney U-test)
Modified Bonferroni correction
Factor analysis
Bayesian (WinBUGS estimated w/ Markov-Chain Monte Carlo processes)
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Q-1 Q-2 Q-3 Q-4 Q-5 Q-6 E F G H I J K Using TOC Not Using TOC
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 Q-1 Q-2 Q-3 Q-4 Q-5 Q-6 Q-7 Q-8 Q-9 Q-10 Using ToC PM Not Using ToC PM
0 5 10 15 20 25 Fa vors ToC Fa vors Non-ToC Tie Critical Chain Duration Critical Path Duration
Days Saved Savings
Project 1 123 162 39 24% Project 2 57 72 15 21% Project 3 63 66 3 10%
Significance
Cost
Line managers support rejection (r(62) = −3.521, p = .0008) Bayesian statistical analysis supported rejection of the null
hypothesis (p < .05)
Senior managers did not support rejection (U = 13.5, p = .841)
Schedule
Line managers not significant (r(62) = 1.71, p = .0919)
Bayesian statistical analysis supported rejection of the null hypothesis (p < .05)
Significance
Performance
Line managers not significant: r(62) = 0.796, p = .429
Bayesian statistical analysis supported rejection of the null hypothesis (p < .05)
Significance
Overall Effectiveness (pooled data)
Line managers significant (r(615) = 2.194, p = .029)
Bayesian statistical analysis supported rejection of the null
hypothesis (p < .05)
Significant for senior managers (1-6: U = 698.5, p < .001; E-K
U = 601, p = .014)
Implications
Literature and this study support ToC PM as improving project cost
Mixed study results addressing schedule; in contrast to the literature which strongly supports ToC PM
Limited study support for performance; again, in contrast to the literature
Literature and this study support ToC PM as improving project overall effectiveness
Power: high value for sensitivity and specificity
Recommendations
Future research
Schedule & performance not consistent with literature Bayesian analysis original contribution to the lierature
Continuous vs likert scale Address age/gender/rank etc
Empirical study with larger group
Continue the use of ToC PM at the 412th Test Wing
Expand ToC PM across the AFFTC
Consider ToC PM for any highly volatile project management scenario
Recommendations, continued
Complete one or more comparative studies:
Bayesian analysis of results, using current study as prior
information
Use Edwards as well as other AF bases Consider inclusion in the PM BOK
