PRICE TRUEPLANNING PROBABILISTIC ESTIMATES

Similar to SEER, PRICE too has the ability to perform uncertainty analysis. For every input in PRICE the user has the option to input an optimistic, most likely, and pessimistic value. Unlike SEER, in PRICE the user selects what inputs to apply uncertainty to, and what the inputs should be. Whereas SEER has default least/likely/most assumptions for nearly every input which can be adjusted by the user as needed. Within PRICE, the user must select what inputs to apply uncertainty to. In this study, the cost estimators applied uncertainty to weight and new design. PRICE then quantifies uncertainty in a similar way to SEER, except it uses a triangular distribution instead of a beta distribution. Details of their methods can be found in "FRISK-Formal Risk Assessment of System Cost Estimates"[54].

PRICE outputs the values of the CDF from the 5% to 95% confidence levels. It is therefore possible to generate a 90% confidence interval, however, for consistency with SEER, the 80% confidence interval is presented instead. The 80% confidence intervals generated from PRICE for the twelve missions in this study can be seen in Figure 3. It is immediately clear that none of the PRICE estimates fall within their confidence intervals. With an 80%

confidence interval, it would be expected that nine or ten of the twelve missions would fall within the bounds of the estimate.

Another significant difference between PRICE and SEER is the confidence level at which the point estimate falls. In SEER, the point estimate is the median value of the distribution, meaning the uncertainty inputs drive the point estimate. However, in PRICE the point estimate is generated using only the most likely input. Since the input distributions

Figure 3. PRICE Uncertainty Analysis

are typically right skewed the end result is that the confidence level of the point estimate is substantially lower in PRICE compared to SEER. The distributions produced by PRICE are so right skewed that the majority of the point estimates fall below the 10% confidence level and outside of the 80% confidence interval.

3. DISCUSSION

The estimates developed using the Space Missions Catalog within PRICE True-Planning in this study were relatively inaccurate. The most likely explanation for this is related to the dependence of the tool’s CERs on mass, particularly for electronic compo-nents. The three subsystems where PRICE most severely overestimated costs were those with the highest number of electronic components: C&DH, Power, and Communications.

The overestimation of these subsystems then effects the estimates of the systems level cost.

PRICE’s estimate of PM, SE, and S&MA (project and payload/spacecraft level) and IAT

costs are strongly correlated with the subsystem level hardware costs. Further, many of the missions had large electronic boxes without sufficient information to break down the mass into individual boards. In SEER, where electronics cost depend on other factors such as clock speed, number of integrated circuits per circuit board, and number of pins per circuit board, the information was available on specification sheets. However, this information did not include mass and the estimators had to make educated guesses on not only the breakdown of mass between the individual circuit boards, but also the breakdown between structures and electronics mass (a significant cost driver). The reliance on mass is fur-ther exacerbated in the uncertainty analysis, where, as discussed above, the assumed input distribution is heavily right-skewed, potentially more than necessary for this point in the project’s lifecycle. Another, and likely related, potential source of error was leveraging the SEER Space Guidance document for input assumptions when data in CADRe was lacking.

This may not be appropriate in certain areas, particularly in the heritage assumptions of the spacecraft bus. A similar guidance document for PRICE would be a useful addition to the PRICE documentation. Lastly, there is simply more work required to determine, with certainty, the sources of error in the PRICE estimates, particularly considering the Space Missions Catalog’s leveraging of CADRe data (or related data) to develop the CERs.

4. CONCLUSIONS

The present study determined that Price had an average error of 52%, median error of 50%, and standard deviation of 45%. Weighing the errors by the actual cost for each the mission, the weighted average error is 43%. PRICE’s uncertainty quantification tool performed poorly, which may be due to the fact that uncertainty could be applied to far fewer input parameters than in SEER. None of the twelve missions fell within PRICE’s 80%

confidence interval.

(a) (b) (c)

(d) (e) (f)

(g) (h) (i)

(j) (k) (l)

Figure 1. Payload vs. Aerocapture HIAD Ballistic Coefficient Trades.

(a) (b) (c)

(d) (e) (f)

(g) (h) (i)

(j) (k) (l)

Figure 2. Payload vs. EDL HIAD Ballistic Coefficient Trades.

(a) (b) (c)

(d) (e) (f)

(g) (h) (i)

(j) (k) (l)

Figure 3. Payload vs. Specific Impulse Trades.

(a) (b) (c)

(d) (e) (f)

(g) (h) (i)

(j) (k) (l)

Figure 4. Payload vs. Thrust-to-Weight Ratio Trades.

Because of the large amount of time it takes for SEER-H to run a Monte Carlo analysis a convergence study was run to determine an appropriate number of Monte Carlo samples to use. SEER does Monte Carlo simulations in two modes, fully correlated and fully uncorrelated. In the fully correlated mode, each random sample is used to set the probability level of the inputs for all work elements of the model. In the fully uncorrelated mode, a different random sample is used to set the probability level of the inputs to each work element individually. Figure 1 shows the results of the convergence study. The study utilized

(a) (b)

(c) (d)

Figure 1. SEER-H Monte Carlo convergence study.

a SEER model of a spacecraft and varied the number of Monte Carlo iterations. SEER allows the user to select between 100 and 10,000 iterations for a Monte Carlo simulation, so the x-axis of the plots runs from 0 to 10,000 and shows the results in steps of 1000 (with an initial step of 900 from 100 to 1000 iterations). The y-axis of the plots shows

the normalized cost or standard deviation of the CDF of the cost distribution where the normalization constant is the value achieved at 10,000 iterations. To achieve errors of less than 1% in both mean cost and standard deviation it is recommended that the user select a minimum of 2,000 iterations.

Before the author of this dissertation was aware that SAPE existed he attempted to make a similar program that would integrate POST II with EXAMINE and some data on required HIAD TPS thickness. However, this work stopped when the author realized that SAPE was already doing exactly what he was trying to accomplish only in more detail.

SAPE had also already been peer reviewed, used in a number of NASA studies, and it was already shown to be accurate. Recreating SAPE would have been time consuming and not accomplished anything of significant value. Instead the author focused on developing SAPE-C an extension to SAPE. This Appendix details some of the work that was accomplished before the author learned of SAPE and started working on SAPE-C.

A Python script was created that would automatically edit POST II input files. A POST II input file for a baseline HIAD entry vehicle landing 20 t of payload on the surface of Mars was obtained. The python script would then automatically edit vehicle and simulation properties such as ballistic coefficient/HIAD diameter, main engine specific impulse, main engine thrust-to-weight ratio, and the targeted maximum gs experienced during entry. For each combination of vehicle/entry parameters, the script would write a POST II file, call POST II, run the file, and save the output. A separate MATLAB script was then used to assemble the output and plot it.

In its final form the python script was only editing vehicle properties in POST II but had not yet integrated EXAMINE or a similar tool to size the vehicle fuel tanks, structures, etc. Thus, the information shown in the plots of this appendix does not necessarily represent the performance of actual vehicles, but the trends are useful for understanding the design tradespace of human Mars entry vehicles.

Figure 1 shows the effect that the specific impulse of the main propulsion systems of a HIAD entry vehicle has on the total fuel burned during entry. It also shows how changing the HIAD diameter (and therefore the ballistic coefficient) effects required fuel burn. As expected the higher specific impulse propulsion systems require less propellant to be burned. This plot does not take into account the effect that a reduced fuel tank size

Figure 1. The effect of main engine specific impulse on fuel burn for entry vehicles with HIADs ranging from 12 to 25 m.

would have on the vehicle. As a result, the actual propellant mass savings would be slightly greater than shown. Figure 1 (along with Figures. 2 and 3) also show how the size of the HIAD effects the total propellant burned. Initially increasing HIAD decreases the total fuel burn significantly. However, the larger the diameter becomes the less of an effect it has on decreasing propellant consumption. These plots do not take into account the mass of the HIAD which would be increasing as the HIAD diameter continues to increase. Thus, in reality there would be a point where the propellant burned would start to increase as the HIAD became larger. This is observed to be the case in the SAPE-C output presented in Figure 4.4d.

Figure 2 shows the effect that altering the thrust-to-weight ratio has on the total amount of propellant burned. Clearly increasing the thrust-to-weight ratio of the main engines significantly reduces the total propellant necessary. For most cases this amounts to several hundred kilograms of mass saved. This however, does not take into account the increased mass of the descent engines. In addition, there are operational reasons to avoid using high thrust engines. For example, in the final phase of flight the lander which is nearly

Figure 2. The effect of thrust-to-weight ratio on fuel burn for entry vehicles with HIADs ranging from 12 to 25 m.

empty of fuel has to descend at a constant rate to safely land on the surface of Mars. Since the Martian gravity is about a third of Earth’s gravity a spacecraft with a thrust to weight ratio of two would have to throttle its engines to one sixth their maximum thrust in order to descend at a constant rate. Deep throttling like this is technically challenging and may significantly increase the cost of engine development.

Figure 3 shows the effect that a targeted maximum deceleration will have on propel-lant burned. The maximum deceleration is targeted by altering the initial flight path angle of the entry vehicle. A steeper flight path angle will generally result in a higher maximum deceleration. Figure 3 shows that the maximum deceleration experienced by the entry vehicle has a very small effect on the total propellant burned especially with higher HIAD diameters. Note that discontinuities in Figure 3 are likely due to POST II not converging or generating an otherwise erroneous result. Since this work was in the preliminary stages and was soon abandoned, no significant effort was put into figuring out why the discontinuities occurred.

Figure 3. The effect of targeted maximum deceleration on fuel burn for entry vehicles with HIADs ranging from 12 to 25 m.

Figures 4, and 5 show the effect that the maximum deceleration has on the maximum heat rate and total heat load experienced. Figure 4 shows that the maximum heat rate is decreased by utilizing a larger HIAD diameter and by using a shallower entry with a lower

Figure 4. The effect of targeted maximum deceleration on maximum heat rate for entry vehicles with HIADs ranging from 12 to 25 m.

Figure 5. The effect of targeted maximum deceleration on total heat load for entry vehicles with HIADs ranging from 12 to 25 m.

max g load. Decreasing the maximum heat rate allows for thinner layers of refractory fabric to be used on the outer shell of a HIAD. However, Figure 5 shows that the total heat load is increased when flying trajectories with lower maximum decelerations. A higher total heat load requires more layers of insulating fabric within a HIAD.

REFERENCES

[1] W. Von Braun, Das marsprojekt: studie einer interplanetarischen expedition. Um-schau, 1952.

[2] D. S. Portree, “Humans to mars: Fifty years of mission planning, 1950-2000,” 2001.

[3] D. Rapp, “Human missions to mars: Enabling technologies for exploring the red planet,” 2016.

[4] B. G. Drake, “Human exploration of mars design reference architecture 5.0,” tech.

rep., NASA-SP-2009-566, 2009.

[5] B. G. Drake, “Mars human exploration mission dra 5.0,” tech. rep., NASA-SP-2009-566, July, 2009.

[6] B. G. Drake and D. Watts Kevin, “Human exploration of mars design reference architecture 5.0, addendum# 2,” 2014.

[7] S. Himmel, J. Dugan Jr, R. Luidens, and R. Weber, “A study of manned nuclear-rocket missions to mars,” Aero/Space Eng., vol. 20, no. 7, 1961.

[8] R. Zubrin, The case for Mars. Simon and Schuster, 2012.

[9] D. A. Craig, P. Troutman, and N. Herrmann, “Pioneering space through the evolvable mars campaign,” in AIAA Space 2015 Conference and Exposition, p. 4409, 2015.

[10] K. E. Goodliff, P. Troutman, D. A. Craig, J. Caram, and N. Herrmann, “Evolvable mars campaign 2016-a campaign perspective,” in Aiaa Space 2016, p. 5456, 2016.

[11] T. A. May and S. D. Creech, “Nasa’s space launch system (sls) program: Mars program utilization,” 2012.

[12] T. Percy, M. McGuire, and T. Polsgrove, “In-space transportation for nasa’s evolvable mars campaign,” in AIAA SPACE 2015 Conference and Exposition, p. 4519, 2015.

[13] K. HACK, J. George, and L. DUDZINSKI, “Nuclear electric propulsion mission performance for fast piloted mars missions,” in Conference on Advanced SEI Tech-nologies, p. 3488, 1991.

[14] T. Percy, M. McGuire, and T. Polsgrove, “Combining solar electric propulsion and chemical propulsion for crewed missions to mars,” in 2015 IEEE Aerospace Confer-ence, pp. 1–10, IEEE, 2015.

[15] J. R. Wertz and R. Bell, “Autonomous rendezvous and docking technologies: status and prospects,” in Space Systems Technology and Operations, vol. 5088, pp. 20–31, International Society for Optics and Photonics, 2003.

[16] M. Simon, K. Latorella, J. Martin, J. Cerro, R. Lepsch, S. Jefferies, K. Goodliff, D. Smitherman, C. McCleskey, and C. Stromgren, “Nasa’s advanced exploration systems mars transit habitat refinement point of departure design,” in 2017 IEEE Aerospace Conference, pp. 1–34, IEEE, 2017.

[17] J. Chancellor, G. Scott, and J. Sutton, “Space radiation: the number one risk to astronaut health beyond low earth orbit,” Life, vol. 4, no. 3, pp. 491–510, 2014.

[18] J. Wilson, M. Kim, W. Schimmerling, F. Badavi, S. Thibeault, F. Cucinotta, J. Shinn, and R. Kiefer, “Issues in space radiation protection: galactic cosmic rays,” 1995.

[19] A. D. Cianciolo and T. T. Polsgrove, “Human mars entry, descent, and landing architecture study overview,” 2016.

[20] A. S. Howe, M. Simon, D. Smitherman, R. Howard, L. Toups, and S. J. Hoffman,

“Mars surface habitability options,” in 2015 IEEE Aerospace Conference, pp. 1–9, IEEE, 2015.

[21] L. Mason, D. Poston, and L. Qualls, “System concepts for affordable fission surface power,” 2008.

[22] M. A. Rucker, S. Oleson, P. George, G. A. Landis, J. Fincannon, A. Bogner, R. E.

Jones, E. Turnbull, J. McNatt, M. C. Martini, et al., “Solar versus fission surface power for mars,” 2016.

[23] C. A. Jones, Risk-value optimization of performance and cost for propellant produc-tion on Mars. PhD thesis, Georgia Institute of Technology, 2016.

[24] R. Zubrin, T. Muscatello, B. Birnbaum, K. Caviezel, G. Snyder, and M. Berggren,

“Progress in mars isru technology,” in 40th AIAA Aerospace Sciences Meeting &

Exhibit, p. 461, 2002.

[25] A. F. Davila, D. Willson, J. D. Coates, and C. P. McKay, “Perchlorate on mars: a chemical hazard and a resource for humans,” International Journal of Astrobiology, vol. 12, no. 4, pp. 321–325, 2013.

[26] J. Agui and M. Nakagawa, “Dust on the moon and mars,” in 43rd AIAA Aerospace Sciences Meeting and Exhibit, p. 539, 2005.

[27] T. P. Polsgrove, H. D. Thomas, W. Stephens, T. Collins, M. Rucker, M. Gernhardt, M. R. Zwack, and P. D. Dees, “Human mars ascent vehicle configuration and perfor-mance sensitivities,” in 2017 IEEE Aerospace Conference, pp. 1–15, IEEE, 2017.

[28] A. Owens, O. d. Weck, B. Mattfeld, C. Stromgren, and W. Cirillo, “Comparison of spares logistics analysis techniques for long duration human spaceflight,” 45th International Conference on Environmental Systems, 2015.

[29] M. Sargusingh, M. Anderson, J. Perry, R. Gatens, J. Broyan, A. Macatangay, W. Schneider, and N. Toomarian, “Nasa environmental control and life support technology development and maturation for exploration: 2017 to 2018 overview,”

48th International Conference on Environmental Systems, 2018.

[30] E. Antonsen, A. Hanson, R. Shah, R. Reed, and M. Canga, “Conceptual drivers for an exploration medical system,” 2016.

[31] A. Owens, S. Do, A. Kurtz, and O. d. Weck, “Benefits of additive manufacturing for human exploration of mars,” 45th International Conference on Environmental Systems, 2015.

[32] L. J. Hastings, D. Plachta, L. Salerno, and P. Kittel, “An overview of nasa efforts on zero boiloff storage of cryogenic propellants,” Cryogenics, vol. 41, no. 11-12, pp. 833–839, 2001.

[33] B. R. Hollis, K. T. Berger, T. J. Horvath, J. J. Coblish, J. D. Norris, R. P. Lillard, and B. S. Kirk, “Aeroheating testing and predictions for project orion crew exploration vehicle,” Journal of Spacecraft and Rockets, vol. 46, no. 4, pp. 766–780, 2009.

[34] N. Kanas, G. Sandal, J. Boyd, V. Gushin, D. Manzey, R. North, G. Leon, P. Suedfeld, S. Bishop, E. Fiedler, et al., “Psychology and culture during long-duration space missions,” Acta Astronautica, vol. 64, no. 7-8, pp. 659–677, 2009.

[35] N. Kanas, “Psychological, psychiatric, and interpersonal aspects of long-duration space missions,” Journal of spacecraft and rockets, vol. 27, no. 5, pp. 457–463, 1990.

[36] C. Sagan, “Why send humans to mars?,” Issues in science and technology, vol. 7, no. 3, pp. 80–85, 1991.

[37] A. J. Coates, “Limited by cost: The case against humans in the scientific exploration of space,” Earth, Moon, and Planets, vol. 87, no. 3, pp. 213–219, 1999.

[38] F. A. Prince, “Why nasa’s management doesn’t believe the cost estimate,” Engineer-ing Management Journal, vol. 14, no. 1, pp. 7–12, 2002.

[39] R. Prakash, P. D. Burkhart, A. Chen, K. A. Comeaux, C. S. Guernsey, D. M. Kipp, L. V. Lorenzoni, G. F. Mendeck, R. W. Powell, T. P. Rivellini, et al., “Mars science laboratory entry, descent, and landing system overview,” in 2008 IEEE Aerospace Conference, pp. 1–18, IEEE, 2008.

[40] D. Cadogan, J. Stein, and M. Grahne, “Inflatable composite habitat structures for lunar and mars exploration,” Acta Astronautica, vol. 44, no. 7-12, pp. 399–406, 1999.

[41] A. M. DwyerCianciolo, J. L. Davis, D. R. Komar, M. M. Munk, J. A. Samareh, R. W.

Powell, J. D. Shidner, D. O. Stanley, A. W. Wilhite, D. J. Kinney, et al., “Entry, descent and landing systems analysis study: Phase 1 report,” 2010.

[42] T. Zang, A. Dwyer Cianciolo, D. Kinney, A. Howard, G. Chen, M. Ivanov, R. Sostaric, and C. Westhelle, “Overview of the nasa entry, descent and landing systems analysis stud,” in AIAA SPACE 2010 Conference & Exposition, p. 8649, 2010.

[43] T. Polsgrove, J. Chapman, S. Sutherlin, B. Taylor, E. Robertson, B. Studak, S. Vi-talpur, L. Fabisinski, A. Y. Lee, T. Collins, et al., “Human mars lander design for nasa’s evolvable mars campaign,” in 2016 IEEE Aerospace Conference, pp. 1–15, IEEE, 2016.

[44] R. R. Sostaric, C. J. Cerimele, E. A. Robertson, and J. A. Garcia, “A rigid mid lift-to-drag ratio approach to human mars entry, descent, and landing,” in AIAA Guidance, Navigation, and Control Conference, p. 1898, 2017.

[45] H. W. Price, R. D. Braun, R. Manning, and E. Sklyanski, “A high-heritage blunt-body entry, descent, and landing concept for human mars exploration,” in 54th AIAA Aerospace Sciences Meeting, p. 0219, 2016.

[46] E. Venkatapathy, K. Hamm, I. Fernandez, J. Arnold, D. Kinney, B. Laub, A. Makino, M. McGuire, K. Peterson, D. Prabhu, et al., “Adaptive deployable entry and placement technology (adept): a feasibility study for human missions to mars,” in 21st AIAA Aerodynamic Decelerator Systems Technology Conference and Seminar, p. 2608, 2011.

[47] A. M. Cassell, C. A. Brivkalns, J. V. Bowles, J. A. Garcia, D. J. Kinney, P. F.

Wercinski, A. D. Cianciolo, and T. T. Polsgrove, “Human mars mission design study utilizing the adaptive deployable entry and placement technology,” in 2017 IEEE Aerospace Conference, pp. 1–16, IEEE, 2017.

[48] J. R. Wertz, “Preface,” Journal of Reducing Space Mission Cost, vol. 1, no. 1, p. 1, 1998.

[49] J. R. Wertz, “Aims and scope,” Journal of Reducing Space Mission Cost, vol. 1, no. 1, p. 3, 1998.

[50] J. R. Wertz, “The end of j. rsmc,” Journal of Reducing Space Mission Cost, vol. 1, no. 4, pp. 275–276, 1998.

[51] NASA, “Cost estimating handbook,” Washington, DC, 2008.

[52] J. R. Wertz and W. J. Larson, Reducing space mission cost. Microcosm Press Torrance, CA, 1996.

[53] O. Trivailo, M. Sippel, and Y. A. Şekercioğlu, “Review of hardware cost estimation methods, models and tools applied to early phases of space mission planning,”

Progress in Aerospace Sciences, vol. 53, pp. 1–17, 2012.

[54] P. Young, “Frisk-formal risk assessment of system cost estimates,” in Aerospace Design Conference, p. 1054, 1992.

[55] S. D. Winn and J. W. Hamcher, “Nasa/air force cost model: Nafcom,” 2002.

[56] B. Alford and A. Prince, “Nasa project cost estimating capability: New analyses for spacecraft estimating,” 2016.

[57] PRICE Systems, “A Framework for Achieving a Parametric Basis of Estimate,” 2019.

[58] Galorath Inc., “Product Brief: SEER for Hardware, Electronics and Systems,” 2019.

[59] J. R. Wertz, “Editorial: Show us the data,” Journal of Reducing Space Mission Cost, vol. 1, no. 1, pp. 5–7, 1998.

[60] H. W. Jones, “Underestimation of project costs,” 2015.

[61] S. Keller, P. Collopy, and P. Componation, “What is wrong with space system cost models? a survey and assessment of cost estimating approaches,” Acta Astronautica, vol. 93, pp. 345–351, 2014.

[62] NASA Associate Deputy Administrator for Policy, “Falcon 9 launch vehicle nafcom cost estimates,” Washington, DC, 2011.

[63] S. Sanchez and K. Kha, “Seer validation study results for nasa space science mis-sions,” 2015.

[64] J. Swaren, “Trueplanning®validation studies for estimating unmanned space projects,” 2017.

[65] Galorath Inc., “SEER-H Space Guidance: Revision 2.2,” 2016.

[66] P. D. Friz, J. J. Klovstad, B. B. Leser, B. C. Towle, and S. Hosder, “Blind study validating parametric costing tools price trueplanning and seer-h for nasa science

[66] P. D. Friz, J. J. Klovstad, B. B. Leser, B. C. Towle, and S. Hosder, “Blind study validating parametric costing tools price trueplanning and seer-h for nasa science