Risk Identification Experiment

To confirm whether the MLD is able to help undergraduate students identify failure modes, an experiment was carried out at MIT. Thirteen mechanical, operations research, and aerospace

subsystem to help guide them on the level of detail desired. The students were allowed to use any resources except the MLD, and they were asked to return the experiment within a few days. To see how well the students could identify risks, the results from the students were compared with the MLD section shown in Figure 11, which has 56 failure modes identified for the end state of “No Power.” When comparing the student work and the MLD, the student was counted as identifying a failure mode correctly when their lowest level of failure mode in a chain also occurred for the same reason in the MLD, even if their level of detail was not the same as the MLD. For example, if the student specified that an interface could fail, but in reality you have three different interfaces (mechanical, data, and electrical), the student was given one point. If they correctly identified all three interfaces, then they got three points. An incorrect identification of a failure mode resulted in a point in the “Incorrect” column. This occurred when the student flowed events incorrectly, gave the wrong reason for failure, or included parts not used on satellites (e.g. turbine blades). If a student identified the same risk more than once, that was ok, but if they only included the risk twice because the organization was poor, they did not receive points for the second time they identified the same risk. Table 8 shows the results when the students performed this risk identification task.

Table 8. Student Experience & Number of Failure Modes Identified for “No Power”

Student Schooling Student Experience #

Correct

# Incorrect

1. Undergrad Freshman Undecided 8 1

2. Undergrad Freshman Undecided 18 1

3. Undergrad Freshman Aerospace Eng major 14 1

4. Undergrad Freshman Aerospace Eng major 19 0

5. Undergrad Junior Mech Eng major, 1 summer on satellite project 14 0 6. Undergrad Senior Mech Eng major, 1 summer on satellite project 7 0

7. Bachelor’s Degree BS in Operations Engineering 10 0

8. Master’s Student BS in Mechanical Eng, MS in Aerospace Engineering (in progress)

9 0

9. Master’s Student BS in Aerospace Engineering, MS in Aerospace Engineering (in progress)

6 2

10. Master’s Student BS in Aerospace Engineering, MS in Aerospace Engineering (in progress)

17 2

11. Master’s Student BS in Mechanical Eng, MS in Aerospace Engineering (in progress), 1 year on satellite project

23 0

It should be noted first that neither age nor experience seem to help identify failure modes. This is counterintuitive, but it is probably due to the small sample size of students that participated in the study. This result is not as important, though, as how the students responded and what their strengths and weaknesses were. Some of the students showed good insight into technical problems and the design of a satellite, but this was not true for the majority. In addition to the fact that students could not identify many of the failure modes, they also fell into a number of pitfalls.

While Student 1 focused solely on the solar panels not working, he was able to identify other parts of the subsystem as being a cause of the failure, including regulation and distribution failures. He was also able to see that the problem might be recursive. Students 1 and 2 understood that the power subsystem connects to other subsystems, but they did not know what those subsystems were. These two students showed a characteristic of beginning to understand the complexity of the system, but they could not yet see the big picture.

The third student not only identified major components of the power subsystem, but he was also able to identify other aspects of the power management system as well. His knowledge of the entire satellite was lacking, so his ability to identify failure modes in the power subsystem could have been from previous experience and not from general satellite design knowledge. This student demonstrated another common student problem – organization. The students were given an example of an MLD and told how to go through the process, but some students had a lot of repetition and were not well organized.

Student 5 was an undergraduate that had worked for a summer on the power subsystem of a satellite project. He identified most of the general failure methods but did not go into enough detail (even though the level of detail was shown in the instructions). It was expected that this student would have been able to identify many of the failure modes because of his previous experience with the power system, but it was surprising that he did not. This could have potentially been either because he did not have enough time or enough knowledge.

Student 6 did not include a number of major components, including the solar panels, and he only identified a few of the smaller components (such as converters, but not regulators). This

experience and was unable to identify the breadth of failure modes. Finally, the list of failure modes had very little organization.

Student 7 did not have much experience with space systems, although he does follow its major news, and he admitted to knowing little about the internal workings of the power subsystem. However, this person is familiar with cars and performed his failure mode identification using cars as a reference, which helped to at least identify some of the failure modes. The major problem in this case was that he did not of know the components that comprise the power subsystem nor of the resources in which to find this information. He did, though, logically step through many of the potential power options and some of the subsystems they were connected to.

Student 8 was the first student to not identify any of the interfaces to other subsystems. Most students do see some connections and include those, but it is also likely that some students are not able to identify connections to other parts of the satellite at all. Student 9 has extensive experience in the airline industry, and his failure mode identifications were highly skewed toward that knowledge. In this case, he did not seek further information on satellites and created his failure mode tree with incorrect knowledge, resulting in a nearly useless list.

Student 11 was able to identify many of the failure modes and had good grasp on how the power subsystem can fail due to other subsystems. However, he was missing a number of the components of the power subsystem. His diagram did have decent organization and a sensible hierarchy. Jumping ahead to Student 13, he was unable to identify many of the components of satellite’s power system, but this might been because of his background or perhaps a hurried attempt to complete the form, even though the students were given a few days to fill out their fault tree.

Most likely due to his education and experience with satellite projects, Student 12 was able to identify many more of the failure modes of the satellite. There were no major errors, omissions, or biases in the identification of power failures. In a couple of places, the student missed a few risks or did not go into enough detail, and that is why he wasn’t able to get the full set of risks.

While this experiment had a small pool of subjects, it still shows a number of interesting trends for when students try to identify failure modes. First, students need a structure to help

with inconsistent organization led to a hodgepodge identification of failure modes. Second, if the students did have experience in a field outside aerospace engineering, it biased their results when asked to identify failure modes for a satellite’s power system. Sometimes this knowledge helped them to identify any failure modes at all, but other times it hindered their thinking about satellite missions.

Third, students could not fully identify interfaces to other parts of the satellite. Oftentimes, they understood the mission’s complexity, but they were not able to list out all of those relations. This is a major problem because many failures come from other subsystems, so those interfaces must be identified. Lastly, it’s not surprising that students couldn’t identify all of the components of a power system. Many of the students hadn’t studied this subsystem, and some of the students only had in classes. However, in all cases, the students could have used any resources for identifying failure modes, but they chose not to or did not know where to find the information.

While students in satellite projects will have more time and potentially more subject-area knowledge compared with the subjects in this study, this experiment still shows that an MLD can be a useful tool in identifying failure modes. While the Mars Gravity Biosatellite has not yet flown, these two case studies prove that a general yet adaptable tool for failure mode identification, such as a master logic diagram, is needed for student-run satellite programs.