The final three survey questions were taken from Shane Frederick’s Cognitive Reflection Test (CRT; 2005). For example, “A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? (in cents).” Frederick (2005) discusses the use of the CRT to determine how individuals answer questions, namely whether they answer with what initially comes to mind, in this case $0.10, or if they have patience and are able to report the correct answer, in this case $0.05. These two methods of answering the CRT are exemplified by
“System 1” and “System 2” processes. Frederick (2005) and Stanovich and West (2000) discuss how System 1 represents processes which are done almost instantaneously and require little conscious thought, for example recognizing the face of a friend in a classroom. On the other hand, System 2 represents more of a conscious level of thought which requires one to dedicate thinking, effort and concentration. Frederick (2005) goes on to discuss how subjects taking the
-30-
CRT who answer with the predicted intuitive answer are using System 1 as opposed to those who have cognitive inhibition and answer CRT correctly use System 2. These two systems each present similarities to Fuzzy Trace Theory in that System 1 is similar to gist memory and System 2 to verbatim memory. Although not identical, the comparison between System 1 and 2, and gist and verbatim representations, respectively, allows CRT data to be evaluated as a secondary method to determine how subjects make decisions in instances outside of the pre-hospital realm.
The data from the three questions included in the CRT was analyzed individually and summed as a composite variable. Data from the participants who provided the intuitive answer was also analyzed.
Procedure
As stated in the “Participants” section, specific EMS/Fire/Rescue agencies were contacted by the investigator and permission was requested through an officer or leader of the department to distribute the survey to their members or employees. The participants enrolled in the study voluntarily and anonymously. Some departments or agencies may have provided their members with credit for Continuing Medical Education (CMEs), however the hours were not connected with the investigators of the survey. Participants received a secure link to the web-based survey administered through Qualtrics Web Survey Tool via Cornell University. Each subject was asked to read and affirm that they understood the different aspects, risks and any consequences of the study via an official consent form. The consent form also contained contact information for the investigator, Cornell’s Institutional Review Board, and Ethicspoint, an anonymous complaint hotline. Upon completion of the survey, participants received a confirmation message that their survey was complete.
-31- Results
Upon completion of the testing period, the data gathered from each participant’s survey was analyzed in a number of sections and then between each section. The sections were analyzed in the order that the parts of the survey were presented in the methods. The first analysis that was completed for each of the sections included checking the general frequency values for every question to determine if participants showed any significant results when conditions were varied. Following the frequency results, depending on the type of variable, categorical, ordinal, or continuous, analyses of variance (ANOVA), correlations and/or repeated measures analyses were completed. The main results from this study are based on repeated measures ANOVAs found in Tables 1-8. The additional values and data discussed can be found in Appendices B, C, D, E, and F. Scales of variables were also calculated and then placed within calculations against other variables against scales. Additionally, variables from different parts of the survey were then compared. Below the results are discussed in full and in the order that the sections appear in the survey itself, which is found in Appendix A.
Medical Case-Based Scenarios Treatment Questions
In this first part of the survey, participants were asked to make treatment decisions based on a medical case scenario they were given. As discussed in Methods, participants were given nine sets of treatment questions to complete, with the six control scenarios being based straight off of protocol knowledge, while the last three sets provided participants with a conflict scenario.
Overall, participants showed strong knowledge of protocols by selecting the proper answer
-32-
choice a majority of the time. In Table C.1, the frequency that participants chose each answer is shown for each of the six knowledge scenarios. The correct treatment answer for each question is highlighted. Of the six questions, the question with the lowest number of participants
selecting the correct answer was Scenario 1- Treatment 2 with 72.2% of participants selecting the correct answer, while the highest proportion of correct answers was Scenario 5- Patient 1 with 96.8%. The four remaining scenarios had 88.5%, 79.4%, 84.4%, and 88.5% of participants getting the correct answer. These percentages show that providers who completed this survey show strong knowledge for the protocols on a single case by case basis.
Each provider’s score among all six scenarios was also analyzed to determine the number of questions out of the six that they got right. Table C.2 shows the number of questions that participants got correct. The results show that almost half of all participants (46.8%) got all six knowledge treatment questions correct and only 7.5% got three or less questions right (worse than half of the questions correct). The mean of questions out of six correct was 5.09 with a standard deviation of 1.04.
With these results showing such high numbers of correct responses both across all scenarios and for each individual scenario, it is clear that when participants weren’t forced to make decisions between protocol-based answers (verbatim) or answers that indicate desirable deviations from protocols (gist) and they are given only one correct, verbatim answer out of the four possible choices, they are able to discern the correct treatment. This creates an interesting setting to analyze the results of the conflict scenarios.
In general, a large number of the participants selected the gist-based answer in the three conflict scenarios. The frequency for each answer choice selected for the conflict scenarios is shown in Table C.3. In the first conflict scenario (Scenario 5- Patient 2), 62.4% of participants
-33-
selected the gist answer choice while 37.1% of participants selected one of two verbatim choices.
The second conflict scenario followed a similar pattern with 69.3% of providers selecting the gist based answer and 29.4% of the providers selecting the verbatim answer. The final conflict scenario had roughly equivalent breakdowns of gist and verbatim answers chosen with 47.2%
and 42.2%, respectively.
Similar to the six knowledge scenarios, each participant’s answers for all three questions were summed into a gist and a verbatim score. Each participant’s sum was out of three and reflected the number of times the participant selected a gist or verbatim answer during the three scenarios. The breakdown of participants who chose the gist treatment option for the scenarios is shown in Table C.4, with 19.8% choosing the gist answer for each of the three conflict scenarios.
The mean number of gist answers chosen was 1.79 with a standard deviation of 0.81. A
histogram of gist answer sum frequency is shown in Figure G.4. The breakdown of participants who chose a verbatim treatment option is shown in Table C.5, with only 3.7% choosing a
verbatim answer for each of the three conflict scenarios. The mean number of verbatim answers chosen was 1.09 with a standard deviation of 0.82. A histogram of verbatim answer sums frequency is shown in Figure G.5. As shown by Table C.4 and C.5, there was a higher number of participants who showed patterns of selecting multiple gist answers as well as a higher number of gist answers chosen overall. This data reflects the idea that when required to choose between a gist answer which reflects a desirable deviation from protocol and a verbatim answer, a large number of providers show preference towards the gist.
With a solid foundation of the knowledge of participants as well as their general tendency to use gist and verbatim based answers, it was then analyzed what types of EMS demographic information if any plays a factor in how participants selected their answers. Thus, a repeated
-34-
measures analysis of variance (rmANOVA) was completed comparing the average mean of those selecting the six verbatim answers in the control scenarios versus the average mean of those selecting the three verbatim answers in the conflict scenarios; means were used to ensure equal weighting values between the six scenarios and the three scenarios. Gender, highest education level and highest certification level were used as between subject factors after finding significant correlations between the verbatim average in the conflict scenarios and highest education and highest certification level.
In these analyses, highest certification was grouped into three levels due to some of the certifications having low n numbers. For this reason, certifications were grouped based on general standards of EMS in the United States. The first group was Minimal Certification and included those with no certification, a CPR certification or a Certified First Responder
certification. All three certifications were grouped together because even though they are standard certifications recognized in EMS they do not meet BLS or ALS standards. However, once the three levels were grouped, the n-value was still low. With a low n-value and these participants not meeting minimum EMS requirements to be in charge of an ambulance, they were not included in the analysis. The second group was BLS providers which include EMT- Basics, the national standard for BLS, and EMT- Intermediates. Although EMT-Is are
technically a level of care above EMT-Bs, they are not nationally recognized in every state and exist only in certain areas and they do not meet the standards of ALS providers. For this reason, EMT-Is are often viewed as BLS providers and were thus grouped in this manner. The third and final group is composed of ALS providers including EMT- Critical Cares (EMT-CCs),
Paramedics and Critical Care (CC) Paramedics. Although national standards for ALS are based around Paramedics, EMT-CCs and CC Paramedics are also viewed as ALS providers. Highest
-35-
education level was also grouped into three levels including, High School or Associate’s Degree, Bachelor’s or Nursing Degree (RN, PA, or NP), and Graduate or Physician’s Degree (MD, DO, etc.). The test of within- subjects effect’s results of the rmANOVA are found in Table 1 and show significant interactions with Scenarios (Control vs. Conflict; sig = 0.000), Scenario with Highest Certification Level (sig = 0.008), and Scenario and Highest Education Group (sig = 0.032).
Upon closer analysis of the significance between scenario (control and conflict) means, found in Table 2, there is a significant difference between the percent of participants that selected the verbatim answer in the control scenarios (0.859) and the conflict scenarios (0.383). As discussed earlier, these results show similar findings to the analysis of general frequencies with the addition of showing that the difference is significant.
The second group of significant results was found in the interaction of scenario with highest certification level (BLS or ALS) found in Table 3. This interaction shows that overall, BLS and ALS providers showed little difference in selecting the verbatim answer in the control scenarios, however ALS providers showed significantly lower level of selecting the verbatim answer than BLS providers in the conflict scenarios. The mean value of ALS providers in choosing the verbatim answer in the conflict scenarios is 0.320 with a standard error of 0.042, while the mean value for BLS providers in conflict scenarios is 0.446 with a standard error of 0.027. This significant difference shows that BLS and ALS providers show an insignificant difference in general knowledge of the protocols (control scenarios), but when given a scenario that causes them to choose between a statement representing a rote view of the protocol or a desirable deviation from the protocol, ALS providers show decreased uses of verbatim
-36-
processing. This finding supports previous data that more experience and more highly trained providers use less verbatim when making decisions (Reyna & Lloyd, 2006).
Additionally, there was a significant interaction between scenario and highest education level shown in Table 4. This interaction shows that all education groups (High School or Associate’s, Bachelor’s or Nursing, and Graduate or Physician) showed no major differences in mean values for selecting the verbatim answer in the control scenarios. However, Graduate and Physicians had a significantly higher mean (0.484, standard error = 0.053) than those with High School/Associate’s (0.331, standard error = 0.041) or Bachelor’s/Nursing degrees (0.334, standard error = 0.032). This increase in use of verbatim answer choices in those with the highest education level is contrary to standard views of fuzzy-trace theory, which indicates that as education increases verbatim level normally decreases.
With these data showing interesting results that are contrary to normal beliefs of
education (and gist versus verbatim), a second repeated measures ANOVA was completed and shown in Tables 5-8. This second repeated measures (rm) ANOVA followed similar analysis procedures as the first rmANOVA, except that highest education was regrouped into
Nursing/Medicine (RN, NP, PA, MD, DO, etc.) degrees versus all other levels of education.
Despite a small n-value of participants with Nursing and Medicine degrees (n = 12), significant interactions were found between scenarios; scenario and highest education level; and scenario, highest education level and highest certification level.
Similar to the rmANOVA discussed previously (Table 1), there was a significant
difference between mean values of participants who selected the verbatim answer in the control and conflict scenario, with means equaling 0.790 (standard error = 0.032) and 0.399 (standard error = 0.49), respectively. A full set of estimate values can be found in Table 6. These
-37-
differences show that there was a significant drop in the number of participants who selected the verbatim response in the conflict scenarios. Just as in the first rmANOVA, this shows that when participants were put in a position where the verbatim answer choice may not be the best
possible treatment for the patient, many participants moved away from the rote interpretation of the protocol.
Table 7 shows the second significant interaction which was between scenario and highest education level. In this analysis, highest education level was broken into two groups as
discussed above. One group was composed of participants with a medical education outside of EMS including Nursing or Physician degrees and the second group was composed of all other participants without a medical education outside of EMS. In the table, it can be seen that for both education groups use of the verbatim answer decreases as one moves from the control to the conflict scenarios. Those with medical degrees had a significantly lower mean (0.722, standard error = 0.062) for selecting the verbatim (correct answer) in the control scenario when compared with participants with no medical education (0.859, standard error = 0.015). This finding is extremely interesting and contradictory to the idea that those with a more extensive medical education would know the treatment protocols better than those with less extensive medical educations. A possible explanation for these results could lie in the quantity of EMS that each participant does each year, however these values were not collected in this study.
Last, there was a significant interaction between scenario, highest certification and highest education level, which is shown in table 8. In this interaction, all BLS providers and ALS providers with no medical education showed relatively consistent means for selecting the verbatim (correct answer) in the control scenarios. However, ALS providers with a
Nursing/Physician degree showed dramatically lower means of selecting the correct answer in
-38-
control scenarios with a mean of 0.583, standard error = 0.106. Also, all BLS providers and ALS providers with no medical education showed significant decreases in selecting the verbatim answer in the conflict scenarios when compared to control scenarios. This finding, which is consistent with the first rmANOVA and general fuzzy-trace theory, is not found in the ALS provider with a Nursing/Physician degree. Instead, this group showed no significant difference in the mean of those who selected the verbatim response in the control and the conflict scenarios.
These significant differences reflect the idea that ALS providers with additional
Nursing/Physician degrees know the protocols less than all BLS providers, and ALS providers with no additional medical education. These results also contradict general fuzzy-trace theory which would indicate that having additional education with higher levels of certification would lead to greater knowledge of protocols and less use of verbatim responses in conflict scenario.
One possible explanation for this is the small number of pre-hospital providers with additional medical education, who are not reflective of the general population of providers with similar certification and education levels.
After analyzing the repeated measures ANOVAs, the mean number of participants who selected the correct answer for each of the control scenarios along with the gist and verbatim answer for the conflict scenarios was examined. The specific means for each treatment question can be found in Table C.7. There was no statistical significance between grouped certifications for the six knowledge treatment questions. In Scenario 5- patient 2, ALS providers chose the first verbatim answer less than BLS providers with means of 0.18 and 0.28, respectively. ALS providers chose the gist answer more than BLS providers. However, the difference in means was not significant. There was a significant difference between BLS and ALS providers for the gist answer of Scenario 6 with the means being 0.61 and 0.85, respectively (F =13.206, p =
-39-
0.00). The verbatim answer for Scenario 6 also showed significant difference between BLS and ALS with means being 0.37 and 0.14, respectively (F = 13.491, p = 0.00). Scenario 7 showed ALS providers using gist more than BLS providers and verbatim less than BLS providers in trend, however the data was not significantly different. All of these trends, both the significant and insignificant ones, provide evidence of increased gist and decreased verbatim in higher certified providers, who are viewed as having more expertise.
Decision Questions
Following each of the treatment questions in the nine scenarios, participants were asked to determine how they decided that the treatment chosen was the correct one. These nine
questions (one decision question per scenario) allowed participants to meta-cognitively select the rationale for the decision they chose. As discussed in the methods, all nine decision questions had the same five possible answer choices which were composed of two verbatim answers, two gist and one other with the ability to fill in another reason. When analyzing the frequency values of the number of participants, who selected each of the five possible answer choices for each of the nine questions, there was a clear change in verbatim and gist reasoning between the control medical case scenarios and the conflict scenarios. As seen in Tables C.34-35, the frequency of the choice, “Based on specific memory for National/State/Local Protocols,” a verbatim reasoning, decreased by about 15-20% from control to conflict scenarios. For example, 73.3% of the participants selected this as the basis for their treatment for Scenario 3, however the use of this basis dropped to 52.3% for Scenario 5- Patient 2. Along with this decrease in
verbatim, the gist reasoning, “Past experience with a similar patient,” was used more often in the conflict scenarios. For example, Scenario 3 had a frequency of 11.5% and Scenario 5- Patient 2
-40-
was 26.1%. Table C.36 shows the combined frequency and percent values for the sums of the two verbatim reasons and the two gist reasons. This increase in gist rationale for treatment agrees with the idea that in the six control scenarios the questions were intended to get
participants to use verbatim knowledge of protocols to make decisions. In the conflict scenarios,
participants to use verbatim knowledge of protocols to make decisions. In the conflict scenarios,