III.4.1 Previous Task Experience
Previous research in attribution and self-efficacy has found a ceiling effect with high levels of pre-existing self-efficacy precluding an effect of attributions (Stajkovic & Sommer, 2000). Gist and Mitchell (1992) evaluated the development of self-efficacy across different levels of task experience with a novel task. They found that with low experience self-efficacy is informed by multiple cues in an assessment of requirements, resources, and constraints of the task. In comparison, those with high task experience form self-efficacy based on past
performance and motivation rather than aspects of the task. High task experience, particularly experience with the specific damage guideline used in this study, would also greatly decrease the cognitive load. To account for this potential, participants were given three items to assess their previous experience in aerial image damage assessment: “How would you rate your experience in damage assessment of aerial images prior to this study?” with 7-point response options “None” (coded as 0) and “Very limited” (1) to “Highly Experienced” (6); “How often do you rate damage to structures in aerial images?” with 7-point response options from “Never” (coded as 0) and “Yearly, or less” (1) to “Daily” (6); and “Have you previously used a damage guideline to make ratings?” with three response options: “No” (coded as 0), “Yes – But not the same one” (3) and “Yes – The same guideline as here” (6). If “None” was answered to the first item, the remaining two items were not presented. The three components were added into an index value for the measure.
III.4.2 Trust in the Intelligent Agent
Trust has been the primary measure of participant attitudes in human-automation research including XAI. Since the simulated agent provides erroneous classifications in half of the
scenarios, lack of trust is an appropriate response. However, even if the worker does not fully trust the simulated agent, they may have greater trust in agents that are better able to explain erroneous performance. To measure trust for comparison against previous studies a single item from Dzindolet, Peterson, Pomranky, Pierce, and Beck (2003) (“I believe I can trust the
automated assessment”) was adapted here and rated on a 7-point Likert-Type scale from “Strongly Disagree” to “Strongly Agree” with only the ends of the scale labeled.
III.4.3 Dispositional and Learned Trust for AI
The three-level model of information systems and technology trust proposed by Marsh and Dibben (2003) defines three types of trust: “learned,” based on experience with similar systems; “situational,” where dispositions are adjusted by cues from the environment; and “dispositional” where attitudes are based on pre-existing attitudes about technology. Though participants may have previous experience in the damage assessment task, they are unlikely to have had experience making assessments with an intelligent agent. The three-level model predicts that trust in the intelligent agent will be determined by pre-existing and stable
dispositions towards automation and AI. To evaluate these pre-existing attitudes, dispositional trust for automation was measured using three items adapted from Nees (2016). These items were rated on a 7-point Likert-Type scale from “Strongly Disagree” to “Strongly Agree” with only the ends of the scale labeled. For participants that have experience working with AI, the inclination to trust the intelligent agent may be better informed by learned trust (or distrust) attained from working with other AI systems. Participants rated learned trust for AI using a 5-
point scale of how well AI had met their expectations in their profession with “Far Short of My Expectations” and “Far Exceeded My Expectations” as the ends of the scale, with an option for not having used AI in their work.
III.4.4 Perceived Interdependence
The final rating in this task is fully dependent on the human’s judgment, and the human must actively accept the input by changing their selections after reviewing the agent’s output. As such, the task is not truly interdependent. However, explanations increase observability and predictability of the agent, which joint activity theory predicts improves performance (Johnson et al., 2012). Providing the cause of a disagreeing classification can also produce a greater
understanding of the guideline, creating some level of “directability” by the agent. To assess attitudes about task interdependence, three items from Morgeson and Humphrey (2006) were adapted: “My ratings were affected by ADAM's input,” “Assessments depend on the both the human and ADAM for accuracy,” and “My ratings benefitted by working with ADAM.” Items were rated on a 7-point Likert-Type scale from “Strongly Disagree” to “Strongly Agree” with only the ends of the scale labeled.
III.4.5 Perceived Level of Automation
Participants that perceive a high degree of system automation are likely to rely on the agent’s assessment in erroneous cases without independently assessing the scenario themselves. Greater detail in explanation may be perceived as lower automation by creating a burden to the worker to evaluate the agent. One item was included to measure the participant’s perception of the level of automation of the microtask. Participants rated a single item “The damage
assessment was” on a 7-point Likert-Type scale from “Highly Manual” to “Highly Automated” with only the ends of the scale labeled.