Experimental Manipulation of Explanation

Among the earliest experiments with explanations in intelligent systems were those conducted with expert and decision support systems. Suermondt and Cooper (1993) examined explanations in a decision support system in a medical context, finding that explanations improved diagnostic accuracy. Rook and Donnell (1993) compared two means of presenting information to justify a decision. That study identified that cognitive limitations in interpreting output was critical in the design of explanations. Template explanations of procedures were examined in Gregor (2001), which found relationships between their use decreasing cognitive effort and increasing usage of explanations in cooperative tasks. Rose (2005) identified that when accounting students were provided a decision-aid system to assist with decision making regarding tax-related rules, students with low cognitive loading learned the rules similarly to those without the system, but those with high load knew less about the rules than if they did not have the system. These early cases of research identified performance increases and the

foundations of many of the challenges in explanation that remain today.

The recent increase in interest in explanation has resulted in reviews collecting the past findings across intelligent system technologies. The review by Nunes and Jannach (2017) was specific to experiments with explanation and analyzed 217 studies of explanation in decision support and recommender systems. They found that the performance benefits identified by early studies did not hold over time, and that the persuasiveness of explanations was a potential confound to understand the presented output. Abdul et al. (2018) analyzed the connections and isolation of areas of research and Adadi and Berrada (2018) surveyed the body of XAI research,

and both identified a lack of articles which focused on the human factor. The review of the interpretability of explanations by Doshi-Velez and Kim (2017) identified that experiments with humans have preferred simplified tasks over realistic settings to focus on algorithm and

explanation performance (Doshi-Velez & Kim, 2017).

Some recent studies have focused on the subjective quality of explanations. Narayanan et al. (2018) manipulated the length and complexity of explanations and measured a subjective score from users on an artificial task, finding that long explanations were less preferred, though not as sensitive to the number of concepts in an explanation. The study by van der Waa, van Diggelen, van den Bosch, and Neerincx (2018) requested that users evaluate multiple

explanations, and asked them to measure them in dimensions of preference for long or short, strategy versus policy, and sufficiency of information, with the highest preference for

explanations related to policy and greater information. In the study by Lakkaraju, Bach, and Leskovec (2016), humans were employed to evaluate the interpretability of machine-generated explanation in logic form (rather than natural language) and to recreate the explanation in natural language. The responses were evaluated qualitatively by two judges, finding widely much better performance with interpretable decision sets which applied rules independently than Bayesian decision lists, which required interpreting an explanation sequentially.

Several studies evaluated compliance with system decisions based on providing an

explanation. Gedikli, Jannach, and Ge (2014) examined explanations within explanation systems, identifying transparency as a key antecedent of trust and compliance. Arnold, Clark, Collier, Leech, and Sutton (2006) looked at definition, justification, rule-trace, and strategic justifications with the performance metric of acceptance of the recommendation, finding that explanations did increase compliance with differences in the types of explanations that were persuasive to novices

as compared to experts. In a classification task for microblogs, the explanation, control, and metadata available to the user were manipulated with the primary finding that system

recommendations should be provided ahead of search results and tools to achieve compliance (Schaffer et al., 2015). In a robotic experiment Nikolaidis, Kwon, Forlizzi, and Srinivasa (2018) found an increase in adaptation to the robot’s intentions when an explanation was provided, but with decreasing trust, and belief that the robot was not being truthful.

Among more interactive uses of explanation, three studies were identified. Soundness of recommendations and completeness of explanations in recommendations was examined by Kulesza et al. (2013) in a music preference context. They tested the impact of completeness and soundness of explanations on user’s mental models, recommending that both completeness and soundness are required, but only if users believe their input is improving the intelligent agent. In Sklar and Azhar (2018) the researchers experimented with coordination of a robot that could provide an explanation in dialogue through argumentation. They found that subjective preference and objective performance criteria did not vary significantly between explanation and black box conditions. Holliday, Wilson, and Stumpf (2013) compared the effect of users being provided an explanation that they could then correct to a system that did not provide explanations. They found that participants in the condition where they were not providing explanations exhibited more control-exerting behaviors and reported such while “thinking aloud,” but there were no perceptual differences in control when the same concept was tested in a questionnaire. They deduced that there were differences between perception and behavior related to perception of control.

Several experiments introduced intentional errors to evaluate conditions where users accepted erroneous output and the effects on other measures. Ribeiro et al. (2016) manipulated

the quality of the model being explained in human experiments of trust between unexplained and explained classifications, and found that explanations of erroneous performance decreased trust but increased accuracy of user perceptions about the quality of the classification model.

Poursabzi-Sangdeh, Goldstein, Hofman, Vaughan, and Wallach (2018) examined the effect of modifying the information interpretable from a linear regression model which predicted apartment prices in New York. They also introduced mistakes and measured trust. They found that the timing of offering explanations and the design of optimal interpretability was not intuitive to designers of the interface, and that different levels of local explanation did not produce statistically significant differences in perception. In the medical context a mixed- methods analysis was conducted with an experiment involving a simulated clinical decision support system. Half of the eight scenarios provided to the subjects had an incorrect

recommendation. They found that that confidence statements did not sway trust but that

“comprehensive why” explanations induced over-reliance, and “selective why” explanations led to better confidence in the user’s decisions (Bussone et al., 2015).

The closest experiment identified to the design of this study was conducted by Tan, Tan, and Teo (2012). They evaluated how differences in explanation impacted cognitive measures by manipulating explanations into forms of trace, justification, and strategic explanations. They measured perceived confidence in the decision and cognitive capacity (preferences on how much to think) in a consumer context with types of recommendations. No single configuration of explanations produced an optimal outcome and they instead identified trade-offs in decision quality, confidence, and speed. For instance, conditions with explanations based on a table of metrics and benchmark comparisons resulted in the fastest decision times and highest

experiment. They proposed combining multiple explanation approaches into the same interaction as the solution; however, such a configuration was not experimentally evaluated.

II.3.6 Open Issues

There has been relatively little study done on what explanations should contain to balance persuasive power and use of logic. The most logically rigorous models consider the ability for explanations to be able to defend claims against those of competing explanations. Models based on more subjective criteria do not reject the value of compelling logic, but instead prioritize human outcomes of interpreting explanations. This is not without risk as well, as persuasive explanations may be more accepted based on subjective criteria than complete, accurate, and transparent explanations (Gilpin et al., 2018). The review by Nunes and Jannach (2017) identified open concerns over the lack of clarity in the differences between stakeholder goals, user goals, and the purposes of explanation, challenges in selecting the best content for

explanations, user interface options for revealing explanations, responsive explanations that were adjusted to the needs and requirements of users, and objective evaluation protocols and metrics which do not rely on stated behavioral intentions or subjective assessment of the explanations. Additionally, while the explanations generated for images have been tested in terms of

preference and trust, they have not been evaluated in terms of to what extent users process them beyond those attitudes. Experiments have found widely varying results with the most consistent “successful” outcomes based on achieving trust and compliance. Systems that introduced errors identified that compliance and trust were not always the ideal outcomes for explainable systems to achieve partnership with the human user.

III RESEARCH MODEL AND HYPOTHESIS DEVELOPMENT