Creating rationales from classification models

Without loosing its generality, the problem can be narrowed down two a two dimensional classification task. Instances are then described by two features denoted f₁ and f₂, and the space of possible variations is limited to a plane. Fig. B.1 (Left panel) schematises the initial problem where the a new instance (red circle) have been classified by a black-box model in the

”circle” category. The previously trained black-box model has generated an invisible (in the sense that it is implicit and not accessible) delimitation of the space between both categories.

The aim of the methodology is to explicit locally this delimitation. For that we aim for the smallest distance (i.e. the minimal variation) necessary for the new instance to switch its forecasted label to the ”triangles” class (red arrow pointing toward the delimitation frontier).

The information would therefore contain information about a local portion of this frontier and help understand a piece of the black-box algorithm’s unknown rationale.

The first intuitive and non optimal solution would be exhaustive scanning of the space. That

is abruptly create a large set of virtual instances (i.e. not resulting from an actual observation, but rather synthetically created for the purpose of the method) uniformly spaced all over the feature space, to then study their forecasted class. This would allow for the description of the whole frontier function. However, such brute force approach becomes intractable when more dimensions are considered. Indeed, the computational cost of this first solution is in the order of O(n^dc), n being the sampling resolution for each dimension, d the number of dimensions, and c the cost of performing one single classification. Furthermore, such method guarantees the extraction of the whole frontier, but not the precise changes necessary for our red circle to change of label. In our scenario, any many others, it is more interesting to have the least number of changes, even of higher magnitudes, to maintain the interpretability.

So, we propose a more adapted solution than the complete sampling of the feature space consists in an progressive increase of the number of independent features allowed to changes (denoted n_f) until finding the least dimensional solution. The process steps are the following:

1. Initially set n_f to 1.

2. Create a vector C with the (_d

n_f

) possible combinations of n_f elements out of d - d being

the dimension of the problem. Note that, when n_f = 1, C consists is merely the list of all the features in the problem.

3. Generate n_p new virtual instances for every feature combination c in C. The new point is created by replacing in the target observation (see red dot in Fig. B.1) the value of the features contained in c by randomly drawn values from the empirical distribution observed for such features within the original (training) data set. When n_f = 1 the task basically consists in randomly changing one feature at the time, and save each result as a new virtual instance.

4. Sort the ensemble of generated virtual instances according to their distance to the target instance.

5. Sequentially apply the classification model until one of the virtual instances falls into the desired class. If no forecasted class complies, increment n_f and go back to step (2).

Fig. B.1 middle and right panels schematise the afore-explained process. Specifically, the process starts with setting n_f to 1 and modify only one feature at a time (see middle panel;

squares represent the set of virtual created instances.). Two potential solutions (black squares) are found along the two axis. The algorithm would have stopped at the closest one, then saved the corresponding virtual instance. If no solutions would have been found, n_f would have been incremented to 2 (right panel) where virtual elements are sampled from the complete f₁ − f2

plane space. Again, the closest solution (black square) would have been saved by the algorithm.

It is important to specify three customisable aspects of the process:

a The priority is given to low dimensional solution following Occam’s Razor principle.

High dimensional solutions are only explored when no simpler solution has been found.

However, such propriety might be customised by the user of the system in order to save multiple feasible solutions for each set of modified set of feature (resulting in a Pareto front), or prioritising large dimensional changes. The former situation might be of interest in our MDDS example where the system might propose a different treatment in both cases where the patient had a different age, or the same age and a different physical condition. Therefore, the saving of various possible solutions might help improving the understanding of the rationale of the system.

b The sampling strategy in step (3) have been designed to avoid creating virtual points in irrelevant, sparsely populated regions of the feature space therefore optimising the search.

Here again the strategy might be adapted to suit more complex situation (e.g. Simulated Annealing, Genetic Algorithm, etc.).

c A simple Euclidean distance have been computed between the virtual points and the target instance to perform the sorting of step (4). This definition of the distance suggest an uniform importance of the features. However, some situation might call for a weighting scheme where all feature importance differs in function of a given characteristic. For example, some feature might be difficult to change (or very costly) and therefore the user might want to minimise its contribution to the best solution by affecting it with a low weight.

This approach, whilst still computationally intensive, is still substantially quicker than the first intuitive brute force analysis. The resulting complexity is dominated by the maximum dimension explored, therefore scales as O(n^max_f · np· c).