Finding the Maximal Pattern

The minimal blanket is defined by the most specialized subnetworkS0 which is defined

by the maximal pattern (Definition 4.4). To find the maximal pattern is to find the decision pattern with the largest number of constraints on neuron activations, so that the blanket defined by the subnetwork still covers more of the semantic phenomenon P

than the minimum coveragecP. Because the distribution ofP is inferred from the data, the minimal blanket is not guaranteed to cover P precisely, but it is an approximation by nature.

If the subnetworks are required to cover all samples that embody P (cP = 1) then the minimal blanket is the intersection of blankets B. The proof is provided in Appendix B.1. However, if the minimal coverage is smaller than one the intersection of blankets is not guaranteed to either 1) be a blanket, nor 2) cover the phenomena more than the required cP. Because of this, finding the minimal blanket requires a more sophisticated algorithm than simply taking the intersection of blankets, when cP <1. However, the minimal blanket which covers 100% of the samples that embody P

can be utilized when constructing the smaller blankets withcP <1. Since the samples that embody P support the constraints in the minimal blanket with cP = 1, then all smaller blankets can have these constraints as well. Exploiting this, we can start searching for the maximal pattern by taking the intersection of blankets with cP = 1 as the starting point.

Finding the maximal pattern with cP < 1 is similar to the set cover problem, which is known to be NP-complete [43]. In the set cover problem the goal is to find the smallest collection of sub-sets that covers all the samples in the union of all the sets (the universe). Finding the maximal pattern requires identifying the activation con- straints for which at least cP proportion of the samples embodying the given semantic phenomenon are covered by the corresponding blanket.

Evaluating the effect of adding new activation constraints is computationally expensive because it requires computing the hypervolume of all the activation regions that are not included in the blanket already. A simple way to solve this problem is a greedy approach: add neuron constraints with the largest individual coverage to the

5.1. Minimal Blanket Hypervolume Algorithm 59 pattern one by one until the blanket does no longer cover enough inputs that embody the phenomenon.

Pseudocode for finding the maximal pattern with a greedy strategy is presented as Algorithm 3. The algorithm picks input samples in the data D that embody a semantic phenomenon P (line 3). Then the filtered data is run through a FFNN F

while recording the activation patterns produced by the processing (line 4).

Instead of using the activation patterns as a whole, it is possible for the user to observe only some subset of the neurons. For example, the layer patterns of the last hidden layer are expected to encode knowledge of the highest abstraction level features learned by the network [4, 24, 48], and therefore the scope of the pattern can be limited to the last hidden layer’s decision pattern. The scope could be anything, and is expected to depend on the use case. In this work, the specialized subnetworks are searched by observing the last hidden layer’s layer patterns (line 5).

Algorithm 3: Find the Maximal Pattern

Result: Maximal Pattern mp

1 F, P, c_P, D = a FFNN, a semantic phenomenon, minimum coverage, data 2 mp = { } // mapping for ’neuron index → activation status’ 3 D_P = filter(D,P) // choose inputs that embody P 4 aps = record_activation_patterns(F,DP)

5 lps = extract_decision_patterns(aps) // last h-layer’s d-patterns 6 mp = choose_constant_neurons(lps) // this is mp for cP = 1...

7 cc = 1 // ...so the current coverage of mp is 100%.

8 new_mp = copy(mp)

9 while cc >c_P and not all neurons belong to the mp do 10 mp =new_mp

11 new_mp = add_neuron_with_largest_coverage(mp, lps) 12 cc= current_coverage(new_mp, D_P)

After acquiring the decision patterns, which are produced byF processing inputs that embody P, the algorithm begins to search for the maximal pattern by adding all neurons with constant activation status to the pattern (line 6). If the minimum coveragecP <1, the greedy search is needed.

From the neurons that are not yet in the pattern, the neuron constraint with the largest coverage is added to the pattern on each iteration (line 11). If adding the new constraint did not reduce the coverage too much, then the maximal pattern is updated (line 10) and the search continues. However, if the coverage of the new blanket is too small, or all the neurons are already in the pattern, the search terminates (line 9).

to make the blanket smaller with every added constraint. The former follows from the fact that the hypervolume of the blanket, i.e. the requirement for its minimality, is not directly dependent on the number of inputs that activate the subnetwork. Therefore adding the neuron with the largest coverage might reduce the size of the blanket less than another neuron with a smaller coverage. An example to illustrate this with images can be found in the Appendix C.1.

The greedy approach is, however, guaranteed to make the blanket smaller with every added constraint. Since the minimal blanket is the union of activation regions which support the maximal pattern1_{, adding a new constraint cannot bring new regions}

to the blanket.

Lemma 5.1. Adding a neuron constraint to a decision pattern cannot make the

related blanket larger.

Proof. Adding a constraint makes blanket B larger, if it adds at least one non empty activation region to the blanket. Let σ be a decision pattern which defines a blanket B in the input space. All the regions that belong to B support all the activation constraints of the pattern σ. Since all regions in the input space which support the existing constraints already belong to the blanket, there is no region outsideB which would support the existing constraints. Therefore adding a region to the blanket requires changing the existing constraints of σ. Because adding a new constraint does not change the existing constraints inσ, adding a new neuron to the pattern cannot make the blanket larger.

Because the greedy algorithm adds neurons in the order of their individual cover- age, the smaller blankets found by the greedy approach include the constraints present in the larger blankets. More formally, let B1 be a minimal blanket that covers a

phenomenon with minimum coverage c1. Let B2 be a blanket that covers the same

phenomenon with smaller coverage c2 < c1. Now the constraints that define subnet-

work S1 also defineS2, because the greedy algorithm adds the constraints in the same

order, and B2 is equal to or smaller than B1 because of c2 < c1. This guarantees that

all activation regions which belong to B2 also belong to the blanket B1. An example

can be seen in figures 4.6, B.1, and B.2: every blanket of the same phenomenon consists of regions which belong also to the blankets with larger minimum coverage.

The greedy approach is an efficient way to obtain reliably the optimal or nearly optimal solution in cases whencP <1. This emphasizes the splitting of the input space as the bottleneck of the MBH algorithm in terms of computational resources.

5.2. Approximating the Specialization Measure 61