The Two-Sample-Problem

We present the two-sample-problem in Section 3.1.1, and introduce the MMD test statistic, proving that it is zero only when the two distributions being tested are identical. In Section 3.1.2, we give a brief background to statistical hypothesis testing, and describe prior approaches to the two-sample-problem in the multivariate domain.

Maximum Mean Discrepancy

Our goal is to formulate a statistical test that answers the following question:

Problem 1 Let p and q be distributions defined on a domain X. Given observations

X := {x1, . . . , xm1} and Y := {y1, . . . , ym2}, drawn independently and identically dis- tributed (i.i.d.) from p and q respectively, does p6=q?

To start with, we wish to determine a criterion that, in the population setting, takes on a unique and distinctive value only when p = q. It will be defined based on Lemma 9.3.2 of [Dudley, 2002].

Lemma 28 Let (X, d) be a separable metric space, and let p, q be two Borel probability measures defined on X. Then p = q if and only if Ep[f(x)] = Eq[f(x)] for all f ∈ C(X),

where C(X) is the space of continuous bounded functions on X.

AlthoughC(X) in principle allows us to identifyp=q uniquely, it is not practical to work with such a rich function class in the finite sample setting. We thus define a more general class of statistic, for as yet unspecified function classesF, to measure the disparity between

p and q, as proposed by [Fortet and Mourier, 1953].

Definition 29 (Maximum Mean Discrepancy) Let F be a class of functions f :X→

R and let p, q, X, Y be defined as above. Then we define the Maximum Mean Discrepancy

(MMD) and its empirical estimate as

MMD(F, p, q) := sup f∈F (Ex∼p[f(x)]−Ey∼q[f(y)]), (3.1) MMD(F, X, Y) := sup f∈_F 1 m1 m1 X i=1 f(xi)− 1 m2 m2 X i=1 f(yi) ! . (3.2)

We must now identify a function class that is rich enough to uniquely identify whether

p=q, yet restrictive enough to provide useful finite sample estimates (the latter property will be established in subsequent sections).

We have a large degree of freedom in selecting F. The function class determines our prior knowledge as to where we expect p and q to differ most. Also, F will be determined by the problems we wish to solve given the datasets X and Y: for instance, if we are only

84 3. Two-Sample Tests on Graphs

interested in linear estimates of the data afterwards, it will suffice to ensure thatX andY

agree within the class of bounded linear functions.

A large class of functions used in Machine Learning can be described by Banach spaces

B. Consequently we will selectFto be the unit ball inB,i.e., F ={f| kfk_B ≤1 and f ∈B}. If Bis dense in C(X) we have the following theorem (proved in Appendix B):

Theorem 30 Denote by B a Banach space which is dense in C(X) and let F be a unit ball in a B. Then MMD(F, p, q) = 0 if and only if p=q.

We next express the MMD in a more easily computable form. For this purpose denote byB∗ the dual space ofB, and letφ(x) be the evaluation functionals inB. They are defined by f(x) =: hf, φ(x)i. This allows us to find a more concise expression for MMD(F, p, q) and MMD(F, X, Y).

Theorem 31 Let B be a Banach space of functions on X and denote by φ(x) ∈ B∗ _the

evaluation functionals on B. Let F be the unit ball in B. Moreover, let

µ[p] :=Ex∼p[φ(x)] and µ[X] := 1 |X| X x∈X φ(x). (3.3) Then MMD(F, p, q) = kµ[p]−µ[q]k and MMD(F, X, Y) = kµ[X]−µ[Y]k.

Proof [Theorem 31] By construction we can express Ex∼p[f(x)] = Ex∼p[hf, φ(x)i] =

hµ[p], fi. Hence

MMD(F, p, q) = sup

kfk≤1

hµ[p]−µ[q], fi=kµ[p]−µ[q]k_B∗. (3.4)

The first equality follows from the linearity of the expectation, the second one follows from the definition of the dual norm. An analogous derivation proves the second claim regarding MMD(F, X, Y).

A sufficient condition for the existence of µ[p] is that kφ(x)k ≤C for some C∈_R and for all x∈X. In other words, the evaluation operator needs to be bounded.

The next lemma will prove a result that is at the core of our novel approach. It establishes that under certain conditions, we can establish a one-to-one correspondence between a distribution p and its expectation in feature spaceµ[p].

Lemma 32 Denote by P(X)the set of distributions on X. The operatorµ[p]is linear inp. The set M:={µ[p] where p∈P(X)}, often referred to as the marginal polytope, is convex. If B is dense in C(X) then µ:P(X)→M is bijective.

Proof [Lemma 32] The expectation is a linear operation in p, hence µ[p] is linear. Since

P(X) is convex, also its image M under µ must be convex. Finally, by construction µ :

3.1 Maximum Mean Discrepancy 85

What remains to show is injectivity forBdense inC(X): by Theorem 30 MMD(F, p, q) only vanishes for p=q. By Theorem 31 we can express MMD in terms of the means µ[p] and µ[q]. Hence µ[p] =µ[q] immediately implies p=q.

This means that MMD(F, p, q) defines ametricon the space of probability distributions, induced by the Banach spaceB. As we shall see, it is often easier to compute distances in this metric, as it will not require density estimation as an intermediate step.

Reproducing Kernel Hilbert Spaces

IfB is a Reproducing Kernel Hilbert Space H many of the aforementioned quantities can be computed very efficiently. We will henceforth use F only to denote unit balls in H. Moreover, we will refer to Has universal, whenever H, defined on a compact metric space

X and with associated kernel k :X2 _→

R, is dense inC(X) with respect to the L∞ norm.

It is shown in [Steinwart, 2002] that Gaussian and Laplace kernels are universal. As a specialization of Theorem 30 we immediately have the following result:

Theorem 33 Let F be a unit ball in a universal RKHS H, defined on the compact metric space X, with associated kernel k(·,·). Then MMD(F, p, q) = 0 if and only if p=q.

We obtain an improved condition for the existence ofµ[p] viakµ[p]k2 =Ex,x0_∼p[k(x, x0)]<

∞. Here x, x0 are drawn independently from p. Exploiting Theorem 31 we are able to obtain a more accessible formulation for MMD(F, p, q) and MMD(F, X, Y).

Theorem 34 Let F be a unit ball in a RKHS H with kernel k then MMD(F, p, q) and

MMD(F, X, Y) can be computed as follows:

MMD(F, p, q) = [Ex,x0_∼p[k(x, x0)]−2E_x∼p,x0_∼q[k(x, x0)] +E_x,x0_∼q[k(x, x0)]] 1 2 _(3.5) MMD(F, X, Y) = " 1 m2 1 m1 X i,j=1 k(xi, xj)− 2 m1m2 m1,m2 X i,j=1 k(xi, yj) + 1 m2 2 m2 X i,j=1 k(yi, yj) #1₂ (3.6)

Proof [Theorem 34] We only prove (3.5), as the proof of (3.6) is completely analogous. By virtue of Theorem 31 we have

MMD(F, p, q)2 =kµ[p]−µ[q]k2_H

=hEx∼p[φ(x)]−Ex∼q[φ(x)],Ex0_∼p[φ(x0)]−E_x0_∼q[φ(x0)]i (3.7) In an RKHS we have hφ(x), φ(x0)i = k(x, x0). Plugging this into (3.7) and pulling the expectations out of the inner product proves the claim.

Eq. (3.6) provides us with a test statistic for p = q. We shall see in Section 3.1.3 that this estimate is biased, although it is straightforward to upper bound the bias (we give an unbiased estimate, and an associated test, in Section 3.1.4). Intuitively we expect MMD(F, X, Y) to be small if p =q, and the quantity to be large if the distributions are

86 3. Two-Sample Tests on Graphs

far apart. Also, since MMD(F, X, Y) ≥ 0 we intuitively expect the discrepancy measure to be positive, even when the underlying distributions p = q agree. Note that it costs

O((m1+m2)2) time to compute the statistic.