Regularity Estimates with CoCo

In the following, we defined the principles of ICA, EPD, data compression and linked them to the parameter-free outlier detection withCoCo_.

7.3 Unsupervised Outlier Detection with CoCo 155

Algorithm 7.1CoCo

inputDatabaseD

OF ∶= {} # Outlier Factors

fordata objectp∈Ddo

costp ∶= {}

costMinimumnnp ∶= {}

nnp ∶= initial set of nearest neighbors

not_nnp ∶= D/nn_p # remaining points

icannp ∶=ICA(nnp)

nnp,ica ∶= transform(nn_p; ica_nnp)

epdnnp ∶= estimate EPD(nnp,ica)

whilenot_nnp ≠ {} do

costp ∪= coding_cost(p_ica; epd_nnp)

costMinimumnnp ∪= min(coding_cost(nnp,ica; epdnnp))

not_nnp,ica ∶= transform(not_nn_p; ica_nnp)

costnot_nnp,ica ∶=coding_cost(not_nnp,ica; epdnnp)

nnp ∪= {not_nn_p,icawith lowest cost_{not_nnp,ica}}

not_nnp ∶= D/nn_p

update icannp

nnp,ica ∶= transform(nn_o; ica_nnp)

update epdnnp

end while

j∶=min(costMinimum_nnp) # index best compressed cluster

OF(p) ∶= (cost_p(j) −costMinimum_nnp(j))

end for

Independent Component Analysis

It was observed that mixtures of signals were best de-mixed when search- ing for non-Gaussianity. Note, that the mixing of several source signals of arbitrary distribution types are always more Gaussian than the source signals themselves. The entropy of a Gaussian distribution was maximal, whereby, all other distributions had a lower entropy. Only minimization of the coding costs, measured by the entropy, guaranteed maximal compres- sion efficiency. Thus, we applied the ICA to maximize non-Gaussianity as a measure of statistical independence. Its algorithm favored the directions in the data which were not similar to the Gaussian distribution.

We assumed that most datasets in experimental data usually did not follow equally dense distributions. They were rather distorted datasets with respect to the Cartesian coordinate system. The ICA first transformed the data into a so-called white space. Whitening involved de-correlation and normalization of the data to unit variance which enabled to implicitly handle unequally dense clusters.

Whitening may be performed with the Principal Component Analysis (PCA) that identified the directions of maximal variance ⃗y, given a set of

coordinatesx⃗∈ Cin ad-dimensional space. First, the data was centered ⃗

c= ⃗x− ⃗maround the empirical mean

⃗ m= 1 ∣C∣ ∑ ⃗ x∈C ⃗ x

of the datasetC_{. Second, the centered data}c⃗had to be normalized to unit

variance in all directions. The eigenvalue decomposition of the covariance matrix Σ is Σ ∶= V ×Λ×VT, whereV and Λ were orthogonal matrices

containing the eigenvectors and eigenvalues of Σ, respectively. Finally, the PCA transformation ofx⃗was determined by

⃗

y∶= √

7.3 Unsupervised Outlier Detection with CoCo 157

Note, that Λ = diag(λ₁, . . . ,λd)and √ Λ−1 = diag( √ 1/λ₁, . . . , √ 1/λd)

were both diagonal matrices.

For optimal projection of the data we next had to determine the direc- tions of minimal entropy (determined with ICA) rather than the one of maximal variance (yielded by PCA). After transforming the data to white space, the FastICA algorithm (Hyvarinen 1999, Hyvärinen et al. 2001) de- termined a weighting matrixW_{containing the desired independent com-} ponents (ICs). Regarding the original space, the ICs were not orthonormal in contrast to the principal components (PCs). The fix point iteration op- timizedW = ( ⃗w₁, . . . ,w⃗_d), whereby the weight vectors were updated with

the following rule:

⃗

wi ∶=E{ ⃗y×g( ⃗w_iT× ⃗y)} −E{g′( ⃗w_iT × ⃗y)} × ⃗w_i

We used tanh(s) for the non-linear contrast function g(s). Note that

g′_{(s) =} dg(s)

ds was the derivative ofg(s)andE{. . .}was the expected value.

W_{was updated until convergence and then orthonormalized. The overall} projection of the original data into the space of ICs was achieved by the de-mixing matrixM−1_{. WithM=V×}√_{Λ×W we denoted}

M−1_=WT 1 √

ΛV

T_.

W _andV _{were orthonormal matrices, thus the determinant was simply} det(M−1) = ∏₁≤i≤d

√

1/λi. Recall that the rotation performed in the white

space was expressed byW_{, and whitening was achieved by multiplying the} coordinate vector by the scaled eigenvector matrix.

After the ICs were determined, we simply projected the datax⃗into the

IC space with

⃗

Exponential Power Distribution

The EPD is a generalization of the Gaussian distribution in such a way, that it also included the Laplacian and the uniform distribution, depending on the parameter setting by exploiting the third-order statistics. Thus, its PDF has three different parameters to be fitted. Beside the location parameter µ_{, and the scale parameter}σ_{, a shape parameter}β_{is introduced (Mineo} and Ruggieri 2005). For a random variableX_{, the EPD is defined as:}

fEPD(x;µ,σ,β) = exp(−∣x−µ∣ β βσβ ) 2σ β1βΓ₍1₊ 1 β) .

Note that Γ(s) =

∫

₀∞ts−1exp(−t)dtis the gamma function as an extension

of the factorial operator for real numbers.

The shape parameter β _{determines kurtosis or the sharpness of the} distribution. For β > 2, the EPD is platykurtic, with β → ∞mimicking

a uniform distribution. For β = 2, the EPD corresponds to a Gaussian

distribution. For 2<β <0, the EPD is leptokurtic, including a Laplacian

distribution forβ=1 (Figure 7.2a).

EPD after ICA

After projection of the coordinates into the white space and ICA, the data

⃗

z _{was de-correlated and independent. This allowed to describe each di-} mension independently by an own EPD. Typically, a multi-dimensional data space contained d _{different PDF representations} fEPD(z_i;µ_i,σ_i,β_i)

with 1≤i ≤d. Allddistributions were combined in a mixing matrixM,

where the data pointsx⃗corresponded tox⃗=M× ⃗z+ ⃗m, withm⃗ being the

shifting vector andM_{determined by PCA, as described above.} M_allows the IC vectors to be not orthogonal. The EPD in a d_{-dimensional space}

7.3 Unsupervised Outlier Detection with CoCo 159 -4 -2 0 2 4 0.0 0.2 0.4 ß = 1 ß = 2 ß ∞

ICA + EPD PCA + Gaussian ICA

b a

c

Figure 7.2: EPD coupled to ICA to Estimate Regularities. a. Different

shapes of the EPD for different choices of parameter β. b. ICA generated

redundancy in the data by centering, whitening and de-mixing. c. Dataset

(after ICA) was defined for a pointx⃗as

fEPD(x;M−1,m⃗,µ,σ,β) =

∏_1≤i≤d fEPD(zi;µi,σi,βi) ∣det(M−1)∣

Figure 7.2c illustrates the effect of the approximation of a dataset with an EPD after ICA. The approximation of the same data with a Gaussian dis- tribution (with PCA) would not achieve the same precision.

EPD Approximation

The estimation of the three parameters was a non-trivial problem. Al- though,µi =0 andσi =1 were defined for β =2 (Gaussian distribution)

after ICA, µi and σi were no longer identical to the empirical mean and

standard deviation, respectively. All three parametersµi,σiandβimay be

optimized by estimating the maximum likelihood, given a datasetC_{. Only} a simultaneous approximation of all parameters ensured that the deriva- tives of the likelihood of the EPD vanished with respect toµi,σi andβi.

Assuming µi and βi to be given, the parameter σi was determined

by differentiating the likelihood function with respect to σi of the EPD ∑⃗z∈C fEPD( ⃗zi;µi,σi,βi): d fEPD(C;µi,σi,βi) dσi = − ∣C∣ σi +∑⃗z∈C ∣zi−µi∣βi σβi+1 i =0. ⇒σi = ( 1 ∣C∣ ∑ ⃗ z∈C ∣zi−µi∣ 1 βi₎

The parameters µi and βi were to be optimized explicitly. We used

a nested bisecting search as optimization technique to find βi and µi in

their parameter space. The direction to browse through the space was de- termined by the derivatives of the log-likelihood function with respect to

7.3 Unsupervised Outlier Detection with CoCo 161 µi d fEPD(C;µi,σi,βi) dµi = − 1 σ_iβ ∑⃗z∈C ∣zi−µi∣1βi−1sign(zi−µi) andβi d fEPD(C;µi,σi,βi) dβi = − ∣C∣ β2 i (logβi+Ψ(1+ 1 βi ) −1) + ∑⃗z∈Csβii +βlogσi∑⃗z∈Csiβi −β∑⃗z∈C(sβ_ii −logsi) β2 iσ βi i ,

with si = ∣zi −µi∣. Ψ(s) = dln Γ_ds(s) was the digamma function being the

logarithmic derivative of the gamma function. The EPD was estimated by this maximum likelihood approach until convergence ofβi.

In document Müller, Nikola (2012): Finding correlations and independences in omics data. Dissertation, LMU München: Fakultät für Mathematik, Informatik und Statistik (Page 168-175)