Monday Morning Data Mining

(1)

Fakultät Physik

Experimentelle Physik V

Monday Morning

Data Mining

Tim Ruhe

(2)

Fakultät Physik

Outline:

- data mining - IceCube

(3)

Fakultät Physik

(4)

Fakultät Physik

(5)

Fakultät Physik

(6)

Fakultät Physik

(7)

Fakultät Physik

1. Find representation of data 2. Find a good algorithm

(8)

Fakultät Physik

IceCube in a nutshell:

- completed in December 2010 - located at the geographic

South Pole

- 5160 Digital Optical Modules on 86 strings

- instrumented volume of 1 km3

- subdetectors DeepCore and IceTop

(9)

Fakultät Physik

IceCube in a nutshell:

- Detection principle: Cherenkov light

- Look for events of the form:

ν

+ X

e,µ,

τ

(10)

Fakultät Physik

IceCube: Scientific goals

- detection of astrophysical

neutrinos

- atmospheric neutrino energy spectrum

- neutrino oscillations - CR-anisotropy

(11)

Fakultät Physik

(12)

Fakultät Physik

(13)

Fakultät Physik

(14)

Fakultät Physik

(15)

Fakultät Physik

(16)

Fakultät Physik

(17)

Fakultät Physik

(18)

Fakultät Physik

Data Mining in IceCube:

- app. 2600 reconstructed attributes

- Data and MC do not necessarily agree - signal/background ratio ~ 10-3

interesting for studies within the scope

of machine learning

(19)

Fakultät Physik

(20)

Fakultät Physik

Make sure you understand your input:

Attributes can be: nominal green, blue, red, yellow

ordinal cool, mild, hot cool < mild < hot

numerical 1,2,3,4,....

labels can be: polynominal red, green, yellow, blue

binominal signal, background

(21)

Fakultät Physik

Data Preprocessing: Preselection of parameters

1. Check for consistency (data vs.Signal MC vs. Backgr. MC) 2. Check for missing values (nans, infs)

How to handle the nans? (see next slide)

3. Eliminate the “obvious“ (Azimuth angle, timing information...) 4. Eliminate highly correlated and constant parameters

(22)

Fakultät Physik

Data and MC preprocessing: How to handle nans?

Several possibilities:

- Exclude attributes that exceed a certain number of nans - Replace by: - minimum

- maximum - average

- nothing at all - (median...)

(23)

Fakultät Physik

Data and MC preprocessing: Feature Selection

1. Forward Selection

start with empty selection

add each unused attribute

estimate performance

(24)

Fakultät Physik

Data and MC preprocessing: Feature Selection

2. Backward Elimination

start with a full set of attributes

Remove each of the attributes

Estimate performance for each removed attribute

The attribute giving the least decrease in performance is removed

(25)

Fakultät Physik

Backward Elimination in RapidMiner:

(26)

Fakultät Physik

Data and MC preprocessing: Feature Selection

3. M

ininmum

R

edundancy

M

aximum

R

elevance

iteratively add features with biggest relevance and least redundancy

Quality criterion Q:

∑

′

−

=

j F in x

x

D

j

y

x

R

(27)

Fakultät Physik

MRMR in RapidMiner:

(28)

Fakultät Physik

Evaluating the Stability of the Parameter Selection:

- Data and MC is subject to a certain variance

(29)

Fakultät Physik

Stability of the MRMR Selection:

Jaccard Index: Kuncheva‘s Index:

B

A

B

A

J

∪

∩

=

)

(

)

,

(

2

k

n

k

rn

B

A

I

_C

−

=

(30)

Fakultät Physik

(31)

Fakultät Physik

(32)

Fakultät Physik

Learners:

1. Decision Trees 2. Naive Bayes

3. k - Nearest Neighbours 4. Random Forests

(33)

Fakultät Physik

A bit more technically speaking:

set of vectors x = (x₁,x₂,...,x_n); x_i = attribute (attributes = features, variables, parameters) labels y₁,y₂,...,y_nlabels = target class

(34)

Fakultät Physik

(35)

Fakultät Physik

(36)

Fakultät Physik

Naive Bayes:

]

Pr[

]

Pr[

]

|

Pr[

]

|

Pr[

E

H

E

H

=

×

- based on Bayes theorem:

(37)

Fakultät Physik

(38)

Fakultät Physik

Naive Bayes:

Play?

outlook = sunny, temperature = cool,

humidity = high, windy = true

(39)

Fakultät Physik

Naive Bayes:

]

Pr[

14 /

9

9 /

3

9 /

3

9 /

3

9 /

2 ]

|

Pr[

E

(40)

Fakultät Physik

Naive Bayes:

0206

.

0 ]

|

Pr[

0053

.

0 ]

|

Pr[

]

Pr[

14 /

9

9 /

3

9 /

3

9 /

3

9 /

2 ]

|

Pr[

=

×

=

E

no

E

yes

E

yes

795 .

0 ]

|

Pr[

205 .

0 ]

|

Pr[

=

E

no

E

yes

(41)

Fakultät Physik

Naive Bayes: What if Pr[E

_i

|yes]=0?

12 /

6 ]

|

Pr[

9 /

5 ]

|

Pr[

12 /

5 ]

|

Pr[

9 /

4 ]

|

Pr[

=

→

=

→

=

yes

overcast

yes

overcast

yes

sunny

yes

sunny

Let‘s assume we don not have positive examples

for outlook = rainy

(42)

Fakultät Physik

k-Nearest Neighbours (k-NN)

- memory based classifier

- unsupervised

- find the k neighbours closest to x and classify by

majority vote

- all features should be normalized

(43)

Fakultät Physik

Random Forests:

- ensemble of decision trees

- developed by Leo Breiman (2001) - no boosting between individual trees - events are classified by individual trees - uses average for final classification

(44)

Fakultät Physik

Random Forests: Output

(45)

Fakultät Physik

(46)

Fakultät Physik

(47)

Fakultät Physik

Boosting:

- uses an ensemble of weak classifiers (decision trees) - weights are increased for

false classified events - weighted vote is applied

(48)

Fakultät Physik

(49)

Fakultät Physik

(50)

Fakultät Physik

(51)

Fakultät Physik

(52)

Fakultät Physik

(53)

Fakultät Physik

(54)

Fakultät Physik

(55)

Fakultät Physik

Cut Nugen Corsika Sum

0.990 4817 ± 44 93 ± 38 4910 ± 58 0.992 4633 ± 43 80 ± 30 4633 ± 52 0.994 4414 ± 41 57 ± 30 4414 ± 51 0.996 4122 ± 32 49 ± 26 4122 ± 41

Cross validated predictions:

(56)

Fakultät Physik

Cross Validation for a limited number of examples?

(57)

Fakultät Physik

(58)

Fakultät Physik

Change the Scaling of the Corsika:

(59)

Fakultät Physik

(60)

Fakultät Physik

Cut Nugen Corsika Sum Data

0.990 4817 ± 44 93 ± 38 4910 ± 58 4988

0.992 4633 ± 43 80 ± 30 4633 ± 52 4757 0.994 4414 ± 41 57 ± 30 4414 ± 51 4476

0.996 4122 ± 32 49 ± 26 4122 ± 41 4134 0.998 3695 ± 46 18 ± 17 3695 ± 49 3638

(61)

Fakultät Physik

(62)

Fakultät Physik

(63)

Fakultät Physik

(64)

Fakultät Physik

k-means Clustering:

- Pick mean at random

- Calculate distance of examples to mean - assign examples to cluster

- recalculate mean of the cluster

- reiterate until mean does not change any longer

Significantly faster than hierarchical clustering

(65)

Fakultät Physik

(66)

Fakultät Physik

Summary:

- IceCube is interesting for detailed studies in machine learning

- studies can be carried out using RapidMiner - MRMR for Feature Selection

- Simple learners are good for benchmarks - Cross Validation is good for you!

(67)

Fakultät Physik

(68)

Fakultät Physik