Paul Cohen ISTA 370
Spring, 2012
Data, revisited
The purpose of exploratory data analysis Learning to see
Things and Data
Things and Data
Where Data Come From
Data are measurements of individuals (people, trees, countries, ecosystems...).
56 years old 70" tall 180 lbs Brown eyes
Moderately presbyo8c Good health
Married One child ...
Data An Individual
A Data Table
What is Exploratory Data Analysis (EDA)?
In terms of the Fundamental Model of Data, y = f (x, ):
EDA infers which factors strongly and weakly influence y and the functions that combine these factors
EDA examines to see whether it contains evidence of other important but unmeasured (“hidden”) factors
Confirmatory studies test whether x really is a causal factor that influences y
Exploratory studies are to confirmatory studies as test kitchens are to cookbook recipes.
EDA generally doesn’t test hypotheses, but, rather, “helps the data tell its story”
EDA helps you understand phenomena, and suggests hypotheses to test in confirmatory studies.
What is Exploratory Data Analysis?
Learning to See
Data have structure that is evidence of causal influences. EDA uncovers, exposes, clarifies this structure.
EDA is like hunting for fossils – it’s a skill, and you must “learn to see” not only what’s in front of you, but what lies within data.
EDA asks, “what do I see, and what does it mean?”
Like any other skill, EDA takes a lot of practice.
Load Some Data
> read.table.ISTA370<-function(filename){
dataURL<-"http://www.sista.arizona.edu/~cohen/ISTA%20370/Data/"
# Reads a data frame from a URL path rooted at ISTA370 data page read.table(paste(dataURL,filename,sep=""))
}
>
> # taheri<-read.table.ISTA370("taheri1.txt")
> # iris<-read.table.ISTA370("iris.txt")
> # heightC<-read.table.ISTA370("heightC.txt")
> # treering<-read.table.ISTA370("treering1.txt")
> # blast<-read.table.ISTA370("blastSummary.txt")
> # kinect<-read.table.ISTA370("onemovie.txt")
> # readability<-read.table.ISTA370("readability.txt")
What Do You See? What Does it Mean?
> hist(iris$Petal.Length,col="grey",main=NULL)
iris$Petal.Length
Frequency
1 2 3 4 5 6 7
0102030
What Do You See? What Does it Mean?
> ipl<-iris$Petal.Length
> hist(ipl,prob="true",ylim=c(0,1),main=NULL)
> lines(density(ipl[iris$Species=="setosa"]),col="red")
> lines(density(ipl[iris$Species=="versicolor"]),col="green")
> lines(density(ipl[iris$Species=="virginica"]),col="blue")
Looking at density curves for each species, we see that the histogram did indeed indicate two or more separate populations (species).
ipl
Density
1 2 3 4 5 6 7
0.00.20.40.60.81.0
What Do You See? What Does it Mean?
> boxplot(iris$Petal.Length~iris$Species, ylab="Petal.Length",xlab="Species")
A boxplot by species con- firms, and summarizes the petal length statistics for each species.
●
●
setosa versicolor virginica
1234567
Species
Petal.Length
Boxplots
median
upper quar,le (75% quan,le)
lower quar,le (25% quan,le) whisker (various interpreta1ons) outliers
outliers
interquar,le range
whisker
Median, Quartiles, Interquartile Range
If you sort the values in a sample from lowest to highest, the median is the middle value, or the average of the two middle values when the sample contains an even number of points.
The median is the 50th quantile, or the value for which 50% of the values are greater.
The lower quartile is the 25th quantile, above which 75% of the values are found.
The upper quartile is the 75th quantile, above which 25% of the values are found.
The interquartile range is a measure of variability and is the difference between the upper and lower quartiles.
Median, Quartiles, Interquartile Range
The median is robust against outliers; the mean is not.
Suppose 100 families in a neighborhood each make $40,000/year.
When a millionaire moves in the mean jumps from $40,000 to
$49,504/year. What happens to the median?
Before the millionaire arrived, the variance in income was zero.
Afterwards the variance is over nine billion!!! What happens to the interquartile range?
Suppose you have a sorted sample of 9 elements; the median is the fifth element. If you add another element, what will the median be? By how many locations in the sorted distribution can the median shift?
Symmetry and Skew
> with(blast, hist(Test0,breaks=20,col="grey",main=NULL))
> with(treering, hist(width,breaks=20,col="grey",main=NULL))
Test0 is skewed to the right, meaning it has a long tail of higher values, while Treering is nearly symmetric.
width
Frequency
40 60 80 100 120
010203040
Test0
Frequency
0.1 0.2 0.3 0.4 0.5 0.6 0.7
010203040
Transformations
> attach(blast)
> hist(Train0,breaks=20,col="grey",main=NULL)
> Train0Squared<-Train0^2 #square the Train0 data
> hist(Train0Squared,breaks=20,col="grey",main=NULL)
A simple transformation amplifies an otherwise hidden feature
Train0Squared
Frequency
0.2 0.4 0.6 0.8 1.0
051015202530
Train0
Frequency
0.4 0.6 0.8 1.0
010203040
Transformations
> Train0Squared<-with(blast,Train0^2)
> with(blast,plot(density(Train0Squared)))
> with(blast,lines(density(Train0Squared[gender=="female"]),col="red"))
> with(blast,lines(density(Train0Squared[gender=="male"]),col="blue"))
Does gender explain the bump?
0.0 0.2 0.4 0.6 0.8 1.0
0.00.51.01.52.02.5
density.default(x = Train0Squared)
N = 187 Bandwidth = 0.04378
Density
What Explains the Bump New Topics
> plot(Train0Squared,NewSkills0,col=gender)
> plot(NewSkills0,Train0Squared,col=gender)
The number of topics to which students were exposed (NewSkills0) seems to explain the bump, but gender doesn’t.
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
● ●
●
●
●
●
●
● ●●
2 4 6 8 10 12 14
0.20.40.60.81.0
NewSkills0
Train0Squared
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●● ●●● ●
●
●
● ●● ●
●
●
●
●
●
●● ●●● ● ●
● ●
●
●
●
● ●● ●● ●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
●
● ●
●
●●●
● ●● ●
●
●
●●●
● ● ●
●
● ●
●
● ●
● ●● ●●
●
●
● ● ●●
●
●
●
● ●●
●● ●●
●
●
● ●
● ●
●
● ● ●
●
●●
●
●
●
●
●
●
●
●● ●
●
● ●
●
●
0.2 0.4 0.6 0.8 1.0
2468101214
Train0Squared
NewSkills0
What Explains the Bump New Topics
> precocious<-NewSkills0>8
> plot(density(Train0Squared))
> lines(density(Train0Squared[precocious=="TRUE"]),col="red")
> lines(density(Train0Squared[precocious=="FALSE"]),col="blue")
So the students who saw too many subjects account for the bump.
0.0 0.2 0.4 0.6 0.8 1.0
0.00.51.01.52.02.5
density.default(x = Train0Squared)
N = 187 Bandwidth = 0.04378
Density
Boxplots instead of density plots
> boxplot(Train0Squared~precocious, xlab="precocious",
ylab="proportion training items correct"
)
●
●
●
●
●
●
●
●
●
●
●
●
FALSE TRUE
0.20.40.60.81.0
precocious
proportion training items correct
What Explains the Bump
Why did some students see hard problems?
Exploratory data analysis helped us find and amplify an odd feature of data: Some students saw too many hard problems while training for the first test. How did this happen?
> table(precocious, policy) policy
precocious DBN_11 EXPERT RANDOM
FALSE 39 119 4
TRUE 0 0 25
All these “precocious” students were in one condition of the experiment. In the RANDOM condition, training problems were selected at random, so we shouldn’t be surprised that students in
Outline
Data, revisited
The purpose of exploratory data analysis
Learning to see: Histograms, boxplots, median and other robust statistics, transforming data...missing values, tips.
What Do You See? What Does it Mean?
> ts.plot(kinect$lhand.y,ylim=c(-500,1000),col="red")
Time
kinect$lhand.y
0 50 100 150
−50005001000
What Do You See? What Does it Mean?
> ts.plot(kinect$lhand.y,ylim=c(-500,1000),col="red")
> lines(rep(0,150))
Constants and lack of change are rare in biometric data.
Zero is a special num- ber. Perhaps the Kinect codes missing data as“0”.
What is really happen- ing around time 115?
Time
kinect$lhand.y
0 50 100 150
−50005001000
Missing Data
Data can be missing for many reasons (e.g., subjects in BLAST experiment were allowed to leave once they hit a maximum score, so didn’t take all tests)
Missing data is usually given a code, such as -999 or NA. The Kinect code of zero is unhelpful. Why?
R sometimes refuses to do math on random variables with missing values. Is this
unhelpful?
Missing Data in R
> # won't work, missing values:
> with(blast,mean(Test3)) [1] NA
> # this time, exclude missing values:
> with(blast,mean(Test3,na.rm=TRUE)) [1] 0.3843575
> # get their indices:
> with(blast,which(is.na(Test3))) [1] 17 39 41 55 73 78 142 179
Missing Data: It Matters Why!
Most experiment results are based on the assumption of “random sampling” or “random assignment to conditions.”
When data are Missing Completely At Random (MCAR), missing data don’t violate these assumptions. MCAR means missingness is independent of measured and unmeasured factors.
Data are Missing at Random (MAR) when the reason they are missing has nothing to do with the data themselves. If food poisoning is proportional to fast food consumption (FFC), but FFC is unrelated to enrollment in ISTA 370, then if you’re absent due to food poisoning, then you are MAR.
NMAR data are not missing at random. If students ordinarily take four tests, but are excused from future tests if they get a perfect score on a test, are they MAR or NMAR?
How might NMAR introduce errors in analysis?
Not Missing At Random (NMAR) Data
Participants in the BLAST experiment took four tests, but those who scored 18 or more on any test were excused from later tests.
> which(is.na(Test3)) # who didn't take Test3 [1] 17 39 41 55 73 78 142 179
> Test2[which(is.na(Test3))] # what were their Test2 scores [1] 1.00 NA 0.95 NA 1.00 0.90 1.00 1.00
> T3<-Test3
> mean(T3,na.rm=TRUE) # Mean Test3 scores [1] 0.3843575
> T3[is.na(T3)]<-1 # Replace NAs with perfect scores
> mean(T3) # Mean T3 scores [1] 0.4106952
Not Missing At Random (NMAR) Data Does it Matter?
If NMAR data are “evenly” distributed over experimental conditions, then they might not matter so much. So let’s check:
> notTest3<-which(is.na(Test3)) # who didn't take Test3
> gender[notTest3]
[1] female male male male female male male male Levels: female male
> cond.plus.policy[notTest3]
[1] DBN_11_NO_CHOICE DBN_11_NO_CHOICE DBN_11_NO_CHOICE [4] EXPERT_NO_CHOICE EXPERT_NO_CHOICE EXPERT_NO_CHOICE [7] EXPERT_CHOICE EXPERT_CHOICE_ZPD
6 Levels: DBN_11_NO_CHOICE ... RANDOM_NO_CHOICE
Not Missing At Random (NMAR) Data Does it Matter?
> aggregate.data.frame(Test3,by=list(condition,gender), FUN=mean, na.rm=TRUE)
Group.1 Group.2 x
1 CHOICE female 0.3939024 2 NO_CHOICE female 0.3139535 3 CHOICE male 0.3901961 4 NO_CHOICE male 0.4375000
> aggregate.data.frame(T3,by=list(condition,gender), FUN=mean, na.rm=TRUE)
Group.1 Group.2 x
1 CHOICE female 0.3939024 2 NO_CHOICE female 0.3444444 3 CHOICE male 0.4132075
T3 sets all the NAs to 1.0 so it’s what students would get if they maxed out tests they were allowed to skip.
Note useful aggregate.data.frame command, which applies FUN to subsets of a variable de- fined by by=...
Missing and Censored Data
When a small fraction of your data is missing, you can ignore it or impute its values.
Common imputation methods involve matching records that have missing data to records that don’t, and using one or more of the complete records to infer the missing value.
Censored data is a more challenging problem. Examples: Inferring average runtime of an algorithm for a batch of jobs that’s allowed to run for a fixed time. Inferring age at death of a treatment sample when some people are still alive when the study ends.
What Do You See? What Does it Mean?
> hist(heightC$weight,main=NULL,xlab="Weight of 33 College Students")
Weight of 33 College Students
Frequency
50 100 150 200
0246810
What Do You See? What Does it Mean?
Common sense tells us that a weight of 50lbs or less is unlikely and is probably an errorful datum.
Weight of 33 College Students
Frequency
50 100 150 200
0246810
What Do You See? What Does it Mean?
What Do You See? What Does it Mean?
The vertical axes are different, so it’s hard to tell what’s happening.
What Do You See? What Does it Mean?
What Do You See? What Does it Mean?
Not all apparent differences are real differences. Mentally group your data to see if something might explain apparent differences.
What Do You See? What Does it Mean?
> m<-c(0.93,0.95,0.94,0.86,NA,0.86, 0.89, 0.85, 0.90, 0.85, 0.87)
> s<-c(0.26, 0.23, 0.25, 0.34,NA,0.34, 0.31, 0.35, 0.30, 0.35, 0.33)
> barx <- barplot(m,ylim=c(0,1.5),names.arg=1:11,axis.lty=1,xlab="Condition")
> error.bar(barx,m,s)
Not all differences are
real differences 1 2 3 4 5 6 7 8 9 10 11
Condition
0.00.40.81.2
What Do You See? What Does it Mean?
> plot(taheri$A,col="red",type="l",ylim=c(0,1))
> lines(taheri$B,col="blue")
2 4 6 8
0.00.20.40.60.81.0
Index
taheri$A
What Do You See? What Does it Mean?
> plot(taheri$A,col="red",type="l",ylim=c(0,1))
> lines(taheri$B,col="blue")
> cor(taheri$A,taheri$B) [1] -0.93024
Although features A and B were supposed to be independent, they were normalized to sum to a constant, rendering them dependent.
2 4 6 8
0.00.20.40.60.81.0
Index
taheri$A
What Do You See? What Does it Mean?
> with(mtcars,plot(disp,mpg))
> cor(disp,mpg) [1] -0.8475514
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
100 200 300 400
1015202530
disp
mpg
What Do You See? What Does it Mean?
> palette(c("blue","red","forestgreen"))
> with(mtcars,plot(disp,mpg,col=cyl))
> cor(disp,mpg) [1] -0.8475514
Coloring by a third vari- able shows that the cor- relation between disp and mpg depends on cyl.
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
100 200 300 400
1015202530
disp
mpg
Tips: Not All Data are Real Data
Data can be noisy, missing, contaminated, or even intentionally wrong (e.g., perverse subjects).
You rarely know which data are suspicious. You have to look carefully for strange values and decide what to do with them.
Weight of 33 College Students
Frequency
50 100 150 200
0246810
Time
kinect$lhand.y
0 50 100 150
−50005001000
Tips: Not All Differences are Real Differences
Get the right vertical axis scale
Augment your picture with measures of variation
1 2 3 4 5 6 7 8 9 10 11
Condition
0.00.40.81.2
Tips: Look for “holes” in data
Holes are regions that have fewer data.
You ask, “why are so few data there?”
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●● ●●● ●
●
●
● ●● ●
●
●
●
●
●
●● ●●● ● ●
● ●
●
●
●
● ●● ●● ●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
●
● ●
●
●●●
● ●● ●
●
●
●●●
● ● ●
●
● ●
●
● ●
● ●● ●●
●
●
● ● ●●
●
●
●
● ●●
●● ●●
●
●
● ●
● ●
●
● ● ●
●
●●
●
●
●
●
●
●
●
●● ●
●
● ●
●
●
0.2 0.4 0.6 0.8 1.0
2468101214
Train0Squared
NewSkills0
iris$Petal.Length
Frequency
1 2 3 4 5 6 7
0102030
Tips: Don’t rely on summaries. Look at the data!
Means and other summaries are useful but the raw data show patterns obscured by summaries.
1990 1995 2000 2005 2010
456789
year
unemployment
