The interquartile range (IQR) represents the middle 50% of the data, that is, Q3 minus Q1. It is less affected by outliers and skewed data compared to the standard deviation or variance.
The IQR can be quoted as a single value or as the range Q1 to Q3 the same as when quoting the range. The IQR is the main section of a box plot, with the top line being Q3 and the bottom line being Q1. Figure 4-12 shows a box plot with the sections labeled.
Figure 4-12. Box plots of the skewed and normal data
Translating Statistics to Make Decisions 93
As described in Chapter 3, the whiskers will only represent the range if there are no statistical outliers. Here it also highlights that the mean and median are very close together for the normal data, but are quite far apart for the skewed data, in fact the mean is above Q3 for the skewed data.
A less commonly used measure of spread is the semi-interquartile range (SIQR), which represents half the IQR. It is more robust against skewed data than the standard deviation but is not necessary if the data follows a normal distribution.
To calculate the IQR and the SIQR in R, the commands are IQR(x) and IQR(x)/2, respectively, where x represents a column of continuous data.
MAD
The median absolute deviation (MAD) is another good measure of dispersion in the data, and it is more robust to outliers than the standard deviation.
The MAD is calculated by finding the positive difference between each value and the median of the data, and then finding the median of all those results.
Example 4.7 walks through the method for calculating the MAD and then shows a shortcut using an R command.
EXAMPLE 4.7
Calculate the MAD for the following data: 1, 4, 3, 5, 6, 2, 4, 2, 3, 4.
Rearrange the order of the data:
1, 2, 2, 3, 3, 4, 4, 4, 5, 6 Calculate the median of the values:
3.5
Subtract the median from all values in the data:
(1 – 3.5), (4 – 3.5), (3 – 3.5), (5 – 3.5), (6 – 3.5), (2 – 3.5), Rearrange the order of the data:
0.5, 0.5, 0.5, 0.5, 0.5, 1.5, 1.5, 1.5, 2.5, 2.5 Calculate the median of the values:
1
This can be done simply in R with the following command:
Chapter 4 | Descriptive Statistics
The MAD for the data is 1, as this is small it suggests that the median value is a good representation for the other values in the data.
The MAD can also be used to highlight outliers by calculating the absolute deviation from the median for each point, then dividing them by the MAD. This will show the distance of the values from the center of the data in terms of the MAD, see Example 4.8.
EXAMPLE 4.8
Determine if there are any outliers from the previous data set. Repeat the same question replacing the value of 1 with 10.
# Create the data
x = c(1, 4, 3, 5, 6, 2, 4, 2, 3, 4)
# Calculate the distance from the center in terms of the MAD abs(x - median(x)) / mad(x, constant = 1)
[1] 2.5 0.5 0.5 1.5 2.5 1.5 0.5 1.5 0.5 0.5
# Create the data swapping 1 for 10 x2 = c(10, 4, 3, 5, 6, 2, 4, 2, 3, 4)
# Calculate the distance from the center in terms of the MAD for the new data abs(x2 - median(x2)) / mad(x2, constant = 1)
[1] 6 0 1 1 2 2 0 2 1 0
For the first set of data the values are all quite similar and not that different to the MAD of 1, so we can conclude there are no statistical outliers. For the second set of data, which still has a MAD of 1, the new value of 10 is clearly a statistical outlier as the distance of 6 is much larger than all the other values and the MAD itself.
There can be some confusion as MAD also can be used to refer to the mean absolute deviation, which I call the average absolute deviation (AAD) to avoid this issue.
The AAD will tell us on average how far the values are from the mean, and the MAD will tell us the median distance of the values from the median. If the data is skewed the MAD is preferable to the AAD, however the results for both will be similar if the data follows a normal distribution as Example 4.9 highlights.
Translating Statistics to Make Decisions 95
EXAMPLE 4.9
Calculate the MAD and the AAD for the skewed data and the normal data and determine if there are any statistical outliers.
# Calculate the MAD for the skewed data MADs = mad(data3, constant = 1); MADs [1] 0.575
# Calculate the AAD for the skewed data AADs = mean(abs(data3 - mean(data3))); AADs [1] 2.4932
# Calculate the MAD for the normally distributed data MADn = mad(data4, constant = 1); MADn
[1] 0.5043415
# Calculate the AAD for the normally distributed data AADn = mean(abs(data4 - mean(da ta4))); AADn
[1] 0.6003418
# Calculate the distance from the center in terms of the MAD for the skewed data
abs(data3 - median(data3))/ MADs
[1] 1.28695652 0.92173913 0.33043478 1.26956522 1.25217391 [6] 0.26086957 0.93913043 1.56521739 0.01739130 1.87826087 [11] 2.01739130 1.25217391 21.96521739 0.01739130 0.29565217 [16] 0.88695652 0.03478261 20.29565217 1.18260870 1.20000000 [21] 0.33043478 0.36521739 0.45217391 0.08695652 1.04347826 [26] 1.11304348 22.40000000 0.55652174 0.80000000 0.34782609 [31] 0.29565217 1.42608696 0.34782609 0.95652174 21.51304348 [36] 1.60000000 1.21739130 0.81739130 9.96521739 4.88695652
# Calculate the distance from the center in terms of the AAD for the skewed data
abs(data3 - mean(data3))/ AADs
[1] 0.8467030 0.3373175 0.4736884 0.2570993 0.2611102 [6] 0.6100594 0.7664848 0.9108776 0.5539066 0.9830740 [11] 1.0151612 0.8386812 4.5158832 0.5458848 0.6180812 [16] 0.3453393 0.5579175 4.1308359 0.2771539 0.2731429 [21] 0.6261030 0.6341248 0.4456121 0.5699503 0.7905503 [26] 0.8065939 4.6161559 0.4215466 0.7343976 0.6301139 [31] 0.4817103 0.2210011 0.4696775 0.7704957 4.4115996 [36] 0.9188994 0.2691320 0.7384085 1.7483555 0.5771699
Chapter 4 | Descriptive Statistics
96
# Calculate the distance from the center in terms of the MAD for the normally distributed data
abs(data4 - median(data4))/ MADn
[1] 0.09080554 1.26333645 0.05428266 1.61245704 1.51548306 [6] 1.05829284 3.27826681 0.48997753 0.86828270 1.22988689 [11] 0.57145605 1.45430428 0.11157321 0.45287171 1.02429604 [16] 0.25993697 0.05428266 3.25842509 0.09118028 3.19001510 [21] 2.06479142 2.65414407 0.28295510 0.55616482 0.79607567 [26] 1.68002831 1.06347187 0.94021016 2.04742818 2.83061775 [31] 0.05428266 2.32980034 0.84537759 0.64911573 0.17834543 [36] 0.68850967 1.72453982 0.97570396 1.12204528 2.11762467
# Calculate the distance from the center in terms of the AAD for the normally distributed data
abs(data4 - mean(data4))/ AADs
[1] 0.13937556 0.99822635 0.10869304 1.41770062 1.21005234 [6] 0.82597115 2.69095033 0.47471618 0.66634542 0.97012569 [11] 0.41698415 1.28483798 0.15682228 0.44354394 0.79741075 [16] 0.15527991 0.01748833 2.67428149 0.01350901 2.74299226 [21] 1.79770247 2.29281207 0.30079859 0.53031950 0.73186634 [26] 1.34828525 0.95650337 0.72677100 1.65693441 2.44106594 [31] 0.01748833 1.89415260 0.77328441 0.48222532 0.21291699 [36] 0.64150116 1.51186031 0.75658901 1.00571033 1.71590582 Comparing the MAD and the AAD for the skewed data you can see there is quite a difference in results compared to the MAD and the AAD for the normal data, which produce similar values. When looking for outliers: the MAD for the skewed data confirms four extreme outliers and suggests two less extreme outliers, whereas the AAD really only picks out the four extreme outliers, which are highlighted in bold in the R output. There are no outliers in the normal data, which is to be expected.
CV
The coefficient of variation (CV) is very useful if you want to compare data sets with different units. Another common area of use is to compare disper-sion at differing concentration levels.
The CV is calculated as the standard deviation divided by the mean, which means that the value is normalized and can be compared across different groups as it is “unitless.” Due to this, the CV gives a measure of relative disper-sion whereas the standard deviation gives a measure of absolute disperdisper-sion.
It is also commonly converted to a percentage by multiplying the CV by 100, to give the percentage coefficient of variation (%CV).
Example 4.10 shows how to calculate the %CV from the previous datasets and compares this to the standard deviations.
Translating Statistics to Make Decisions 97
EXAMPLE 4.10
Compare the mean, standard deviation, and %CV for both the skewed data and the normal data.
# Calculate the mean for both the skewed and the normally distributed data mean(data3); mean(data4)
[1] 8.721 [1] 6.989539
# Calculate the standard deviation for both the skewed and the normally distributed data
sd(data3); sd(data4) [1] 3.893894
[1] 0.771864
# Calculate the percentage coefficient of variation for both the skewed and the normally distributed data
(sd(data3) / mean(data3))*100; (sd(data4) / mean(data4))*100 [1] 44.64963
[1] 11.04313
The means for the skewed data and the normal data are similar to each other, but we can clearly see from the standard deviation that the skewed data is more variable than the normal data. However we can’t directly compare these values. Calculating the %CV lets us do that; the skewed data has a %CV of 44.6% and the normal data a %CV of 11.0%, these values are now directly comparable and we can say that the skewed data is four times more dispersed than the normal data.
Discrete Data
There are fewer descriptive statistics when dealing with discrete and qualita-tive data due to the nature of the data. In general the only things that can be done are to summarize the counts, turn the counts into proportions or per-centages, or calculate the mode.
For example, if you were collecting data on the colors of passing cars you couldn’t calculate a mean or a standard deviation, you could however calculate the mode. Figure 4-13 shows some example calculations you could make on the data; the mode for this data would be “black.” However the mode isn’t always the most useful piece of information on its own as it may not be a good representation of the data, in the example below under a quarter of the cars are “black.”
Chapter 4 | Descriptive Statistics
98
It is worth noting that the more groups there are, the easier it is to see in a plot as opposed to tables of counts. Likert response data in particular is always clearer to view in a Likert plot as opposed to tables of counts. On the other hand if you just have a binary response for one group, this would be better presented in a table than a plot.
Figure 4-13. Displaying discrete data
Translating Statistics to Make Decisions 99