Statistical inference

Statistical inference is mainly concerned with the induction of general statements on a population of interest, on the basis of the observed sample. First we need to derive the expression of the distribution function for a sample of observations from a statistical model. A sample ofnobservations on a random variable Xis a sequence of random variablesX1, X2, . . . , Xn that are distributed identically

asX. In most cases it is convenient to assume that the sample is a simple ran- dom sample, with the observations drawn with replacement from the population modelled by X. Then it follows that the random variables X1, X2, . . . , Xn are

independent and therefore they constitute a sequence of independent and iden- tically distributed (i.i.d.) random variables. Let X indicate the random vector formed by such a sequence of random variables,X =(X1, X2, . . . , Xn), and let

X =(x1, x2, . . . , xn)indicate the sample value actually observed. It can be shown

that, if the observations are i.i.d., the cumulative distribution ofX simplifies to

F (x)=

n

i=1 F (xi)

with F (xi)the cumulative distribution of X, evaluated for each of the sample

values (x1, x2, . . . , xn). If x =(x1, x2, . . . , xn) are the observed sample values,

this expression gives a probability, according to the assumed statistical model, of observing sample values less than or equal to the observed values. Furthermore, whenXis a continuous random variable

f (x)=

n

i=1 f (xi)

wheref is the density function ofX. And whenXis a discrete random variable

p(x)=

n

i=1 p(xi)

where p is the discrete probability function of X. If x=(x1, x2, . . . , xn) are

the assumed statistical model, of observing sample values exactly equal to the observed values. In other words, it measures how good is the assumed model for the given data. A high value ofp(x), possibly close to one, implies that the data is well described by the statistical model; a low value ofp(x)implies the data is poorly described. Similar conclusions can be drawn forf (x)in the continuous case. The difference is that the sample density f (x) is not constrained to be in [0,1], unlike the sample probabilityp(x). Nevertheless, higher values off (x)

also indicate that the data is well described by the model, and low values indicate the data is poorly described.

In both cases we can say that p(x) or f (x) express the likelihood of the model, for the given data.

These are fundamentals ideas when considering inference. A statistical model is a rather general model, in the sense that once a model is assumed to hold, it remains to specify precisely the distribution function or, if the model is para- metric, the unknown parameters. In general, there remain unknown quantities to be specified. This can seldom be done theoretically, without reference to the observed data. As the observed data is typically observed on a sample, the main purpose of statistical inference is to ‘extend’ the validity of the calcu- lations obtained on the sample to the whole population. In this respect, when statistical summaries are calculated on a sample rather than a whole popula- tion, it is more correct to use the term ‘estimated’ rather than ‘calculated’, to reflect the fact that the obtained values depend on the chosen sample and may therefore be different if a different sample is considered. The summary func- tions that produce the estimates, when applied to the data, are called statistics. The simplest examples of statistics are the sample mean and the sample vari- ance; other examples are the statistical indexes in Chapter 3, when calculated on a sample.

The methodologies of statistical inference can be divided into estimation meth- ods and hypothesis testing procedures. Estimation methods derive statistics, called estimators, of the unknown model quantities that, when applied to the sample data, can produce reliable estimates of them. Estimation methods can be divided into point estimate methods, where the quantity is estimated with a precise value, and confidence interval methods, where the quantity is estimated to have a high frequency of lying within a region, usually an interval of the real line. To pro- vide a confidence interval, estimators are usually supplemented by measures of their sampling variability. Hypothesis testing procedures look at the use of the statistics to take decisions and actions. More precisely, the chosen statis- tics are used to accept or reject a hypothesis about the unknown quantities by constructing useful rejection regions that impose thresholds on the values of the statistics.

I briefly present the most important inferential methods. For simplicity, I refer to a parametric model. Starting with estimation methods, consider some desirable properties for an estimator. An estimator T is said to be unbiased, for a parameter θ, if E(T )=θ. The difference E(T )−θ is called the bias of the estimator and is null if the estimator is unbiased. For example, the

sample mean X= 1 n n i=1 Xi

is always an unbiased estimator of the unknown population meanµ, as it can be shown thatE(X)=µ. On the other hand, the sample variance

S2 = 1 n n i=1 (Xi−X)2

is a biased estimator of the sample varianceσ2_{, asE(S}2₎₌ n−1(n−1)σ2 n . Its

bias is therefore

bias(S2)= −1 nσ

2

This explains why an often used estimator of the population variance is the unbiased sample variance:

S2 = 1 n−1 n i=1 (Xi−X)2

A related concept is the efficiency of an estimator, which is a relative concept. Among a class of estimators, the most efficient estimator is usually the one with the lowest mean squared error (MSE), which is defined on the basis of the Euclidean distance by

MSE(T )=E[(T −θ )2] It can be shown that

MSE(T )=[bias(T )]2+Var(T )

MSE is composed of two components: the bias and the variance. As we shall see in Chapter 6, there is usually a trade-off between these quantities: if one increases, the other decreases. The sample mean can be shown to be the most efficient estimator of the population mean. For large samples, this can be easily seen applying the definition.

Finally, an estimator is said to be consistent (in quadratic mean) if, forn→ ∞, lim(MSE(T ))=0. This implies, for n→ ∞, P (lim|T −θ|)=1; that is, for a large sample, the probability that the estimator lies in an arbitrarily small neighbourhood of θ approximates to 1. Notice that the sample mean and the sample variances we have introduced are consistent.

In practice the two most important estimation methods are the maximum likelihood method and the Bayesian method.

Maximum likelihood methods

Maximum likelihood methods start by considering the likelihood of a model which, in the parametric case, is the joint density ofX, expressed as a function of the unknown parametersθ:

p(x;θ )=

n

i=1

p(xi;θ )

where θ are the unknown parameters and X is assumed to be discrete.

The same expression holds for the continuous case but withp replaced byf. In the rest of the text we will therefore use the discrete notation, without loss of generality.

If a parametric model is chosen, the model is assumed to have a precise form, and the only unknown quantities left are the parameters. Therefore the likelihood is in fact a function of the parametersθ. To stress this fact, the previous expression can also be denoted by L(θ;x). Maximum likelihood methods suggest that, as estimators of the unknown parameter θ, we take the statistics that maximise

L(θ;x) with respect to θ. The heuristic motivation for maximum likelihood is that it selects the parameter value that makes the observed data most likely under the assumed statistical model. The statistics generated using maximum likelihood are known as maximum likelihood estimators (MLEs) and they have many desirable properties. In particular, they can be used to derive confidence intervals. The typical procedure is to assume that a large sample is available (this is often the case in data mining), then the maximum likelihood estimator is approximately distributed as a normal distribution. The estimator can thus be used in a simple way to derive an asymptotic (valid for large samples) confidence interval. Here is an example. Let T be a maximum likelihood estimator and let var(T) be its asymptotic variance. Then a 100(1−α)% confidence interval is given by T −z(1−α/2) VaR(T ), T +z(1−α/2) VaR(T )

where z(1−α/2) is the 100(1−α/2) percentile of the standardised normal distri-

bution, such that the probability of obtaining a value less than z(1−α/2) is equal

to (1−α/2). The quantity (1−α) is also known as the confidence level of the interval, as it gives the confidence that the procedure is correct: in 100(1−α)% of the cases the unknown quantity will fall within the chosen interval. It has to be specified before to the analysis. For the normal distribution, the estimator of

µ is the sample mean, X=n−1_X

i. So a confidence interval for the mean,

assuming we known the varianceσ2_{, is given by} X−z(1−α/2) Var(X), X+z(1−α/2) Var(X) where Var(X)= σ 2 n

When the distribution is normal from the start, as in this case, the expression for the confidence interval holds for any sample size. A common procedure

in confidence intervals is to assume a confidence level of 95%; in a normal distribution this leads to z(1−α/2)=1.96.

Bayesian methods

Bayesian methods use Bayes’ rule, which expresses a powerful framework for combining sample information with (prior) expert opinion to produce an updated (posterior) expert opinion. In Bayesian analysis, a parameter is treated as a ran- dom variable whose uncertainty is modelled by a probability distribution. This distribution is the expert’s prior distribution p(θ ), stated in the absence of the sampled data. The likelihood is the distribution of the sample, conditional on the values of the random variable θ:p(x|θ ). Bayes’ rule provides an algorithm to update the expert’s opinion in light of the data, producing the so-called posterior distribution p(θ|x):

p(θ|x)=c−1p(x|θ )p(θ )

with c=p(x), a constant that does not depend on the unknown parameter θ. The posterior distribution represents the main Bayesian inferential tool. Once it is obtained, it is easy to obtain any inference of interest. For instance, to obtain a point estimate, we can take a summary of the posterior distribution, such as the mean or the mode. Similarly, confidence intervals can be easily derived by taking any two values of θ such the probability of θ belonging to the interval described by those two values corresponds to the given confidence level. As

θ is a random variable, it is now correct to interpret the confidence level as a probabilistic statement: (1−α) is the coverage probability of the interval, namely, the probability that θ assumes values in the interval. The Bayesian approach is thus a coherent and flexible procedure. On the other hand, it has the disadvantage of requiring a more computationally intensive approach, as well as more careful statistical thinking, especially in providing an appropriate prior distribution.

For the normal distribution example, assuming as a prior distribution forθ a constant distribution (expressing a vague state of prior knowledge), the poste- rior mode is equal to the maximum likelihood estimator. Therefore, maximum likelihood estimates can be seen as a special case of Bayesian estimates. More generally, it can be shown that, when a large sample is considered, the Bayesian posterior distribution approaches an asymptotic normal distribution, with the maximum likelihood estimate as expected value. This reinforces the previous conclusion.

An important application of Bayes’ rule arises in predictive classification problems. As explained in Section 4.4, the discriminant rule establishes that an observationxis allocated to the class with the highest probability of occurrence, on the basis of the observed data. This can be stated more precisely by appealing to Bayes’ rule. LetCi, fori=1, . . . , k, indicate a partition of mutually exclusive

and exhaustive classes. Bayes’ discriminant rule allocates each observationx to the classCi that maximises the posterior probability:

p(Ci|x)=c−1p(x|Ci)p(Ci)wherec=p(x)= k

i=1

andxis the observed sample. Since the denominator does not depend onCi, it is

sufficient to maximise the numerator. If the prior class probabilities are all equal tok−1_{, maximisation of the posterior probability is equivalent to maximisation of}

the likelihoodp(x|Ci). This is the approach often followed in practice. Another

common approach is to estimate the prior class probabilities with the observed relative frequencies in a training sample. In any case, it can be shown that the Bayes discriminant rule is optimal, in the sense that it leads to the least possible misclassification error rate. This error rate is measured as the expected error probability when each observation is classified according to Bayes’ rule:

pB=

(1−max

i p(Ci|x))p(x)dx

also known as the Bayes error rate. No other discriminant rule can do better than a Bayes classifier; that is, the Bayes error rate is a lower bound on the misclassi- fication rates. Only rules that derive from Bayes’ rule are optimal. For instance, the logistic discriminant rule and the linear discriminant rule (Section 4.4) are optimal, whereas the discriminant rules obtained from tree models, multilayer perceptrons and nearest-neighbour models are optimal for a large sample size.

Hypothesis testing

We now briefly consider procedures for hypothesis testing. A statistical hypoth- esis is an assertion about an unknown population quantity. Hypothesis testing is generally performed in a pairwise way: a null hypothesis H0 specifies the hypothesis to be verified, and an alternative hypothesisH1 specifies the hypoth- esis with which to compare it. A hypothesis testing procedure is usually built by finding a rejection (critical) rule such that H0 is rejected, when an observed sample statistic satisfies that rule, and vice versa. The simplest way to build a rejection rule is by using confidence intervals. Let the acceptance region of a test be defined as the logical complement of the rejection region. An acceptance region for a (two-sided) hypothesis can be obtained from the two inequalities describing a confidence interval, swapping the parameter with the statistic and setting the parameter value equal to the null hypothesis. The rejection region is finally obtained by inverting the signs of the inequalities. For instance, in our normal distribution example, the hypothesis H0: µ=0 will be rejected against the alternative hypotheses H1: µ=0 when the observed value ofX is outside the interval (0−z(1−α/2)

Var(X), 0+z(1−α/2)

Var(X)).

The probability α has to be specified a priori and is called the significance level of the procedure. It corresponds to the probability of a type I error, namely, the probability of rejecting the null hypothesis when it is actually true. A com- mon assumption is to takeα=0.05, which corresponds to a confidence level of 0.95. The probability is obtained, in this case, by summing two probabilities rela- tive to the random variableX: the probability thatX <0−z(1−α/2)

Var(X)and the probability thatX >0+z(1−α/2)

Var(X). Notice that the rejection region is derived by setting µ=0. The significance level is calculated using the same assumption. These are general facts: statistical tests are usually derived under

the assumption that the null hypothesis is true. For short, it is said that the test holds ‘under the null hypothesis’. The limits of the interval are known as critical values. If the alternative hypothesis were one-sided, the rejection region would correspond to only one inequality. For example, ifH1:µ >0, the rejection region would be defined by the inequality

X >0+z(1−α)

Var(X)

The critical value is different because the significance level is now obtained by considering only one probability.

There are other methods for deriving rejection rules; we will consider them in Section 5.4. An alternative approach to testing the validity of a certain null hypothesis is by calculating thep-value of the test. Thep-value can be described as the probability, calculated under the null hypothesis, of observing a test statistic more extreme than actually observed, assuming the null hypothesis is true, where ‘more extreme’ means in the direction of the alternative hypothesis. For a two- sided hypothesis, thep-value is usually taken to be twice the one-sidedp-value. Note that the p-value is calculated using the null hypothesis. In our normal distribution example, the test statistics isX. Letx be the observed sample value ofX. Thep-value would then be equal to twice the probability thatXis greater than x: p-value=2P (X > x). A smallp-value will indicate thatx is far from the null hypothesis, which is thus rejected; a largep-value will mean that the null hypothesis cannot be rejected. The threshold value is usually the significance level of the test, which is chosen in advance. For instance, if the chosen significance level of the test isα=0.05, ap-value of 0.03 indicates that the null hypothesis can be rejected whereas a p-value of 0.18 indicates that the null hypothesis cannot be rejected.