Statistical analysis testing

Statistical analysis involves descriptive and inferential approaches. Descriptive statistics simply report data in various standardized forms without making conclusions or assumptions. Inferential statistics involve the computation of a number of mathematical procedures from information given in a data sample, in order to make informed guesses about the entire population (Lane, 2011).

3.9.3 Parametric and non-parametric data analysis

The choice of statistical test(s) used is dependent on the nature of data collected. Data distribution can be said to be parametric (normally or evenly distributed) or non-parametric (skewed). Parametric data are best suited to descriptive analysis methods using measures of central tendency (mean, median, mode) and measures of variation (range, percentile, standard deviation), and inferential analysis methods such as regression coefficients, which assume that the variability between compared samples is similar. In parametric data analysis, the absolute value of data is important, as these statistics analyse the distances between numbers in a data set and the mean or each other, and then attempt to fit these distances into a theoretically normal distribution (van Emden, 2008). However, since parametric statistical analysis methods are based on a quantitative theoretical distribution of data, they are likely to produce unreliable results if used on non-parametric data, which is fundamentally not normally distributed.

Conversely in non-parametric statistical analysis the absolute value of data is not important and data are not required to follow a normal distribution. Non-parametric statistical testing is particularly useful in two situations – when groups of data being compared have widely

57

different distributions, and when data are ordinal i.e. presented in numbered ranks different from its original form. Because data are replaced with numbered ranks, some information about the magnitude of the difference between scores may be lost during non-parametric data analysis (Field, 2013).

However, in general, data sets are primarily classified as continuous or categorical, and this classification influences the choice of statistical test used even more that the nature of data distribution. Continuous data can be measured on a scale, and categorical data are presented in categories or groups which are either unrelated (nominal) or ranked (ordinal). As with parametric and non-parametric data distributions, different statistical testing methods are appropriate for data which is categorical or continuous in nature. Continuous data can be statistically analysed with a combination of parametric and non-parametric tests, but categorical data tends to be considered for non-parametric (skewed) data analysis only because it cannot be tested for normality assumptions.

Data collected through questionnaires and presented in research chapters 5 and 7 were primarily categorical in nature, specifically of two categorical types - nominal and ordinal. The questionnaires used Likert scale responses and, as such, were considered appropriate for categorical data analysis. Descriptive statistics for data were therefore presented in the form of frequencies and not measures of central tendency. This is because categorical data have no meaning beyond the categories into which actual data are grouped and, as such, values like the mean and standard deviation cannot be calculated. Furthermore, as data variables were categorical, distribution assumptions could not be tested for normality leading to the preference for non-parametric data analysis methods.

3.9.4 Pearson’s Chi square test (and contingency table analysis)

The Pearson’s Chi-Square test is an appropriate statistical significance testing method for use with categorical variables, and is recommended for its usefulness in determining statistical significance or analyzing two or more groups of categorical variables (McCrum-Gardener, 2008). This test is used to determine statistical significance in data which can be divided into groups or partitions i.e. data in the form of frequencies (van Emden, 2008). The test statistic is

58

defined as the square of the difference between an observed number and an expected number, divided by that expected number. Data are presented in an ‘RxC’ table of rows and columns. Due to its suitability, this test is used in analyzing groups of categorical variables for this type of research.

However, certain assumptions must be met when analyzing with the Pearson’s Chi-square test. The first assumption is independence, requiring that each respondent (in this case) has contributed only once to the contingency table. Data within this research meet the independence criteria, because respondents were required to respond only once to each question. The Chi-square test also has minimum size restrictions for frequencies of expected counts, as the test assumes that samples are densely distributed across cells. For valid results to be produced, no expected counts should be below 5 when analyzing two variables with two categories i.e. in a 2x2 contingency table. If more than two variables are being analysed, then no expected counts should be less than 1, and no more than 20% should be less than 5. A failure to meet the assumptions of the Chi-squared test results in a disintegration of the chi- statistic and a substantial loss of test power. Analyzing categorical variables from this research with the Chi-square test revealed that expected counts below 5 ranged from 40 – 80%, making the p-value emerging from the test unlikely to be trusted.

An alternative (or supplement) to the Chi-squared test which addresses the problem of low frequencies of expected counts is the Fischer’s Exact test, a type of exact test. Exact tests provide accurate significance levels without making assumptions that may not be met by small samples, larger contingency tables or larger samples, which may have low, sparse or zero frequencies in table cells. These tests do not use the asymptotic approach used by other significance tests like the Pearson’s Chi-square test, and always produce an exact and reliable significance value regardless of the nature and distribution of the data. The Fischer’s Exact test is typically used for analyzing two variables with two categories (Field and Wiredu, 2008), but can also be used for larger samples. However, with the Fischer’s Exact test, there are instances when the data set may be too large for the test to produce significance p-values, while not meeting the assumptions for asymptotic tests like Pearson’s Chi-square. This may result in a prolonged delay in software processing times and an inability to calculate results. The Monte Carlo method is useful in these instances for providing accurate significance p-

59

values. It uses a repeated sampling approach by repeatedly checking a specified number of contingency table options to obtain an unbiased and accurate p-value (Mehta and Patel, 2011). A tiered approach was taken in selecting the statistical test for calculating significance p- values in this research, by calculating p-values using all three tests – the Pearson’s Chi-square, Fischer’s Exact and Monte Carlo method. From observation, the exact test qualified the Pearson’s Chi-square p-value by providing a more accurate and reliable p-value. However, the Fischer’s Exact test was not considered the most suitable test as it is computationally intensive and gave inaccurate values or error problems such as long computation delays while obtaining results for every p-value. The Monte Carlo method was found to be less computationally intensive and consistently provided more useable p-values, which were more closely related to the Pearson’s Chi-square p-value. Subsequently, test statistics and p-values computed from both the Pearson’s Chi-square test and the Monte Carlo method were reported in Chapter 5, with the Monte Carlo p-value being selected as the more acceptable result.

3.10 Qualitative data analysis methods