Testing for Statistical Significance…………………………………………………117-120

Keen to avoid what Gorard (2004:219) refers to as “superfluity”, this Chapter has thus far, been presented in such a way that the methods and methodology are accessible, ensuring that explanations are clear, and terminology is explained. This section continues to adopt this approach, discussing how and why confidence intervals featured in the analysis [CI]

and how the statistical test, Chi-squared was also used to explore the data, each to determine possible relationships between groups and over time.

E

Presented at the beginning of each findings section, to understand the significance of changing attitudes over time, confidence intervals were included in the analysis of the DV’s. This included the DV’s: ‘the gap between high and low incomes is too large’ (Section 4.1), ‘the government should redistribute income from the better-off to the less-well-off’

(Section 4.2), ‘many people who get social security don't really deserve any help’ (Section 5.1), ‘most people on the dole are fiddling in one way or another’ (Section 5.2) and ‘if welfare benefits weren't so generous, people would learn to stand on their own feet’

(Section 5.3). Having established the proportions in agreement each year, individually, CI’s were constructed around the point estimates in each year, reporting the 95% confidence intervals and comparing these findings to determine whether the changes between 2009-2012, 2012-2015 and 2009-2015, were significantly different (at the 5.0% level).

The next stage of the analysis involved using crosstabs and employing the statistical test, Pearson’s Chi-square test to determine whether statistically significant relationships

between variables were evident and whether trends were notable (Gorard, 2004; Adeyemi, 2009; Field, 2012). As Adeyemi (2009:48) explains, “the chi-square test is a nominal level non-parametric test of significance…used to test the differences or relationship between two variables”, using the formula:

χ² =∑ ( 0 - E ² )

Here, χ² = Chi-square, O = Observed frequency; E = Expected Frequency. As Adeyemi (2009) and Field (2012) acknowledge, as a convention, the significance value or alpha [α], is often set at less than 0.05, using hypotheses (H0 and H1) to guide the research findings.

H0 = there is no association between the IV (s) and the DV H1 = there is an association between the IV (s) and the DV

If the p value (observed significance level) is equal to or less than the alpha [α], the null hypothesis (that there is no relationship) can be rejected (Adeyemi, 2009; Field, 2012), thus suggesting that there is a relationship between the IV(s) and the DV.

In this research, this was also the case. However, rather than focusing on whether the significance value was less than 0.05 (p<0.05), the values were also identified if these were less than 0.01 (p<0.01), and less than 0.001 (p<0.001). The level of statistical

significance (where applicable) is emphasised within each of the tables (p<0.05*, p<0.01**, p<0.001***). For significant findings, these are further indicated within the narrative,

expressing significant differences in attitudes between groups as ‘more likely’ or ‘less likely’. For results, where p>0.05, these were discussed clearly as not significant findings

(ns), meaning that the null hypothesis (of no relationship) cannot be rejected (Gorard, 2004).

Recent literature, however, not only suggests the need to exercise caution when employing tests of statistical significance, including the use of confidence intervals (Amrhein et al., 2019:306) but also calls for an abandonment of the “concept of statistical significance” entirely. This desire for abandonment rests on the assumption that the use of statistical significance in a dichotomous way results in the production of “misleading”

findings (Amrhein et al., 2019:306). This occurs when researchers are presented with a P value over 0.05 (P>0.05), leading the researcher to declare there is no difference or no association or perhaps that there is no significant difference. For Amrhein et al. (2019:307) researchers should instead “embrace uncertainty”, avoiding making “overconfident claims”

based on statistical analysis. Amrhein et al., (2019:307) thus advocate that P values should be reported with “precision”, rather than indicated with “adornments such as stars or letters to denote statistical significance and not as binary inequalities (P>0.05 or P<0.05)”.

In this research, the language of statistical significance is retained, with the realisation that whilst a test may show no associations between variables, this is not necessarily reflective of social relationships. Thus, caution is exercised when interpreting the findings of

statistical tests, striving to avoid “misleading” findings (Amrhein et al., 2019:306). To promote precision in the presentation and interpretation of these research findings, the results of statistical tests are also presented within the findings chapters and although stars are utilised, with exception to the BLRA findings, so too are specific P values.

The analysis and findings are grouped by chapter, incorporating available literature, alongside theory to understand changing attitudes. Chapter 4, Section 4.1 involves the analysis of BSAS data from 2009, 2012 and 2015, highlighting the proportion of people who agreed that ‘the gap between high and low incomes is too large’. Section 4.2 explored the proportion of people who agreed that ‘the government should redistribute income from the better-off to the less well-off’. Crosstabulation meant that the levels of agreement toward each statement could be understood in relation to socio-economic and

demographic characteristics, thus highlighting which groups were more likely to hold each view and by extension which groups were less likely. To maintain consistency and enable comparison, this was repeated in each dataset.

The findings presented in Section 4.1 and 4.2 extended the analysis further, leading to the introduction of a new question of interest, intent on understanding why support toward income redistribution remained so low, despite so many members of the British public agreeing that income inequality between groups was ‘too large’. Section 4.3, thus, highlights which groups are more likely to support redistribution, amongst those who also agreed that income inequality was too great. This was achieved by filtering the data, using the variable ‘the gap between high and low incomes is too large’, and then using crosstabs to place the variable ‘the government should redistribute income from the better-off to the less well-off’ as the DV, and the socio-economic and demographic characteristics of interest as IV’s. This was repeated in each dataset. Following a discussion of the findings in each table initially, Chapter 6 provides an analytical discussion of the data alongside the literature, enabling interpretation.

Chapter 5 presents the findings from the analysis of three of the DV’s, concerned with understanding what attitudes were held toward social security recipients and how these changed (if at all). The statements included for this chapter include: ‘many people who get social security don't really deserve any help’ (Section 5.1), ‘most people on the dole are fiddling in one way or another’ (Section 5.2) and ‘if welfare benefits weren't so generous, people would learn to stand on their own feet’ (Section 5.3). Having initially sought to understand how socio-economic and demographic characteristics relate to attitudes in the first three sections, Section 5.4 sought to understand these findings further.

Filtering the data by the variable ‘many people who get social security don’t really deserve any help’, the crosstabs function was used to determine what the proportion of people who said people were undeserving, also agreed that ‘the government should redistribute

income from the better-off to the less well off’. The redistribution variable was used as the DV, with each of the socio-economic and demographic variables acting as IV’s. Similarly, Section 5.5, filtered the data by the DV ‘if welfare benefits weren't so generous, people would learn to stand on their own feet’, using crosstabs to determine what proportion of people who felt that welfare generosity resulted in dependency also supported the redistribution of income. To allow for comparative analysis, this was repeated in each dataset.

3.5.2 Binary Logistic Regression Analysis [BLRA]

Following the independent analysis of data from the 2009, 2012 and 2015 datasets as discussed above, the datasets were merged to create one new dataset, ready for the final statistical analyse, BLRA. This section explains this process, noting why BLRA was employed in the research.

As Gayle and Lambert (2009:3) explain, regression can be used to “explore the effects” of multiple (and categorical) IV’s on a binary outcome variable (DV). Accordingly, this

enabled further exploration of the DV’s in the analysis. Rather than modelling for best fit, however, within this research the results of the BLRA are interpreted in relation to odds ratios [OR], p values and 95.0% CI’s. As Szumilas note, OR’s are a measure of the association between an exposure variable (IV) and an outcome variable (DV), in this way the OR “represents the odds that an outcome will occur given a particular exposure, compared to the odds of the outcome occurring in the absence of that exposure”

(2010:227).

In this analysis, the 95% CI is employed to estimate the accuracy of the calculated OR (Connelly et al., 2016). As Szumilas contends, where a “large CI indicates a low level of precision of the OR”, comparatively, a “small CI indicates a higher precision of the OR”

(2010:227). Given that a positive OR does not “necessarily indicate that this association is statistically significant”, both CI’s and p value’s are used to establish whether or not

significant findings are observable (Szumilas, 2010:229), though the latter is not presented in the tables. Due to variation and a lack of firm guidelines, findings from each BLRA are presented in tables, reporting the Exp Beta (ß) or OR and the 95% CI’s (Connelly et al., 2016). Though this form of data analysis is not without its difficulties (Gayle and Lambert, 2009; Szumilas, 2010; Connelly et al., 2016; Amrhein et al., 2019), to enable further exploration of the data, this was considered the most appropriate.

Prior to each BLRA, to ensure that the DV’s were binary (having just two categories), further data management was, however, necessary. Each DV was recoded, using ‘0’ and

‘1’, renaming each variable in each dataset, to ensure these were consistently labelled throughout. To simplify the analysis, the IV income, was also recoded into a two-category variable, comprised of people in receipt of middle-high incomes (merging the two

categories) and people in receipt of low incomes (remaining the same). Having completed

this process, the individual datasets (2009, 2012 and 2015) were merged to create a new dataset, comprising all the data. Following this, the two DV’s presented in Section 4.1 (‘the gap between high and low incomes is too large’ and 4.2 (‘the government should

redistribute income from the better-off to the less well-off’) were selected for analysis. This analysis was also repeated in Chapter 5, with the three main DV’s selected for BLRA (Section 5.1, 5.2 and 5.3). Focusing on six IV’s (year, age, gender, benefit status, employment status and income), BLRA was included to explore what relationships were evident, if any, having taken into account other variables in the analysis. The findings of the BLRA are presented and discussed in each chapter, before inclusion within the analytical discussion section.

This chapter has, thus far, provided a critical discussion of the methods and methodology, intent on highlighting the strengths and weaknesses of this research design and on clearly outlining the variables featured within the analysis. The following section focuses on emphasising the ethical considerations inherent to this research, outlining the ethical guidance and practices adhered to throughout this research

3.6 Ethical considerations, Introduction

This Section outlines the ethical guidance and practices adhered to throughout this post-graduate research. Importantly, this research has been conducted in accordance with the guidance purveyed by three different research bodies. The guidance afforded by the ethical frameworks of both the Social Research Association’s [SRA] and British

Sociological Association’s [BSA] ethical guidelines, as well as those of the organisation who made this research possible through funding, the Economic and Social Research Council [ESRC] has thus been followed. Aside from these organisations, support and guidance was also received by the University’s Ethical Committee, where ethical approval for the purposes of postgraduate study was granted, following review by the University of Leeds Ethical Committee.