Data coding and analysis

The coding of data in content analysis is a process of transforming unedited texts into “analysable representations” (Krippendorff 2004, p84). Weber (1990) describes this transformation as involving the application of measurement techniques to “semantically equivalent textual units such as words, word senses, phrases, issues, or themes” (p72). To achieve this, the analyst must make three interrelated choices: deciding what categories define the research problem (sampling units), what indicators of syntactic or thematic context are used to classify the content (recording units) and what system of measurement will be used (enumeration units) (Holsti 1969; Krippendorff 2004; Ogden and Clarke 2005). There is no single ‘correct’ answer, but rather each set of decisions should be appropriate to the relevant research problem. Within the accounting discipline, themes have emerged as the most commonly employed recording units by which to examine voluntary CSDs (Ogden and Clarke 2005), with many studies seeking to identify the presence or absence of particular CSD themes, such as the GRI indicators (see Al-Tuwaijri et al. 2003; Bozzolan et al. 2006; Frost et al. 2005; Gallego 2006; Pedrini 2007); or to quantify the presence of themes using enumeration units such as the number of words (Campbell 2004; Campbell et al. 2003; Cooke 1989; Deegan and Gordon 1996; Deegan and Rankin 1996; Frost 2007; Neu et al. 1998), number of sentences (Buhr 1998; Frost 2007; Guthrie et al. 2008; see Hackston and Milne 1996; Holland and Foo 2003; Linsley and Shrives forthcoming; Milne and Adler 1999; Ogden and Clarke 2005; Tilt 2001; Tsang 1998) or number of pages – or some proportion thereof (Adams et al. 1995; Adams et al. 1998; Andrew et al. 1989; Cowen et al. 1987; Deegan and Rankin 1996; see Ernst and Ernst 1972- 1978; Gray et al. 1995a, b; Guthrie and Parker 1989; Guthrie and Parker 1990; Harte and Owen 1991; O'Dwyer and Gray 1998). These attempts to identify and quantify disclosure are grounded in the assumption that the amount of a disclosure signifies its importance (Deegan and Rankin 1996; Krippendorff 1980; Neu et al. 1998; Unerman 2000; Weber 1990). Measures of volume, therefore, allow the analyst to draw inferences as to perceptions of importance placed on the theme by the report preparer (Holsti 1969). However, this is not always the objective of accounting CSD research.

Other studies have instead aimed to draw conclusions about the content itself, seeking to describe the content or evaluate the quality of disclosures within a framework of stakeholder accountability, organisational legitimacy or economic efficiency. These studies adopted alternative approaches to enumeration since evidence suggests disclosure quality and quantity are not necessarily synonymous

enumerate particular attributes (or indicators) of disclosure quality. To date, these have included: the location of a theme (or disclosure category) within a text (Guthrie and Parker 1990; Guthrie et al. 2004); its ‘transparency’ or completeness as demonstrated by the presence or absence of ‘good’ (positive) and ‘bad’ (negative) information (Beck 2007; Gray et al. 1995a; Hackston and Milne 1996); or its level of detail

(Deloitte Touche Tohmatsu 2002; SustainAbility 2006; Webb et al. 2008); external comparability (Webb et al. 2008); or ‘evidence’ (auditability) in terms of the provision of quantitative/monetary, quantitative/non-monetary or qualitative (declarative/narrative) information (Aerts et al. 2006; Andrew et al. 1989; Gray et al. 1995a; Guthrie et al. 2008; Guthrie and Parker 1990; Guthrie et al. 2004; Hackston and Milne 1996).

Furthermore, some analysts have sought to bring greater definition to the multifaceted construct of quality (Beck 2007; Brown and Butcher 2005; Morhardt 2001; Warsame et al. 2002) by examining multiple attributes within a single study. For example, Guthrie et al. (2004) and Guthrie and Parker (1990) examined both evidence and location, while Gray et. al. (1995a) examined evidence (accountability), transparency (news) and volume. Others sought to provide greater definition of ‘evidence’ by means of rating scales that captured different levels of detail provided within a classification of evidence (see for example Cormier and Gordon 2001; Warsame et al. 2002; Wiseman 1982) or a combination of detail, evidence and transparency (see for example, Beck 2007).

Nevertheless, Cormier et al (2005) suggest that the quality of a voluntary disclosure is determined by an organisation’s accountability to particular stakeholders. Within this context quality is defined as a function of “precision, relevance and usefulness” (p6). To this end, critics of the above rating scales have drawn attention to the subjectivity inherent in coding schemes that required arbitrary distinctions between, for example, ‘general’, ‘specific’ and ‘detailed’ information and lack attention to the relevance of information captured within scoring systems (Cormier et al. 2005; Morhardt 2002).

Offering an alternative scoring framework, Morhardt (2001), sought to bring greater rigour and objectivity to the process of scoring CSDs through the development of the Pacific Sustainability Index (PSI)73, a scoring template for rating a wide range of environmental and social issues within corporate disclosures (see Morhardt 2001; Morhardt 2002; Morhardt et al. 2002; Morhardt et al. 2006). This instrument sought to provide clear rules for rating disclosure quality and transparency independent of underlying corporate performance and has since been used by academics and

73

government agencies to evaluate CSDs (for example see Brown and Butcher 2005; Morhardt et al. 2006; NZ Dept Labour 2005). As Brown and Butcher (2005) observed,

The PSI scale included clear and complete descriptions of categories and items along with comprehensive criteria and guidelines for scoring each item (p9).

The present study

The current PhD study seeks to evaluate OHS disclosure by describing and critiquing the content and looking for institutionalised templates, or patterns, of OHS reporting. Consequently, an examination of information quality, rather than quantity, was deemed appropriate. Building on prior research, disclosure quality was enumerated with reference to key indicators of theme, level of detail, transparency and evidence. These indicators were captured within a disclosure matrix (see Appendix 5). The detailed scoring rules and principles offered by the PSI provided a clear process for capturing the quality of the themes (disclosure categories) identified in section 4.4.2.4 and thereby reduced the subjectivity of the coding process74.

The modified PSI rating scale

By rating the quality and depth of the information provided, apart from its mere inclusion, Brown and Butcher (2005) observed that the PSI can address the concerns of Milne et al. (2003) regarding scoring systems that obscure the distinction between firms providing vague disclosures on a number of topics and those providing detailed disclosures on a few issues. However, as noted by Brown and Butcher (2005), some modification to the PSI’s coding protocol was required because, in its original form (see Appendix 7), the PSI is designed to cover a broad range of topics and is not sufficiently detailed to pay attention to the various OHS categories under examination.

The modified coding instructions used to examine the OHS disclosures for the purposes of this study are identified in Table 4-7. Shown in red italics are the amendments and clarifications made by this author to the original PSI scoring rules for Social Intent, Social Reporting and Social Performance (as flagged in Appendix 7). These amendments were based primarily on a need to ensure rating criteria were appropriate for an OHS context. The justification for each modification made to the original PSI scoring protocol is as follows:

74

This study adopted the scoring rules rather than the full instrument since, as identified by Brown and Butcher (2005), the breadth of topics covered by the PSI failed to provide sufficiently detailed attention to categories of OHS.

Changes:

1. Owing to the detailed and specific topic area under study, external recognition (through receipt of awards) is not relevant (available) for all categories of OHS identified in this section. Consequently, internal recognition of improvement or excellence, such as reporting of quantitative outcomes that demonstrates evidence of improvement is also captured.

2. The original PSI item, which stated “+ 1 point if there is a discussion on the benefits or advantages from the program”, was removed because benefits from improved OHS are self-evident and generally well understood by users. In its place discussion of programs relating to the identification and the management of OHS risks were identified separately.

Clarifications:

3. Coding needs to discriminate between plans to implement programs and programs already implemented. For example, claims in successive reports about plans to implement a program without any evidence of actually attempting to do so cannot be counted as high quality disclosure.

4. Mention or discussion of the topic may include statements such as “our performance was better this year” or “we improved by 5% on last year”. Neither, however, provides any indication of what level of performance the organisation is operating at. For this reason numerical data is interpreted as data which reveals a particular level of performance. Typically this will include the absolute number or frequency of an event, incident or cost.

5. The focus of this analysis is quality of reporting not quality of performance. Consequently, a positive data trend is not interpreted to mean improvement in the reported performance outcomes but rather a positive trend in terms of the presentation of data (e.g. completeness or sophistication of metrics).

Table 4-7 presents the modified PSI coding instructions used in this study. Like the original PSI coding protocol, this modified instrument allowed for a maximum disclosure score of 15 for each sampled report.

Pacific Sustainability Index (PSI) coding instructions

Philosophy: Maximum 5 points

• 1 point if there is a mention of the ideology (i.e. any mention of OHS); • + 1 point if there is a discussion of the company’s position on the issue (includes corporate commitment to OHS or the provision of OHS policy);

• + 1 point if the company subscribes to at least one internal or external social program or policy that deals with this issue (i.e. OHS hazard or exposure policies or standards); • + 1 point if there is an active (action

required) program / policy the company uses to enforce this principle (i.e. governance mechanisms including Board committee);

• + 1 point if the company explicitly states that these guidelines or principles are being followed (includes both explicit statements of compliance or external OHS audit).

Narrative: Maximum 5 points

• 1 point if there is a mention of the topic (includes planned

programs) 3;

• + 1 point if there is a discussion ofa program / policy the company uses to identify OHS hazards (programs must be implemented) 2;

• + 1 point if there is a discussion of a program / policy the

company uses to manage known OHS hazards (programs must be implemented) 2;

• + 1 point if the program is continuously being monitored or improved by the company; • + 1 point if the company is a

leader or role model as

evidenced by external recognition or awards – or if internal

evidence of leadership is provided via independently verified data)1.

Quantification: Maximum 5 points

• 1 point if there is a mention of the topic (includes general statements regarding metrics used but results not disclosed or those disclosed simply as percentage changes in performance)4; • + 1 point if there is discussion of the

topic that includes numerical data. (must show actual result i.e. quantify as number, frequency rate or cost)4; • + 1 point if historical data are

presented;

• + 1 point if there is a positive data trend (in terms of increasing number /

variety of related data metrics. For example showing both a number and a rate)5;

• + 1 point if data are better than peer average, if the company is taking a leadership position in the sector or if data are at maximum possible performance (e.g. comparison to relevant industry averages – preferably national rather than global). Table 4-7: PSI scoring criteria

Notes 1 to 5 in this table refer to the justification provided on the previous page.

The coding process

In addition to maximising the objectivity of the score assignment process by providing clear and unambiguous instructions for rating disclosures (as shown above), it is also important to ensure a systematic and reliable approach to data collection by confirming that categories within the disclosure index are clearly and operationally defined (Guthrie and Mathews 1985; Holsti 1969). This not only requires the researcher to decide which questions to code as open and closed responses and what the coding method will be (as outlined in the discussion of the disclosure index provided above), but also to develop a written protocol for interpreting passages of text applying this protocol to a set of accounts so that the question and answer options can be interactively tested and refined with realistic data (Hodson 1999).

Consequently, the process for coding the data into the disclosure index matrix was undertaken as follows. First, testing was undertaken on a sample of two reports which were not part of the research sample. Clarification of the rules for categorising disclosures was needed and the Disclosures Classification Rules template (see, Appendix 4) updated accordingly. Next, the index was tested on the first ten reports (five annual reports and five sustainability reports). Again, some minor clarifications

and additions to the rules needed to be made. More importantly, the initial coding sheet (based on the first column of Table 4-6 above) was found to lack ‘user friendliness’. The layout was then redesigned into a Microsoft Excel worksheet. This proforma coding sheet provided a master template for collecting and organising the data (see Appendix 5). Coding instructions, notes and reminders were entered into worksheet cells as comments on the master file so they would appear each time the cursor was moved over the cell (see Appendix 6 for an example).

The data for this study was then captured by creating a separate Excel file for each of the 15 companies. Within each file, one worksheet was created for each year from 1997 to 2007. The coding template was then copied from the master file onto each of the 11 worksheets in three places. The first template was copied to the top of each sheet (rows 1 to 44) and used to code the annual report issued in the relevant year. A second copy was placed underneath (rows 50 to 94) and used to code the sustainability report issued in that year. The third copy was located below (rows 100 to 147) and used formulae to capture disclosure content from the above templates and in doing so provided information about OHS disclosures irrespective of reporting media. Beside each template were formulae to calculate the relevant PSI score for the disclosure. Two additional worksheets were then created within each of the 15 files, the first for making notes and recording comments or quotes from within the company’s reports and the second for creating a detailed list of the various different injury and illness performance metrics used. The initial ten reports were then recoded using this revised coding system.

A second coder was then trained in the research method and re-tested the original sample of ten reports. Differences were discussed and clarifications updated on the template. A separate line item was entered into the master template and into each worksheet template to capture disclosures of contingent OHS liabilities. Once this was complete, coding of the sample commenced. Two queries regarding unexpected disclosures arose during the coding process and were discussed and resolved. Each report was coded independently by both researchers. Examples of the coding process illustrating the capture of items ‘mentioned’ versus ‘discussed’ are provided in Appendix 8. Section 4.5 provides details of tests performed to evaluate the reliability and validity of this study.

Once the sampled reports were coded into the 15 company files, a separate Excel file was created which used formulae to extract the data onto a single worksheet (reflecting coded data from one annual or sustainability report per row). Descriptive information relating to firm, year and report were captured in the first three columns,

allowing comparative analyses of the data to be performed. A copy of this summarised data was uploaded into SPSS for statistical analysis. The first level of data analysis summarised results for each category of disclosure across the sample. Further analysis then examined the results by year, industry and organisational size. A more detailed analysis of the quality of information relating to OHS outcome metrics and to management strategies was then undertaken as these issues appeared to provide the greatest variability in results across the sample. Presentation and analysis of the results of the final content analysis results and the PSI scores is provided in Chapter 6.