The analysis of the research designs are shown in Table 3.8 (on p. 81, as drawn from Appendix 8a). The results are quite stark, the bulk (26) of the 32 cases are extensive in form. The majority of the research de- picts causal events through regularities across variables. As has been noted, there is nothing inherently wrong with such an approach by itself. Problems though can emerge when it has become all but the only way in which private forest landowners are understood. Here, an explana- tory strategy, through sheer dominance, can become restrictive of ways in which knowing or knowledge is shaped and which in the end may in- fluence action (policy). A better outcome would have seen a more even mix of extensive and intensive designs alongside diversity in methods (something that is also lacking, as will be discussed to follow). Further, it is not enough to include the mixed designs with the intensive so as to more adequately balance the number of extensive designs. Of the three mixed methodology designs, one is stronger on the extensive side, one
Chapter 3: Epistemic analysis
stronger on the intensive side and one is difficult to determine, so the mixed designs - in totality - do not automatically add to the diversity of the reviewed literature. All of this adds to the sense that the literature has little diversity, with an attendant lack in different ways of knowing throughout the reported results.
Extensive and intensive research Number of times used
Extensive 26 Intensive 3 Mixed methodological (model) 3
Total 32
Table 3.8: Extensive, intensive and mixed research designs of the reviewed literature
The methods of data collection/selection in the reviewed literature are shown in Table 3.9 (drawn from Appendix 8a).
Method: data collection/selection Number of times used
mail 16 telephone 9 interview 5 Questionnaire:
Total 30 Face to face interview 9 Focus group 3 Document analysis 2 Delphi 1 Ecological measure of forest condition 1 Field inspection 1 Participatory observation 1 Secondary data 1 Unknown 1
Total *45
Table 3.9: Methods of data collection/selection in the reviewed literature. *As each research design can use several methods, the total exceeds the number of studies (45 listed methods versus 32 case studies).
Questionnaires dominate with 30 applications (67%) out of a total of 45 methods. This overt commitment to questionnaires carries a set of con- sequences which again restricts how private forest landowners and for- ests might be known.
A questionnaire is a form containing fixed questions. The question- naire form is administered to a person so as to elicit information perti- nent to a research question/topic. Generally, questionnaires can be considered as inherently restrictive of communication (especially mail questionnaires), requiring much pre-planning and thought prior to the point of communication so as to minimise respondents misinterpreting some facet of the process. Questionnaires can allow for a degree of re- spondent and interviewer flexibility (notably telephone and personal in- terview types), but the strength of a well-made questionnaire is in the capacity to meet an epistemological commitment to producing a certain type of valid, reliable, objective knowledge. This, in practice, involves a technical protocol which seeks to remove error (Dillman 1991), but which tends, as a by-product, to reduce the complexity of social life. Consequently, creating and administering a questionnaire is a compli- cated and laborious task and a good questionnaire is not at all easy to construct or administer (Dillman 2000), requiring a high level of skill across a diverse range of practices. An appropriate way perhaps of ex- pressing this issue is through briefly looking at poor questionnaire con- struction.
Poor design of questionnaires and questions per se, is an enduring and widespread problem in survey research (Fowler 1995). Further and rarely, in the reporting of research in journals for instance, can ques- tionnaires be easily accessed (as there is usually no room to include them), so it can be nigh on impossible to easily gauge one. Amongst all of the reviewed literature, only with the Northern NSW Forestry Ser- vices (1999) study was the questionnaire available for assessment9 and it was poor when reflected against the standards set by Dillman (2000).
9
Both Jenkins (1998) and Hamilton, Dettmann and Curtis (2000) provide examples of a questionnaire in their research reports, but it is not evident if they are presented in the format respondents received them or are artefacts of the format of the reports. They were left aside because of this.
Chapter 3: Epistemic analysis
For instance, it deployed socio-demographics as the first set of ques- tions, but they should be asked last because they usually contain inva- sive questions (like income) which generate respondent resistance to completion. Nor are they linked directly to the apparent goals of the survey, so bewildering respondents at the start of the process as to their relevance. Further, the form was laid out in a confusing way, with sets of multiple questions presented in such a way that the respondents had to navigate themselves through sub-questions, any of which may not necessarily be relevant to each respondents own circumstances. The focus on questions about timber harvesting was likely to have meant that those who were not predisposed in some way to harvesting were unlikely to have completed the survey. The form also asked about clearing of forest. Considering the resistance to and fear of government amongst some landowners, those who have harvested timber and cleared forest are likely to be wary of completing such a form that asks them about clearing, without some further contextualisation. In other words, it is likely to be only those who have forest on their property, trust government and who are active timber managers and harvesters who filled out the survey. These examples go a long way to undermin- ing the link between questionnaire and epistemological commitment (lethally, it could be argued in this case).
The only remaining method with a reasonably high rate of usage (9) was the face-to-face interview, but a more accurate assessment of face-to-face interviews in the reviewed literature reduces this category to 4 (it is not straight forward to identify each method used, as it de- pends so much on the writing skills of the author/s involved, how much they write-out and how much they know about the method).
An interview is essentially a conversation driven by a specific inten- tion (Burgess 1984). This definition is loose enough to include the ques- tionnaire interview category, but it can be further modified by incorporating the sense that either:
• the interview is conducted within a longer term relationship between interviewer and interviewee; and/or,
• to obtain an insiders understanding of a persons’ life or general so- cial context; and/or,
• the interview itself is structured not through particular questions, but through broad topics or at times unstructured in terms of particular topic areas.
Other definitions can be applied (see Minichiello et al., 1995) but the general sense should be apparent here.
The parameters for assessing interviews as either questionnaire in- terviews or as semi/unstructured interviews, for instance, were fairly loose. Five of the face-to-face interview category could reasonably be moved to the questionnaire category, but for the difficulty determining the extent to which the interviews conformed to one or the other type. Strictly speaking, this author had confidence that only four of the face- to-face interviews could be considered as interviews well away from the questionnaire type (those performed by Bliss and Martin 1989; Bliss 1992; Egan and Jones 1993; Lönnstedt 1997).
The general dominance of interviewing and questionnaire ap- proaches over all other methods of data collection/selection leads to the further conclusion that data is being generated in such a way as to con- sistently deny diversity in the presented results. The Delphi, focus group, participatory observation and document analysis are, for exam- ple, major forms of data collection/selection10, and it is a significant is- sue that they are simply not well used (let alone any other methods). Their theoretical assumptions provide significantly enhanced ways in which the world may be known and would have been well worth under- taking to either a greater extent or in drawing on other techniques.
The methods of data analysis/synthesis in the reviewed literature are shown in Table 3.10 (on p. 85, as drawn from Appendix 8a).
10
There is also a Q-sort technique in amongst the interviews, although it was not ana- lysed as separate to a questionnaire process.
Chapter 3: Epistemic analysis
Method: data analysis/synthesis Number of times used
Statistical 29 Thematic 4 Unknown (narrative?) 1
Total *34
Table 3.10: Methods of data analysis/synthesis in the reviewed literature. *This total of 34 accounts for the two mixed methodological designs, which use two data analysis/synthesis forms.
Not surprisingly, considering the emphasis on extensive designs, statis- tics dominates as the method of choice in data analysis/synthesis. Given the lack of engagement with concepts, theory and design diver- sity, previously noted, statistical techniques appear to have taken the place of methodology (after Morrow and Brown 1994). Further evidence for this is in the way that the weight of design qualification and planning within the reviewed literature generally falls on the nuances and specif- ics of statistical analysis.
When broken down marginally further, in order to look at a little more complexity in the use of different methods of data analysis/synthesis (see Table 3.11 on p. 86, as drawn from Appendix 8a), there is a significant gap between the number of what might be called basic statistical manipulations (such as frequency tables and the like), expected in all statistical analysis designs, and more complex associational and typological analyses (29 versus 16 associational and 4 typological). Around one third of the analyses rely solely on basic statistics. There is certainly a role for basic statistical analyses (and such description is absolutely necessary in extensive designs), but it is arguable if they should so commonly be the sole practice. From reading the various cases, comes a sense of obtaining just numbers or percentages doing x or y as representative of landowners’ lives. This also obliquely echoes back to the kind of questions asked in the research, because what statistics are used is motivated by the type of questions asked and many of these questions are rarely tackling the
complexity of landowners lives. Tables 3.10 and 3.11 re-enforce the earlier perspectives about the tendency towards social arithmetic in the reviewed literature.
Form of results presented Number of times used
Ranking 29 Associational 16 Typological 4 Thematic 3 Unknown (narrative?) 1 Total *53
Table 3.11: Form of presentation of results in the reviewed literature. *This number exceeds the number of reviewed literature pieces (32), as results could be presented in more than one way.
The temporal scope is shown in Table 3.12 (drawn from Appendix 8b). The results are unsurprising, as there are no experimental studies in the literature, and there is little focus on change over time in research questions or topics. There is some potential for using diachronic studies in assessing the shift in forest use over time, especially related to the rise of later-day environmentalism since the 1960s.
Temporal scope Number of times used
Unknown 1 Synchronic 29 Diachronic 2
Total 32
Table 3.12: Temporal scope of the reviewed literature