I NSTRUMENTS AND PROCEDURES

W RITING YOUR METHOD FOR AN

Data are, of course, the raw material of the research. The researcher must decide what types of data are most relevant to answer their research question. Data may be scores on a test, reaction times to a stimulus, rankings on a performance, prefer- ences and attitudes and observations. Given this, the researcher needs to decide how the data are to be obtained, quantified and analysed. The decisions made will influence the quality of the research and its validity. A research question being answered with poor data will founder. Your article must, therefore, effectively describe any instruments, equipment and procedures you used to obtain your data.

Surveys and questionnaires

Often in psychological research the main technique used is a survey or questionnaire consisting of items to which the partici- pant makes a response. It is appropriate in your article to describe the nature of the survey instrument, its antecedents, and give exemplars of items used, unless it is a commonly used instrument with well-established validity and reliability. In the following excerpt, my co-authors and I indicate how we used a well- established instrument, the Adult Sources of Self-Esteem Inven- tory (ASSEI), to examine cultural dimensions of self-concept.

If you make modifications to the well-established instrument these should be noted in your description. If your instrument is new, you might need to report some details on its validity and reliability at this point. Conversely, the reliability of a new instrument is often reported in the results section of an article. This is particularly the case if validating the instrument is a major focus of the research. In the next excerpt I describe the instrument used in the study on beliefs about success, cited earlier.

ASSEI is a 20-item inventory that requests each respon- dent to rate on a 1 (very low) to 10 (very high) scale the importance for him- or herself and his or her satis- faction with different aspects of a person’s self-concept such as the physical, social, ethical, familial, and intellec- tual (see Appendices). For the Chinese, Indian, and Malaysian respondents each item was translated into the local language by teams of bilingual social scientists using the approved translation/back translation method (Brislin, 1986). In Nepal pilot studies showed that greater reliability was found with an English rather than Nepali version of ASSEI (English was the medium of instruction).

From Watkins, D., Adair, J., Akande, A., Cheng, C., Fleming, J.,

Ismail, M., Gerong, A. & McInerney, D. M. (1998). Cultural dimensions, gender, and the nature of self-concept: a fourteen country study. International Journal of Psychology, 33, 17–31.

Instruments

In the two-part questionnaire, participants were asked, first, to think about times when they had felt personally pleased with themselves at school (eg, ‘because you tried really hard to do better at your work,’ ‘because you set out to beat someone in a test, and did’ and ‘because you helped others with their schoolwork’) and, second, to indicate what qualities they thought other students needed to be successful at school (eg, ‘they are always trying to improve in their work,’ ‘they like to beat others at tests,’ and ‘they like to help others with their schoolwork’). There were 16 paired questions targeting mastery, performance, or social goal orientations (see Appendix B).

Physiological reactions

Your data might consist of physiological reactions to stimuli as was the case in the articles on music recognition and sexual arousal cited above. For example, the following excerpt from the music study shows how reaction time to stimuli were used as the data for analytic purposes.

Observations

Your data may consist of observations and in this case you need to indicate the categories that you enumerated, and the schedule

All items used a 3-point response scale indicating level of agreement, ranging from no (1) to not sure (2) to yes (3). A range of achievement data also were obtained, including the following: English and mathematics grades for the subsample of students completing their final compulsory year of schooling (Year 10), absenteeism, intention to complete schooling, and preferred occupation after leaving school. These variables were used as outcome measures in multiple regression analyses.

From McInerney, D. M., Hinkley, J., Dowson, M. & Van Etten, S.

(1998). Aboriginal, Anglo, and immigrant Australian students’ motivational beliefs about personal academic success: are there cultural differences? Journal of Educational Psychology, 90, 621–9. Copyright (1998) by the American Psychological Association; reprinted with permission.

The reaction time data comprised measures of the time taken from the onset of the comparison sequence to the response ‘same’ or ‘different’ in the experimental trials. The data were converted into response latencies to give an indication of the amount of time that had elapsed from the point at which the first feature manipulation occurred in a modified sequence to subjects’ responses.

From Stevens, C. & Latimer, C. (1997). Music recognition: an

illustrative application of a connectionist model. Psychology of

by which the observations were conducted. In the following excerpt, my co-author and I indicate how we conducted the observational studies as part of a research project.

Observational studies

Observational studies were conducted concurrently with the interviews described earlier. Two types of observational methods were used: structured classroom schedules and field notes.

Structured classroom schedules. Twenty-four (24) class-

room schedules were completed in twelve (12) classes from which students participating in the interviews were drawn. Two observation periods (typically lasting between thirty and forty minutes) were completed for each class. The structured classroom schedules were developed, as the interviews progressed, to focus on key ideas identified in the interviews. This meant that each structured classroom schedule used a series of (up to eight) interview responses which acted as focus points for the observations.

The specific content of these observations focused on students’ actual work avoidance behavior in various learning situations, and with respect to various learning tasks. These behaviors included students’ conversations with each other (and, on occasion, with themselves) as they worked on specific tasks; any questions, answers, or other interactions students had with their teachers; and observations of the apparent intensity (or otherwise) with which students engaged in their work ie. whether they appeared distracted from, or focused on, academic tasks at hand and in what circumstances.

Field notes. Field notes were recorded concurrently with

the interviews and classroom observations. The field notes were a more unstructured method of observation and were used in a more open-ended fashion. Thirty-seven field note entries were made. Entries in the field notes typically included notations concerning students’ social and work avoidance behaviors and reactions to various learning situations, and the research processes themselves (eg, whether a student appeared to be comfortable and open in an interview or observation situation).

Special equipment

At times in experimental and other research some special equipment or materials may be used. These should be described in detail (sometimes in the appendix) so that the reader can visualise what took place, and other researchers can replicate the experi- ment or study. If the researcher was presenting visual or oral stimuli the nature of these should be described, including the number, order and any other relevant details. At times diagrams or photographs may be appropriate. Commonly used equipment such as stopwatches need only be referred to. In the following example, you will see how the authors summarised some of the special equipment they used in their study of male sexual arousal. I am including this example as it is somewhat more interesting than usual descriptions of equipment in psychological articles. But really, it provides an excellent example of appropriate detail in such a section, when it is warranted.

From Dowson, M. & McInerney, D. M. (2001). Psycho-

logical parameters of students’ social and work avoidance goals: a qualitative investigation. Journal of Educational Psychology, 93. Copyright 2001 by the American Psychological Association. Reprinted with permission.

Equipment

Penile circumference was monitored throughout the ses- sion using a Parks Electronics mercury-in-rubber strain in the manner described by Julien and Over (1984). Changes in resistance of the gauge resulting from variation in penile tumescence were amplified by a Grass preamplifier (Model 7P1) and recorded on a Grass polygraph (Model 7). Paper speed was 5mm/sec for all records. The strain gauge was sterilised in activated gluteraldehyde (Cidex 7) before and after each use. Calibration permitting each man’s responses to be expressed in millimetres of penile circumference was undertaken using the method described by Julien and Over (1984).

From Koukoumas, E. & Over, R. (1997). Male sexual arousal elicited

by film and fantasy matched in content. Australian Journal of

Psychology, 49, 1–5. Copyright (1997) The Australian Psychological

PROCEDURES

It is very important for you to describe in detail how your study was conducted. In many cases you will describe group testing procedures, in other cases, individual testing procedures. You need to describe in sufficient detail how the samples were organised, such as randomisation, counterbalancing, and other relevant features, as well as the experimental procedures used. At times authors also describe what was said to the participants to prepare them for their involvement. These features, together with those above describing the participants and materials, are given for two main reasons. First, so that readers understand clearly the way in which the research was conducted in order that they can make some assessment of its quality. Second, to provide other researchers with the necessary information needed to replicate the work, or to extend and improve it.

I include below details on the administration of a survey on comparative attitudes to health education of Catholic and Muslim students that I administered some time ago.

ANALYSES

In many articles there is no separate analysis section and the analyses are reported under results. However, in articles which

Administration

The survey, having been approved by the Department of Education and Training, School Committees, and the Human Ethics Research Committee at the University, was administered to students who had completed informed consent forms from themselves and their parents. The survey was administered by a team of researchers and assistants to either intact classes or groups of classes. Each item was read aloud in English while the students responded to it. To ensure confidentiality, the teachers were not involved in the administration procedures.

From McInerney, D. M., Davidson, N., Suliman, R. & Tremayne, B.

(2000). Personal development, health and physical education in context: Muslim and Catholic perspectives. Australian Journal of

Educational Research Ltd. Reproduced by permission of the Australian Council for Educational Research Ltd.

include an analysis section, you briefly describe key elements of your analytic approach. This might include a brief overview of the nature of your data, for example, whether your scores are nominal, ordinal, or ratio, or a combination. It should also include a brief overview of the parametric or non-parametric statistical techniques utilised and the reasons for this. These reasons should relate to their appropriateness to the particular research question being addressed, and to the nature of the data obtained. Some data can be analysed by both parametric and non-parametric approaches, some only by non-parametric approaches. Some statistical approaches are considered strong, others relatively weak. Some techniques are robust with small sample sizes, others need large sample sizes. Some techniques need an examination of the distribution of the data to ensure that it conforms to normal distribution parameters (the assumption of equality of variances), while other techniques may not need this. You will also consider your use of both descriptive and inferential statistics and their various purposes. You need to clearly justify, therefore, your approach in terms of the nature of your data, its strengths and limitations. It is not uncommon for editors of journals to reject articles because inappropriate statistical analyses were performed. Analyses might also include qualitative approaches, and again, you need to describe what you did and why. In the following excerpt I include how my co-authors and I described the analyses performed in an article on self-concept.

Statistical Analyses

The items were coded such that higher scores reflected more favorable self-concepts. In preliminary analyses we examined the internal consistency of the domain-specific self-concept measures. Applying confirmatory factor analysis (CFA), we first examined the multidimensionality of the self-concept responses that has been widely supported in previous research. Then we examined the possibility of a higher order factor to represent the domain-specific self-concepts which in turn represent skill-specific self- concepts. The conduct of CFA has been described else- where (eg, Bollen, 1989; Byrne, 1989, 1998; Joreskog & Sorbom, 1993; Marsh, 1992, 1994; Pedhazur & Schmelkin, 1991) and is not further detailed here. In essence, CFA models were posited such that the designed items were allowed to load on the respective a priori

factors only. Support for the multidimensionality of self- concepts requires a good fit of the multidimensional model to the data and correlations among factors to be reason- ably low for each factor to be distinguishable from other factors. Support for a hierarchical representation requires the domain-specific factors to be correlated such that their relations can be explained by a higher order factor.

All analyses throughout this paper were conducted with the SPSS 6.1.3 version of PRELIS and LISREL 7 (Joreskog & Sorbom, 1988), using maximum likelihood estimation. The goodness of fit of models is evaluated based on suggestions of Marsh, Balla, and McDonald (1988) and Marsh, Balla, and Hau (1996) with an empha- sis on the Tucker–Lewis index (TLI) which takes into account model parsimony, but we present also the chi- square test statistic and the relative noncentrality index (RNI). For an acceptable model fit (typically TLI > .9), for parallel items (with similar wording) correlated uniquenesses were included in the models a priori (Joreskog, 1979; Marsh, 1993b). All models reported here had correlated uniquenesses included for the parallel items. Whereas the goodness-of-fit indexes are useful in assessing model fit, it is also important to evaluate model fit on the basis of comparison between alternative models (typically by comparing their TLI values).

The CFA models presented here were based on a 40 × 40 covariance matrix (4 × 4 = 16 domain-specific and 4 × 6 = 24 subdomain self-concept items) with a sample of 249 for all models after listwise deletion of missing data. We first tested the hypothesis that there is a hierarchical relation among the domain-specific self- concepts so that they can be represented by a higher order factor. Then we tested further the hierarchical relation of a higher order Creative Arts factor with the domain- specific and skill-specific (subdomain) factors. Although we tested numerous alternative models, we report only the most critical models in two sections below (see Table 1).

From Yeung, A. S., Cui, H. C., Kau, I. C., McInerney, D. M.,

Russell-Bowie, D. & Suliman, R. (2000). Where is the hierarchy of academic self-concept? Journal of Educational Psychology 92, 556–7. Copyright by the American Psychological Association. Reprinted with permission.

You will note from this example that the statistical procedures used are relatively complex and therefore require some detailed description, although also note that we refer the reader to other articles and texts on the conduct of CFA so that we do not ‘over dwell’ on this issue. Even when the researcher uses more standard approaches, it is helpful to describe for the reader how and why they were used, as exemplified in the following excerpt.

In the next chapter I consider the results and discussion sections in detail.

Analyses

We conducted separate principal–components analyses on the 16 ‘success for others’ and ‘success for self ’ items for each of the three groups independently. As a quasi- confirmatory approach we set the analyses to extract three factors and examined the results in terms of the ability of the factors to define the targeted dimensions. The three extracted factors targeted were Mastery, Performance and Social goal orientations. We performed multivariate analyses of variance (MANOVAs), analysis of variance (ANOVAs), and paired t tests to consider differences between and within groups on answers to the two part questionnaire, and multiple regression analyses to examine the relationship between self-descriptions and academic achievement outcomes across the three groups in which the three goal orientations were entered as a single block. Because we believed that students’ gender might interact with cultural group in influencing the main effects, we included gender as a factor in the MANOVA analyses. The categories of interest, therefore, were membership of a particular cultural group and gender.

From McInerney, D. M., Hinkley, J., Dowson, M. & Van Etten, S.

WRITING RESULTS AND

In document Publishing Your Psychology Research (Page 89-98)