Trends and Analysis of Research on Teaching in Doctoral Programs

(1)

Trends and Analysis of Research

on Teaching in Doctoral Programs

Stephen Silverman

The University

of Texas

at Austin

The purpose of this study was to describe and analyze research on teaching in doctoral programs. Research on teaching was differentiated from research on teacher education. Abstracts of research were obtained from Dissenm'on Abstracts International for dissertations completed from 1975 to 1984. Data were recorded for the year the dissertation was completed, primary and secondary (if any) focus of the study, observation instruments used, population, and the type of statistics used to analyze the data. Some 120 studies were identified as research on teaching in physical education. A majority of the studies (55) were comparisons of teaching methods using no observation instrument. Methods research was followed in frequency by descriptive research (22), instrument development (16), and comparisons among student subgroups (15). Systematic observation was not used in 60 studies. Of those studies where systematic observation was used, specific instruments were developed for research and other common instruments were employed. Most research occurred in elementary and secondary schools. A large proportion of the studies used univariate statistics to complete the data analysis.

In recent years those interested in the areas generally categorized as pedagogy or teacher education have seen large gains both in the number of university faculty members classifying themselves in this area and in the respect of colleagues outside the area. An increase in research and scholarship is at least partly respon- sible for these gains. The Journal of Teaching in Physical Education has expand- ed outlets for scholarship, and older more generic physical education journals such as the Research Quarterly for Exercise and Sport typically include more papers than in the past from teaching and teacher education researchers.

Although a great deal of the published research on teaching is being conducted by young scholars after they have completed the doctorate, there is a body of research that goes unpublished. As has been noted before (Locke, 1977, 1982; Pieron, 1986), doctoral students have conducted the majority of research on teaching in physical education. Much of this research is not published through tradi-

Request reprints from Stephen Silverman, Department of Physical and Health Edu- cation, The University of Texas at Austin, Austin, TX 78712.

(2)

tional outlets and therefore remains unavailable to, or is not conveniently located for, many of our colleagues.

Though it may seem that this research has little impact on the image of scholarship within the area, it is important to understand its focus and the paradigms used for conducting it. Those completing the doctoral dissertation in the related area of pedagogy will influence the image of the professional group in a variety of ways. First, young scholars that obtain academic positions in a university setting will likely influence graduate students whose interest is the area of teaching and teacher education. It may be that for many young scholars who have recently completed their degrees, the dissertation was the only research conducted as part of the graduate program (Locke, 1977; Pibron, 1986). What is communicated about the research enterprise in pedagogy may depend on the ex- periences the new professor had during the dissertation research.

Second, the image of our area perceived by our colleagues, both in other areas of physical education and in the social sciences, may be based largely on their interactions with those who are interested in teaching and teacher education in physical education. If the paradigms of these young scholars are inappropriate for their research, or if they themselves do not understand the paradigm or the research, the image of professionals interested in pedagogy in physical education may be tarnished.

Finally, many who classify themselves as teacher educators do not con- tinue to conduct research (Metzler & Freedman, 1985). However, they do often play the vital role of communicating research results to preservice and inservice physical educators. If the experience of conducting the dissertation research was not positive, leading either to increased understanding about teaching or about the role research plays in education, these negative feelings about the research enterprise may be passed on to a new generation.

This study was conducted to more fully understand research on teaching in physical education that was conducted as a part of doctoral programs. For dissertations completed during the decade from 1975 to 1984, a detailed analysis of the foci of the research, the observation instruments used, the population of the subjects, and the type of statistical analysis employed was completed.

Method

This study focused on research on teaching in physical education. Abstracts from Dissertation Abstracts International were selected and coded for the primary and secondary focus of the study, the type of observation instrument (if any) that was used, the type of student subjects in the study, and the level of statistical analysis employed.

Selection of Abstracts

All

abstracts listed under the headings Education, Elementary; Education, Physical; and Education, Secondary in Dissertation Abstracts International be- ginning with Volume 35, Number 7 (January, 1975) to Volume 46, Number 7 (January, 1986) were surveyed for inclusion in the study. Because of the time lag between completion of the dissertation and its appearance in Dissertation Ab- stracts Intemutional, this represents complete information for the most recently

(3)

available 10-year time period. In addition, in these same volumes, abstracts listed under the keyword physical also were surveyed to make certain that appropriate abstracts were not missed.

From those surveyed, only abstracts of dissertations that specifically focused on research on teaching in physical education were selected. The working defini- tion of research on teaching (ROT) was that presented by Locke (1984). Each study was characterized by a focus on the teaching and learning process. The independent measure was a process variable related to teaching (teaching method, teacher behavior, student behavior, etc.). The study may have been descriptive or may have looked at the effect of these processes on student behavior or student achievement in the psychomotor, cognitive, or affective domains. Disserta- tions not focusing on ROT in physical education were excluded from this study. For example, studies focusing on teacher education (where the dependent measure was an attempt to change the teacher process variable), curriculum, and needs assessment in adapted physical education were not included. Also studies focusing on elementary or secondary education subjects other than physical education, physical education studies from another subdiscipline (physiology, biomechanics, etc.), or studies that employed paradigms similar to those used in ROT but were conducted with one subject at a time (i.e., motor behavior studies) were excluded. To make certain that the selection of abstracts was reliable, three complete volumes of Dissertation Abstracts International were randomly selected to compare with abstracts selected earlier. The reliability was performed after deter- mining it was not likely that any further abstracts would appear in subsequent issues. Reliability was calculated using the following formula:

agreements Reliability =

agreements

+

disagreements

During reliability coding only one abstract was coded that was not included in the initial coding. For each volume reliability was above .92 with the total reliability being greater than .98.

Coding of Selected Abstracts

Each abstract was coded for the primary and secondary focus of the study, the population of the student subjects, the observation instrument (if any), and the type of statistical analysis.

Focus of Study. For the first level of coding the primary and secondary focus of the study were determined from the abstract. Although design or research method might be viable alternatives for the term focus as used here, it is likely that focus and design are interrelated. For this study, focus will imply the categories presented below. Foci categories were developed (n = 16) based on a pilot study synthesizing the various abstracts. The primary focus of the study was coded first, then any secondary focus. The following categories were developed to code the foci of the studies:

1. Descriptive;

2. Process-product with motor skill as product; 3. Process-product with cognitive or affective product;

(4)

4. Process-product with motor and another product; 5. Process-process;

6. Comparison of methods on teacher behavior;

7. Comparison of methods on student attitudes about movement; 8. Comparison of methods on motor skill;

9. Comparison of methods on cognitive or affective variables; 10. Comparison of methods on motor and another product variable; 11. Comparison of methods on fitness;

12. Comparisons of process or product among teachers;

13. Comparisons of process or product among student subgroups; 14. Observation instrument development;

15. Aptitude-treatment interaction (ATI); and 16. Qualitative study.

For greater discussion of the focus categories readers should consult Brophy and Good (1986), Dunkin and Biddle (1974), Shaver (1983), Shulman (1986), and Silverman (1985).

For presenting data, the categories that were coded as exploring process- product relationships (Categories 2, 3, & 4) were combined into one category called process-product and those categories coded as comparing different teaching methods (Categories 6- 1 1) were combined into one category called methods comparison. Therefore a total of 9 focus categories will be used for presenting the data.

Observation Instrument. Five categories were used to code the obser- vation instrument used for measuring process during data collection for the dissertation. The following categories were used: (a) none (no observation occurred), (b) Cheffers Adaptation of the Flanders Interaction Analysis System (CAFIAS; Cheffers, Mancini, & Martinek, 1980), (c) interaction system-another coding system (not CAFIAS) that addresses teacher-student interaction, (d) Academic Learning Time-Physical Education (ALT-PE; Siedentop, Birdwell, & Metzler, 1979; Siedentop, Tousignant, & Parker, 1982), or (e) another coding system not falling into one of the other categories.

Population. Six categories were developed for coding the population of student subjects. Population was coded as follows: (a) preschool students, (b) elementary school students, (c) secondary school students, (d) both elementary and secondary school students, (e) college students and, (f) adults enrolled in community education programs.

Statistical Analysis. Three categories were used to code the type of in- ferential statistical analysis employed in completing the dissertation. The following categories were used: (a) none (Virtually all studies report means and other descriptive statistics; however, where no inferential statistics are reported in the abstract this category was coded.), (b) univariate (e.g., r, t test, ANOVA), or (c) multivariate (e.g., multiple regression, factor analysis, discriminant analysis, MANOVA). If both univariate and multivariate statistics are presented in the abstract, then multivariate was coded. If the author did not present the analysis used either directly or indirectly, the study was coded as using no statistics. Reliability of Abstract Coding. After all studies were selected for inclusion and each abstract was coded for focus, observation instrument, population, and

(5)

statistical analysis, a 10% sample was recorded to measure reliability for the coding. Intraclass correlation was calculated via repeated measures ANOVA to de- termine reliability for each factor. In all cases the reliability between the first and second coding was above .95.

Data Analysis

Data for primary and secondary focus were combined into one category and cross-tabulations were performed by year for each of the coding categories. In addition, multiway cross-tabulations were performed. Since the four-way cross- tabulations are difficult to present and understand, lower order cross-tabulations are presented.

Results

From 1975 to 1984, 120 dissertations' that specifically focused on research on teaching in physical education were completed. The greatest number of ROT dissertations completed in a single year was 17 in 1975 and again in 1979. The fewest number completed in a single year was 6 in 1981. As indicated in Figure 1, the last few years have seen a trend toward slightly fewer ROT dissertations being completed.

Nearly half (55) of all the ROT dissertations completed during the period sampled were comparisons of teaching method on a dependent variable in which no observation instrument was used to collect data or verify the treatmentlteach- ing method. Instrument development in which an observation instrument other than an interaction system, CAFIAS, or ALT-PE was developed occurred with the next greatest frequency (14). This was followed in frequency by descriptive studies using instruments other than the three specific categories (1 1). Some 17 studies were found that had an identifiable secondary focus. All combinations of focus and observation instrument that were represented three or more times are listed in Table 1.

Trends by Coding Category

The data for each of the coding categories (focus, observation instrument, population, and statistical analysis) will be presented by category. Table 2 in- cludes an analysis for each coding category by year.

Focus. As indicated in Table 2, comparison of teaching methods was the focus occurring most frequently (65). For every year except 1979 the most frequent focus of a dissertation was a method comparison. Descriptive studies were the next most frequent (22). From 1977 to 1980, 18 of the descriptive dissertations were completed. Instrument development for classified and other instruments (16), comparisons of student groups (15), comparisons of teachers

(7),

and process-product studies (6) were the next most frequent.

Observation Instruments. Half of the studies (60) did not use systematic observation of teaching. The category of other, that is, instruments that were not

'A listing of the dissertations included in the study can be obtained by writing the author.

(6)

YEAR

Figure 1

-

Yearly research on teaching dissertations.

an interaction system, CAFIAS, or ALT-PE, was the predominant response when an observation instrument was employed. Researchers used a variety of instruments to collect their data or to verify treatment. CAFIAS was employed in 11

studies, 6 of those dissertations being completed in 1980 and 1981. In 8 studies

non-CAFIAS interaction systems were used, and the ALT-PE instrument was used in 6 studies beginning in 1979.

(7)

Table 1

Most Frequent Combinations of Focus and Observation Instrument

Focus Obse~ation instrument Frequency

Method comparison Instrument development Descriptive Student comparison Process-product Descriptive Method comparison Student comparison Descriptive Method comparison Teacher comparison Qualitative None Other Other Other Other CAFIAS Other None Interaction system CAFIAS CAFIAS Other

Population. Greater than 60% of the studies (74 out of 120) had student subjects of elementary or secondary school age. College age students were the subjects for 29 of the studies. In 11 instances the population of the students could not be determined from the abstract.

Statistical Analysis. In 75 studies the investigator used some form of univariate inferential statistics to complete the data analysis. For 28 studies descriptive or no statistics were used. Multivariate analysis was employed less frequently (17 times).

Discussion

As indicated in Figure 1, the past few years have seen a slight but notice- able decrease in the number of physical education dissertations that can be classified as research on teaching. This may indicate that fewer doctoral students are interested in the areas generally termed professional preparation. However, it is more probable that research on teacher education (Locke, 1984) is occurring more often, and therefore fewer doctoral students are focusing their research on teaching.

This study's most striking result is that most of what can legitimately be called research on teaching in physical education is a comparison of teaching methods with no systematic observation. Most dissertations studied used a statistical test to compare means on a dependent variable (quite often this was a multiple t test among three or more means). Although this research may make some interesting comparisons, often it adds little to our knowledge. This is not because a comparison of teaching methods is an especially futile tack but because, as Shaver (1983) has suggested, virtually all methods research that does not confirm the independent variable (the teaching method) by systematic observation suffers from a weak treatment effect (Silverman, 1985).

(8)

Table 2

Yeariy Analysis for Focus, Observation Instrument, Population, and Type of Statistical Analysis

Focus of study Descriptive 0 Process-product 0 Process-process 0 Methods comparison 15 Teacher comparison 1 Student comparison 0 Instrument devel. 1 AT1 0 Qualitative 0 Observation instruments None 15 CAFIAS 0 Interaction sys. 1 ALT-PE 0 Other 1 Population Preschool 1 Elementary 4 Secondary 3

Elem. & secondary 0

College 7

Community (adults) 0 Not reported 2

Statistical analysis employed

None 1

Univariate 16 Multivariate 0

aThe overall total for focus is a number greater than the total number of dissertations (120). Focus combines the 120 primary foci with 17 studies that also had a secondary focus.

In this same vein it is interesting that half of the studies used no systematic observation at all. As Locke (1977) implored, it is important to enter the "black box" of the gymnasium when conducting research on teaching. Although there is a long history of these comparisons in physical education (Nixon & Locke, 1973) in which teaching is not observed, knowledge has not been advanced in proportion to the effort expended in conducting these studies.

The second most frequent focus of study was describing the physical education environment. This focus occurred approximately one third as frequently as methods comparisons. Almost all dissertations were completed between 1977 and 1980 when research on teaching in physical education was beginning to use systematic observation. Descriptive research using systematic observation fol-

(9)

lowed by a move to other teacher effectiveness research reflects the recommen- dations of Rosenshine and Furst (1973). It is reasonable that after describing the status quo and finding similar results, researchers will move on to other questions related to the area. It is likely that descriptive research conducted by more experienced investigators has seen a similar decline in recent years.

Very few of the dissertations are teacher effectiveness studies using the process-product (6 of 120) or AT1 (2 of 120) paradigms for research. Although an average of 12 dissertations per year focused on ROT in physical education, the powerful designs that have yielded much information over the years for researchers interested in the classroom teacher's effectiveness (Brophy & Good, 1986; Shulman, 1986) are not well represented in physical education. Shulman (1986) has noted that many researchers of the classroom have moved on to other perhaps more esoteric paradigms. Pikron noted in 1986 that physical educators often may have moved on without studying the relationships between process and the fundamental variable of student achievement.

In about 40% of the dissertations in which an observation instrument was used, an interaction system, CAFIAS, or one of the versions of the ALT-PE system was chosen. Studies using similar instruments will likely help researchers understand the interconnections among various situations. As Brophy & Good (1986) have noted, it is equally positive that research is being conducted in which the investigator is designing observation instruments that have been developed to answer a specific question.

Most research occurred with elementary and secondary school children or college students enrolled in physical education classes. This would be expected since this is the population to which researchers may generalize. Few studies have investigated physical education in the community, such as that conducted by the YMCA, the American Red Cross, and health clubs. This may become a thrust in the future as pedagogical researchers see the need for examining this expand- ing option for physical education.

As noted, 75 of the 120 studies (62.5%) used some form of univariate statistical analysis. In many instances this was quite appropriate. However, in other instances multiple t tests were used in lieu of ANOVA, and multiple ANO- VAs were used when it appeared that a MANOVA would be more appropriate. The option of using a less sophisticated test makes interpretation of the data easi- er but will inflate the Type I error rate and lead to erroneous conclusions (Bock

& Haggard, 1968; Silverman, 1985). Fortunately many of the researchers reported

descriptive statistics that often are valuable in understanding the data. Locke's call (1977) to bring back descriptive statistics is being heard by many doctoral candidates in reporting their research.

The qualitative paradigm has been used in only a few (3) ROT studies in physical education. As more research advisors become familiar with this power- ful tool and the questions for which it is appropriate, it is likely that more doctoral students will be trained to pursue qualitative research.

Research on teaching physical education has advanced in recent years. As research in this area continues, research advisors and young scholars must be attuned to the possibility of their answering important questions with weak research designs. We scholars, as a group, should train the future generation of researchers and practitioners to appropriately plan, understand, and value the research enterprise.

(10)

References

Bock, R.D., & Haggard, E.A. (1968). The use of multivariate analysis of variance in behavioral research. In D.K. Whitla (Ed.), Handbook of measurement and assess-

ment in the behavioral sciences (pp. 100-142). Reading, MA: Addison-Wesley.

Brophy, J.E., & Good, T.L. (1986). Teacher behavior and student achievement. In M.C. Wittrock (Ed.), Handbook of research on teaching (3rd ed., pp. 328-375). New York: Macrnillan.

Cheffers, J., Mancini, V., & Martinek, T. (1980). Interaction analysis: An application

to nonverbal activity (2nd ed.). Minneapolis: Association for Productive Teaching.

Dunkin, M.J., & Biddle, B.J. (1974). The study of teaching. New York: Holt, Rinehart & Winston.

Locke, L.F. (1977). Research on teaching physical education: New hope for a dismal science. Quest, 28, 2-16.

Locke, L.F. (1982, September). Research on teaching physical activity: A modest cele-

bration. Paper presented at the VIIth Commonwealth and International Conference

on Sport, Physical Education, Recreation, and Dance, Brisbane, Australia. Locke, L.F. (1984). Research on teaching teachers: Where are we now? Journal of

Teaching in Physical Education, Monograph 2 .

Metzler, M.W., & Freedman, M.S. (1985). Here's looking at you, PETE: A profde of physical education teacher education faculty. Journal of Teaching in Physical Edu-

cation, 4 , 123-133.

Nixon, J., & Locke, L. (1973). Research on teaching in physical education. In M.W. Travers (Ed.), Second handbook of research on teaching (pp. 1210-1242). Chica- go: Rand McNally.

PiCron, M. (1986). Analysis of the research based on observation of the teaching of physical education. In M. Pikron & G . Graham (Eds.), Sport Pedagogy: 1984 Olym- pic Scient$c Congress Proceedings, Volume 6 (pp. 193-202). Champaign, IL: Hu-

man Kinetics.

Rosenshine, B., & Furst, N. (1973). The use of direct observation to study teaching. In M.W. Travers (Ed.), Second handbook of research on teaching @p. 122-183). Chica- go: Rand McNally.

Shaver, J.P. (1983). The verification of independent variables in teaching methods re- search. Educational Researcher, 12(8), 3-9.

Shulman, L.S. (1986). Paradigms and research programs in the study of teaching. In M.C. Wittrock (Ed.), Handbook of research on teaching (3rd ed., pp. 3-36). New York: Macmillan.

Siedentop, D., Birdwell, D., & Metzler, M. (1979, April). Aprocess approach to measuring

teacher effectiveness in physical education. Paper presented at the annual confer-

ence of the AAHPERD, New Orleans.

Siedentop, D., Tousignant, M., & Parker, M. (1982). Academic learning time-physical

education, 1982 revision: Coding manual. Columbus: The Ohio State University,

School of Health, Physical Education and Recreation.

Silverman, S. (1985). Critical considerations in the design and analysis of teacher effec- tiveness research in physical education. International Journal of Physical Educa-