Doctoral assessment: process, issues and the concept of originality

34 

Loading....

Loading....

Loading....

Loading....

Loading....

Full text

(1)

Doctoral assessment:

process, issues and the concept of originality

Gill Clarke

Department of Education, University of Oxford Vice-chair, UK Council for Graduate Education

European Universities Association 7th EUA-CDE Workshop

Dokuz Eylül University, Izmir 23-24 January 2014

Theme:

Outcomes of Doctoral Education – Mindset, Research, Innovation

(2)

Summary

• Evolution of the PhD as a ‘global brand’

• Different assessment models: are they

comparable, do they affect outcomes, do differences matter?

• Paper on the UK PhD: examines concept of

‘originality’ in doctoral assessment and its interpretation at subject level

(3)

Bologna Declaration and earlier

• German/Prussian PhD influenced development of all doctorates (17th century)

• ‘Adoption of a system of easily readable and

comparable degrees’ (Bologna Declaration,

19.06.99)

• PhD as a qualification was way ahead: a ‘global brand’ for

around a century

Wilhelm von Humboldt ________________________________________________________________________

Alexander von Humboldt

References: Noble (1994:6);, QAA (2012: 31); Stanford Encyclopaedia of Philosophy [online]

(4)

A variety of assessment models

1

Similarities

• Thesis (or equivalent) common to all?

• Viva or defence – used in all countries except Australia2?

• Formal requirement for

‘originality’ or ‘contribution to knowledge’

• Licence to become an academic practitioner?

Differences

• Timing and nature of

disclosure of final outcome to candidate

• Nature of the oral defence: public, private, or none

• Number of examiners

• Whether or not supervisor can be present

• Requirement for professional practice in some subjects

• Pass/fail or graded

1Kyvik, S. (2014) Assessment procedures of Norwegian PhD theses as viewed by examiners

from the USA, the UK and Sweden. Assessment & Evaluation in HE, 39:2, 140-153

2Group of Eight (2013) The Changing PhD: discussion paper [Online] :

http://www.go8.edu.au/university-staff/go8-policy-_and_-analysis/2013/the-changing-phd

(5)

Assessment of the PhD in the UK

• Judgement of thesis plus viva

voce in all cases

A private process – no public

defence

• At least two examiners, sometimes three

Independent chair/convenor

may be present

• Supervisor may attend with candidate’s permission

Length of viva: 1.25 – 3.5

hours, depending on subject

(6)

Some topical issues

• Do doctoral graduates need more skills than those they use in research?

• Education, or training for employment, or both?

• Importance of doctorate in universities’ research effort and knowledge exchange ?3

________________________________________________________________________

3Moreno-Navarro, J. J. (2010). New Regulation for Doctoral Studies in Spain: presentation at 3rd annual

meeting of EUA-CDE, Berlin, June 2010. [Online] Available from:

http://www.eua.be/events/past/2010/third-eua-cde-annual-meeting/Presentations.aspx

(7)

Assessment of the PhD: ‘Originality’ and its interpretation

The concept of ‘originality’ in the PhD: how is it

interpreted by examiners?

• Joint authors: Gillian Clarke and Ingrid Lunt

• Taylor and Francis online, recent articles: published 02.01.14

• Explores ways examiners and others interpret the concept of originality when judging candidates’ achievements in the final PhD exam

• Compares two data sets

______________________________________________________________________

Clarke, G. and Lunt, I. (2014) The concept of ‘originality ‘ in the PhD: how is it interpreted by examiners? Assessment and Evaluation in Higher Education. Published online 02.01.14

(8)

Structure and content

• Introduction

• Originality and how it is addressed in the literature

• UK and international doctoral qualification descriptors

• Responses to a 2007 discussion paper (QAA)

• Emerging data from a PhD study

• Conclusions

(9)

Literature (1)

Australian Qualifications Framework Council (2013)

Australian Qualifications Framework. 2nd edition.

Bourke, S. and Holbrook, A. (2013) Examining PhD and Research Masters Theses. Assessment and Evaluation in Higher Education

Delamont, S., Atkinson, P. and Parry, O. (2000) The Doctoral Experience: Success and Failure in Graduate School

Denicolo, P.M. (2003) Assessing the PhD: a constructive view of criteria. Quality Assurance in Education

Johnston, S. (1997) Examining the examiners: An analysis of examiners’ reports on doctoral theses. Studies in Higher

Education

Lovitts, B. (2007) Making the Implicit Explicit: creating performance expectations for the dissertation.

(10)

Literature (2)

Mullins, G. and Kiley, M. (2002) ‘It’s a PhD, not a Nobel

Prize’: How experienced examiners assess research theses,

Studies in Higher Education

Ostriker, Jeremiah P., Holland, Paul W., Kuh,Charlotte V. and Voytuk, James A.(eds) (2010) A Data-Based Assessment of Research-Doctorate programmes in the United States

Quality Assurance Agency for Higher Education (2001) The framework for higher education qualifications in Scotland. Glasgow: QAA.

Quality Assurance Agency for Higher Education (2008) The framework for higher education qualifications in England, Wales and Northern Ireland. Gloucester: QAA.

(11)

Literature (3)

Tinkler, P. and Jackson, C. (2000) Examining the

Doctorate: institutional policy and the PhD examination process in Britain, Studies in Higher Education

Tinkler, P. and Jackson, C. (2004) The Doctoral Examination Process

Trafford, V (2002) Questions in a Doctoral Viva: Views from the Inside. Paper presented at the UK Council for Graduate Education Research Degree Examining

Symposium

George E. Walker, Chris M. Golde, Laura Jones, Andrea Conklin Bueschel and Pat Hutchings (2008), The

Formation of Scholars: Rethinking Doctoral Education for the Twenty-First Century

(12)

Formal guidance and criteria: FHEQ and FQHES UK doctoral qualification descriptor:

• Creation and interpretation of new

knowledge, through original research or other advanced scholarship

• Systematic acquisition and understanding of a substantial body of knowledge at the forefront of an academic discipline or area of

professional practice

• General ability to conceptualise, design and

implement a project for the generation of new knowledge…at the forefront of the discipline

(13)

2007 discussion paper about doctoral degrees

Total of 72 respondents, 65 of whom (90%) answered Question 8: How do you/does your institution define ‘originality’ in the context of doctoral study?

_______________________________________________________________________________

Group (a): 31 (43%) of respondents provided their own

definitions of originality

Group (b): 16 (22%) linked originality to publishability

Group (c): 13 (18%) said definitions of originality should be discipline-specific

Group (d): 10 (14%) did not wish to define the concept of originality

Group (e): 3 (4%) emphasised the importance of a common

understanding of originality

Group (f): 2 (3%) wanted a reference to originality to remain within the doctoral qualification descriptor

(14)

Group (a) – some of the definitions of originality:

‘a contribution to knowledge, specifically, the extent to which the candidate’s work provides insights into and increases understanding of their field’

‘new knowledge/discovery of new facts arising from an individual’s research or creativity’

‘the application of existing knowledge in a way that

provides new insights into the subject, e.g. through using different approaches or methodology’

‘forms a distinct contribution to knowledge of the subject and affords evidence of originality by the discovery of new facts and/or by the exercise of critical power’

‘the ability to think independently, find solutions to

difficulties and offer fresh insights into existing situations’

(15)

Group (b) – respondents linking originality to publishability

‘[Originality] is a red herring. The key is ‘publishable’ – and we are not talking of format but content.’

‘Work that meets international standards of published research’

‘…making a novel contribution to the subject such that it is likely to be publishable in an appropriate journal…’ Other comments: the outcomes of a doctorate should include a requirement to produce work of publishable quality; it should be clear examiners are looking for

thesis containing work of this quality rather than having the expectation the thesis itself should be publishable; a basic quantifier for the quality of the PhD is how

many publications or papers come out of the research

(16)

Group (c): definitions of originality should be subject-specific

Various views:

• What expectations and assessment criteria are used in different disciplines?

• Defining originality is a matter for disciplines, not institutions, because the contribution is to the body of knowledge in the relevant subject Differences between STEM and arts, humanities and social sciences: in STEM, originality often

linked to publication during PhD, whereas in arts etc. more emphasis may be placed on an original research concept / intellectual originality?

(17)

Groups (d), (e) and (f)

• Group (d): institutional-level definition of

originality not appropriate; originality should only be interpreted at subject level

• Group (e): emphasised the need for a common understanding of originality, especially for new examiners

• Group (f): reference to originality must remain in the FHEQ/FQHES doctoral qualification

descriptor

(18)

Emerging data from the PhD study • Title: A study of how examiners judge the

achievement of PhD candidates in the final

examination: perspectives, process and outcomes • How do examiners in a range of subjects make

judgements when they assess PhD candidates (thesis and viva)?

• Possible reference points used: subject custom and practice; examiners’ previous experience; external criteria (institutional and other)

(19)

Case study design

Stage 1: Negotiate observations and/or interviews. with participating

universities. Obtain copies of

assessment regulations governing the procedure, including any relevant documents used by the institution to guide examiners, candidates and independent chairs, if used.

The case studies

Case studies were carried out on a phenomenon: the doctoral examination process. This design is based on a triangulation approach to each case study, using research methods sequentially.

Stage 3: analyse interview transcripts and examiners’ reports linked to the oral examinations . Note any similarities and differences in the examiners’ decision-making and between university

regulations and guidance.

Stage 2: Observe and take notes

at two to three vivas at five universities. Conduct interviews with candidates, supervisors and examiners (and independent chair, if in attendance), preferably just after final PhD examinations.* Request examiners’ reports for relevant examinations.

Triangulation

(20)

Progress with case studies and individual interviews

Candidate Examiner Int Examiner

Ext Supervisor Independent Chair/ Convenor Non-case study examiners University 1 1: SS 1: SS 1: SS 2: SS - University 2 2: B, A 1: B 2: A, B 1: A 2: A, B

University 3 tbi 1: SS tbi tbi - 1: MB

University 4 1: E

University 5 1: MB

Key A = arts MB = molecular biosciences B = biological sciences SS= social sciences

(21)

Questions for examiners

As an examiner, what attributes/

characteristics/abilities/skills are you seeking in PhD candidates? For example, what

questions did you have in mind when

considering the recent candidate’s work (thesis or equivalent) and during the viva? Did you

benchmark the person with other candidates you’ve examined? To what extent, if at all, did you have in mind any external criteria

(including guidance at subject level)?

(22)

Questions for supervisors and independent chairs

How do you think examiners decide whether a

candidate has or has not achieved what is necessary to be awarded a PhD (thesis and viva)? What questions do they have in mind? What attributes /characteristics

/abilities/skills do you think examiners are looking for in the candidate? Do examiners benchmark the person with others they have examined? How do they make their

judgement(s)?

(23)

Questions for candidates

Before the final assessment of your thesis, and

before the viva, what did you think you would have to do to be awarded a PhD? For example, on what basis did you think examiners would be making

their judgements, about your thesis and about you as an individual researcher? What formal

guidance, if any, are you aware of that suggests what examiners should take into account? And what attributes /characteristics/abilities/skills do you think examiners are looking for in PhD

candidates?

(24)

Overview of responses

• Participants’ refer to the question of ‘orginality’, which is often mentioned in the context of ‘an original contribution’ (Lovitts, 2007: 32)

• They address what ‘originality’ and/or ‘a

contribution to the field’ means for their own subject

• They make it clear that the concept of original work or a contribution to knowledge is only one of the criteria examiners use to assess PhD

candidates

(25)

Categories of responses to questions Eight groups:

• Originality and/or a contribution to knowledge • Academic level and intellectual rigour

• Quality of data and its analysis • Methodological approach

• Knowledge and understanding (of the student’s own work and the field of study)

• Publication and publishability

• Candidate’s ability to analyse their own work critically and to defend it

• Quality of thesis and ownership of the work

(26)

Originality/contribution to knowledge

• To fulfil criteria set by the university, e.g. an

original contribution

• What is the candidate’s contribution to the field and does s/he have a grasp of the body of literature?

• Has the candidate generated new knowledge

(produced something not done before or added to the understanding in/made a contribution to the field)

• Is it an original contribution and does it tell a coherent story?

(27)

Academic level/rigour and quality of data

Academic level/rigour

• Is the work worthy of a PhD – has the candidate created something the examiners can pass? • Does the candidate

show intellectual rigour; does the evidence and argument stand up to challenge?

Quality of data and analysis

• What is the quality of data and its analysis and is there enough in-depth analysis?

• Has the candidate carried out the right statistical

analysis and have they interpreted their data correctly?

(28)

Methodological approach

• How has the candidate conceptualised the research idea and is the methodology

appropriate and rigorous?

• Can the student defend their methodological choices and argue why they have chosen the methodology used rather than a different

approach; have they considered the alternatives and does their approach withstand scrutiny?

• Has the student understood what they’ve done and can they justify their approach?

(29)

Knowledge and understanding

• What is the candidate’s understanding and

awareness of the broader field and the literature? • Have they produced a body of work that is

integrated in the field and shows how the literature has a bearing on what they’ve done?

• Does the candidate understand the importance of controls in their experiments and have they been repeated to check validity and reliability?

• Is the candidate’s coverage and review of the literature sufficient and appropriate? Does the review demonstrate insight rather than just the

ability to summarise? Have the research questions emerged from the literature review?

(30)

Publication and publishability

• Has the candidate already published in peer-reviewed publications?

• Is the work ‘publishable’ – of publishable quality?

• In some subjects, expectation that the

candidate will present portfolio of published papers as the thesis

(31)

Candidate’s ability to analyse and critique own work

• Is the candidate aware of any weaknesses in their approach and have they considered alternative interpretations?

• At the viva, can the candidate engage in

intellectual defence of the thesis and can s/he cope with unexpected questions?

• Can the candidate think analytically, draw on theory when analysing their research and does s/he bring to bear on it a wider knowledge of the field?

• Does the candidate have the ability to critically appraise their work within the field?

(32)

Quality of thesis and ownership of the work

• Is the student’s work independent – their own

rather than their supervisor’s, using ideas they’ve largely developed themselves?

• Has the candidate done the work and do they acknowledge others’ contributions

appropriately?

• Has the candidate been honest: do they have integrity?

• Have the papers cited in the thesis been read? • What is the overall quality of the thesis:

presentation and writing?

(33)

Candidates’ responses

• One candidate did not mention

originality in the PhD, instead focusing on the quality of research being

assessed

Another said ‘I think the only criterion

I was really aware of was originality, that you had to make some

substantive contribution to knowledge

• The third candidate confirmed s/he was aware the research had to be original because of guidance by the university, which defined originality in terms of a contribution to knowledge

_____________________________________________________________________________

(34)

Conclusions / questions

Is ‘originality’ the same as ‘a contribution to

knowledge’ or are they different?

What is a ‘significant’ contribution to knowledge?

• Are there ‘degrees’ of originality and do they relate to the candidate’s overall achievement level?

• How does originality / a contribution to knowledge relate to publishability?

Do we agree that ‘originality’ can only be defined at

subject level; should we try to define the way iit is interpreted by different subjects?

Do all PhD candidates who are awarded the degree

meet the ‘originality’ / ‘contribution to knowledge’ criterion, as well as criteria?

Figure

Updating...

Related subjects :