Source text evaluation: a
Source text evaluation: a
case study and its
case study and its
theoretical explanation
theoretical explanation
Holistic evaluation or quantifiable criteria? Holistic evaluation or quantifiable criteria?
Florika
Florika GriessnerGriessner Heike
Heike LambergerLamberger-Felber-Felber Universit
Why source text evaluation?
Why source text evaluation?
ST are a decisive factor in interpreter
ST are a decisive factor in interpreter
performance
performance
Comparability of ST difficulty is
Comparability of ST difficulty is
indispensable for meaningful
indispensable for meaningful
assessment
assessment
•
• If ST is too difficult, assessment is If ST is too difficult, assessment is impossible
impossible
Variability in ST evaluation may be
Variability in ST evaluation may be
relevant for performance assessment
relevant for performance assessment
Source text evaluation
Source text evaluation
A case study
A case study
Corpus:
Corpus:
•
• 3 speeches (EN) 3 speeches (EN) •
• 12 professional conference interpreters 12 professional conference interpreters (German A)
(German A) •
• ENEN--DE interpretationsDE interpretations •
• 3 working conditions3 working conditions
Without text (O)Without text (O)
With text, but without preparation (T)With text, but without preparation (T)
Aim of the case study
Aim of the case study
Does availability of speaker
Does availability of speaker
´
´
s
s
manuscript influence interpreting
manuscript influence interpreting
performance?
performance?
•
• Semantic deviationsSemantic deviations •
• Time lagTime lag •
• Linguistic variabilityLinguistic variability •
• Length of interpretationsLength of interpretations •
Experimental setup
Experimental setup
Group A
Group A Group BGroup B Group CGroup C Speech Speech SISI TVTV TT OO Speech Speech HIHI TT OO TVTV Speech Speech SLSL OO TVTV TT
Case study
Case study
Experimental setup requires comparable Experimental setup requires comparable input speeches
input speeches
““objectiveobjective”” comparability of 3 speeches comparability of 3 speeches o.k. (
o.k. (PPöchhackeröchhacker‘‘ss text profile)text profile)
•
• ReadRead--out vs. freeout vs. free •
• Speed of deliverySpeed of delivery •
• Intonation/rhythm/dynamicsIntonation/rhythm/dynamics •
• Voice qualityVoice quality •
Case study
Case study
Comparison of
Comparison of
“
“
objective
objective
”
”
evaluation
evaluation
of source texts and
of source texts and
“
“
subjective
subjective
”
”
evaluation by 12 interpreters
evaluation by 12 interpreters
•
• Questionnaire after interpreting each Questionnaire after interpreting each speech
speech
ReadRead--out out vsvs freefree
Overall difficulty of speechOverall difficulty of speech SpeedSpeed TerminologyTerminology Articulation/accentArticulation/accent
Speech
Speech HIHI SpeedSpeed Term.Term. AccentAccent A1 A1 33 22 22 A2 A2 44 22 11 A3 A3 33 11 11 A4 A4 44 44 44 B1 B1 44 22 22 B2 B2 44 11 22 B3 B3 33 22 44 B4 B4 44 22 11 C1 C1 22 11 11 C2 C2 44 22 22 C3 C3 44 22 11 C4 C4 33 11 11
Speech
Speech SLSL SpeedSpeed Term.Term. AccentAccent A1 A1 33 22 33 A2 A2 44 22 44 A3 A3 44 11 22 A4 A4 44 44 44 B1 B1 44 22 22 B2 B2 33 11 33 B3 B3 33 22 44 B4 B4 33 11 33 C1 C1 22 11 11 C2 C2 33 11 11 C3 C3 22 22 11 C4 C4 33 22 33
Difficulty
Difficulty SpeechSpeech SISI SpeechSpeech HIHI SpeechSpeech SLSL A1 A1 11 22 22 A2 A2 22 44 44 A3 A3 11 22 22 A4 A4 22 44 44 B1 B1 22 22 22 B2 B2 22 11 33 B3 B3 11 33 44 B4 B4 11 22 33 C1 C1 11 22 11 C2 C2 22 33 22 C3 C3 22 22 33 C4 C4 11 22 44
Case study
Case study
-
-
results
results
Ranking of speeches/difficulty
Ranking of speeches/difficulty
• • 1. Speech SI (100%)1. Speech SI (100%) • • 2. Speech HI (75%)2. Speech HI (75%) • • 3. Speech SL (75%)3. Speech SL (75%)
Case study
Case study
-
-
results
results
Objective comparability of input text Objective comparability of input text parameters does not assure subjective parameters does not assure subjective
comparability from interpreters
comparability from interpreters’’ viewpointviewpoint
subjective evaluation of different input subjective evaluation of different input parameters is highly variable
parameters is highly variable But
But
In spite of variability in evaluation of In spite of variability in evaluation of
individual parameters, overall evaluation individual parameters, overall evaluation
of difficulty is homogeneous of difficulty is homogeneous
A spontaneous explanation
A spontaneous explanation
Similarities in overall evaluation
Similarities in overall evaluation
•
• Professional expertise (expert Professional expertise (expert knowledge > holistic approach) knowledge > holistic approach)
Differences in individual parameter
Differences in individual parameter
evaluation:
evaluation:
•
• Different ways of interpreting itemsDifferent ways of interpreting items •
• Different ways of ranking individual Different ways of ranking individual parameters
parameters
Further research Further research
Holistic approaches in TS
Holistic approaches in TS
Early Translation studies: Early Translation studies:
•
• Hermeneutic approach to texts, Hermeneutic approach to texts, oversummativity
oversummativity (e.g. (e.g. PaepckePaepcke, 1986) , 1986) •
• PrototypologyPrototypology: (Snell: (Snell--HornbyHornby)) •
• Snell-Snell-HornbyHornby: : “Text “Text alsals Gestalt”Gestalt” (1992)(1992)
Less holistic approaches in interpreting Less holistic approaches in interpreting studies:
studies:
•
• cognition, information processing, pattern cognition, information processing, pattern recognition, expertise (e.g.
Gestalt Theory
Gestalt Theory
Holistic theory of Holistic theory of perception and perception and information information processing: processing: •
• Figure andFigure and ground ground (relevance) (relevance) ¾ ¾Relativity of Relativity of assessment assessment process process (context
Gestalt Theory
Gestalt Theory
“The whole is “The whole is more more
than and different
than and different
from
from the individual the individual parts
parts””
The human brain The human brain
tends to organize tends to organize information in
information in meaningful sets,
meaningful sets, even even if parts of the if parts of the information are information are missing. missing.
In addition to the basic principles
In addition to the basic principles
GestaltistsGestaltists ((Wertheimer, Wertheimer, KofkaKofka, , KKööhlerhler) )
discovered and demonstrated that: discovered and demonstrated that:
•
• FIRST we perceive the whole and then the FIRST we perceive the whole and then the single parts
single parts •
• The whole is something like the The whole is something like the ““intrinsic intrinsic nature of things
nature of things”” •
• Subjective factors are not considered Subjective factors are not considered
disturbing factors, they are important for disturbing factors, they are important for
perception perception
Gestalt principles in EMCI context
Gestalt principles in EMCI context
Implicit consensus among EMCI
Implicit consensus among EMCI
schools already in place
schools already in place
Prevalent criteria for source texts
Prevalent criteria for source texts
are:
are:
•
• Not read, spontaneous presentationNot read, spontaneous presentation •
• Coherent development of argumentsCoherent development of arguments •
• General topic (consensus)General topic (consensus)
Speeches are custom made for EMCI
Speeches are custom made for EMCI
exam situations
exam situations
Conclusions
Conclusions
Evaluation of individual parameters
Evaluation of individual parameters
does not necessarily lead to the
does not necessarily lead to the
same conclusions as overall
same conclusions as overall
evaluation
evaluation
If this is true for source texts, it may
If this is true for source texts, it may
also be relevant for target text
also be relevant for target text
evaluation
evaluation
Holistic criteria may thus be useful
Holistic criteria may thus be useful
for both choice of ST and assessment
for both choice of ST and assessment
of interpretations
of interpretations
Conclusions: Target text
Conclusions: Target text
assessment
assessment
Assessment of the Assessment of the ““target text Gestalttarget text Gestalt””
rather than on the basis of isolated target rather than on the basis of isolated target
text parameters text parameters
Isolation of target text parameters highly Isolation of target text parameters highly useful for didactic assessment
useful for didactic assessment (e.g.interims)
(e.g.interims)
Objectivity in holistic assessment assured Objectivity in holistic assessment assured by
by
•
• composition and size of exam panel and composition and size of exam panel and qualifications of panel members
qualifications of panel members •
Conclusions
Conclusions
Holistic peer assessment as
Holistic peer assessment as
commonly practised is justifiable
commonly practised is justifiable
from a scientific point of view
from a scientific point of view
