Automatic evaluation
Paraphrasing for Automatic Evaluation
... Automatic Paraphrasing and Entailment Our work is closely related to research in automatic para- phrasing, in particular, to sentence level paraphras- ing (Barzilay and Lee, 2003; Pang et al., 2003; Quirk ...
8
Comparing Automatic Evaluation Measures for Image Description
... In this paper we estimate the correlation of hu- man judgements with five automatic evaluation measures on two image description data sets. Our work extends previous studies of evaluation mea- sures ...
6
Kernel based Approach for Automatic Evaluation of Natural Language Generation Technologies: Application to Automatic Summarization
... accurate automatic evaluation method that is close to human evalua- ...ing automatic summarization evaluation data developed for Text Summarization Challenge 3 ...
8
Towards Automatic Evaluation of Question/Answering Systems
... semi-automatic evaluation techniques are of great interest for QA for several ...Performance evaluation of such systems re- quires a huge amount of ...the automatic as- sessment of an answer ...
7
GLEU: Automatic Evaluation of Sentence Level Fluency
... of automatic evaluation, we are not aware of any technique that measures only fluency or sim- ilar characteristics, ignoring content, apart from that of Wan et ...
8
ParaMetric: An Automatic Evaluation Metric for Paraphrasing
... The goal of this work is to develop a set of scores that both allows different paraphrasing tech- niques to be compared objectively and provides an easily repeatable method for automatically evalu- ating paraphrases. ...
8
An Assessment of the Accuracy of Automatic Evaluation in Summarization
... standard automatic evaluation procedures com- pare a summary with a set of human gold-standards, it is reasonable to expect that they will be more accu- rate in reproducing results from Pyramid ...
9
Beyond Class A: A Proposal for Automatic Evaluation of Discourse
... Beyond Class A A Proposal for Automatic Evaluation of Discourse B e y o n d Class A A Proposal for A u t o m a t i c Evaluation of Discourse L y n e t t e H i r s c h m a n , D e b o r a h A D a h l ,[.] ...
5
Automatic Evaluation of Information Ordering: Kendall’s Tau
... We next used correlation analysis to explore the linear relationship between sub- jects’ ratings and Kendall’s τ . The comparison yielded a Pearson correlation coefficient of r = 0 . 45 (p < 0 . 01, N = 64). Figure 3 ...
14
Automatic Evaluation of Summary Using Textual Entailment
... manual evaluation is a very hectic process, specially when the summary generated from multiple ...an automatic evaluation method needs to evaluate the ...based automatic evaluation ...
8
A Report on the Automatic Evaluation of Scientific Writing Shared Task
... In this work we have reported and described the re- sults of the AESW Shared Task (Automatic Eval- uation of Scientific Writing), which focuses on the problem of identifying sentences in scientific works that ...
10
Automatic Evaluation of Relation Extraction Systems on Large scale
... In order to scale, a method for evaluation of open RE must have no human involvement. One way to automatically produce a benchmark is to use an ex- isting database as ground truth (Agichtein and Gra- vano, 2000; ...
6
Automatic evaluation of spoken summaries: the case of language assessment
... computed over the output of the automated speech recognizer showed good agreement with expert ratings and performed better than the cosine sim- ilarity measure commonly used for the evaluation content of spoken ...
11
Automatic Evaluation of Commonsense Knowledge for Refining Japanese ConceptNet
... for automatic common sense knowledge evaluation for Japanese entries in ConceptNet ...for evaluation of generality on natural sentences using com- mercial search engines and simpler input: one with ...
8
Bleu: a Method for Automatic Evaluation of Machine Translation
... A candidate translation should be neither too long nor too short, and an evaluation metric should en- force this. To some extent, the n-gram precision al- ready accomplishes this. N-gram precision penal- izes ...
8
'Automatic' evaluation? Strategic effects on affective priming
... Referring to findings in the semantic priming literature (Neely, 1977; but see Hutchison, 2007) it is often argued that affective priming reveals automatic evaluations when stimulus onset asynchrony (SOA) is below ...
21
Automatic Evaluation of Topic Coherence
... We have proposed the novel task of topic coher- ence evaluation as a form of intrinsic topic evalu- ation with relevance in document search/discovery and visualisation applications. We constructed a gold-standard ...
9
Automatic Evaluation of Local Topic Quality
... Once again aiming for a wide variety of topic models for our evaluation, for each of these datasets, we train three types of topic models. As a baseline, we train Latent Dirichlet Alloca- tion (Blei et al., 2003) ...
9
An Information Theoretic Approach to Automatic Evaluation of Summaries
... Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 463–470, New York, June 2006.c©2006 Association for Computational Linguistics.[r] ...
8
EASSE: Easier Automatic Sentence Simplification Evaluation
... We introduce EASSE, a Python package aim- ing to facilitate and standardise automatic evaluation and comparison of Sentence Sim- plification (SS) systems. EASSE provides a single access point to a broad ...
6