TU Darmstadt / ULB / TUbiblio

Learning to Score System Summaries for Better Content Selection Evaluation

Peyrard, Maxime and Botschen, Teresa and Gurevych, Iryna :
Learning to Score System Summaries for Better Content Selection Evaluation.
[Online-Edition: http://www.aclweb.org/anthology/W17-4510]
Proceedings of the EMNLP workshop "New Frontiers in Summarization" Association for Computational Linguistics
[Conference or Workshop Item] , (2017)

Official URL: http://www.aclweb.org/anthology/W17-4510

Abstract

The evaluation of summaries is a challenging but crucial task of the summarization field. In this work, we propose to learn an automatic scoring metric based on the human judgements available as part of classical summarization datasets like TAC-2008 and TAC-2009. Any existing automatic scoring metrics can be included as features, the model learns the combination exhibiting the best correlation with human judgments. The reliability of the new metric is tested in a further manual evaluation where we ask humans to evaluate summaries covering the whole scoring spectrum of the metric. We release the trained metric as an open-source tool.

Item Type: Conference or Workshop Item
Erschienen: 2017
Creators: Peyrard, Maxime and Botschen, Teresa and Gurevych, Iryna
Title: Learning to Score System Summaries for Better Content Selection Evaluation
Language: English
Abstract:

The evaluation of summaries is a challenging but crucial task of the summarization field. In this work, we propose to learn an automatic scoring metric based on the human judgements available as part of classical summarization datasets like TAC-2008 and TAC-2009. Any existing automatic scoring metrics can be included as features, the model learns the combination exhibiting the best correlation with human judgments. The reliability of the new metric is tested in a further manual evaluation where we ask humans to evaluate summaries covering the whole scoring spectrum of the metric. We release the trained metric as an open-source tool.

Title of Book: Proceedings of the EMNLP workshop "New Frontiers in Summarization"
Publisher: Association for Computational Linguistics
Uncontrolled Keywords: Natural Language Processing;AIPHES_corpus;AIPHES_area_c3;AIPHES_area_b2
Divisions: DFG-Graduiertenkollegs
DFG-Graduiertenkollegs > Research Training Group 1994 Adaptive Preparation of Information from Heterogeneous Sources
Event Location: Copenhagen, Denmark
Event Dates: September 2017
Date Deposited: 04 Jul 2017 10:32
Official URL: http://www.aclweb.org/anthology/W17-4510
Identification Number: TUD-CS-2017-0202
Related URLs:
Export:

Optionen (nur für Redakteure)

View Item View Item