TU Darmstadt / ULB / TUbiblio

On the Role of Summary Content Units in Text Summarization Evaluation

Nawrath, Marcel ; Nowak, Agnieszka ; Ratz, Tristan ; Walenta, Danilo C ; Opitz, Juri ; Ribeiro, Leonardo FR ; Sedoc, João ; Deutsch, Daniel ; Mille, Simon ; Liu, Yixin ; Zhang, Lining ; Gehrmann, Sebastian ; Mahamood, Saad ; Clinciu, Miruna ; Chandu, Khyathi ; Hou, Yufang (2024)
On the Role of Summary Content Units in Text Summarization Evaluation.
doi: 10.48550/arXiv.2404.01701
Report, Bibliographie

WarnungEs ist eine neuere Version dieses Eintrags verfügbar.

Kurzbeschreibung (Abstract)

At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs are concise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate summary, possibly partially automated via natural language inference (NLI) systems. Interestingly, with the aim to fully automate the Pyramid evaluation, Zhang and Bansal (2021) show that SCUs can be approximated by automatically generated semantic role triplets (STUs). However, several questions currently lack answers, in particular: i) Are there other ways of approximating SCUs that can offer advantages? ii) Under which conditions are SCUs (or their approximations) offering the most value? In this work, we examine two novel strategies to approximate SCUs: generating SCU approximations from AMR meaning representations (SMUs) and from large language models (SGUs), respectively. We find that while STUs and SMUs are competitive, the best approximation quality is achieved by SGUs. We also show through a simple sentence-decomposition baseline (SSUs) that SCUs (and their approximations) offer the most value when ranking short summaries, but may not help as much when ranking systems or longer summaries.

Typ des Eintrags: Report
Erschienen: 2024
Autor(en): Nawrath, Marcel ; Nowak, Agnieszka ; Ratz, Tristan ; Walenta, Danilo C ; Opitz, Juri ; Ribeiro, Leonardo FR ; Sedoc, João ; Deutsch, Daniel ; Mille, Simon ; Liu, Yixin ; Zhang, Lining ; Gehrmann, Sebastian ; Mahamood, Saad ; Clinciu, Miruna ; Chandu, Khyathi ; Hou, Yufang
Art des Eintrags: Bibliographie
Titel: On the Role of Summary Content Units in Text Summarization Evaluation
Sprache: Englisch
Publikationsjahr: 2 April 2024
Verlag: arXiV
Reihe: Computation and Language
Auflage: 1. Version
DOI: 10.48550/arXiv.2404.01701
Kurzbeschreibung (Abstract):

At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs are concise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate summary, possibly partially automated via natural language inference (NLI) systems. Interestingly, with the aim to fully automate the Pyramid evaluation, Zhang and Bansal (2021) show that SCUs can be approximated by automatically generated semantic role triplets (STUs). However, several questions currently lack answers, in particular: i) Are there other ways of approximating SCUs that can offer advantages? ii) Under which conditions are SCUs (or their approximations) offering the most value? In this work, we examine two novel strategies to approximate SCUs: generating SCU approximations from AMR meaning representations (SMUs) and from large language models (SGUs), respectively. We find that while STUs and SMUs are competitive, the best approximation quality is achieved by SGUs. We also show through a simple sentence-decomposition baseline (SSUs) that SCUs (and their approximations) offer the most value when ranking short summaries, but may not help as much when ranking systems or longer summaries.

Zusätzliche Informationen:

Preprint

Fachbereich(e)/-gebiet(e): 01 Fachbereich Rechts- und Wirtschaftswissenschaften
01 Fachbereich Rechts- und Wirtschaftswissenschaften > Betriebswirtschaftliche Fachgebiete
01 Fachbereich Rechts- und Wirtschaftswissenschaften > Betriebswirtschaftliche Fachgebiete > Wirtschaftsinformatik
01 Fachbereich Rechts- und Wirtschaftswissenschaften > Betriebswirtschaftliche Fachgebiete > Fachgebiet Software Business & Information Management
20 Fachbereich Informatik
20 Fachbereich Informatik > Ubiquitäre Wissensverarbeitung
Hinterlegungsdatum: 23 Apr 2024 10:06
Letzte Änderung: 13 Aug 2024 10:08
PPN: 52061142X
Export:
Suche nach Titel in: TUfind oder in Google

Verfügbare Versionen dieses Eintrags

Frage zum Eintrag Frage zum Eintrag

Optionen (nur für Redakteure)
Redaktionelle Details anzeigen Redaktionelle Details anzeigen