Nawrath, Marcel ; Nowak, Agnieszka Wiktoria ; Ratz, Tristan ; Walenta, Danilo Constantin ; Opitz, Juri ; Ribeiro, Leonardo F. R. ; Sedoc, João ; Deutsch, Daniel ; Mille, Simon ; Liu, Yixin ; Gehrmann, Sebastian ; Zhang, Lining ; Mahamood, Saad ; Clinciu, Miruna ; Chandu, Khyathi ; Hou, Yufang (2024)
On the Role of Summary Content Units in Text Summarization Evaluation.
2024 Conference of the North American Chapter of the Association for Computational Linguistics. Mexico City, Mexico (16.06.2024 - 21.06.2024)
Konferenzveröffentlichung, Bibliographie
Dies ist die neueste Version dieses Eintrags.
Kurzbeschreibung (Abstract)
At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs areconcise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate summary, possibly partially automated via natural language inference (NLI) systems. Interestingly, with the aim to fully automate the Pyramid evaluation, Zhang and Bansal (2021) show that SCUs can be approximated by automatically generated semantic role triplets (STUs). However, several questions currently lack answers, in particular: i) Are there other ways of approximating SCUs that can offer advantages?ii) Under which conditions are SCUs (or their approximations) offering the most value? In this work, we examine two novel strategiesto approximate SCUs: generating SCU approximations from AMR meaning representations (SMUs) and from large language models (SGUs), respectively. We find that while STUs and SMUs are competitive, the best approximation quality is achieved by SGUs. We also show through a simple sentence-decomposition baseline (SSUs) that SCUs (and their approximations) offer the most value when rankingshort summaries, but may not help as much when ranking systems or longer summaries.
| Typ des Eintrags: | Konferenzveröffentlichung |
|---|---|
| Erschienen: | 2024 |
| Autor(en): | Nawrath, Marcel ; Nowak, Agnieszka Wiktoria ; Ratz, Tristan ; Walenta, Danilo Constantin ; Opitz, Juri ; Ribeiro, Leonardo F. R. ; Sedoc, João ; Deutsch, Daniel ; Mille, Simon ; Liu, Yixin ; Gehrmann, Sebastian ; Zhang, Lining ; Mahamood, Saad ; Clinciu, Miruna ; Chandu, Khyathi ; Hou, Yufang |
| Art des Eintrags: | Bibliographie |
| Titel: | On the Role of Summary Content Units in Text Summarization Evaluation |
| Sprache: | Englisch |
| Publikationsjahr: | Juni 2024 |
| Verlag: | ACL |
| Buchtitel: | The 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Proceedings of the Conference - Volume 2: Short Papers |
| Veranstaltungstitel: | 2024 Conference of the North American Chapter of the Association for Computational Linguistics |
| Veranstaltungsort: | Mexico City, Mexico |
| Veranstaltungsdatum: | 16.06.2024 - 21.06.2024 |
| URL / URN: | https://aclanthology.org/2024.naacl-short.25/ |
| Zugehörige Links: | |
| Kurzbeschreibung (Abstract): | At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs areconcise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate summary, possibly partially automated via natural language inference (NLI) systems. Interestingly, with the aim to fully automate the Pyramid evaluation, Zhang and Bansal (2021) show that SCUs can be approximated by automatically generated semantic role triplets (STUs). However, several questions currently lack answers, in particular: i) Are there other ways of approximating SCUs that can offer advantages?ii) Under which conditions are SCUs (or their approximations) offering the most value? In this work, we examine two novel strategiesto approximate SCUs: generating SCU approximations from AMR meaning representations (SMUs) and from large language models (SGUs), respectively. We find that while STUs and SMUs are competitive, the best approximation quality is achieved by SGUs. We also show through a simple sentence-decomposition baseline (SSUs) that SCUs (and their approximations) offer the most value when rankingshort summaries, but may not help as much when ranking systems or longer summaries. |
| Fachbereich(e)/-gebiet(e): | 01 Fachbereich Rechts- und Wirtschaftswissenschaften 01 Fachbereich Rechts- und Wirtschaftswissenschaften > Betriebswirtschaftliche Fachgebiete 01 Fachbereich Rechts- und Wirtschaftswissenschaften > Betriebswirtschaftliche Fachgebiete > Wirtschaftsinformatik 01 Fachbereich Rechts- und Wirtschaftswissenschaften > Betriebswirtschaftliche Fachgebiete > Fachgebiet Software Business & Information Management 20 Fachbereich Informatik 20 Fachbereich Informatik > Ubiquitäre Wissensverarbeitung |
| Hinterlegungsdatum: | 11 Jul 2024 06:51 |
| Letzte Änderung: | 11 Jul 2024 06:51 |
| PPN: | |
| Export: | |
| Suche nach Titel in: | TUfind oder in Google |
Verfügbare Versionen dieses Eintrags
-
On the Role of Summary Content Units in Text Summarization Evaluation. (deposited 23 Apr 2024 10:06)
- On the Role of Summary Content Units in Text Summarization Evaluation. (deposited 11 Jul 2024 06:51) [Gegenwärtig angezeigt]
 |
Frage zum Eintrag |
Optionen (nur für Redakteure)
 |
Redaktionelle Details anzeigen |
Drucken
Impressum