Petrak, Dominic ; Moosavi, Nafise Sadat ; Gurevych, Iryna (2023)
Arithmetic-Based Pretraining Improving Numeracy of Pretrained Language Models.
Konferenzveröffentlichung, Bibliographie
Kurzbeschreibung (Abstract)
State-of-the-art pretrained language models tend to perform below their capabilities when applied out-of-the-box on tasks that require understanding and working with numbers (usually referred to as numeracy). Recent work suggests two main reasons for this: (1) popular tokenisation algorithms have limited expressiveness for numbers, and (2) common pretraining objectives do not target numeracy. Approaches that address these shortcomings usually require architectural changes or pretraining from scratch. In this paper, we propose a new extended pretraining approach called Arithmetic-Based Pretraining that jointly addresses both in one extended pretraining step without requiring architectural changes or pretraining from scratch. Arithmetic-Based Pretraining combines contrastive learning to improve the number representation, and a novel extended pretraining objective called Inferable Number Prediction Task to improve numeracy. Our experiments show the effectiveness of Arithmetic-Based Pretraining in three different tasks that require improved numeracy, i.e., reading comprehension in the DROP dataset, inference-on-tables in the InfoTabs dataset, and table-to-text generation in the WikiBio and SciGen datasets.
| Typ des Eintrags: | Konferenzveröffentlichung |
|---|---|
| Erschienen: | 2023 |
| Autor(en): | Petrak, Dominic ; Moosavi, Nafise Sadat ; Gurevych, Iryna |
| Art des Eintrags: | Bibliographie |
| Titel: | Arithmetic-Based Pretraining Improving Numeracy of Pretrained Language Models |
| Sprache: | Englisch |
| Publikationsjahr: | 10 Juli 2023 |
| Ort: | Toronto, Canada |
| Verlag: | Association for Computational Linguistics |
| Buchtitel: | Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023) |
| URL / URN: | https://aclanthology.org/2023.starsem-1.42/ |
| Kurzbeschreibung (Abstract): | State-of-the-art pretrained language models tend to perform below their capabilities when applied out-of-the-box on tasks that require understanding and working with numbers (usually referred to as numeracy). Recent work suggests two main reasons for this: (1) popular tokenisation algorithms have limited expressiveness for numbers, and (2) common pretraining objectives do not target numeracy. Approaches that address these shortcomings usually require architectural changes or pretraining from scratch. In this paper, we propose a new extended pretraining approach called Arithmetic-Based Pretraining that jointly addresses both in one extended pretraining step without requiring architectural changes or pretraining from scratch. Arithmetic-Based Pretraining combines contrastive learning to improve the number representation, and a novel extended pretraining objective called Inferable Number Prediction Task to improve numeracy. Our experiments show the effectiveness of Arithmetic-Based Pretraining in three different tasks that require improved numeracy, i.e., reading comprehension in the DROP dataset, inference-on-tables in the InfoTabs dataset, and table-to-text generation in the WikiBio and SciGen datasets. |
| Freie Schlagworte: | UKP_p_crisp_senpai, UKP_p_INCEpTION |
| Fachbereich(e)/-gebiet(e): | 20 Fachbereich Informatik 20 Fachbereich Informatik > Ubiquitäre Wissensverarbeitung |
| Hinterlegungsdatum: | 30 Aug 2023 11:48 |
| Letzte Änderung: | 30 Aug 2023 11:59 |
| PPN: | 511164742 |
| Export: | |
| Suche nach Titel in: | TUfind oder in Google |
 |
Frage zum Eintrag |
Optionen (nur für Redakteure)
 |
Redaktionelle Details anzeigen |
Drucken
Impressum