Waldis, Andreas ; Hou, Yufang ; Gurevych, Iryna (2024)
Dive into the Chasm: Probing the Gap between In- and Cross-Topic Generalization.
18th Conference of the European Chapter of the Association for Computational Linguistics. St. Julian's, Malta (17.-22.03.2024)
Konferenzveröffentlichung, Bibliographie
Kurzbeschreibung (Abstract)
Pre-trained language models (PLMs) perform well in In-Topic setups, where training and testing data come from the same topics. However, they face challenges in Cross-Topic scenarios where testing data is derived from distinct topics. This paper analyzes various PLMs with three probing-based experiments to better understand the reasons behind such generalization gaps. For the first time, we demonstrate that the extent of these generalization gaps and the sensitivity to token-level interventions vary significantly across PLMs. By evaluating large language models (LLMs), we show the usefulness of our analysis for these recent models. Overall, we observe diverse pre-training objectives and architectural regularization contribute to more robust PLMs and mitigate generalization gaps. Our research contributes to a deeper understanding and comparison of language models across different generalization scenarios.
| Typ des Eintrags: | Konferenzveröffentlichung |
|---|---|
| Erschienen: | 2024 |
| Autor(en): | Waldis, Andreas ; Hou, Yufang ; Gurevych, Iryna |
| Art des Eintrags: | Bibliographie |
| Titel: | Dive into the Chasm: Probing the Gap between In- and Cross-Topic Generalization |
| Sprache: | Englisch |
| Publikationsjahr: | März 2024 |
| Verlag: | ACL |
| Buchtitel: | Findings of the Association for Computational Linguistics: EACL 2024 |
| Veranstaltungstitel: | 18th Conference of the European Chapter of the Association for Computational Linguistics |
| Veranstaltungsort: | St. Julian's, Malta |
| Veranstaltungsdatum: | 17.-22.03.2024 |
| URL / URN: | https://aclanthology.org/2024.findings-eacl.146/ |
| Kurzbeschreibung (Abstract): | Pre-trained language models (PLMs) perform well in In-Topic setups, where training and testing data come from the same topics. However, they face challenges in Cross-Topic scenarios where testing data is derived from distinct topics. This paper analyzes various PLMs with three probing-based experiments to better understand the reasons behind such generalization gaps. For the first time, we demonstrate that the extent of these generalization gaps and the sensitivity to token-level interventions vary significantly across PLMs. By evaluating large language models (LLMs), we show the usefulness of our analysis for these recent models. Overall, we observe diverse pre-training objectives and architectural regularization contribute to more robust PLMs and mitigate generalization gaps. Our research contributes to a deeper understanding and comparison of language models across different generalization scenarios. |
| Fachbereich(e)/-gebiet(e): | 20 Fachbereich Informatik 20 Fachbereich Informatik > Ubiquitäre Wissensverarbeitung |
| Hinterlegungsdatum: | 23 Apr 2024 10:04 |
| Letzte Änderung: | 23 Apr 2024 10:04 |
| PPN: | |
| Export: | |
| Suche nach Titel in: | TUfind oder in Google |
 |
Frage zum Eintrag |
Optionen (nur für Redakteure)
 |
Redaktionelle Details anzeigen |
Drucken
Impressum