Lu, Sheng ; Hendrik, Schuff ; Gurevych, Iryna (2024)
How are Prompts Different in Terms of Sensitivity?
2024 Conference of the North American Chapter of the Association for Computational Linguistics. Mexico City, Mexico (17-21.06.2024)
Konferenzveröffentlichung, Bibliographie
Kurzbeschreibung (Abstract)
In-context learning (ICL) has become one of the most popular learning paradigms. While there is a growing body of literature focusing on prompt engineering, there is a lack of systematic analysis comparing the effects of prompt techniques across different models and tasks. To address this, we present a comprehensive prompt analysis based on sensitivity. Our analysis reveals that sensitivity is an unsupervised proxy for model performance, as it exhibits a strong negative correlation with accuracy. We use gradient-based saliency scores to empirically demonstrate how different prompts affect the relevance of input tokens to the output, resulting in different levels of sensitivity. Furthermore, we introduce sensitivity-aware decoding which incorporates sensitivity estimation as a penalty term in the standard greedy decoding. We show that this approach is particularly helpful when information in the input is scarce. Our work provides a fresh perspective on the analysis of prompts, and contributes to a better understanding of the mechanism of ICL.
| Typ des Eintrags: | Konferenzveröffentlichung |
|---|---|
| Erschienen: | 2024 |
| Autor(en): | Lu, Sheng ; Hendrik, Schuff ; Gurevych, Iryna |
| Art des Eintrags: | Bibliographie |
| Titel: | How are Prompts Different in Terms of Sensitivity? |
| Sprache: | Englisch |
| Publikationsjahr: | Juni 2024 |
| Ort: | Mexico City, Mexico |
| Verlag: | Association for Computational Linguistics |
| Buchtitel: | Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) |
| Veranstaltungstitel: | 2024 Conference of the North American Chapter of the Association for Computational Linguistics |
| Veranstaltungsort: | Mexico City, Mexico |
| Veranstaltungsdatum: | 17-21.06.2024 |
| URL / URN: | https://aclanthology.org/2024.naacl-long.325 |
| Kurzbeschreibung (Abstract): | In-context learning (ICL) has become one of the most popular learning paradigms. While there is a growing body of literature focusing on prompt engineering, there is a lack of systematic analysis comparing the effects of prompt techniques across different models and tasks. To address this, we present a comprehensive prompt analysis based on sensitivity. Our analysis reveals that sensitivity is an unsupervised proxy for model performance, as it exhibits a strong negative correlation with accuracy. We use gradient-based saliency scores to empirically demonstrate how different prompts affect the relevance of input tokens to the output, resulting in different levels of sensitivity. Furthermore, we introduce sensitivity-aware decoding which incorporates sensitivity estimation as a penalty term in the standard greedy decoding. We show that this approach is particularly helpful when information in the input is scarce. Our work provides a fresh perspective on the analysis of prompts, and contributes to a better understanding of the mechanism of ICL. |
| Freie Schlagworte: | UKP_p_crisp_senpai |
| Fachbereich(e)/-gebiet(e): | 20 Fachbereich Informatik 20 Fachbereich Informatik > Ubiquitäre Wissensverarbeitung |
| Hinterlegungsdatum: | 24 Jun 2024 11:31 |
| Letzte Änderung: | 02 Aug 2024 09:20 |
| PPN: | 520293916 |
| Export: | |
| Suche nach Titel in: | TUfind oder in Google |
 |
Frage zum Eintrag |
Optionen (nur für Redakteure)
 |
Redaktionelle Details anzeigen |
Drucken
Impressum