Tunstall, Lewis ; Reimers, Nils ; Jo, Unso Eun Seo ; Bates, Luke ; Korat, Daniel ; Wasserblat, Moshe ; Pereg, Oren (2022)
Efficient Few-Shot Learning Without Prompts.
2nd Workshop on Efficient Natural Language and Speech Processing. New Orleans, USA (02.12.2022-02.12.2022)
Konferenzveröffentlichung, Bibliographie
Kurzbeschreibung (Abstract)
Recent few-shot learning methods, such as parameter-efficient fine-tuning (PEFT) and pattern exploiting training (PET), have achieved impressive results in label scarce settings. However, they are difficult to employ since they are highly sensitive to handcrafted prompts, and typically require billion-parameter language models to achieve high accuracy. To address these shortcomings, we propose SETFIT (Sentence Transformer Fine-tuning), an efficient and prompt-free framework for few-shot fine-tuning of Sentence Transformers (ST). SETFIT works by first finetuning a pretrained ST on a small number of labeled text pairs, in a contrastive Siamese manner. The resulting model is then used to generate rich text embeddings, which are used to train a classification head. This simple framework requires no prompts or verbalizers, and achieves high accuracy with orders of magnitude less parameters and runtime than existing techniques. Our experiments show that SETFIT1 achieves results competitive with PEFT and PET techniques, and outperforms them on a variety of classification tasks.
| Typ des Eintrags: | Konferenzveröffentlichung |
|---|---|
| Erschienen: | 2022 |
| Autor(en): | Tunstall, Lewis ; Reimers, Nils ; Jo, Unso Eun Seo ; Bates, Luke ; Korat, Daniel ; Wasserblat, Moshe ; Pereg, Oren |
| Art des Eintrags: | Bibliographie |
| Titel: | Efficient Few-Shot Learning Without Prompts |
| Sprache: | Englisch |
| Publikationsjahr: | 29 November 2022 |
| Veranstaltungstitel: | 2nd Workshop on Efficient Natural Language and Speech Processing |
| Veranstaltungsort: | New Orleans, USA |
| Veranstaltungsdatum: | 02.12.2022-02.12.2022 |
| URL / URN: | https://neurips2022-enlsp.github.io/index.html |
| Kurzbeschreibung (Abstract): | Recent few-shot learning methods, such as parameter-efficient fine-tuning (PEFT) and pattern exploiting training (PET), have achieved impressive results in label scarce settings. However, they are difficult to employ since they are highly sensitive to handcrafted prompts, and typically require billion-parameter language models to achieve high accuracy. To address these shortcomings, we propose SETFIT (Sentence Transformer Fine-tuning), an efficient and prompt-free framework for few-shot fine-tuning of Sentence Transformers (ST). SETFIT works by first finetuning a pretrained ST on a small number of labeled text pairs, in a contrastive Siamese manner. The resulting model is then used to generate rich text embeddings, which are used to train a classification head. This simple framework requires no prompts or verbalizers, and achieves high accuracy with orders of magnitude less parameters and runtime than existing techniques. Our experiments show that SETFIT1 achieves results competitive with PEFT and PET techniques, and outperforms them on a variety of classification tasks. |
| Freie Schlagworte: | UKP_p_seditrah_factcheck |
| Fachbereich(e)/-gebiet(e): | 20 Fachbereich Informatik 20 Fachbereich Informatik > Ubiquitäre Wissensverarbeitung Zentrale Einrichtungen Zentrale Einrichtungen > hessian.AI - Hessisches Zentrum für Künstliche Intelligenz |
| Hinterlegungsdatum: | 06 Jun 2023 10:27 |
| Letzte Änderung: | 20 Dez 2023 10:20 |
| PPN: | |
| Export: | |
| Suche nach Titel in: | TUfind oder in Google |
 |
Frage zum Eintrag |
Optionen (nur für Redakteure)
 |
Redaktionelle Details anzeigen |
Drucken
Impressum