TU Darmstadt / ULB / TUbiblio

AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification

Huang, Yongxin ; Wang, Kexin ; Dutta, Sourav ; Patel, Raj Nath ; Glavaš, Goran ; Gurevych, Iryna (2023)
AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification.
2023 Conference on Empirical Methods in Natural Language Processing. Singapore (06.-10.12.2023)
doi: 10.18653/v1/2023.emnlp-main.208
Conference or Workshop Item, Bibliographie

Abstract

Recent work has found that few-shot sentence classification based on pre-trained Sentence Encoders (SEs) is efficient, robust, and effective. In this work, we investigate strategies for domain-specialization in the context of few-shot sentence classification with SEs. We first establish that unsupervised Domain-Adaptive Pre-Training (DAPT) of a base Pre-trained Language Model (PLM) (i.e., not an SE) substantially improves the accuracy of few-shot sentence classification by up to 8.4 points. However, applying DAPT on SEs, on the one hand, disrupts the effects of their (general-domain) Sentence Embedding Pre-Training (SEPT). On the other hand, applying general-domain SEPT on top of a domain-adapted base PLM (i.e., after DAPT) is effective but inefficient, since the computationally expensive SEPT needs to be executed on top of a DAPT-ed PLM of each domain. As a solution, we propose AdaSent, which decouples SEPT from DAPT by training a SEPT adapter on the base PLM. The adapter can be inserted into DAPT-ed PLMs from any domain. We demonstrate AdaSent’s effectiveness in extensive experiments on 17 different few-shot sentence classification datasets. AdaSent matches or surpasses the performance of full SEPT on DAPT-ed PLM, while substantially reducing the training costs. The code for AdaSent is available.

Item Type: Conference or Workshop Item
Erschienen: 2023
Creators: Huang, Yongxin ; Wang, Kexin ; Dutta, Sourav ; Patel, Raj Nath ; Glavaš, Goran ; Gurevych, Iryna
Type of entry: Bibliographie
Title: AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification
Language: English
Date: December 2023
Place of Publication: Singapore
Publisher: Association for Computational Linguistics
Book Title: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Event Title: 2023 Conference on Empirical Methods in Natural Language Processing
Event Location: Singapore
Event Dates: 06.-10.12.2023
DOI: 10.18653/v1/2023.emnlp-main.208
URL / URN: https://aclanthology.org/2023.emnlp-main.208
Abstract:

Recent work has found that few-shot sentence classification based on pre-trained Sentence Encoders (SEs) is efficient, robust, and effective. In this work, we investigate strategies for domain-specialization in the context of few-shot sentence classification with SEs. We first establish that unsupervised Domain-Adaptive Pre-Training (DAPT) of a base Pre-trained Language Model (PLM) (i.e., not an SE) substantially improves the accuracy of few-shot sentence classification by up to 8.4 points. However, applying DAPT on SEs, on the one hand, disrupts the effects of their (general-domain) Sentence Embedding Pre-Training (SEPT). On the other hand, applying general-domain SEPT on top of a domain-adapted base PLM (i.e., after DAPT) is effective but inefficient, since the computationally expensive SEPT needs to be executed on top of a DAPT-ed PLM of each domain. As a solution, we propose AdaSent, which decouples SEPT from DAPT by training a SEPT adapter on the base PLM. The adapter can be inserted into DAPT-ed PLMs from any domain. We demonstrate AdaSent’s effectiveness in extensive experiments on 17 different few-shot sentence classification datasets. AdaSent matches or surpasses the performance of full SEPT on DAPT-ed PLM, while substantially reducing the training costs. The code for AdaSent is available.

Uncontrolled Keywords: UKP_p_MISRIK, UKP_p_HUAWEI
Divisions: 20 Department of Computer Science
20 Department of Computer Science > Ubiquitous Knowledge Processing
Date Deposited: 18 Jan 2024 13:49
Last Modified: 19 Mar 2024 17:40
PPN: 516403923
Export:
Suche nach Titel in: TUfind oder in Google
Send an inquiry Send an inquiry

Options (only for editors)
Show editorial Details Show editorial Details