TU Darmstadt / ULB / TUbiblio

IMPLI: Investigating NLI Models’ Performance on Figurative Language

Stowe, Kevin ; Utama, Prasetya ; Gurevych, Iryna (2022)
IMPLI: Investigating NLI Models’ Performance on Figurative Language.
60th Annual Meeting of the Association for Computational Linguistics. Dublin, Ireland (22.-27.05.2022)
Conference or Workshop Item, Bibliographie

Abstract

Natural language inference (NLI) has been widely used as a task to train and evaluate models for language understanding. However, the ability of NLI models to perform inferences requiring understanding of figurative language such as idioms and metaphors remains understudied. We introduce the IMPLI (Idiomatic and Metaphoric Paired Language Inference) dataset, an English dataset consisting of paired sentences spanning idioms and metaphors. We develop novel methods to generate 24k semiautomatic pairs as well as manually creating 1.8k gold pairs. We use IMPLI to evaluate NLI models based on RoBERTa fine-tuned on the widely used MNLI dataset. We then show that while they can reliably detect entailment relationship between figurative phrases with their literal counterparts, they perform poorly on similarly structured examples where pairs are designed to be non-entailing. This suggests the limits of current NLI models with regard to understanding figurative language and this dataset serves as a benchmark for future improvements in this direction.

Item Type: Conference or Workshop Item
Erschienen: 2022
Creators: Stowe, Kevin ; Utama, Prasetya ; Gurevych, Iryna
Type of entry: Bibliographie
Title: IMPLI: Investigating NLI Models’ Performance on Figurative Language
Language: English
Date: 17 May 2022
Publisher: ACL
Book Title: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Event Title: 60th Annual Meeting of the Association for Computational Linguistics
Event Location: Dublin, Ireland
Event Dates: 22.-27.05.2022
URL / URN: https://aclanthology.org/2022.acl-long.369/
Abstract:

Natural language inference (NLI) has been widely used as a task to train and evaluate models for language understanding. However, the ability of NLI models to perform inferences requiring understanding of figurative language such as idioms and metaphors remains understudied. We introduce the IMPLI (Idiomatic and Metaphoric Paired Language Inference) dataset, an English dataset consisting of paired sentences spanning idioms and metaphors. We develop novel methods to generate 24k semiautomatic pairs as well as manually creating 1.8k gold pairs. We use IMPLI to evaluate NLI models based on RoBERTa fine-tuned on the widely used MNLI dataset. We then show that while they can reliably detect entailment relationship between figurative phrases with their literal counterparts, they perform poorly on similarly structured examples where pairs are designed to be non-entailing. This suggests the limits of current NLI models with regard to understanding figurative language and this dataset serves as a benchmark for future improvements in this direction.

Divisions: 20 Department of Computer Science
20 Department of Computer Science > Ubiquitous Knowledge Processing
Date Deposited: 19 May 2022 10:01
Last Modified: 14 Nov 2022 14:31
PPN: 501650954
Export:
Suche nach Titel in: TUfind oder in Google
Send an inquiry Send an inquiry

Options (only for editors)
Show editorial Details Show editorial Details