Liu, Chen ; Koto, Fajri ; Baldwin, Timothy ; Gurevych, Iryna (2024)
Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings.
2024 Conference of the North American Chapter of the Association for Computational Linguistics. Mexico City, Mexico (17-21.06.2024)
Konferenzveröffentlichung, Bibliographie
Kurzbeschreibung (Abstract)
Large language models (LLMs) are highly adept at question answering and reasoning tasks, but when reasoning in a situational context, human expectations vary depending on the relevant cultural common ground. As languages are associated with diverse cultures, LLMs should also be culturally-diverse reasoners. In this paper, we study the ability of a wide range of state-of-the-art multilingual LLMs (mLLMs) to reason with proverbs and sayings in a conversational context. Our experiments reveal that: (1) mLLMs “know” limited proverbs and memorizing proverbs does not mean understanding them within a conversational context; (2) mLLMs struggle to reason with figurative proverbs and sayings, and when asked to select the wrong answer (instead of asking it to select the correct answer); and (3) there is a “culture gap” in mLLMs when reasoning about proverbs and sayings translated from other languages. We construct and release our evaluation dataset MAPS (MulticulturAl Proverbs and Sayings) for proverb understanding with conversational context for six different languages.
Typ des Eintrags: | Konferenzveröffentlichung |
---|---|
Erschienen: | 2024 |
Autor(en): | Liu, Chen ; Koto, Fajri ; Baldwin, Timothy ; Gurevych, Iryna |
Art des Eintrags: | Bibliographie |
Titel: | Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings |
Sprache: | Englisch |
Publikationsjahr: | Juni 2024 |
Ort: | Mexico City, Mexico |
Buchtitel: | Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) |
Veranstaltungstitel: | 2024 Conference of the North American Chapter of the Association for Computational Linguistics |
Veranstaltungsort: | Mexico City, Mexico |
Veranstaltungsdatum: | 17-21.06.2024 |
URL / URN: | https://aclanthology.org/2024.naacl-long.112/ |
Kurzbeschreibung (Abstract): | Large language models (LLMs) are highly adept at question answering and reasoning tasks, but when reasoning in a situational context, human expectations vary depending on the relevant cultural common ground. As languages are associated with diverse cultures, LLMs should also be culturally-diverse reasoners. In this paper, we study the ability of a wide range of state-of-the-art multilingual LLMs (mLLMs) to reason with proverbs and sayings in a conversational context. Our experiments reveal that: (1) mLLMs “know” limited proverbs and memorizing proverbs does not mean understanding them within a conversational context; (2) mLLMs struggle to reason with figurative proverbs and sayings, and when asked to select the wrong answer (instead of asking it to select the correct answer); and (3) there is a “culture gap” in mLLMs when reasoning about proverbs and sayings translated from other languages. We construct and release our evaluation dataset MAPS (MulticulturAl Proverbs and Sayings) for proverb understanding with conversational context for six different languages. |
Freie Schlagworte: | UKP_p_emergencity,UKP_p_MISRIK |
Fachbereich(e)/-gebiet(e): | 20 Fachbereich Informatik 20 Fachbereich Informatik > Ubiquitäre Wissensverarbeitung |
Hinterlegungsdatum: | 24 Jun 2024 12:33 |
Letzte Änderung: | 05 Aug 2024 08:53 |
PPN: | 520326776 |
Export: | |
Suche nach Titel in: | TUfind oder in Google |
![]() |
Frage zum Eintrag |
Optionen (nur für Redakteure)
![]() |
Redaktionelle Details anzeigen |