Schiller, Benjamin ; Daxenberger, Johannes ; Gurevych, Iryna (2021)
Stance Detection Benchmark: How Robust Is Your Stance Detection?
In: KI - Künstliche Intelligenz, 35 (3-4)
doi: 10.1007/s13218-021-00714-w
Artikel, Bibliographie

Kurzbeschreibung (Abstract)

Stance detection (StD) aims to detect an author’s stance towards a certain topic and has become a key component in applications like fake news detection, claim validation, or argument search. However, while stance is easily detected by humans, machine learning (ML) models are clearly falling short of this task. Given the major differences in dataset sizes and framing of StD (e.g. number of classes and inputs), ML models trained on a single dataset usually generalize poorly to other domains. Hence, we introduce a StD benchmark that allows to compare ML models against a wide variety of heterogeneous StD datasets to evaluate them for generalizability and robustness. Moreover, the framework is designed for easy integration of new datasets and probing methods for robustness. Amongst several baseline models, we define a model that learns from all ten StD datasets of various domains in a multi-dataset learning (MDL) setting and present new state-of-the-art results on five of the datasets. Yet, the models still perform well below human capabilities and even simple perturbations of the original test samples (adversarial attacks) severely hurt the performance of MDL models. Deeper investigation suggests overfitting on dataset biases as the main reason for the decreased robustness. Our analysis emphasizes the need of focus on robustness and de-biasing strategies in multi-task learning approaches. To foster research on this important topic, we release the dataset splits, code, and fine-tuned weights.

Typ des Eintrags:	Artikel
Erschienen:	2021
Autor(en):	Schiller, Benjamin ; Daxenberger, Johannes ; Gurevych, Iryna
Art des Eintrags:	Bibliographie
Titel:	Stance Detection Benchmark: How Robust Is Your Stance Detection?
Sprache:	Englisch
Publikationsjahr:	26 März 2021
Verlag:	Springer
Titel der Zeitschrift, Zeitung oder Schriftenreihe:	KI - Künstliche Intelligenz
Jahrgang/Volume einer Zeitschrift:	35
(Heft-)Nummer:	3-4
DOI:	10.1007/s13218-021-00714-w
URL / URN:	https://link.springer.com/article/10.1007/s13218-021-00714-w
Zugehörige Links:	Verwandtes Werk TUprints Zweitveröffentlichung
Kurzbeschreibung (Abstract):	Stance detection (StD) aims to detect an author’s stance towards a certain topic and has become a key component in applications like fake news detection, claim validation, or argument search. However, while stance is easily detected by humans, machine learning (ML) models are clearly falling short of this task. Given the major differences in dataset sizes and framing of StD (e.g. number of classes and inputs), ML models trained on a single dataset usually generalize poorly to other domains. Hence, we introduce a StD benchmark that allows to compare ML models against a wide variety of heterogeneous StD datasets to evaluate them for generalizability and robustness. Moreover, the framework is designed for easy integration of new datasets and probing methods for robustness. Amongst several baseline models, we define a model that learns from all ten StD datasets of various domains in a multi-dataset learning (MDL) setting and present new state-of-the-art results on five of the datasets. Yet, the models still perform well below human capabilities and even simple perturbations of the original test samples (adversarial attacks) severely hurt the performance of MDL models. Deeper investigation suggests overfitting on dataset biases as the main reason for the decreased robustness. Our analysis emphasizes the need of focus on robustness and de-biasing strategies in multi-task learning approaches. To foster research on this important topic, we release the dataset splits, code, and fine-tuned weights.
Freie Schlagworte:	UKP_p_ArgumenText, UKP_p_sppratio, UKP_p_OAM
Fachbereich(e)/-gebiet(e):	20 Fachbereich Informatik 20 Fachbereich Informatik > Ubiquitäre Wissensverarbeitung
Hinterlegungsdatum:	02 Mär 2021 09:05
Letzte Änderung:	03 Apr 2024 05:18
PPN:
Zugehörige Links:	Verwandtes Werk TUprints Zweitveröffentlichung
Export:	EP3 XMLASCII CitationHTML CitationSimple MetadataMultiline CSVAtomEndNoteIBW_RDAMODST2T_XMLJSONDublin CoreRDF+XMLBibTeX
Suche nach Titel in:	TUfind oder in Google

Verfügbare Versionen dieses Eintrags

• Stance Detection Benchmark: How Robust is Your Stance Detection? (deposited 02 Apr 2024 11:20)
  • Stance Detection Benchmark: How Robust Is Your Stance Detection? (deposited 02 Mär 2021 09:05) [Gegenwärtig angezeigt]

Frage zum Eintrag

Optionen (nur für Redakteure)

Redaktionelle Details anzeigen