TU Darmstadt / ULB / TUbiblio

Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging

Reimers, Nils and Gurevych, Iryna :
Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging.
[Online-Edition: http://aclweb.org/anthology/D17-1035]
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP)
[Conference or Workshop Item] , (2017)

Official URL: http://aclweb.org/anthology/D17-1035

Abstract

In this paper we show that reporting a single performance score is insufficient to compare non-deterministic approaches. We demonstrate this for common sequence tagging tasks that the seed value for the random number generator can result in statistically significant (p < 10^{-4}) differences for state-of-the-art systems. For two recent systems for NER, we observe an absolute difference of one percentage point F1-score depending on the selected seed value, making these systems perceived either as state-of-the-art or mediocre. Instead of publishing and reporting single performance scores, we propose to compare score distributions based on multiple executions. Based on the evaluation of 50.000 LSTM-networks for five sequence tagging tasks, we present network architectures that perform superior as well as produce results with higher stability on unseen data.

Item Type: Conference or Workshop Item
Erschienen: 2017
Creators: Reimers, Nils and Gurevych, Iryna
Title: Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging
Language: English
Abstract:

In this paper we show that reporting a single performance score is insufficient to compare non-deterministic approaches. We demonstrate this for common sequence tagging tasks that the seed value for the random number generator can result in statistically significant (p < 10^{-4}) differences for state-of-the-art systems. For two recent systems for NER, we observe an absolute difference of one percentage point F1-score depending on the selected seed value, making these systems perceived either as state-of-the-art or mediocre. Instead of publishing and reporting single performance scores, we propose to compare score distributions based on multiple executions. Based on the evaluation of 50.000 LSTM-networks for five sequence tagging tasks, we present network architectures that perform superior as well as produce results with higher stability on unseen data.

Title of Book: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Divisions: Department of Computer Science
Department of Computer Science > Ubiquitous Knowledge Processing
DFG-Graduiertenkollegs
DFG-Graduiertenkollegs > Research Training Group 1994 Adaptive Preparation of Information from Heterogeneous Sources
Event Location: Copenhagen, Denmark
Date Deposited: 04 Jul 2017 09:53
Official URL: http://aclweb.org/anthology/D17-1035
Identification Number: TUD-CS-2017-0150
Related URLs:
Export:

Optionen (nur für Redakteure)

View Item View Item