TU Darmstadt / ULB / TUbiblio

DKPro Agreement: An Open-Source Java Library for Measuring Inter-Rater Agreement

Meyer, Christian M. and Mieskes, Margot and Stab, Christian and Gurevych, Iryna (2014):
DKPro Agreement: An Open-Source Java Library for Measuring Inter-Rater Agreement.
In: Proceedings of the 25th International Conference on Computational Linguistics: System Demonstrations (COLING), Dublin, Ireland, ISBN 978-1-941643-27-3,
[Online-Edition: http://aclweb.org/anthology/C14-2023],
[Conference or Workshop Item]

Abstract

In this paper, we introduce a novel Java implementation of multiple inter-rater agreement measures, which we make available as open-source software. Besides assessing the reliability of coding tasks using S, pi, kappa, alpha, etc., we particularly support unitizing tasks by measuring alpha_U as the agreement of the boundaries of the identified annotation units. We provide a unified interface and data model for both tasks as well as multiple diagnostic devices for analyzing the results.

Item Type: Conference or Workshop Item
Erschienen: 2014
Creators: Meyer, Christian M. and Mieskes, Margot and Stab, Christian and Gurevych, Iryna
Title: DKPro Agreement: An Open-Source Java Library for Measuring Inter-Rater Agreement
Language: English
Abstract:

In this paper, we introduce a novel Java implementation of multiple inter-rater agreement measures, which we make available as open-source software. Besides assessing the reliability of coding tasks using S, pi, kappa, alpha, etc., we particularly support unitizing tasks by measuring alpha_U as the agreement of the boundaries of the identified annotation units. We provide a unified interface and data model for both tasks as well as multiple diagnostic devices for analyzing the results.

Title of Book: Proceedings of the 25th International Conference on Computational Linguistics: System Demonstrations (COLING)
ISBN: 978-1-941643-27-3
Uncontrolled Keywords: UKP_reviewed;UKP_p_DKPro;UKP_a_WALL
Divisions: 20 Department of Computer Science
20 Department of Computer Science > Ubiquitous Knowledge Processing
Event Location: Dublin, Ireland
Date Deposited: 31 Dec 2016 14:29
Official URL: http://aclweb.org/anthology/C14-2023
Identification Number: TUD-CS-2014-0863
Export:
Suche nach Titel in: TUfind oder in Google

Optionen (nur für Redakteure)

View Item View Item