TU Darmstadt / ULB / TUbiblio

DKPro Agreement: An Open-Source Java Library for Measuring Inter-Rater Agreement

Meyer, Christian M. ; Mieskes, Margot ; Stab, Christian ; Gurevych, Iryna (2014)
DKPro Agreement: An Open-Source Java Library for Measuring Inter-Rater Agreement.
Dublin, Ireland
Conference or Workshop Item, Bibliographie

Abstract

In this paper, we introduce a novel Java implementation of multiple inter-rater agreement measures, which we make available as open-source software. Besides assessing the reliability of coding tasks using S, pi, kappa, alpha, etc., we particularly support unitizing tasks by measuring alpha_U as the agreement of the boundaries of the identified annotation units. We provide a unified interface and data model for both tasks as well as multiple diagnostic devices for analyzing the results.

Item Type: Conference or Workshop Item
Erschienen: 2014
Creators: Meyer, Christian M. ; Mieskes, Margot ; Stab, Christian ; Gurevych, Iryna
Type of entry: Bibliographie
Title: DKPro Agreement: An Open-Source Java Library for Measuring Inter-Rater Agreement
Language: English
Date: August 2014
Book Title: Proceedings of the 25th International Conference on Computational Linguistics: System Demonstrations (COLING)
Event Location: Dublin, Ireland
URL / URN: http://aclweb.org/anthology/C14-2023
Abstract:

In this paper, we introduce a novel Java implementation of multiple inter-rater agreement measures, which we make available as open-source software. Besides assessing the reliability of coding tasks using S, pi, kappa, alpha, etc., we particularly support unitizing tasks by measuring alpha_U as the agreement of the boundaries of the identified annotation units. We provide a unified interface and data model for both tasks as well as multiple diagnostic devices for analyzing the results.

Uncontrolled Keywords: UKP_reviewed;UKP_p_DKPro;UKP_a_WALL
Identification Number: TUD-CS-2014-0863
Divisions: 20 Department of Computer Science
20 Department of Computer Science > Ubiquitous Knowledge Processing
Date Deposited: 31 Dec 2016 14:29
Last Modified: 24 Jan 2020 12:03
PPN:
Export:
Suche nach Titel in: TUfind oder in Google
Send an inquiry Send an inquiry

Options (only for editors)
Show editorial Details Show editorial Details