TU Darmstadt / ULB / TUbiblio

Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspective

Dycke, Nils ; Simpson, Edwin ; Gurevych, Iryna (2021)
Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspective.
doi: 10.48550/arXiv.2109.01190
Report, Bibliographie

Kurzbeschreibung (Abstract)

Peer review is the primary means of quality control in academia; as an outcome of a peer review process, program and area chairs make acceptance decisions for each paper based on the review reports and scores they received. Quality of scientific work is multi-faceted; coupled with the subjectivity of reviewing, this makes final decision making difficult and time-consuming. To support this final step of peer review, we formalize it as a paper ranking problem. We introduce a novel, multi-faceted generic evaluation framework for ranking submissions based on peer reviews that takes into account effectiveness, efficiency and fairness. We propose a preference learning perspective on the task that considers both review texts and scores to alleviate the inevitable bias and noise in reviews. Our experiments on peer review data from the ACL 2018 conference demonstrate the superiority of our preference-learning-based approach over baselines and prior work, while highlighting the importance of using both review texts and scores to rank submissions.

Typ des Eintrags: Report
Erschienen: 2021
Autor(en): Dycke, Nils ; Simpson, Edwin ; Gurevych, Iryna
Art des Eintrags: Bibliographie
Titel: Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspective
Sprache: Englisch
Publikationsjahr: 2 September 2021
Verlag: arXiv
Reihe: Computation and Language
Kollation: 8 Seiten
DOI: 10.48550/arXiv.2109.01190
URL / URN: https://arxiv.org/abs/2109.01190
Kurzbeschreibung (Abstract):

Peer review is the primary means of quality control in academia; as an outcome of a peer review process, program and area chairs make acceptance decisions for each paper based on the review reports and scores they received. Quality of scientific work is multi-faceted; coupled with the subjectivity of reviewing, this makes final decision making difficult and time-consuming. To support this final step of peer review, we formalize it as a paper ranking problem. We introduce a novel, multi-faceted generic evaluation framework for ranking submissions based on peer reviews that takes into account effectiveness, efficiency and fairness. We propose a preference learning perspective on the task that considers both review texts and scores to alleviate the inevitable bias and noise in reviews. Our experiments on peer review data from the ACL 2018 conference demonstrate the superiority of our preference-learning-based approach over baselines and prior work, while highlighting the importance of using both review texts and scores to rank submissions.

Zusätzliche Informationen:

1.Version

Fachbereich(e)/-gebiet(e): 20 Fachbereich Informatik
20 Fachbereich Informatik > Ubiquitäre Wissensverarbeitung
Hinterlegungsdatum: 12 Jun 2023 12:39
Letzte Änderung: 19 Dez 2024 11:36
PPN: 510471757
Export:
Suche nach Titel in: TUfind oder in Google
Frage zum Eintrag Frage zum Eintrag

Optionen (nur für Redakteure)
Redaktionelle Details anzeigen Redaktionelle Details anzeigen