Puzikov, Yevgeniy (2021)
Principled Approach to Natural Language Generation.
Technische Universität Darmstadt
doi: 10.26083/tuprints-00019115
Dissertation, Erstveröffentlichung, Verlagsversion
Kurzbeschreibung (Abstract)
The research field of Natural Language Generation offers practitioners a wide range of techniques for producing texts from a variety of data types. These techniques find their way into various real- world applications and help many people to automate time-consuming tasks of text production in many areas. At the moment, the design and evaluation of text generation approaches is largely empirical. Many systems are being developed to solve one particular task and work on a single data type, which makes it hard to compare the approach to any other technique and critically evaluate its performance. Some systems employ complex machine learning algorithms to learn rich data representations and perform joint modeling of the steps involved in the process of text generation. Such approaches offer an attractive trade-off between the development costs and output quality, but often lack transparency in terms of the reasoning about the behavior of the system. The number of the proposed approaches constantly grows, but the methodology lags behind and sometimes fails to solicit a better understanding of which approaches work, and the reasons for it. In this thesis we present our view on the task of text production from a methodological point of view. We analyze the existent scientific literature, examine common text generation approaches and the established evaluation protocols. We further propose a principled view on the problem: we break it into components, examine their interaction and develop a set of recommendations which are envisioned to offer assistance during the design or analysis of a study. We further conduct a range of experiments to test this framework in several text generation tasks. First, we show that task specification analysis sometimes allows one to solve the problem at hand with very simple techniques, without resorting to the complex machinery of advanced statistical learning methods. We further demonstrate the potential of the developed framework to find discrepancies in the established evaluation protocols. We show that sometimes neither metric, nor conventional human evaluation is sufficient to draw conclusions about system performance. We demonstrate how a system can fit the data to achieve high automatic metric scores, while falling short in terms of actual output quality. Finally, we use the framework to demonstrate how one can develop effective text generation systems without sacrificing the transparency of the inner working logic, making the developed systems both accurate and reliable.
Typ des Eintrags: | Dissertation | ||||
---|---|---|---|---|---|
Erschienen: | 2021 | ||||
Autor(en): | Puzikov, Yevgeniy | ||||
Art des Eintrags: | Erstveröffentlichung | ||||
Titel: | Principled Approach to Natural Language Generation | ||||
Sprache: | Englisch | ||||
Referenten: | Gurevych, Prof. Dr. Iryna ; Dagan, Prof. Ido ; Gardent, Prof. Dr. Claire | ||||
Publikationsjahr: | 2021 | ||||
Ort: | Darmstadt | ||||
Kollation: | viii, 172 Seiten | ||||
Datum der mündlichen Prüfung: | 22 Juni 2021 | ||||
DOI: | 10.26083/tuprints-00019115 | ||||
URL / URN: | https://tuprints.ulb.tu-darmstadt.de/19115 | ||||
Kurzbeschreibung (Abstract): | The research field of Natural Language Generation offers practitioners a wide range of techniques for producing texts from a variety of data types. These techniques find their way into various real- world applications and help many people to automate time-consuming tasks of text production in many areas. At the moment, the design and evaluation of text generation approaches is largely empirical. Many systems are being developed to solve one particular task and work on a single data type, which makes it hard to compare the approach to any other technique and critically evaluate its performance. Some systems employ complex machine learning algorithms to learn rich data representations and perform joint modeling of the steps involved in the process of text generation. Such approaches offer an attractive trade-off between the development costs and output quality, but often lack transparency in terms of the reasoning about the behavior of the system. The number of the proposed approaches constantly grows, but the methodology lags behind and sometimes fails to solicit a better understanding of which approaches work, and the reasons for it. In this thesis we present our view on the task of text production from a methodological point of view. We analyze the existent scientific literature, examine common text generation approaches and the established evaluation protocols. We further propose a principled view on the problem: we break it into components, examine their interaction and develop a set of recommendations which are envisioned to offer assistance during the design or analysis of a study. We further conduct a range of experiments to test this framework in several text generation tasks. First, we show that task specification analysis sometimes allows one to solve the problem at hand with very simple techniques, without resorting to the complex machinery of advanced statistical learning methods. We further demonstrate the potential of the developed framework to find discrepancies in the established evaluation protocols. We show that sometimes neither metric, nor conventional human evaluation is sufficient to draw conclusions about system performance. We demonstrate how a system can fit the data to achieve high automatic metric scores, while falling short in terms of actual output quality. Finally, we use the framework to demonstrate how one can develop effective text generation systems without sacrificing the transparency of the inner working logic, making the developed systems both accurate and reliable. |
||||
Alternatives oder übersetztes Abstract: |
|
||||
Status: | Verlagsversion | ||||
URN: | urn:nbn:de:tuda-tuprints-191154 | ||||
Sachgruppe der Dewey Dezimalklassifikatin (DDC): | 000 Allgemeines, Informatik, Informationswissenschaft > 004 Informatik | ||||
Fachbereich(e)/-gebiet(e): | 20 Fachbereich Informatik 20 Fachbereich Informatik > Ubiquitäre Wissensverarbeitung |
||||
TU-Projekte: | Bund/BMBF|01IS17050|Software Campus 2.0 DFG|GRK1994|Gurevych_GRK_1994_Au DFG|GU798/17-1|Deutsch-Israelische |
||||
Hinterlegungsdatum: | 19 Jul 2021 11:33 | ||||
Letzte Änderung: | 27 Jul 2021 09:18 | ||||
PPN: | |||||
Referenten: | Gurevych, Prof. Dr. Iryna ; Dagan, Prof. Ido ; Gardent, Prof. Dr. Claire | ||||
Datum der mündlichen Prüfung / Verteidigung / mdl. Prüfung: | 22 Juni 2021 | ||||
Export: | |||||
Suche nach Titel in: | TUfind oder in Google |
Frage zum Eintrag |
Optionen (nur für Redakteure)
Redaktionelle Details anzeigen |