TU Darmstadt / ULB / TUbiblio

Expected Information Maximization: Using the I-Projection for Mixture Density Estimation

Becker, Philipp ; Arenz, Oleg ; Neumann, Gerhard (2022)
Expected Information Maximization: Using the I-Projection for Mixture Density Estimation.
8. International Conference on Learning Representations (ICLR 2020). Virtual Conference (26.-30.04.2020)
doi: 10.26083/tuprints-00022969
Conference or Workshop Item, Secondary publication, Publisher's Version

Abstract

Modelling highly multi-modal data is a challenging problem in machine learning. Most algorithms are based on maximizing the likelihood, which corresponds to the M(oment)-projection of the data distribution to the model distribution. The M-projection forces the model to average over modes it cannot represent. In contrast, the I(nformation)-projection ignores such modes in the data and concentrates on the modes the model can represent. Such behavior is appealing whenever we deal with highly multi-modal data where modelling single modes correctly is more important than covering all the modes. Despite this advantage, the I-projection is rarely used in practice due to the lack of algorithms that can efficiently optimize it based on data. In this work, we present a new algorithm called Expected Information Maximization (EIM) for computing the I-projection solely based on samples for general latent variable models, where we focus on Gaussian mixtures models and Gaussian mixtures of experts. Our approach applies a variational upper bound to the I-projection objective which decomposes the original objective into single objectives for each mixture component as well as for the coefficients, allowing an efficient optimization. Similar to GANs, our approach employs discriminators but uses a more stable optimization procedure, using a tight upper bound. We show that our algorithm is much more effective in computing the I-projection than recent GAN approaches and we illustrate the effectiveness of our approach for modelling multi-modal behavior on two pedestrian and traffic prediction datasets.

Item Type: Conference or Workshop Item
Erschienen: 2022
Creators: Becker, Philipp ; Arenz, Oleg ; Neumann, Gerhard
Type of entry: Secondary publication
Title: Expected Information Maximization: Using the I-Projection for Mixture Density Estimation
Language: English
Date: 2022
Place of Publication: Darmstadt
Collation: 16 Seiten
Event Title: 8. International Conference on Learning Representations (ICLR 2020)
Event Location: Virtual Conference
Event Dates: 26.-30.04.2020
DOI: 10.26083/tuprints-00022969
URL / URN: https://tuprints.ulb.tu-darmstadt.de/22969
Corresponding Links:
Origin: Secondary publication service
Abstract:

Modelling highly multi-modal data is a challenging problem in machine learning. Most algorithms are based on maximizing the likelihood, which corresponds to the M(oment)-projection of the data distribution to the model distribution. The M-projection forces the model to average over modes it cannot represent. In contrast, the I(nformation)-projection ignores such modes in the data and concentrates on the modes the model can represent. Such behavior is appealing whenever we deal with highly multi-modal data where modelling single modes correctly is more important than covering all the modes. Despite this advantage, the I-projection is rarely used in practice due to the lack of algorithms that can efficiently optimize it based on data. In this work, we present a new algorithm called Expected Information Maximization (EIM) for computing the I-projection solely based on samples for general latent variable models, where we focus on Gaussian mixtures models and Gaussian mixtures of experts. Our approach applies a variational upper bound to the I-projection objective which decomposes the original objective into single objectives for each mixture component as well as for the coefficients, allowing an efficient optimization. Similar to GANs, our approach employs discriminators but uses a more stable optimization procedure, using a tight upper bound. We show that our algorithm is much more effective in computing the I-projection than recent GAN approaches and we illustrate the effectiveness of our approach for modelling multi-modal behavior on two pedestrian and traffic prediction datasets.

Uncontrolled Keywords: density estimation, optimization
Status: Publisher's Version
URN: urn:nbn:de:tuda-tuprints-229690
Additional Information:

Recording of the presentation and slides: https://iclr.cc/virtual_2020/poster_ByglLlHFDS.html

Reviews: https://openreview.net/forum?id=ByglLlHFDS

Classification DDC: 000 Generalities, computers, information > 004 Computer science
Divisions: 20 Department of Computer Science
20 Department of Computer Science > Intelligent Autonomous Systems
Date Deposited: 07 Dec 2022 13:11
Last Modified: 09 Dec 2022 08:45
PPN:
Corresponding Links:
Export:
Suche nach Titel in: TUfind oder in Google
Send an inquiry Send an inquiry

Options (only for editors)
Show editorial Details Show editorial Details