TU Darmstadt / ULB / TUbiblio

Receding Horizon Curiosity

Schultheis, Matthias ; Belousov, Boris ; Abdulsamad, Hany ; Peters, Jan (2022)
Receding Horizon Curiosity.
3rd Conference on Robot Learning (CoRL 2019). Osaka, Japan (30.10.- 1.11.2019)
doi: 10.26083/tuprints-00020578
Conference or Workshop Item, Secondary publication, Publisher's Version

Abstract

Sample-efficient exploration is crucial not only for discovering rewarding experiences but also for adapting to environment changes in a task-agnostic fashion. A principled treatment of the problem of optimal input synthesis for system identification is provided within the framework of sequential Bayesian experimental design. In this paper, we present an effective trajectory-optimization-based approximate solution of this otherwise intractable problem that models optimal exploration in an unknown Markov decision process (MDP). By interleaving episodic exploration with Bayesian nonlinear system identification, our algorithm takes advantage of the inductive bias to explore in a directed manner, without assuming prior knowledge of the MDP. Empirical evaluations indicate a clear advantage of the proposed algorithm in terms of the rate of convergence and the final model fidelity when compared to intrinsic-motivation-based algorithms employing exploration bonuses such as prediction error and information gain. Moreover, our method maintains a computational advantage over a recent model-based active exploration (MAX) algorithm, by focusing on the information gain along trajectories instead of seeking a global exploration policy. A reference implementation of our algorithm and the conducted experiments is publicly available.

Item Type: Conference or Workshop Item
Erschienen: 2022
Creators: Schultheis, Matthias ; Belousov, Boris ; Abdulsamad, Hany ; Peters, Jan
Type of entry: Secondary publication
Title: Receding Horizon Curiosity
Language: English
Date: 2022
Place of Publication: Darmstadt
Publisher: PMLR
Series: Proceedings of Machine Learning Research
Series Volume: 100
Collation: 11 Seiten
Event Title: 3rd Conference on Robot Learning (CoRL 2019)
Event Location: Osaka, Japan
Event Dates: 30.10.- 1.11.2019
DOI: 10.26083/tuprints-00020578
URL / URN: https://tuprints.ulb.tu-darmstadt.de/20578
Corresponding Links:
Origin: Secondary publication service
Abstract:

Sample-efficient exploration is crucial not only for discovering rewarding experiences but also for adapting to environment changes in a task-agnostic fashion. A principled treatment of the problem of optimal input synthesis for system identification is provided within the framework of sequential Bayesian experimental design. In this paper, we present an effective trajectory-optimization-based approximate solution of this otherwise intractable problem that models optimal exploration in an unknown Markov decision process (MDP). By interleaving episodic exploration with Bayesian nonlinear system identification, our algorithm takes advantage of the inductive bias to explore in a directed manner, without assuming prior knowledge of the MDP. Empirical evaluations indicate a clear advantage of the proposed algorithm in terms of the rate of convergence and the final model fidelity when compared to intrinsic-motivation-based algorithms employing exploration bonuses such as prediction error and information gain. Moreover, our method maintains a computational advantage over a recent model-based active exploration (MAX) algorithm, by focusing on the information gain along trajectories instead of seeking a global exploration policy. A reference implementation of our algorithm and the conducted experiments is publicly available.

Uncontrolled Keywords: Bayesian exploration, artificial curiosity, model predictive control
Status: Publisher's Version
URN: urn:nbn:de:tuda-tuprints-205782
Classification DDC: 000 Generalities, computers, information > 004 Computer science
Divisions: 20 Department of Computer Science
20 Department of Computer Science > Intelligent Autonomous Systems
TU-Projects: EC/H2020|640554|SKILLS4ROBOTS
Date Deposited: 18 Nov 2022 14:27
Last Modified: 25 Apr 2023 12:06
PPN: 502453915
Corresponding Links:
Export:
Suche nach Titel in: TUfind oder in Google
Send an inquiry Send an inquiry

Options (only for editors)
Show editorial Details Show editorial Details