TU Darmstadt / ULB / TUbiblio

The Inadequacy of Discrete Scenarios in Assessing Deep Neural Networks

Mori, Ken T. ; Liang, Xu ; Elster, Lukas ; Peters, Steven (2022)
The Inadequacy of Discrete Scenarios in Assessing Deep Neural Networks.
In: IEEE Access, 10
doi: 10.1109/ACCESS.2022.3220904
Article, Bibliographie

Abstract

Many recent approaches for automated driving (AD) functions currently include components relying on deep neural networks (DNNs). One approach in order to test AD functions is the scenario-based approach. This work formalizes and evaluates the parameter discretization process required in order to yield concrete scenarios for which an AD function can be tested. Using a common perception algorithm for camera images, a simulation case study is conducted for a simple static scenario containing one other vehicle. The results are analyzed with methods akin to those applied in the domain of computational fluid dynamics (CFD). The performance of the perception algorithm shows strong fluctuations even for small input changes and displays unpredictable outliers even at very small discretization steps. The convergence criteria as known from CFD fail, meaning that no parametrization is found which is sufficient for the validation of the perception component. Indeed, the results do not indicate consistent improvement with a finer discretization. These results agree well with theoretical attributes known for existing neural networks. However, the impact appears to be large even for the most basic scenario without malicious input. This indicates the necessity of directing more attention towards the parameter discretization process of the scenario-based testing approach to enable the safety argumentation of AD functions.

Item Type: Article
Erschienen: 2022
Creators: Mori, Ken T. ; Liang, Xu ; Elster, Lukas ; Peters, Steven
Type of entry: Bibliographie
Title: The Inadequacy of Discrete Scenarios in Assessing Deep Neural Networks
Language: English
Date: 2022
Publisher: IEEE
Journal or Publication Title: IEEE Access
Volume of the journal: 10
DOI: 10.1109/ACCESS.2022.3220904
URL / URN: https://ieeexplore.ieee.org/document/9943537
Abstract:

Many recent approaches for automated driving (AD) functions currently include components relying on deep neural networks (DNNs). One approach in order to test AD functions is the scenario-based approach. This work formalizes and evaluates the parameter discretization process required in order to yield concrete scenarios for which an AD function can be tested. Using a common perception algorithm for camera images, a simulation case study is conducted for a simple static scenario containing one other vehicle. The results are analyzed with methods akin to those applied in the domain of computational fluid dynamics (CFD). The performance of the perception algorithm shows strong fluctuations even for small input changes and displays unpredictable outliers even at very small discretization steps. The convergence criteria as known from CFD fail, meaning that no parametrization is found which is sufficient for the validation of the perception component. Indeed, the results do not indicate consistent improvement with a finer discretization. These results agree well with theoretical attributes known for existing neural networks. However, the impact appears to be large even for the most basic scenario without malicious input. This indicates the necessity of directing more attention towards the parameter discretization process of the scenario-based testing approach to enable the safety argumentation of AD functions.

Divisions: 16 Department of Mechanical Engineering
16 Department of Mechanical Engineering > Institute of Automotive Engineering (FZD)
Date Deposited: 20 Dec 2022 13:56
Last Modified: 28 Dec 2022 09:49
PPN: 503202789
Export:
Suche nach Titel in: TUfind oder in Google
Send an inquiry Send an inquiry

Options (only for editors)
Show editorial Details Show editorial Details