TU Darmstadt / ULB / TUbiblio

Executing cyclic scientific workflows in the cloud

Krämer, Michel ; Würz, Hendrik M. ; Altenhofen, Christian (2021)
Executing cyclic scientific workflows in the cloud.
In: Journal of Cloud Computing, 10 (1)
doi: 10.1186/s13677-021-00229-7
Article, Bibliographie

This is the latest version of this item.

Abstract

We present an algorithm and a software architecture for a cloud-based system that executes cyclic scientific workflows whose structure may change during run time. Existing approaches either rely on workflow definitions based on directed acyclic graphs (DAGs) or require workarounds to implement cyclic structures. In contrast, our system supports cycles natively, avoids workarounds, and as such reduces the complexity of workflow modelling and maintenance. Our algorithm traverses workflow graphs and transforms them iteratively into linear sequences of executable actions. We call these sequences process chains. Our software architecture distributes the process chains to multiple compute nodes in the cloud and oversees their execution. We evaluate our approach by applying it to two practical use cases from the domains of astronomy and engineering. We also compare it with two existing workflow management systems. The evaluation demonstrates that our algorithm is able to execute dynamically changing workflows with cycles and that design and maintenance of complex workflows is easier than with existing solutions. It also shows that our software architecture can run process chains on multiple compute nodes in parallel to significantly speed up the workflow execution. An implementation of our algorithm and the software architecture is available with the Steep Workflow Management System that we released under an open-source license. The resources for the first practical use case are also available as open source for reproduction.

Item Type: Article
Erschienen: 2021
Creators: Krämer, Michel ; Würz, Hendrik M. ; Altenhofen, Christian
Type of entry: Bibliographie
Title: Executing cyclic scientific workflows in the cloud
Language: English
Date: 6 April 2021
Publisher: Springer Nature
Journal or Publication Title: Journal of Cloud Computing
Volume of the journal: 10
Issue Number: 1
DOI: 10.1186/s13677-021-00229-7
Corresponding Links:
Abstract:

We present an algorithm and a software architecture for a cloud-based system that executes cyclic scientific workflows whose structure may change during run time. Existing approaches either rely on workflow definitions based on directed acyclic graphs (DAGs) or require workarounds to implement cyclic structures. In contrast, our system supports cycles natively, avoids workarounds, and as such reduces the complexity of workflow modelling and maintenance. Our algorithm traverses workflow graphs and transforms them iteratively into linear sequences of executable actions. We call these sequences process chains. Our software architecture distributes the process chains to multiple compute nodes in the cloud and oversees their execution. We evaluate our approach by applying it to two practical use cases from the domains of astronomy and engineering. We also compare it with two existing workflow management systems. The evaluation demonstrates that our algorithm is able to execute dynamically changing workflows with cycles and that design and maintenance of complex workflows is easier than with existing solutions. It also shows that our software architecture can run process chains on multiple compute nodes in parallel to significantly speed up the workflow execution. An implementation of our algorithm and the software architecture is available with the Steep Workflow Management System that we released under an open-source license. The resources for the first practical use case are also available as open source for reproduction.

Uncontrolled Keywords: Cloud computing, Distributed systems
Identification Number: Artikel-ID: 25
Additional Information:

Art.No.: 25

Divisions: 20 Department of Computer Science
20 Department of Computer Science > Interactive Graphics Systems
20 Department of Computer Science > Fraunhofer IGD
Date Deposited: 25 May 2021 07:59
Last Modified: 11 Apr 2024 09:48
PPN:
Corresponding Links:
Export:
Suche nach Titel in: TUfind oder in Google

Available Versions of this Item

Send an inquiry Send an inquiry

Options (only for editors)
Show editorial Details Show editorial Details