.png){kind=icon}
.png){kind=icon}
Return
Validity constraints for data analysis workflows
.png){kind=icon}
OA
AI
DOI：10.1016/j.future.2024.03.037.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Porting a scientific data analysis workflow ( DAW ) to a cluster infrastructure, a new software stack, or even only a new dataset with some notably different properties is often challenging. Despite the structured definition of the steps (tasks) and their interdependencies during a complex data analysis in the DAW specification, relevant assumptions may remain unspecified and implicit. Such hidden assumptions often lead to crashing tasks without a reasonable error message, poor performance in general, non -terminating executions, or silent wrong results of the DAW , to name only a few possible consequences. Searching for the causes of such errors and drawbacks in a distributed compute cluster managed by a complex infrastructure stack, where DAW s for large datasets typically are executed, can be tedious and time-consuming. We propose validity constraints ( VC s) as a new concept for DAW languages to alleviate this situation. A VC is a constraint specifying logical conditions that must be fulfilled at certain times for DAW executions to be valid. When defined together with a DAW , VC s help to improve the portability, adaptability, and reusability of DAW s by making implicit assumptions explicit. Once specified, VC s can be controlled automatically by the DAW infrastructure, and violations can lead to meaningful error messages and graceful behavior (e.g., termination or invocation of repair mechanisms). We provide a broad list of possible VC s, classify them along multiple dimensions, and compare them to similar concepts one can find in related fields. We also provide a proof -of -concept implementation for the workflow system Nextflow.
Keywords:
Scientific workflow systems
Workflow specification languages
Validity constraints
Dependability
Integrity and conformance checking
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
F
IF:
6.1
Papers:
6.8K
Citations:
2.3W
Organization
View more organizations