.png){kind=icon}
.png){kind=icon}
Return
Scythe: A Python toolkit for orchestrating embarrassingly parallel experiments via distributed queues
.png){kind=icon}
PRE
AI
DOI：10.1016/j.softx.2026.102701.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Researchers across scientific and engineering disciplines frequently need to execute large-scale embarrassingly parallel experiments but lack the distributed computing expertise to manage task queues, artifact storage, and result aggregation. Scythe is an open-source Python library that addresses this gap by providing a lightweight, schema-driven orchestration layer on top of the Hatchet distributed task engine. Users define typed input and output specifications as fully typesafe Pydantic models, register simulation functions via a decorator, and launch versioned experiment runs with a single call. Scythe automatically handles S3 artifact management, recursive scatter/gather fanout for high-throughput task allocation, and bottom-up aggregation of results into structured Parquet datasets. The toolkit supports deployment from local Docker Compose setups to cloud-scale containerized clusters with spot capacity. Scythe is in active use at the MIT Sustainable Design Lab for urban building energy modeling research involving millions of simulations.
Keywords:
Distributed computing
Embarrassingly parallel
Experiment orchestration
Python
Building energy modeling
Workflow automation
Journal
IF:
2.4
Papers:
325
Citations:
7.3K