Machine learning-driven predictive resource management in complex science workflows

doi:10.1142/S0217751X26500259

2025-12-01

PRE

Verena Ingrid Martinez Outschoorn

Steele, John Rembrandt

DOI：10.1142/S0217751X26500259

原文链接

原文求助

摘要

En 中文

大型科学实验中全球性社区的合作努力，通常包含数千名成员，体现了对探索和发现的巨大承诺。近年来，先进且复杂的数据处理在科学实验中日益重要。数据处理工作流通常由多个复杂步骤组成，每个步骤精确的资源需求规格对于分配有效处理所需的最佳资源至关重要。由于分析场景多样、社区成员技能水平不一以及计算选项持续扩展，预先估算资源需求具有挑战性。一种缓解这些挑战的实用方法是在完成整个步骤之前，先处理每个步骤的子集，通过实际处理剖面测量精确的资源使用情况。尽管这种两阶段方法使大部分工作流能够在最优资源上运行，但其缺点包括初始不准确性可能导致失败和资源使用次优，以及等待初始处理完成的开销，这对快速周转分析至关重要。在此背景下，本研究在全面的工作流管理系统——生产与分布式分析（PanDA）系统中引入了一种新型机器学习模型管道。这些模型采用先进的机器学习技术预测关键资源需求，克服了因每个步骤特性前期知识有限而带来的挑战。精确的资源需求预测能够支持工作流管理中的明智且前瞻性的决策，提升在异构资源上处理多样化、复杂工作流的效率。

Keyword:

Resource prediction

workflow management

computational efficiency

scout jobs

PanDA system

resource allocation

high-performance computing (HPC)

scientific workflows

distributed computing

deep learning

multi-class classification

data-driven resource management

期刊

International Journal of Modern Physics A

IF:

1.2

论文数:

211

被引数:

机构

university of massachusetts amherst

学者数:

838

论文数: 448

被引数: 0

university of massachusetts system

学者数:

3.9W

论文数: 3.6W

被引数: 42

united states department of energy (doe)

学者数:

11.3W

论文数: 9.6W

被引数: 246

pennsylvania commonwealth system of higher education (pcshe)

学者数:

12.9W

论文数: 11.7W

被引数: 177

Brookhaven National Laboratory

学者数:

6.4K

论文数: 4.9K

被引数: 1.9W

university of pittsburgh

学者数:

6.3K

论文数: 2.8K

被引数: 1

查看更多机构

引用论文

XGBoost

Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

IF0

2016-08-13

OAAI

Tianqi Chen; Carlos Guestrin

Author Correction: Elemental analysis of single ambient aerosol particles using laser-induced breakdown spectroscopy

Scientific Reports

IF0

PREAI

Heikkilä,Paavo; Rostedt,Antti; Toivonen,Juha; Keskinen,Jorma

LSST: From Science Drivers to Reference Design and Anticipated Data Products LSST: 从科学驱动到参考设计和预期数据产品

ASTROPHYSICAL JOURNAL

IF5.4

2019-03-11

1.1K

OAAI

Ivezic, Zeljko; Kahn, Steven M.; Tyson, J. Anthony; Abel, Bob; Acosta, Emily; Allsman, Robyn; Alonso, David; AlSayyad, Yusra; Anderson, Scott F.; Andrew, John; Angel, James Roger P.; Angeli, George Z.; Ansari, Reza; Antilogus, Pierre; Araujo, Constanza; Armstrong, Robert; Arndt, Kirk T.; Astier, Pierre; Aubourg, Eric; Auza, Nicole; Axelrod, Tim S.; Bard, Deborah J.; Barr, Jeff D.; Barrau, Aurelian; Bartlett, James G.; Bauer, Amanda E.; Bauman, Brian J.; Baumont, Sylvain; Bechtol, Ellen; Bechtol, Keith; Becker, Andrew C.; Becla, Jacek; Beldica, Cristina; Bellavia, Steve; Bianco, Federica B.; Biswas, Rahul; Blanc, Guillaume; Blazek, Jonathan; Blandford, Roger D.; Bloom, Josh S.; Bogart, Joanne; Bond, Tim W.; Booth, Michael T.; Borgland, Anders W.; Borne, Kirk; Bosch, James F.; Boutigny, Dominique; Brackett, Craig A.; Bradshaw, Andrew; Brandt, William Nielsen; Brown, Michael E.; Bullock, James S.; Burchat, Patricia; Burke, David L.; Cagnoli, Gianpietro; Calabrese, Daniel; Callahan, Shawn; Callen, Alice L.; Carlin, Jeffrey L.; Carlson, Erin L.; Chandrasekharan, Srinivasan; Charles-Emerson, Glenaver; Chesley, Steve; Cheu, Elliott C.; Chiang, Hsin-Fang; Chiang, James; Chirino, Carol; Chow, Derek; Ciardi, David R.; Claver, Charles F.; Cohen-Tanugi, Johann; Cockrum, Joseph J.; Coles, Rebecca; Connolly, Andrew J.; Cook, Kem H.; Cooray, Asantha; Covey, Kevin R.; Cribbs, Chris; Cui, Wei; Cutri, Roc; Daly, Philip N.; Daniel, Scott F.; Daruich, Felipe; Daubard, Guillaume; Daues, Greg; Dawson, William; Delgado, Francisco; Dellapenna, Alfred; de Peyster, Robert; de Val-Borro, Miguel; Digel, Seth W.; Doherty, Peter; Dubois, Richard; Dubois-Felsmann, Gregory P.; Durech, Josef; Economou, Frossie; Eifler, Tim; Eracleous, Michael; Emmons, Benjamin L.; Neto, Angelo Fausti; Ferguson, Henry; Figueroa, Enrique; Fisher-Levine, Merlin; Focke, Warren; Foss, Michael D.; Frank, James; Freemon, Michael D.; Gangler, Emmanuel; Gawiser, Eric; Geary, John C.; Gee, Perry; Geha, Marla; Gessner, Charles J. B.; Gibson, Robert R.; Gilmore, D. Kirk; Glanzman, Thomas; Glick, William; Goldina, Tatiana; Goldstein, Daniel A.; Goodenow, Iain; Graham, Melissa L.; Gressler, William J.; Gris, Philippe; Guy, Leanne P.; Guyonnet, Augustin; Haller, Gunther; Harris, Ron; Hascall, Patrick A.; Haupt, Justine; Hernandez, Fabio; Herrmann, Sven; Hileman, Edward; Hoblitt, Joshua; Hodgson, John A.; Hogan, Craig; Howard, James D.; Huang, Dajun; Huffer, Michael E.; Ingraham, Patrick; Innes, Walter R.; Jacoby, Suzanne H.; Jain, Bhuvnesh; Jammes, Fabrice; Jee, M. James; Jenness, Tim; Jernigan, Garrett; Jevremovic, Darko; Johns, Kenneth; Johnson, Anthony S.; Johnson, Margaret W. G.; Jones, R. Lynne; Juramy-Gilles, Claire; Juric, Mario; Kalirai, Jason S.; Kallivayalil, Nitya J.; Kalmbach, Bryce; Kantor, Jeffrey P.; Karst, Pierre; Kasliwal, Mansi M.; Kelly, Heather; Kessler, Richard; Kinnison, Veronica; Kirkby, David; Knox, Lloyd; Kotov, Ivan V.; Krabbendam, Victor L.; Krughoff, K. Simon; Kubanek, Petr; Kuczewski, John; Kulkarni, Shri; Ku, John; Kurita, Nadine R.; Lage, Craig S.; Lambert, Ron; Lange, Travis; Langton, J. Brian; Le Guillou, Laurent; Levine, Deborah; Liang, Ming; Lim, Kian-Tat; Lintott, Chris J.; Long, Kevin E.; Lopez, Margaux; Lotz, Paul J.; Lupton, Robert H.; Lust, Nate B.; MacArthur, Lauren A.; Mahabal, Ashish; Mandelbaum, Rachel; Markiewicz, Thomas W.; Marsh, Darren S.; Marshall, Philip J.; Marshall, Stuart; May, Morgan; McKercher, Robert; MichelleMcQueen; Meyers, Joshua; Migliore, Myriam; Miller, Michelle; Mills, David J.; Miraval, Connor; Moeyens, Joachim; Moolekamp, Fred E.; Monet, David G.; Moniez, Marc; Monkewitz, Serge; Montgomery, Christopher; Morrison, Christopher B.; Mueller, Fritz; Muller, Gary P.; Arancibia, Freddy Munoz; Neill, Douglas R.; Newbry, Scott P.; Nief, Jean-Yves; Nomerotski, Andrei; Nordby, Martin; O'Connor, Paul; Oliver, John; Olivier, Scot S.; Olsen, Knut; O'Mullane, William; Ortiz, Sandra; Osier, Shawn; Owen, Russell E.; Pain, Reynald; Palecek, Paul E.; Parejko, John K.; Parsons, James B.; Pease, Nathan M.; Peterson, J. Matt; Peterson, John R.; Petravick, Donald L.; Petrick, M. E. Libby; Petry, Cathy E.; Pierfederici, Francesco; Pietrowicz, Stephen; Pike, Rob; Pinto, Philip A.; Plante, Raymond; Plate, Stephen; Plutchak, Joel P.; Price, Paul A.; Prouza, Michael; Radeka, Veljko; Rajagopal, Jayadev; Rasmussen, Andrew P.; Regnault, Nicolas; Reil, Kevin A.; Reiss, David J.; Reuter, Michael A.; Ridgway, Stephen T.; Riot, Vincent J.; Ritz, Steve; Robinson, Sean; Roby, William; Roodman, Aaron; Rosing, Wayne; Roucelle, Cecille; Rumore, Matthew R.; Russo, Stefano; Saha, Abhijit; Sassolas, Benoit; Schalk, Terry L.; Schellart, Pim; Schindler, Rafe H.; Schmidt, Samuel; Schneider, Donald P.; Schneider, Michael D.; Schoening, William; Schumacher, German; Schwamb, Megan E.; Sebag, Jacques; Selvy, Brian; Sembroski, Glenn H.; Seppala, Lynn G.; Serio, Andrew; Serrano, Eduardo; Shaw, Richard A.; Shipsey, Ian; Sick, Jonathan; Silvestri, Nicole; Colin, T.; Smith, J. Allyn; Smith, R. Chris; Shahram; Soldahl, Christine; Storrie-Lombardi, Lisa; Stover, Edward; Strauss, Michael A.; Street, Rachel A.; Stubbs, Christopher W.; Sullivan, Ian S.; Sweeney, Donald; Swinbank, John D.; Szalay, Alexander; Takacs, Peter; Tether, Stephen A.; Thaler, Jon J.; Thayer, John Gregg; Thomas, Sandrine; Thornton, Adam J.; Thukral, Vaikunth; Tice, Jeffrey; Trilling, David E.; Turri, Max; Van Berg, Richard; Vanden Berk, Daniel; Vetter, Kurt; Virieux, Francoise; Vucina, Tomislav; Wahl, William; Walkowicz, Lucianne; Walsh, Brian; Walter, Christopher W.; Wang, Daniel L.; Wang, Shin-Yawn; Warner, Michael; Wiecha, Oliver; Willman, Beth; Winters, Scott E.; Wittman, David; Wolff, Sidney C.; Wood-Vasey, W. Michael; Wu, Xiuqin; Xin, Bo; Yoachim, Peter; Zhan, Hu

Learning from Imbalanced Data 从不平衡数据中学习

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING

IF10.4

2009-09-01

5.5K

PREAI

He, Haibo; Garcia, Edwardo A.

PanDA: Production and Distributed Analysis System 前川拓也；阿列克谢耶夫·亚历山大；巴雷罗·梅吉诺，F.H.；德，K.；关，W.；卡拉瓦基斯，E.；克利门托夫，A.；科尔丘加诺娃，T.；林，F.；尼尔松，P.；等。PanDA：生产与分布式分析系统。计算机软件与大数据科学，2024，8，4。 [谷歌学术] [交叉引用]

Computing and Software for Big Science

IF0

2024-01-23

PREAI

Tadashi Maeno; Aleksandr Alekseev; Fernando Harald Barreiro Megino; Kaushik De; Wen Guan; Edward Karavakis; Alexei Klimentov; Tatiana Korchuganova; FaHui Lin; Paul Nilsson; Torre Wenaus; Zhaoyu Yang; Xin Zhao

Machine Learning Evaluation

IF0

2024-10-31

PREAI

Japkowicz,Nathalie; Boukouvalas,Zois

scRNA-seq assessment of the human lung, spleen, and esophagus tissue stability after cold preservation

GENOME BIOLOGY

IF9.4

2019-12-31

188

OAAI

Madissoon, E.; Wilbrey-Clark, A.; Miragaia, R. J.; Saeb-Parsy, K.; Mahbubani, K. T.; Georgakopoulos, N.; Harding, P.; Polanski, K.; Huang, N.; Nowicki-Osuch, K.; Fitzgerald, R. C.; Loudon, K. W.; Ferdinand, J. R.; Clatworthy, M. R.; Tsingene, A.; van Dongen, S.; Dabrowska, M.; Patel, M.; Stubbington, M. J. T.; Teichmann, S. A.; Stegle, O.; Meyer, K. B.

没有更多内容