.png){kind=icon}
返回
A Task-Oriented General-Purpose Distributed Computing System Based on CLTS Scheduling Algorithm
.png){kind=icon}
OA
AI
DOI：10.1109/ACCESS.2023.3298049.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Most current research on commonly used distributed computing frameworks neglects the difficulty of learning these frameworks. These commonly used distributed computing frameworks are designed for professionals, and users must understand them well to use them for calculation tasks. In order to solve the challenging balance problem between versatility and user learning difficulty, a new distributed computing system is proposed. The design of this system is based on the idea of divide and conquer in distributed computing. The paper uses Netty, Zookeeper, Hadoop Distributed File System (HDFS) and other tools to implement the system functions. Firstly, the paper analyzes the research status of distributed computing frameworks at home and abroad; Then, according to the idea of divide and conquer, the paper uses Netty to implement the basic architecture of the system, uses Zookeeper and Netty to implement relevant distributed mechanisms, uses HDFS as the distributed storage, designs and implements a Task-Oriented General-Purpose (TOGP) distributed computing system, TOGP; Finally, the paper intensely studies the task scheduling problem of the system, proposes a scheduling algorithm, Node Processing Capacity and Distributed Lock Task Scheduling (CLTS) algorithm. The system scalability test proves that the TOGP has good horizontal scalability. In the comparative experiment, the processing capacity of the TOGP using the round robin is about 16% higher than that of Hadoop, which proves that the TOGP is available and efficient. A series of experiments prove that the CLTS scheduling algorithm can effectively adapt to the heterogeneous cluster environment and efficiently schedule tasks.
Keyword:
Distributed computing
load balance task-oriented
task scheduling
期刊
IF:
3.6
论文数:
9.8W
被引数:
29.4W
机构
引用论文
Development of a Methodology and Software Package for Predicting the Formation of Organic Deposits Based on the Results of Laboratory Studies.png)
FluidsOAAI
FluidsIF0
OAAI分享收藏A Survey on Spark Ecosystem: Big Data Processing Infrastructure, Machine Learning, and Applications
OAAI
OAAI分享收藏分享收藏
查看更多内容