.png){kind=icon}
返回
SPsync: Lightweight multi-terminal big spatiotemporal data synchronization solution
.png){kind=icon}
PRE
AI
DOI：10.1016/j.future.2022.11.003.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
With the rapid increase in mobile device usage and the popularity of 5G communication technology, rich Big spatiotemporal Data has become an important research resource. How to quickly synchronize the Big spatiotemporal Data collected by various types of devices to multiple distributed data centers has become the key to further utilizing spatio-temporal data. Therefore, in this paper, a new lightweight Big Spatiotemporal Data synchronization scheme called SPsync, based on a traditional file synchronization algorithm, is proposed. The contributions of SPsync are as follows. (1) SPsync leads to the optimization of the file processing strategy, changing the algorithm from a serial algorithm to a parallel one. (2) Combined with Spark, a distributed Big Data processing framework is advanced to achieve optimization of distributed tasks and further improvement of algorithm performance. Further, we demonstrate the excellent performance of SPsync through rich experiments using multiple simulated and real datasets. Compared to the best incremental synchronization algorithms currently available, the synchronization speedup is 30% faster and CPU usage 20% lower. The algorithm itself is 20% faster and CPU usage reduced by more than 25% in a multi-node scenario.(c) 2022 Elsevier B.V. All rights reserved.
Keyword:
Content -based chunking
Data sync
Parallelization
Spark
期刊
F
IF:
6.1
论文数:
6.9K
被引数:
2.3W
机构
引用论文
.png){kind=icon}
Reconstructing 1-km-resolution high-quality PM2.5 data records from 2000 to 2018 in China: spatiotemporal variations and policy implications重建1公里分辨率的高质量PM2.5数据记录2000年中国2018年: 时空变化和政策含义
PREAI
分享收藏A Parallel Random Forest Algorithm for Big Data in a Spark Cloud Computing EnvironmentSpark云计算环境下的大数据并行随机森林算法
OAAI
OAAI分享收藏
查看更多内容