.png){kind=icon}
返回
A service-oriented framework for large-scale documents processing and application via 3D models and feature extraction
.png){kind=icon}
PRE
AI
DOI：10.1016/j.simpat.2024.102903.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Educational big data analysis is facilitated by the significant amount of unstructured data found in education institutions. Python has various toolkits for both structured and unstructured data processing. However, its ability for processing large-scale data is limited. On the other hand, Spark is a big data processing framework, but it does not have the needed toolkits for processing unstructured rich text documents, 3D model and image processing. In this study, we develop a generic framework that integrates Python toolkits and Spark based on service-oriented architecture. The framework automatically extends the serial algorithm written in Python to distributed algorithm to accomplish parallel processing tasks seamlessly. First, our focus is on achieving non-intrusive deployment to Spark servers and how to run Python codes in Spark environment to process rich text documents. Second, we propose a compression-based schema to address the poor performance of small sized files in HDFS. Finally, we design a generic model that can process different types of poly-structured data such as 3D models and images. We published the services used in the system for sharing them at https level for constructing different systems. It is evaluated through simulation experiments using large-scale rich text documents, 3D models and images. According to the results, the speedup is 49 times faster than the standalone Python-docx in the simulations of extracting 232 GB docx files when eight physical nodes with 128 cores are used. It reaches about 89 times after further compression schema is applied. In addition, simulations for 3D model descriptors' extraction show that the simulation achieves a speedup of about 116 times. In the large-scale image's HOG features extraction simulation task of up to 256.7 GB (6,861,024 images), a speedup of up to 110 times is achieved.
Keyword:
Service -oriented computing
Spark
Generic parallel framework
HDFS small files problem
Poly -structured data processing and simulation
期刊
IF:
4.6
论文数:
2.6K
被引数:
4.8K
机构
引用论文
A comprehensive performance analysis of Apache Hadoop and Apache Spark for large scale data sets using HiBench.png)
JOURNAL OF BIG DATA
OAAI
JOURNAL OF BIG DATAIF6.4
OAAI分享收藏分享收藏Distributed fuzzy clustering algorithm for mixed-mode data in Apache SPARKJOURNAL OF BIG DATA
OAAI
JOURNAL OF BIG DATAIF6.4
OAAI分享收藏Multi-view 3D model retrieval based on enhanced detail features with contrastive center loss基于增强细节特征的对比中心缺失多视图三维模型检索
PREAI
分享收藏Microwave Dielectric Properties of (111) Oriented Ferroelectric (Ba 1 - x Sr x )TiO 3 Thin Films(111)取向铁电(Ba₁₋ₓSrₓ)TiO₃薄膜的微波介电性能
PREAI
分享收藏
查看更多内容