.png){kind=icon}
返回
Efficient processing of complex XSD using Hive and Spark
.png){kind=icon}
OA
AI
DOI：10.7717/peerj-cs.652.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The eXtensible Markup Language (XML) files are widely used by the industry due to their flexibility in representing numerous kinds of data. Multiple applications such as financial records, social networks, and mobile networks use complex XML schemas with nested types, contents, and/or extension bases on existing complex elements or large real-world files. A great number of these files are generated each day and this has influenced the development of Big Data tools for their parsing and reporting, such as Apache Hive and Apache Spark. For these reasons, multiple studies have proposed new techniques and evaluated the processing of XML files with Big Data systems. However, a more usual approach in such works involves the simplest XML schemas, even though, real data sets are composed of complex schemas. Therefore, to shed light on complex XML schema processing for real-life applications with Big Data tools, we present an approach that combines three techniques. This comprises three main methods for parsing XML files: cataloging, deserialization, and positional explode. For cataloging, the elements of the XML schema are mapped into root, arrays, structures, values, and attributes. Based on these elements, the deserialization and positional explode are straightforwardly implemented. To demonstrate the validity of our proposal, we develop a case study by implementing a test environment to illustrate the methods using real data sets provided from performance management of two mobile network vendors. Our main results state the validity of the proposed method for different versions of Apache Hive and Apache Spark, obtain the query execution times for Apache Hive internal and external tables and Apache Spark data frames, and compare the query performance in Apache Hive with that of Apache Spark. Another contribution made is a case study in which a novel solution is proposed for data analysis in the performance management systems of mobile networks.
Keyword:
Hive
Spark
Performance management
Mobile network
Complex XSD
XML
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
2.5
论文数:
3.4K
被引数:
6.9K
机构
引用论文
Development and Evaluation of a Big Data Framework for Performance Management in Mobile Networks用于移动网络性能管理的大数据框架的开发和评估.png)
IEEE ACCESSOAAI
IEEE ACCESSIF3.6
OAAI分享收藏分享收藏Phase Space Reconstruction from a Biological Time Series: A Photoplethysmographic Signal Case StudyOAAI
OAAI分享收藏Effects of magnesium oxide and copper oxide nanoparticles on biofilm formation of Escherichia coli and Listeria monocytogenes氧化镁和氧化铜纳米颗粒对大肠杆菌和单核细胞增生李斯特菌生物膜形成的影响
PREAI
分享收藏
查看更多内容