.png){kind=icon}
返回
Parallel naive Bayes algorithm for large-scale Chinese text classification based on spark
.png){kind=icon}
PRE
AI
DOI：10.1007/s11771-019-3978-x.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The sharp increase of the amount of Internet Chinese text data has significantly prolonged the processing time of classification on these data. In order to solve this problem, this paper proposes and implements a parallel naive Bayes algorithm (PNBA) for Chinese text classification based on Spark, a parallel memory computing platform for big data. This algorithm has implemented parallel operation throughout the entire training and prediction process of naive Bayes classifier mainly by adopting the programming model of resilient distributed datasets (RDD). For comparison, a PNBA based on Hadoop is also implemented. The test results show that in the same computing environment and for the same text sets, the Spark PNBA is obviously superior to the Hadoop PNBA in terms of key indicators such as speedup ratio and scalability. Therefore, Spark-based parallel algorithms can better meet the requirement of large-scale Chinese text data mining.
Keyword:
Chinese text classification
naive Bayes
spark
hadoop
resilient distributed dataset
parallelization
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
4.4
论文数:
5.2K
被引数:
1.0W
机构
引用论文
Safety and Efficacy of Stereotactic Body Radiotherapy for Ultra-Central Thoracic Tumors立体定向体部放射治疗在超中心型胸肿瘤中的安全性与有效性
OAAI
.png){kind=icon}
OAAI分享收藏分享收藏DSM5 : quels changements pour les troubles spécifiques du langage oral ?DSM5 : quels changememes倾诉麻烦？
PREAI
分享收藏A comparative study of TF*IDF, LSI and multi-words for text classificationTF * IDF、LSI和多词文本分类的比较研究
PREAI
分享收藏 没有更多内容
没有更多内容