Improved performance optimization for massive small files in cloud computing environment

doi:10.1007/s10479-016-2376-0

2016-11-17

PRE

DOI：10.1007/s10479-016-2376-0

原文链接

原文求助

摘要

En 中文

Hadoop uses the Hadoop distributed file system for storing big data, and uses MapReduce to process big data in cloud computing environments. Because Hadoop is optimized for large file sizes, it has difficulties processing large numbers of small files. A small file can be defined as any file that is significantly smaller than the Hadoop block size, which is typically set to 64 MB. Hadoop is optimized to store data in relatively large files, and thus suffers from name node memory insufficiency and increased scheduling and processing time when processing large numbers of small files. This study proposes a performance improvement method for MapReduce processing, which integrates the CombineFileInputFormat method and the reuse feature of the Java Virtual Machine (JVM). Existing methods create a mapper for every small file. Unlike these methods, the proposed method reduces the number of created mappers by processing large numbers of files that are combined by a single split using CombineFileInputFormat. Moreover, to improve MapReduce processing performance, the proposed method reduces JVM creation time by reusing a single JVM to run multiple mappers (rather than creating a JVM for every mapper).

Keyword:

Massive small files

Hadoop

MapReduce

JVM reuse

CombineFileInputFormat

AI总结

对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。

期刊

Annals of Operations Research

IF:

4.5

论文数:

8.0K

被引数:

2.1W

机构

Chosun University

学者数:

3.8K

论文数: 4.3K

被引数: 2.9K

引用论文

Properties of Co/Ni codoped ZnO based nanocrystalline DMS

Journal of Magnetism and Magnetic Materials

IF0

2011-12-01

PREAI

Rezq Naji Aljawfi; S. Mollah

A method of DDoS attack detection using HTTP packet pattern and rule engine in cloud computing environment

SOFT COMPUTING

IF2.5

2014-03-11

PREAI

Choi, Junho; Choi, Chang; Ko, Byeongkyu; Kim, Pankoo

Practice of Controlled Temperature Chain (CTC) Technique during a Mass Vaccination Campaign in Côte d’Ivoire

World Journal of Vaccines

IF0

2016-01-01

OAAI

Damus Paquin Kouassi; Lépri Bernadin Nicaise Aka; Bi Vroh Joseph Bénié; Soplé Ruth Coulibaly; Dodji Blaise Tagodé; Daouda Coulibaly; Kouadio Félix Koffi; Patrick Lydon; Anna-Lea Kahn; Olivier Ronveaux

Chemiluminescence determination of chromium(VI), molybdenum(VI), and vanadium(V) with bis-(2,4,6-trichlorophenyl) oxalate and perylene

Collection of Czechoslovak Chemical Communications

IF0

1982-01-01

PREAI

Záviš Holzbecher; Lubomír Kábrt; Lubomír Janšta

Tectonic Units Division of Crystalline Basement in the Tarim Basin, NW China 塔里木盆地（NW China）结晶基底构造单元划分

Acta Geologica Sinica - English Edition

IF0

2015-12-06