Abstract:

In order to obtain single HDFS (Hadoop Distributed File System) file that cannot be provided by classi-cal MapReduce programming model,a new MapReduce programming model named SingleMapReduce is presented.In this mode,all files in an output directory are consolidated into a single HDFS file by intercepting Job Successfulstate.Then,four features of HDFS are summarized,and two concepts including Typical Distribution of Block andAtypical Distribution of Block are proposed,on the basis of which metadata are integrated to obtain integrated files.The results of theoretical analysis and experiments show that (1) one MapReduce computing on the basis of Sin-gleMapReduce helps achieve single output file; (2) the output produced by one MapReduce computing can be splitvia file splitting; (3) one large- scale table or one large- scale file can be imported into HDFS in a parallel manner;and (4) SingleMapReduce supports the scalability of name node in auxiliary.

Key words: distributed computing system, metadata, MapReduce, Hadoop distributed file system, name node;data node, block