关键词: 大数据, MapReduce, 垃圾回收, 内存分配, 性能优化

Abstract:

MapReduce job performance depends heavily on memory configurations.In order to overcome the difficulty in predicting the memory requirement of MapReduce jobs,on the basis of the fact that Java Virtual Machine ( JVM) divides the heap space managed by JVM Garbage Collector into young and old generations,a generational memory prediction method is proposed.In the method,first,a regression model to resolve average garbage collection time for a given young generation size is constructed.Then,the problem of looking for the rational size of young generation is converted into a constrained nonlinear optimization problem,and a fixed-size search algorithm is designed to solve the optimization problem.Moreover,memory models of the Map and Reduce tasks of MapReduce jobs are constructed to solve the memory requirement of optimal performance,thus obtaining reasonable old generation memory size of the Map and Reduce tasks.Finally,a k-means clustering algorithm is used to predict the value of parameter PretenureSizeThreshold,and JVM configurations are tuned to reduce garbage collection pause time.Experimental results show that the proposed method can accurately predict the memory requirements of the Map and Reduce tasks of MapReduce jobs,and it can significantly improve job performance.

Key words: big data, MapReduce, garbage collection, memory allocation, performance tuning