Abstract:

As the malware detection method based on API can only detect the malware running in user mode and is noneffective for the malware running in kernel mode and calling kernel APIs,a novel detection method of unknown malware is proposed based on IRP（I/O Request Packet）.Then,the Nave Bayes,the Bayesian networks,the support vector machine,the C4.5 decision tree,the Boosting,the negative selection algorithm and an improved artificial immune system are used to classify IRP sequences for malware detection,and the detection rates of all the above-mentioned methods with different feature selection algorithms are compared.The results demonstrate that（1） the proposed method is effective in malware detection;（2） the Boosting decision tree with Fisher score feature selection algorithm outperforms other detection methods,with the highest detection rate of 98.3%;（3） the improved artificial immune system,which detects malware by selected IRP subsequences only existing in malware＇s IRP sequences,performs well,with a detection rate of 95.0% and a false detection rate of 0.

Key words: I/O Request Packet, artificial immune system, data mining, malware detection, feature selection, detection rate, false detection rate