Abstract: Unlink attack is a kind of attack against heap-based overflow vulnerability in Linux. Existed detection technology of the buffer overflow attack find the vulnerability trigger point and generate testcase by checking the control flow state. However, the heap-based overflow data seldom lead to the control flow hijack and the protection mechanisms limit the trigger condition, it is hard to judge the program whether or not the conditions of unlink attack is satisfied through existed detection technology. To improve the security of software, and detect the unlink attack, this paper summarized the features of unlink attack according to analyzing the instances, built the detection model of unlink attack, and proposed the unlink detection method based on the model. This method monitored the input data and sensitive cooperation of program by using taint analysis; built the path constraint of tainted data and data constraint which is satisfied the condition of unlink attack by selective symbolic execution; through solving the constraints above, judged if the program can be attacked by unlink and generated testcase. The experiments showed that this method can detect the unlink attack effectively.

Key words: heap-based overflow, buffer overflow, unlink attack, symbol execution, taint analysis