Object tracking has always been an important research topic in the field of computer vision． It is widely used in video surveillance，traffic monitoring，medical diagnosis and other fields． An object tracking algorithm based on binary tree model was proposed． The method divides the target area of the image into several homogene-ous blocks of different sizes，following the rule of the binary tree partition． The pixels in the block are similar and can be represented by a single value or vector whereas the pixels in different blocks differ from each significantly，thus forming the feature description model of the whole object． The CT algorithm，the quadtree model based algo-rithm (QT algorithm) and the proposed binary tree model based algorithm (BT algorithm) were compared from the aspects of accuracy and tracking speed． The results show that compared with the quadtree-based algorithm，the BT-based tracking algorithm can improve the tracking speed significantly without reducing the tracking accuracy．And compared with the discriminant CT-based algorithm，which is known for its fast tracking speed，the BT-based tracking accuracy is even better under the premise that the tracking speed is roughly equal．