Occlusion is one of the most challenging problems in object tracking community. To deal with the occlusion problem, this paper presents a local salient feature based probabilistic graphical model for visual tracking. Combining spatial and temporal constraints among different ROIs and local information contained in each ROI, the object is represented as a probabilistic graphical model. Finally, based on the object model and Mean Shift tracking results of each ROI, Probabilistic inference algorithm is adopted to estimate the probability of each ROI belongs to object region. Comprehensive experiments on several testing videos show, compared with three wellknown trackers, i.e. improved Mean Shift, particle filter and fragmentsbased tracker, the proposed method has a higher tracking accuracy and robustness, especially in object occlusion condition. The proposed tracker, using local salient information and spatial and temporal structure constraint of tracking object effectively, can perform with high robustness in complex realworld scenarios such as object occlusion, changes of posture and illumination etc.