Back to Tao Zhao's Research Page

Segmentation and Tracking of Multiple Humans in Complex Situations

Tao Zhao, Ram Nevatia and Fengjun Lv
{taozhao|nevatia|flv} (at) iris (o) usc (o) edu
CVPR 2001 (IEEE Conference on Computer Vision and Pattern Recognition), Kauai, Hawaii, Dec., 2001.


Segmenting and tracking multiple humans is a challenging problem in complex situations in which extended occlusion, shadow and/or reflection exists. We tackle this problem with a 3D model-based approach. Our method includes two stages, segmentation (detection) and tracking. Human hypotheses are generated by shape analysis of the foreground blobs using human shape model. The segmented human hypotheses are tracked with a Kalman filter with explicit handling of occlusion. Hypotheses are verified while they are tracked for the first second or so. The verification is done by walking recognition using an articulated human walking model. We propose a new method to recognize walking using motion template and temporal integration. Experiments show that our approach works robustly in very challenging sequences.

Tracking results (selected frames):

Blue ellipses are the human objects we tracked; rectangles are the bounding boxes of the blobs (different colors show different tracks).
The positions and orientations of the human objects are warped onto the ground plane (corresponding to the second frame above):
The AVI files of the results are available: tracking result (2M), tracking result in 3D (680K).

Verification results by walking motion:

The blue one got verified while the red one got rejected because it does not exhibit walking like motion. The AVI file is here (100K). Other verification results include: single human (37K), two humans (72K), lady in shirt (80K).

Download PDF file of the paper.