This thesis addresses the multi-person tracking task with two types of representation: body pose and segmentation mask. We explore these scenarios in the semi-supervised setting, where one available annotation is available per person during test time. More complex representations of people (segmentation mask and body pose) can provide richer understanding of visual scenes, and methods that leverage supervision during test time should be developed for the cases when supervision is available. We propose HumanMaskTracker for the task of semi-supervised multi-person mask tracking. Our approach builds on recent techniques proposed for the task of video object segmentation. These include the mask refinement approach, training with syn- thetic data, fine-tuning per object and leveraging optical flow. In addition, we propose leveraging instance semantic segmentation proposals to give the tracker a better notion about the human class. Moreover, we propose modeling people occlusions inside the data synthesis process to make the tracker more robust to the challenges of occlusion and disocclusion. For the task of semi-supervised multi-person pose tracking, we propose the method HumanPoseTracker. We show that the task of multi-person pose tracking can benefit significantly from using one pose supervision per track during test time. Fine-tuning per object and leveraging optical flow, techniques proposed for the task of video object segmentation, prove to be highly effective for supervised pose tracking as well. Also, we propose a technique to remove false positive joint detections and develop tracking stopping criteria. A promising application of our work is presented by extending the method to generate dense from sparse annotations in videos.