PeopleSansPeople: A Synthetic Data Generator for Human-Centric Computer Vision, arXiv

Paper     Source Code     macOS Binary     Linux Binary

Top row: PeopleSansPeople generated images.
Bottom row: corresponding COCO-style bounding box and keypoint labels.

Salehe Erfanian Ebadi, You-Cyuan Jhang, Alex Zook, Saurav Dhakad,
Adam Crespi, Pete Parisi, Steven Borkman, Jonathan Hogins, Sujoy Ganguly
Unity Technologies

---

People + Sans (Middle English for “without”) + People

A data generator for a few human-centric computer vision tasks without needing real-world human data.

---

   

---

Summary

• We introduce PeopleSansPeople, a human-centric privacy-preserving synthetic data generator with highly parametrized domain randomization.
• PeopleSansPeople contains simulation-ready 3D human assets, a parameterized lighting and camera system, and generates 2D and 3D bounding box, instance and semantic segmentation, and COCO pose labels.
• We use naïve ranges for the domain randomization and generate a synthetic dataset with labels.
• We provide some guarantees and analysis of human activities, poses, and context diversity on our synthetic data.
• We found that pre-training a network using synthetic data and fine-tuning on target real-world data (COCO-person train) resulted in bbox AP of 57.44 and keypoint AP of 66.83 (COCO-person validation) outperforming models trained with the same real data alone (bbox AP of 56.73 and keypoint AP of 65.12).

Abstract (click to expand)

In recent years, person detection and human pose estimation have made great strides, helped by large-scale labeled datasets. However, these datasets had no guarantees or analysis of human activities, poses, or context diversity. Additionally, privacy concerns may limit the ability to collect more data. An emerging alternative to real-world data that alleviates some of these issues is synthetic data. However, creation of synthetic data generators is incredibly challenging and prevents researchers from exploring their usefulness. Therefore, we release a human-centric synthetic data generator PeopleSansPeople which contains simulation-ready 3D human assets, a parameterized lighting and camera system, and generates 2D and 3D bounding box, instance and semantic segmentation, and COCO pose labels. Using PeopleSansPeople, we performed benchmark synthetic data training using a Detectron2 Keypont R-CNN variant. We found that pre-training a network using synthetic data and fine-tuning on target real-world data (COCO-person train) resulted in bbox AP of 57.44 and keypoint AP of 66.83 (COCO-person validation) outperforming models trained with the same real data alone (bbox AP of 56.73 and keypoint AP of 65.12). This freely available data generator should enable a wide range of research into the emerging field of simulation to real transfer learning in the critical area of human-centric computer vision.

What does PeopleSansPeople provide?

• 28 parameterized simulation-ready 3D human assets
• 39 diverse animation clips
• 21,952 unique clothing textures (from 28 albedos, 28 masks, and 28 normals)
• rameterized lighting
• Parameterized camera system
• Natural backgrounds
• Primitive occluders/distractors
• All packaged in a macOS and Linux binary

A comparison between PeopleSansPeople and the COCO person dataset.

	# train	# validation	# instances (train)	# instances w/ kpts (train)
COCO	64,115	2,693	262,465	149,813
PeopleSansPeople	490,000	10,000	>3,070,000	>2,900,000

Generated data and labels

PeopleSansPeople produces the following types of labels in COCO format: 2D bounding box, human keypoints, semantic and instance segmentation masks. In addition PeopleSansPeople generates 3D bounding boxes which are provided in Unity's Perception format.

           
Generated image and corresponding labels: 2D bounding box, human keypoints, semantic and instance segmentation masks in COCO format. 3D bounding box annotations are provided separately in [Unity Perception](https://github.com/Unity-Technologies/com.unity.perception) format.

Results

Here we show a comparison of gains obtained from pre-training on our synthetic data and fune-tuning on COCO person class over training from scratch on COCO. For each dataset size we show the results of the best performing model.

	bbox AP			keypoint AP
size of real data	scratch	w/ pre-train	Δ	scratch	w/ pre-train	Δ
641	13.82	42.58	+28.76	7.47	46.40	+38.93
6411	37.82	49.04	+11.22	39.48	55.21	+15.73
32057	52.15	55.04	+2.89	58.68	63.38	+4.70
64115	56.73	57.44	+0.71	65.12	66.83	+1.71

Unity Shader Graph randomizer for simulated clothing appearance diversity

Top row: our 3D human assets from RenderPeople with their original clothing textures.
Bottom row: using Shader Graph randomizers we are able to swap out clothing texture albedos, masks, and normals, yielding very diverse-looking textures on the clothing, without needing to swap out the clothing items themselves.

Additional examples

Additional images generated with PeopleSansPeople. Notice the high diversity of lighting, camera perspectives, scene background and occluders, as well as human poses, their proximity to each other and the camera, and the clothing texture variations. Our domain randomization is done here with naïvely-chosen ranges with uniform distributions. It is possible to drastically change the look and the structure of the scenes by varying the randomizer parameters.

Citation

@article{ebadi2021peoplesanspeople,
    title={PeopleSansPeople: A Synthetic Data Generator for Human-Centric Computer Vision},
    author={Salehe Erfanian Ebadi and You-Cyuan Jhang and Alex Zook and Saurav Dhakad 
            and Adam Crespi and Pete Parisi and Steven Borkman and Jonathan Hogins and Sujoy Ganguly},
    journal={arXiv},
    year={2021}
}

Source code

Code is available here

Unity Environment Template here

macOS and Linux binaries here

Unity Tutorial coming soon

Related links

• Unity's Perception Package
• Unity Computer Vision
• Unity's Perception Tutorial
• Unity's Human Pose Labeling and Randomization Tutorial
• SynthDet Project
• Robotics Object Pose Estimation Demo
• Unity Simulation Smart Camera Outdoor Example

License

PeopleSansPeople is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.