• The Sims4Action Dataset: a videogame-based dataset for Synthetic→Real domain adaptation for human activity recognition.

• Goal : Exploring the concept of constructing training examples for Activities of Daily Living (ADL) recognition by playing life simulation video games.

• Sims4Action dataset is created with the commercial game THE SIMS 4 by executing actions-of-interest within the game in a "top-down" manner. It features ten hours of video material of eight diverse characters and multiple environments. Ten actions are selected to have a direct correspondence to categories covered in the real-life dataset Toyota Smarthome [2] to enable the research of Synthetic→Real transfer in action recognition.
• Two benchmarks : Gaming→Gaming (training and evaluation on Sims4Action) and Gaming→Real (training on Sims4Action, evaluation on the real Toyota Smarthome data [2]).
• Main challenge: Gaming→Real domain adaptation
  While ADL recognition on gaming data is interesting from a theoretical perspective, the key challenge arises from transferring knowledge learned from simulated data to real-world applications. Sims4Action specifically provides a benchmark for this scenario since it describes a Gaming→Real challenge, which evaluates models on real videos derived from the existing Toyota Smarthome dataset .

References

[1] Let's Play for Action: Recognizing Activities of Daily Living by Learning from Life Simulation Video Games. Alina Roitberg, David Schneider, Aulia Djamal, Constantin Seibold, Simon Reiß, Rainer Stiefelhagen, In International Conference on Intelligent Robots and Systems (IROS), 2021 (* denotes equal contribution.)

[2] Toyota smarthome: Real-world activities of daily living. Srijan Das, Rui Dai, Michal Koperski, Luca Minciullo, Lorenzo Garattoni, Francois Bremond, Gianpiero Francesca, In International Conference on Computer Vision (ICCV), 2019.