SuperAnimal pretrained pose estimation models for behavioral analysis

Quantification of behavior is critical in applications ranging from neuroscience, veterinary medicine and animal conservation efforts. A common key step for behavioral analysis is first extracting relevant keypoints on animals, known as pose estimation. However, reliable inference of poses currently requires domain knowledge and manual labeling effort to build supervised models. We present a series of technical innovations that enable a new method, collectively called SuperAnimal, to develop unified foundation models that can be used on over 45 species, without additional human labels. Concretely, we introduce a method to unify the keypoint space across differently labeled datasets (via our generalized data converter) and for training these diverse datasets in a manner such that they don't catastrophically forget keypoints given the unbalanced inputs (via our keypoint gradient masking and memory replay approaches). These models show excellent performance across six pose benchmarks. Then, to ensure maximal usability for end-users, we demonstrate how to fine-tune the models on differently labeled data and provide tooling for unsupervised video adaptation to boost performance and decrease jitter across frames. If the models are fine-tuned, we show SuperAnimal models are 10-100$\times$ more data efficient than prior transfer-learning-based approaches. We illustrate the utility of our models in behavioral classification in mice and gait analysis in horses. Collectively, this presents a data-efficient solution for animal pose estimation.

PDF Abstract

Results from the Paper

 Ranked #1 on Animal Pose Estimation on Animal-Pose Dataset (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Animal Pose Estimation Animal-Pose Dataset SuperAnimal-AnimalTokenPose AP 86 # 1
Animal Pose Estimation AP-10K zero-shot SuperAnimal-HRNetw32 AP 68.038 # 10
Animal Pose Estimation AP-10K SuperAnimal-HRNetw32 AP 80.113 # 3
Animal Pose Estimation Horse-10 SuperAnimal-Quadruped HRNet-w32 Normalized Error (OOD) 0.1091 # 1
Animal Pose Estimation Horse-10 mmpose HRNet-w32 (w/ImageNet pretrained weights) Normalized Error (OOD) 0.179 # 2
Animal Pose Estimation TriMouse-161 SuperAnimal HRNetw32 mAP 98.547 # 2
Animal Pose Estimation TriMouse-161 zero-shot SuperAnimal HRNetw32 mAP 76.139 # 7


No methods listed for this paper. Add relevant methods here