FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing
Text-driven motion generation has achieved substantial progress with the emergence of diffusion models. However, existing methods still struggle to generate complex motion sequences that correspond to fine-grained descriptions, depicting detailed and accurate spatio-temporal actions. This lack of fine controllability limits the usage of motion generation to a larger audience. To tackle these challenges, we present FineMoGen, a diffusion-based motion generation and editing framework that can synthesize fine-grained motions, with spatial-temporal composition to the user instructions. Specifically, FineMoGen builds upon diffusion model with a novel transformer architecture dubbed Spatio-Temporal Mixture Attention (SAMI). SAMI optimizes the generation of the global attention template from two perspectives: 1) explicitly modeling the constraints of spatio-temporal composition; and 2) utilizing sparsely-activated mixture-of-experts to adaptively extract fine-grained features. To facilitate a large-scale study on this new fine-grained motion generation task, we contribute the HuMMan-MoGen dataset, which consists of 2,968 videos and 102,336 fine-grained spatio-temporal descriptions. Extensive experiments validate that FineMoGen exhibits superior motion generation quality over state-of-the-art methods. Notably, FineMoGen further enables zero-shot motion editing capabilities with the aid of modern large language models (LLM), which faithfully manipulates motion sequences with fine-grained instructions. Project Page: https://mingyuan-zhang.github.io/projects/FineMoGen.html
PDF Abstract NeurIPS 2023 PDF NeurIPS 2023 AbstractTasks
Datasets
Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
---|---|---|---|---|---|---|
Motion Synthesis | HumanML3D | FineMoGen | FID | 0.151 | # 15 | |
Diversity | 9.263 | # 18 | ||||
Multimodality | 2.696 | # 4 | ||||
R Precision Top3 | 0.784 | # 11 | ||||
Motion Synthesis | KIT Motion-Language | FineMoGen | FID | 0.178 | # 2 | |
R Precision Top3 | 0.772 | # 2 | ||||
Diversity | 10.85 | # 12 | ||||
Multimodality | 1.877 | # 11 |