Search Results for author:
Found 12 papers, 7 papers with code
A Comprehensive Review of Data-Driven Co-Speech Gesture Generation
Finally, we identify key research challenges in gesture generation, including data availability and quality; producing human-like motion; grounding the gesture in the co-occurring speech in interaction with other speakers, and in the environment; performing gesture evaluation; and integration of gesture synthesis into applications.
Lecture Presentations Multimodal Dataset: Towards Understanding Multimodality in Educational Videos
We introduce three research tasks, (1) figure-to-text retrieval, (2) text-to-figure retrieval, and (3) generation of slide explanations, which are grounded in multimedia learning and psychology principles to test a vision-language model's understanding of multimodal content.
Continual Learning for Personalized Co-speech Gesture Generation
However in practical scenarios, speaker data comes sequentially and in small amounts as the agent personalizes with more speakers, akin to a continual learning paradigm.
Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides
As a step toward developing AI to aid in student learning as intelligent teacher assistants, we introduce the Multimodal Lecture Presentations dataset as a large-scale benchmark testing the capabilities of machine learning models in multimodal understanding of educational content.
Low-Resource Adaptation for Personalized Co-Speech Gesture Generation
Personalizing an avatar for co-speech gesture generation from spoken language requires learning the idiosyncrasies of a person's gesture style from a small amount of data.
Crossmodal clustered contrastive learning: Grounding of spoken language to gesture
Crossmodal grounding is a key challenge for the task of generating relevant and well-timed gestures from just spoken language as an input.
No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures
We study relationships between spoken language and co-speech gestures in context of two key challenges.
Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approach
A key challenge, called gesture style transfer, is to learn a model that generates these gestures for a speaking agent 'A' in the gesturing style of a target speaker 'B'.
To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations
In this paper, we introduce a neural architecture named Dyadic Residual-Attention Model (DRAM), which integrates intrapersonal (monadic) and interpersonal (dyadic) dynamics using selective attention to generate sequences of body pose conditioned on audio and body pose of the interlocutor and audio of the human operating the avatar.
Language2Pose: Natural Language Grounded Pose Forecasting
In this paper, we address this multimodal problem by introducing a neural architecture called Joint Language to Pose (or JL2P), which learns a joint embedding of language and pose.
Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling
We evaluate this family on new LRU models on computational convergence rates and statistical efficiency.
Multimodal Machine Learning: A Survey and Taxonomy
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors.
