Search Results for author:
Found 9 papers, 4 papers with code
Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedback
We describe an approach for aligning an LLM-based dialogue agent based on global (i. e., dialogue-level) rewards, while also taking into account naturally-occurring multimodal signals.
HIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory Transformer
Accurately modeling affect dynamics, which refers to the changes and fluctuations in emotions and affective displays during human conversations, is crucial for understanding human interactions.
Multipar-T: Multiparty-Transformer for Capturing Contingent Behaviors in Group Conversations
As we move closer to real-world AI systems, AI agents must be able to deal with multiparty (group) conversations.
Lecture Presentations Multimodal Dataset: Towards Understanding Multimodality in Educational Videos
We introduce three research tasks, (1) figure-to-text retrieval, (2) text-to-figure retrieval, and (3) generation of slide explanations, which are grounded in multimedia learning and psychology principles to test a vision-language model's understanding of multimodal content.
Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides
As a step toward developing AI to aid in student learning as intelligent teacher assistants, we introduce the Multimodal Lecture Presentations dataset as a large-scale benchmark testing the capabilities of machine learning models in multimodal understanding of educational content.
Low-Resource Adaptation for Personalized Co-Speech Gesture Generation
Personalizing an avatar for co-speech gesture generation from spoken language requires learning the idiosyncrasies of a person's gesture style from a small amount of data.
Crossmodal clustered contrastive learning: Grounding of spoken language to gesture
Crossmodal grounding is a key challenge for the task of generating relevant and well-timed gestures from just spoken language as an input.
No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures
We study relationships between spoken language and co-speech gestures in context of two key challenges.
Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approach
A key challenge, called gesture style transfer, is to learn a model that generates these gestures for a speaking agent 'A' in the gesturing style of a target speaker 'B'.
