Search Results for author:
Found 29 papers, 10 papers with code
Interspatial Attention for Efficient 4D Human Video Generation
In this paper, we introduce a new interspatial attention (ISA) mechanism as a scalable building block for modern diffusion transformer (DiT)--based video generation models.
SocialGen: Modeling Multi-Human Social Interaction with Language Models
Leveraging this dataset, we establish the first comprehensive benchmark for multi-human interaction tasks.
Self-Supervised Cross-View Correspondence with Predictive Cycle Consistency
Our approach beats all prior supervised and prior SoTA self-supervised correspondence models on the EgoExo4D correspondence benchmark (+6. 7 IoU Exo Query) and the prior SoTA self-supervised methods SiamMAE and DINO V1&V2 on the DAVIS-2017 and LVOS datasets across large frame gaps.
The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion
In this paper, we propose a novel framework that unifies verbal and non-verbal language using multimodal language models for human motion understanding and generation.
Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos
We propose a novel ambient-aware audio generation model, AV-LDM.
HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness
We study the problem of precisely swapping objects in videos, with a focus on those interacted with by hands, given one user-provided reference object image.
Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Prediction
We propose the first treatment of sim2real for audio-visual navigation by disentangling it into acoustic field prediction (AFP) and waypoint navigation.
ActiveRIR: Active Audio-Visual Exploration for Acoustic Environment Modeling
An environment acoustic model represents how sound is transformed by the physical characteristics of an indoor environment, for any given source/receiver location.
SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos
We propose a novel self-supervised embedding to learn how actions sound from narrated in-the-wild egocentric videos.
F$^3$Loc: Fusion and Filtering for Floorplan Localization
In this paper we propose an efficient data-driven solution to self-localization within a floorplan.
Overview of the L3DAS23 Challenge on Audio-Visual Extended Reality
The primary goal of the L3DAS23 Signal Processing Grand Challenge at ICASSP 2023 is to promote and support collaborative research on machine learning for 3D audio signal processing, with a specific emphasis on 3D speech enhancement and 3D Sound Event Localization and Detection in Extended Reality applications.
Audio Signal Processing
Sound Event Localization and Detection
+1
F3Loc: Fusion and Filtering for Floorplan Localization
In this paper we propose an efficient data-driven solution to self-localization within a floorplan.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
Measuring Acoustics with Collaborative Multiple Agents
As humans, we hear sound every second of our life.
Replay: Multi-modal Multi-view Acted Videos for Casual Holography
We introduce Replay, a collection of multi-view, multi-modal videos of humans interacting socially.
Novel-View Acoustic Synthesis
We introduce the novel-view acoustic synthesis (NVAS) task: given the sight and sound observed at a source viewpoint, can we synthesize the sound of that scene from an unseen target viewpoint?
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning
We introduce SoundSpaces 2. 0, a platform for on-the-fly geometry-based audio rendering for 3D environments.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Few-Shot Audio-Visual Learning of Environment Acoustics
Room impulse response (RIR) functions capture how the surrounding physical environment transforms the sounds heard by a listener, with implications for various applications in AR, VR, and robotics.
Sound Adversarial Audio-Visual Navigation
In this work, we design an acoustically complex environment in which, besides the target sound, there exists a sound attacker playing a zero-sum game with the agent.
Visual Acoustic Matching
We introduce the visual acoustic matching task, in which an audio clip is transformed to sound like it was recorded in a target environment.
Learning Audio-Visual Dereverberation
We introduce Visually-Informed Dereverberation of Audio (VIDA), an end-to-end approach that learns to remove reverberation based on both the observed monaural sound and visual scene.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Semantic Audio-Visual Navigation
We propose a transformer-based model to tackle this new semantic AudioGoal task, incorporating an inferred goal descriptor that captures both spatial and semantic properties of the target.
Learning to Set Waypoints for Audio-Visual Navigation
In audio-visual navigation, an agent intelligently travels through a complex, unmapped 3D environment using both sights and sounds to find a sound source (e. g., a phone ringing in another room).
VisualEchoes: Spatial Image Representation Learning through Echolocation
Several animal species (e. g., bats, dolphins, and whales) and even visually impaired humans have the remarkable ability to perform echolocation: a biological sonar used to perceive spatial layout and locate objects in the world.
SoundSpaces: Audio-Visual Navigation in 3D Environments
Moving around in the world is naturally a multisensory experience, but today's embodied agents are deaf---restricted to solely their visual perception of the environment.
Relational Graph Learning for Crowd Navigation
We present a relational graph learning approach for robotic crowd navigation using model-based deep reinforcement learning that plans actions by looking into the future.
Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning
We propose to (i) rethink pairwise interactions with a self-attention mechanism, and (ii) jointly model Human-Robot as well as Human-Human interactions in the deep reinforcement learning framework.
Constraint-Aware Deep Neural Network Compression
Deep neural network compression has the potential to bring modern resource-hungry deep networks to resource-limited devices.
