Search Results for author:
Found 20 papers, 8 papers with code
SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos
We propose a novel self-supervised embedding to learn how actions sound from narrated in-the-wild egocentric videos.
F$^3$Loc: Fusion and Filtering for Floorplan Localization
In this paper we propose an efficient data-driven solution to self-localization within a floorplan.
Overview of the L3DAS23 Challenge on Audio-Visual Extended Reality
The primary goal of the L3DAS23 Signal Processing Grand Challenge at ICASSP 2023 is to promote and support collaborative research on machine learning for 3D audio signal processing, with a specific emphasis on 3D speech enhancement and 3D Sound Event Localization and Detection in Extended Reality applications.
Audio Signal Processing
Sound Event Localization and Detection
+1
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
Measuring Acoustics with Collaborative Multiple Agents
As humans, we hear sound every second of our life.
Replay: Multi-modal Multi-view Acted Videos for Casual Holography
We introduce Replay, a collection of multi-view, multi-modal videos of humans interacting socially.
Novel-View Acoustic Synthesis
We introduce the novel-view acoustic synthesis (NVAS) task: given the sight and sound observed at a source viewpoint, can we synthesize the sound of that scene from an unseen target viewpoint?
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning
We introduce SoundSpaces 2. 0, a platform for on-the-fly geometry-based audio rendering for 3D environments.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Few-Shot Audio-Visual Learning of Environment Acoustics
Room impulse response (RIR) functions capture how the surrounding physical environment transforms the sounds heard by a listener, with implications for various applications in AR, VR, and robotics.
Sound Adversarial Audio-Visual Navigation
In this work, we design an acoustically complex environment in which, besides the target sound, there exists a sound attacker playing a zero-sum game with the agent.
Visual Acoustic Matching
We introduce the visual acoustic matching task, in which an audio clip is transformed to sound like it was recorded in a target environment.
Learning Audio-Visual Dereverberation
We introduce Visually-Informed Dereverberation of Audio (VIDA), an end-to-end approach that learns to remove reverberation based on both the observed monaural sound and visual scene.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Semantic Audio-Visual Navigation
We propose a transformer-based model to tackle this new semantic AudioGoal task, incorporating an inferred goal descriptor that captures both spatial and semantic properties of the target.
Learning to Set Waypoints for Audio-Visual Navigation
In audio-visual navigation, an agent intelligently travels through a complex, unmapped 3D environment using both sights and sounds to find a sound source (e. g., a phone ringing in another room).
VisualEchoes: Spatial Image Representation Learning through Echolocation
Several animal species (e. g., bats, dolphins, and whales) and even visually impaired humans have the remarkable ability to perform echolocation: a biological sonar used to perceive spatial layout and locate objects in the world.
SoundSpaces: Audio-Visual Navigation in 3D Environments
Moving around in the world is naturally a multisensory experience, but today's embodied agents are deaf---restricted to solely their visual perception of the environment.
Relational Graph Learning for Crowd Navigation
We present a relational graph learning approach for robotic crowd navigation using model-based deep reinforcement learning that plans actions by looking into the future.
Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning
We propose to (i) rethink pairwise interactions with a self-attention mechanism, and (ii) jointly model Human-Robot as well as Human-Human interactions in the deep reinforcement learning framework.
Constraint-Aware Deep Neural Network Compression
Deep neural network compression has the potential to bring modern resource-hungry deep networks to resource-limited devices.
