Search Results for author:
Found 36 papers, 16 papers with code
ControlTac: Force- and Position-Controlled Tactile Data Augmentation with a Single Reference Image
Vision-based tactile sensing has been widely used in perception, reconstruction, and robotic manipulation.
Learning to Highlight Audio by Watching Movies
To train our model, we also introduce a new dataset -- the muddy mix dataset, leveraging the meticulous audio and video crafting found in movies, which provides a form of free supervision.
Differentiable Room Acoustic Rendering with Multi-View Vision Priors
An immersive acoustic experience enabled by spatial audio is just as crucial as the visual aspect in creating realistic virtual environments.
Hearing Anywhere in Any Environment
In mixed reality applications, a realistic acoustic experience in spatial environments is as crucial as the visual experience for achieving true immersion.
Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMs
This performance gain highlights the potential of reasoning-enhanced data generation for advancing AVLLMs in real-world applications.
AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs
With the rapid advancement of Multi-modal Large Language Models (MLLMs), several diagnostic benchmarks have recently been developed to assess these models' multi-modal reasoning proficiency.
Novel View Acoustic Parameter Estimation
The task of Novel View Acoustic Synthesis (NVAS) - generating Room Impulse Responses (RIRs) for unseen source and receiver positions in a scene - has recently gained traction, especially given its relevance to Augmented Reality (AR) and Virtual Reality (VR) development.
Spherical World-Locking for Audio-Visual Localization in Egocentric Videos
Egocentric videos provide comprehensive contexts for user and scene understanding, spanning multisensory perception to behavioral interaction.
Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time
Leveraging Large Language Models' remarkable proficiency in text-based tasks, recent works on Multi-modal LLMs (MLLMs) extend them to other modalities like vision and audio.
Hearing Anything Anywhere
Recent years have seen immense progress in 3D computer vision and computer graphics, with emerging tools that can virtualize real-world 3D environments for numerous Mixed Reality (XR) applications.
FlexLLM: A System for Co-Serving Large Language Model Inference and Parameter-Efficient Finetuning
Finetuning large language models (LLMs) is essential for task adaptation, yet serving stacks today isolate inference and finetuning on separate GPU clusters -- wasting resources and under-utilizing hardware.
The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective
We propose a unified multi-modal framework -- Audio-Visual Conversational Attention (AV-CONV), for the joint prediction of conversation behaviors -- speaking and listening -- for both the camera wearer as well as all other social partners present in the egocentric video.
SoundCam: A Dataset for Finding Humans Using Room Acoustics
A room's acoustic properties are a product of the room's geometry, the objects within the room, and their specific positions.
NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities
We present Neural Signal Operated Intelligent Robots (NOIR), a general-purpose, intelligent brain-robot interface system that enables humans to command robots to perform everyday activities through brain signals.
RealImpact: A Dataset of Impact Sound Fields for Real Objects
Objects make unique sounds under different perturbations, environment conditions, and poses relative to the listener.
Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hear
We introduce Sonicverse, a multisensory simulation platform with integrated audio-visual simulation for training household agents that can both see and hear.
The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects
We introduce the ObjectFolder Benchmark, a benchmark suite of 10 tasks for multisensory object-centric learning, centered around object recognition, reconstruction, and manipulation with sight, sound, and touch.
An Extensible Multimodal Multi-task Object Dataset with Materials
For example, we can train a model to predict the object category from the listing text, or the mass and price from the product listing image.
Learning Object-Centric Neural Scattering Functions for Free-Viewpoint Relighting and Scene Composition
We propose Object-Centric Neural Scattering Functions (OSFs) for learning to reconstruct object appearance from only images.
See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation
Humans use all of their senses to accomplish different tasks in everyday activities.
Differentiable Physics Simulation of Dynamics-Augmented Neural Objects
A robot can use this simulation to optimize grasps and manipulation trajectories of neural objects, or to improve the neural object models through gradient-based real-to-simulation transfer.
ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer
We present ObjectFolder 2. 0, a large-scale, multisensory dataset of common household objects in the form of implicit neural representations that significantly enhances ObjectFolder 1. 0 in three aspects.
Visual Acoustic Matching
We introduce the visual acoustic matching task, in which an audio clip is transformed to sound like it was recorded in a target environment.
Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Video
Binaural audio provides human listeners with an immersive spatial sound experience, but most existing videos lack binaural audio recordings.
ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations
Multisensory object-centric perception, reasoning, and interaction have been a key research topic in recent years.
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
Given a video, the goal is to extract the speech associated with a face in spite of simultaneous background sounds and/or other human speakers.
Learning to Set Waypoints for Audio-Visual Navigation
In audio-visual navigation, an agent intelligently travels through a complex, unmapped 3D environment using both sights and sounds to find a sound source (e. g., a phone ringing in another room).
VisualEchoes: Spatial Image Representation Learning through Echolocation
Several animal species (e. g., bats, dolphins, and whales) and even visually impaired humans have the remarkable ability to perform echolocation: a biological sonar used to perceive spatial layout and locate objects in the world.
Listen to Look: Action Recognition by Previewing Audio
In the face of the video data deluge, today's expensive clip-level classifiers are increasingly impractical.
Ranked #8 on
Action Recognition
on ActivityNet
Co-Separating Sounds of Visual Objects
Learning how objects sound from video is challenging, since they often heavily overlap in a single audio channel.
Ranked #1 on
Audio Denoising
on AV-Bench - Wooden Horse
2.5D Visual Sound
We devise a deep convolutional neural network that learns to decode the monaural (single-channel) soundtrack into its binaural counterpart by injecting visual information about object and scene configurations.
Learning to Separate Object Sounds by Watching Unlabeled Video
Our work is the first to learn audio source separation from large-scale "in the wild" videos containing multiple audio sources per video.
Im2Flow: Motion Hallucination from Static Images for Action Recognition
Second, we show the power of hallucinated flow for recognition, successfully transferring the learned motion into a standard two-stream network for activity recognition.
On-Demand Learning for Deep Image Restoration
While machine learning approaches to image restoration offer great promise, current methods risk training models fixated on performing well only for image corruption of a particular level of difficulty---such as a certain level of noise or blur.
Object-Centric Representation Learning from Unlabeled Videos
Compared to existing temporal coherence methods, our idea has the advantage of lightweight preprocessing of the unlabeled video (no tracking required) while still being able to extract object-level regions from which to learn invariances.
