Search Results for author:
Found 38 papers, 16 papers with code
Towards Understanding the Relation between Gestures and Language
In this paper, we explore the relation between gestures and language.
Factorized Diffusion: Perceptual Illusions by Noise Decomposition
And we explore a decomposition by a motion blur kernel, which produces images that change appearance under motion blurring.
Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark
The dataset includes high-quality and densely captured room impulse response data paired with multi-view images, and precise 6DoF pose tracking data for sound emitters and listeners in the rooms.
Binding Touch to Everything: Learning Unified Multimodal Tactile Representations
We introduce UniTouch, a unified tactile model for vision-based touch sensors connected to multiple modalities, including vision, language, and sound.
Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators
Diffusion models are capable of generating impressive images conditioned on text descriptions, and extensions of these models allow users to edit images at a relatively coarse scale.
Visual Anagrams: Generating Multi-View Optical Illusions with Diffusion Models
During the reverse diffusion process, we estimate the noise from different views of a noisy image, and then combine these noise estimates together and denoise the image.
Generating Visual Scenes from Touch
An emerging line of work has sought to generate plausible imagery from touch.
Conditional Generation of Audio from Video via Foley Analogies
Second, we propose a model for generating a soundtrack for a silent input video, given a user-supplied example that specifies what the video should "sound like".
Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment
The key idea is to enrich the audio features with visual information by learning to align audio to visual latent space.
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models
We present Text2Room, a method for generating room-scale textured 3D meshes from a given text prompt as input.
Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation
In this paper, we use these cues to solve a problem we call Sound Localization from Motion (SLfM): jointly estimating camera rotation and localizing sound sources.
EXIF as Language: Learning Cross-Modal Associations Between Images and Camera Metadata
We learn a visual representation that captures information about the camera that recorded a given photo.
Self-Supervised Video Forensics by Audio-Visual Anomaly Detection
Manipulated videos often contain subtle inconsistencies between their visual and audio signals.
Ranked #3 on
DeepFake Detection
on FakeAVCeleb
Mix and Localize: Localizing Sound Sources in Mixtures
This task requires a model to both group a sound mixture into individual sources, and to associate them with a visual signal.
Touch and Go: Learning from Human-Collected Vision and Touch
The ability to associate touch with sight is essential for tasks that require physically interacting with objects in the world.
Learning Visual Styles from Audio-Visual Associations
Our model learns to manipulate the texture of a scene to match a sound, a problem we term audio-driven image stylization.
Sound Localization by Self-Supervised Time Delay Estimation
We adapt the contrastive random walk of Jabri et al. to learn a cycle-consistent representation from unlabeled stereo sounds, resulting in a model that performs on par with supervised methods on "in the wild" internet recordings.
Learning Pixel Trajectories with Multiscale Contrastive Random Walks
A range of video modeling tasks, from optical flow to multiple object tracking, share the same fundamental challenge: establishing space-time correspondence.
GANmouflage: 3D Object Nondetection with Texture Fields
Our model learns to camouflage a variety of object shapes from randomly sampled locations and viewpoints within the input scene, and is the first to address the problem of hiding complex object shapes.
Structure from Silence: Learning Scene Structure from Ambient Sound
From whirling ceiling fans to ticking clocks, the sounds that we hear subtly vary as we move through a scene.
Comparing Correspondences: Video Prediction with Correspondence-wise Losses
Image prediction methods often struggle on tasks that require changing the positions of objects, such as video prediction, producing blurry images that average over the many positions that objects might occupy.
Strumming to the Beat: Audio-Conditioned Contrastive Video Textures
We learn representations for video frames and frame-to-frame transition probabilities by fitting a video-specific model trained using contrastive learning.
Planar Surface Reconstruction from Sparse Views
The paper studies planar surface reconstruction of indoor scenes from two views with unknown camera poses.
Contrastive Video Textures
By randomly traversing edges with high transition probabilities, we generate diverse temporally smooth videos with novel sequences and transitions.
Self-Supervised Learning of Audio-Visual Objects from Video
Our objective is to transform a video into a set of discrete audio-visual objects using self-supervised learning.
Space-Time Correspondence as a Contrastive Random Walk
We cast correspondence as prediction of links in a space-time graph constructed from video.
CNN-generated images are surprisingly easy to spot... for now
In this work we ask whether it is possible to create a "universal" detector for telling apart real images from these generated by a CNN, regardless of architecture or dataset used.
Detecting Photoshopped Faces by Scripting Photoshop
Most malicious photo manipulations are created using standard image editing tools, such as Adobe Photoshop.
Learning Individual Styles of Conversational Gesture
Specifically, we perform cross-modal translation from "in-the-wild'' monologue speech of a single speaker to their hand and arm motion.
Ranked #4 on
Gesture Generation
on BEAT
MoSculp: Interactive Visualization of Shape and Time
We present a system that allows users to visualize complex human motion via 3D motion sculptures---a representation that conveys the 3D structure swept by a human body as it moves through space.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
This model -- a deep, multimodal convolutional network -- predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions.
Fighting Fake News: Image Splice Detection via Learned Self-Consistency
In this paper, we propose a learning algorithm for detecting visual image manipulations that is trained only using a large dataset of real photographs.
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
The thud of a bouncing ball, the onset of speech as lips open -- when visual and audio events occur together, it suggests that there might be a common, underlying event that produced both signals.
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
The sound of crashing waves, the roar of fast-moving cars -- sound conveys important information about the objects in our surroundings.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
In this work, we investigate the question of whether touch sensing aids in predicting grasp outcomes within a multimodal sensing framework that combines vision and touch.
Ambient Sound Provides Supervision for Visual Learning
We show that, through this process, the network learns a representation that conveys information about objects and scenes.
Camouflaging an Object from Many Viewpoints
We address the problem of camouflaging a 3D object from the many viewpoints that one might see it from.
