Search Results for author:
Found 33 papers, 13 papers with code
Meta-Personalizing Vision-Language Models to Find Named Instances in Video
Large-scale vision-language models (VLM) have shown impressive results for language-guided search applications.
Language-Guided Music Recommendation for Video via Prompt Analogies
A key challenge of this problem setting is that existing music video datasets provide the needed (video, music) training pairs, but lack text descriptions of the music.
Conditional Generation of Audio from Video via Foley Analogies
Second, we propose a model for generating a soundtrack for a silent input video, given a user-supplied example that specifies what the video should "sound like".
Language-Guided Audio-Visual Source Separation via Trimodal Consistency
We propose a self-supervised approach for learning to perform audio source separation in videos based on natural language queries, using only unlabeled video and audio pairs as training data.
Monocular Dynamic View Synthesis: A Reality Check
We study the recent progress on dynamic view synthesis (DVS) from monocular video.
It's Time for Artistic Correspondence in Music and Video
In order to capture the high-level concepts that are required to solve the task, we propose modeling the long-term temporal context of both the video and the music signals, using Transformer networks for each modality.
Neural Volumetric Object Selection
We introduce an approach for selecting objects in neural volumetric 3D representations, such as multi-plane images (MPI) and neural radiance fields (NeRF).
Focal Length and Object Pose Estimation via Render and Compare
We introduce FocalPose, a neural render-and-compare method for jointly estimating the camera-object 6D pose and camera focal length given a single RGB input image depicting a known object.
Look at What I’m Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
Key to our approach is the ability to learn to spatially localize interactions with self-supervision on a large corpus of videos with accompanying transcribed narrations.
Contrastive Feature Loss for Image Prediction
Training supervised image synthesis models requires a critic to compare two images: the ground truth to the result.
Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
Key to our approach is the ability to learn to spatially localize interactions with self-supervision on a large corpus of videos with accompanying transcribed narrations.
Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions
Our task poses unique challenges as a system does not know what types of human-object interactions are present in a video or the actual spatiotemporal location of the human and the object.
Human-Object Interaction Detection
Weakly-supervised Learning
Editing Conditional Radiance Fields
In this paper, we explore enabling user editing of a category-level NeRF - also known as a conditional radiance field - trained on a shape category.
Ranked #1 on
Novel View Synthesis
on PhotoShape
Contact and Human Dynamics from Monocular Video
Existing deep models predict 2D and 3D kinematic poses from video that are approximately accurate, but contain visible errors that violate physical constraints, such as feet penetrating the ground and bodies leaning at extreme angles.
Telling Left from Right: Learning Spatial Correspondence of Sight and Sound
Self-supervised audio-visual learning aims to capture useful representations of video by leveraging correspondences between visual and audio inputs.
FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images
We show that methods trained on our dataset consistently perform well when tested on other datasets.
Ranked #8 on
3D Hand Pose Estimation
on FreiHAND
Neural Re-Simulation for Generating Bounces in Single Images
We introduce a method to generate videos of dynamic virtual objects plausibly interacting via collisions with a still image's environment.
Finding Moments in Video Collections Using Natural Language
We evaluate our approach on two recently proposed datasets for temporal localization of moments in video with natural language (DiDeMo and Charades-STA) extended to our video corpus moment retrieval setting.
Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces
To achieve our results, we introduce the Bounce Dataset comprising 5K RGB-D videos of bouncing trajectories of a foam ball to probe surfaces of varying shapes and materials in everyday scenes including homes and offices.
B-Script: Transcript-based B-roll Video Editing with Recommendations
In video production, inserting B-roll is a widely used technique to enrich the story and make a video more engaging.
Localizing Moments in Video with Temporal Language
To benchmark whether our model, and other recent video localization models, can effectively reason about temporal language, we collect the novel TEMPOral reasoning in video and language (TEMPO) dataset.
BodyNet: Volumetric Inference of 3D Human Body Shapes
Human shape estimation is an important task for video editing, animation and fashion industry.
Ranked #3 on
3D Human Pose Estimation
on Surreal
(using extra training data)
Learning Visual Importance for Graphic Designs and Data Visualizations
Our models are neural networks trained on human clicks and importance annotations on hundreds of designs.
Localizing Moments in Video with Natural Language
A key obstacle to training our MCN model is that current video datasets do not include pairs of localized video segments and referring expressions, or text descriptions which uniquely identify a corresponding moment.
ActionVLAD: Learning spatio-temporal aggregation for action classification
In this work, we introduce a new video representation for action classification that aggregates local convolutional features across the entire spatio-temporal extent of the video.
PixelNet: Representation of the pixels, by the pixels, and for the pixels
We explore design principles for general pixel-level prediction problems, from low-level edge detection to mid-level surface normal estimation to high-level semantic segmentation.
SURGE: Surface Regularized Geometry Estimation from a Single Image
This paper introduces an approach to regularize 2. 5D surface normal and depth predictions at each pixel given a single input image.
PixelNet: Towards a General Pixel-level Architecture
We explore architectures for general pixel-level prediction problems, from low-level edge detection to mid-level surface normal estimation to high-level semantic segmentation.
Marr Revisited: 2D-3D Alignment via Surface Normal Prediction
We introduce an approach that leverages surface normal predictions, along with appearance cues, to retrieve 3D models for objects depicted in 2D still images from a large CAD object library.
Deep Exemplar 2D-3D Detection by Adapting from Real to Rendered Views
This paper presents an end-to-end convolutional neural network (CNN) for 2D-3D exemplar detection.
Understanding deep features with computer-generated imagery
The rendered images are presented to a trained CNN and responses for different layers are studied with respect to the input scene factors.
Localizing 3D cuboids in single-view images
In this paper we seek to detect rectangular cuboids and localize their corners in uncalibrated single-view images depicting everyday scenes.
Segmenting Scenes by Matching Image Composites
In contrast to recent work in semantic alignment of scenes, we allow an input image to be explained by partial matches of similar scenes.
