Search Results for author:
Found 123 papers, 31 papers with code
Proposal-based Video Completion
Video inpainting is an important technique for a wide variety of applications from video content editing to video restoration.
SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning
We introduce SoundSpaces 2. 0, a platform for on-the-fly geometry-based audio rendering for 3D environments.
Few-Shot Audio-Visual Learning of Environment Acoustics
Room impulse response (RIR) functions capture how the surrounding physical environment transforms the sounds heard by a listener, with implications for various applications in AR, VR, and robotics.
Visual Acoustic Matching
We introduce the visual acoustic matching task, in which an audio clip is transformed to sound like it was recorded in a target environment.
Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation
In reinforcement learning for visual navigation, it is common to develop a model for each new task, and train that model from scratch with task-specific interactions in 3D environments.
Active Audio-Visual Separation of Dynamic Sound Sources
We explore active audio-visual separation for dynamic sound sources, where an embodied agent moves intelligently in a 3D environment to continuously isolate the time-varying audio stream being emitted by an object of interest.
DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video
Dexterous multi-fingered robotic hands have a formidable action space, yet their morphological similarity to the human hand holds immense potential to accelerate robot learning.
PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning
We propose Potential functions for ObjectGoal Navigation with Interaction-free learning (PONI), a modular approach that disentangles the skills of `where to look?'
Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Video
Binaural audio provides human listeners with an immersive spatial sound experience, but most existing videos lack binaural audio recordings.
Shaping embodied agent behavior with activity-context priors from egocentric video
For a given object, an activity-context prior represents the set of other compatible objects that are required for activities to succeed (e. g., a knife and cutting board brought together with a tomato are conducive to cutting).
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
Environment Predictive Coding for Visual Navigation
We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents.
Shapes as Product Differentiation: Neural Network Embedding in the Analysis of Markets for Fonts
We then study the causal effects of a merger on the merging firm's creative decisions using the constructed measures in a synthetic control method.
Learning Audio-Visual Dereverberation
The visual environment surrounding a human speaker reveals important cues about the room geometry, materials, and speaker location, all of which influence the precise reverberation effects in the audio stream.
Anticipative Video Transformer
We propose Anticipative Video Transformer (AVT), an end-to-end attention-based video modeling architecture that attends to the previously observed video in order to anticipate future actions.
Egocentric Activity Recognition and Localization on a 3D Map
Our model takes the inputs of a Hierarchical Volumetric Representation (HVR) of the environment and an egocentric video, infers the 3D action location as a latent variable, and recognizes the action based on the video and contextual cues surrounding its potential locations.
Move2Hear: Active Audio-Visual Source Separation
We introduce the active audio-visual source separation problem, where an agent must move intelligently in order to better isolate the sounds coming from an object of interest in its environment.
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos
We introduce an approach for pre-training egocentric video models using large-scale third-person video datasets.
Multiview Pseudo-Labeling for Semi-supervised Learning from Video
We present a multiview pseudo-labeling approach to video learning, a novel framework that uses complementary views in the form of appearance and motion information for semi-supervised learning in video.
Environment Predictive Coding for Embodied Agents
We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents.
From Culture to Clothing: Discovering the World Events Behind A Century of Fashion Images
Fashion is intertwined with external cultural factors, but identifying these links remains a manual process limited to only the most salient phenomena.
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
Given a video, the goal is to extract the speech associated with a face in spite of simultaneous background sounds and/or other human speakers.
Audio-Visual Floorplan Reconstruction
Given only a few glimpses of an environment, how much can we infer about its entire floorplan?
Semantic Audio-Visual Navigation
We propose a transformer-based model to tackle this new semantic AudioGoal task, incorporating an inferred goal descriptor that captures both spatial and semantic properties of the target.
Discovering Underground Maps from Fashion
The fashion sense -- meaning the clothing styles people wear -- in a geographical region can reveal information about that region.
Modeling Fashion Influence from Photos
The discovered influence relationships reveal how both cities and brands exert and receive fashion influence for an array of visual styles inferred from the images.
Learning Dexterous Grasping with Object-Centric Visual Affordances
Our key idea is to embed an object-centric visual affordance model within a deep reinforcement learning loop to learn grasping policies that favor the same object regions favored by people.
Occupancy Anticipation for Efficient Exploration and Navigation
State-of-the-art navigation methods leverage a spatial memory to generalize to new environments, but their occupancy maps are limited to capturing the geometric structures directly observed by the agent.
Ranked #3 on
Robot Navigation
on Habitat 2020 Point Nav test-std
Learning to Set Waypoints for Audio-Visual Navigation
In audio-visual navigation, an agent intelligently travels through a complex, unmapped 3D environment using both sights and sounds to find a sound source (e. g., a phone ringing in another room).
Learning Affordance Landscapes for Interaction Exploration in 3D Environments
We introduce a reinforcement learning approach for exploration for interaction, whereby an embodied agent autonomously discovers the affordance landscape of a new unmapped 3D environment (such as an unfamiliar kitchen).
Learning Patterns of Tourist Movement and Photography from Geotagged Photos at Archaeological Heritage Sites in Cuzco, Peru
The popularity of media sharing platforms in recent decades has provided an abundance of open source data that remains underutilized by heritage scholars.
VisualEchoes: Spatial Image Representation Learning through Echolocation
Several animal species (e. g., bats, dolphins, and whales) and even visually impaired humans have the remarkable ability to perform echolocation: a biological sonar used to perceive spatial layout and locate objects in the world.
From Paris to Berlin: Discovering Fashion Style Influences Around the World
The evolution of clothing styles and their migration across the world is intriguing, yet difficult to describe quantitatively.
Audiovisual SlowFast Networks for Video Recognition
We present Audiovisual SlowFast Networks, an architecture for integrated audiovisual perception.
EGO-TOPO: Environment Affordances from Egocentric Video
We introduce a model for environment affordances that is learned directly from egocentric video.
Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias
Our key idea is to decorrelate feature representations of a category from its co-occurring context.
An Exploration of Embodied Visual Exploration
Embodied computer vision considers perception for robots in novel, unstructured environments.
SoundSpaces: Audio-Visual Navigation in 3D Environments
Moving around in the world is naturally a multisensory experience, but today's embodied agents are deaf---restricted to solely their visual perception of the environment.
ViBE: Dressing for Diverse Body Shapes
Body shape plays an important role in determining what garments will best suit a given person, yet today's clothing recommendation methods take a "one shape fits all" approach.
Listen to Look: Action Recognition by Previewing Audio
In the face of the video data deluge, today's expensive clip-level classifiers are increasingly impractical.
Emergence of Exploratory Look-Around Behaviors through Active Observation Completion
Standard computer vision systems assume access to intelligently captured inputs (e. g., photos from a human photographer), yet autonomously capturing good observations is a major challenge in itself.
Grounded Human-Object Interaction Hotspots from Video (Extended Abstract)
Learning how to interact with objects is an important step towards embodied visual intelligence, but existing techniques suffer from heavy supervision or sensing requirements.
Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback
We provide a detailed analysis of the characteristics of the Fashion IQ data, and present a transformer-based user simulator and interactive image retriever that can seamlessly integrate visual attributes with image features, user feedback, and dialog history, leading to improved performance over the state of the art in dialog-based image retrieval.
Predicting How to Distribute Work Between Algorithms and Humans to Segment an Image Batch
We propose a resource allocation framework for predicting how best to allocate a fixed budget of human annotation effort in order to collect higher quality segmentations for a given batch of images and automated methods.
You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions
The body pose of a person wearing a camera is of great interest for applications in augmented reality, healthcare, and robotics, yet much of the person's body is out of view for a typical wearable camera.
Fashion++: Minimal Edits for Outfit Improvement
We introduce Fashion++, an approach that proposes minimal adjustments to a full-body clothing outfit that will have maximal impact on its fashionability.
Co-Separating Sounds of Visual Objects
Learning how objects sound from video is challenging, since they often heavily overlap in a single audio channel.
Ranked #1 on
Audio Denoising
on AV-Bench - Wooden Horse
Next-Active-Object prediction from Egocentric Videos
Although First Person Vision systems can sense the environment from the user's perspective, they are generally unable to predict his intentions and goals.
Less is More: Learning Highlight Detection from Video Duration
Highlight detection has the potential to significantly ease video browsing, but existing methods often suffer from expensive supervision requirements, where human viewers must manually identify highlights in training videos.
Thinking Outside the Pool: Active Training Image Creation for Relative Attributes
Current wisdom suggests more labeled image data is always better, and obtaining labels is the bottleneck.
Extreme Relative Pose Estimation for RGB-D Scans via Scene Completion
In particular, instead of only performing scene completion from each individual scan, our approach alternates between relative pose estimation and scene completion.
2.5D Visual Sound
We devise a deep convolutional neural network that learns to decode the monaural (single-channel) soundtrack into its binaural counterpart by injecting visual information about object and scene configurations.
Grounded Human-Object Interaction Hotspots from Video
Learning how to interact with objects is an important step towards embodied visual intelligence, but existing techniques suffer from heavy supervision or sensing requirements.
Kernel Transformer Networks for Compact Spherical Convolution
KTNs efficiently transfer convolution kernels from perspective images to the equirectangular projection of 360{\deg} images.
SpotTune: Transfer Learning through Adaptive Fine-tuning
Transfer learning, which allows a source task to affect the inductive bias of the target task, is widely used in computer vision.
Snap Angle Prediction for 360° Panoramas
360° panoramas are a rich medium, yet notoriously difficult to visualize in the 2D image plane.
Retrospective Encoders for Video Summarization
The key idea is to complement the discriminative losses with another loss which measures if the predicted summary preserves the same information as in the original video.
Pixel Objectness: Learning to Segment Generic Objects Automatically in Images and Videos
We propose an end-to-end learning framework for segmenting generic objects in both images and videos.
Sidekick Policy Learning for Active Visual Exploration
We consider an active visual exploration scenario, where an agent must intelligently select its camera motions to efficiently reconstruct the full environment from only a limited set of narrow field-of-view glimpses.
Learning Compressible 360° Video Isomers
Standard video encoders developed for conventional narrow field-of-view video are widely applied to 360° video as well, with reasonable results.
Learning to Separate Object Sounds by Watching Unlabeled Video
Our work is the first to learn audio source separation from large-scale "in the wild" videos containing multiple audio sources per video.
Snap Angle Prediction for 360$^{\circ}$ Panoramas
360$^{\circ}$ panoramas are a rich medium, yet notoriously difficult to visualize in the 2D image plane.
Compare and Contrast: Learning Prominent Visual Differences
We collect instance-level annotations of most noticeable differences, and build a model trained on relative attribute features that predicts prominent differences for unseen pairs.
Attributes as Operators: Factorizing Unseen Attribute-Object Compositions
In addition, we show that not only can our model recognize unseen compositions robustly in an open-world setting, it can also generalize to compositions where objects themselves were unseen during training.
Ranked #5 on
Image Retrieval with Multi-Modal Query
on MIT-States
Compositional Zero-Shot Learning
Image Retrieval with Multi-Modal Query
VizWiz Grand Challenge: Answering Visual Questions from Blind People
The study of algorithms to automatically answer visual questions currently is motivated by visual question answering (VQA) datasets constructed in artificial VQA settings.
Im2Flow: Motion Hallucination from Static Images for Action Recognition
Second, we show the power of hallucinated flow for recognition, successfully transferring the learned motion into a standard two-stream network for activity recognition.
Learning Compressible 360° Video Isomers
Standard video encoders developed for conventional narrow field-of-view video are widely applied to 360{\deg} video as well, with reasonable results.
Creating Capsule Wardrobes from Fashion Images
To permit efficient subset selection over the space of all outfit combinations, we develop submodular objective functions capturing the key ingredients of visual compatibility, versatility, and user-specific preference.
BlockDrop: Dynamic Inference Paths in Residual Networks
Very deep convolutional neural networks offer excellent recognition results, yet their computational expense limits their impact for many real-world applications.
Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks
It is common to implicitly assume access to intelligently captured inputs (e. g., photos from a human photographer), yet autonomously capturing good observations is itself a major challenge.
ShapeCodes: Self-Supervised Feature Learning by Lifting Views to Viewgrids
We introduce an unsupervised feature learning approach that embeds 3D shape information into a single-view image representation.
Learning Spherical Convolution for Fast Features from 360° Imagery
While 360{\deg} cameras offer tremendous new possibilities in vision, graphics, and augmented reality, the spherical images they produce make core feature extraction non-trivial.
Learning the Latent "Look": Unsupervised Discovery of a Style-Coherent Embedding from Fashion Images
Given a collection of unlabeled fashion images, our approach mines for the latent styles, then summarizes outfits by how they mix those styles.
Making 360deg Video Watchable in 2D: Learning Videography for Click Free Viewing
360deg video requires human viewers to actively control "where" to look while watching the video.
FusionSeg: Learning to Combine Motion and Appearance for Fully Automatic Segmentation of Generic Objects in Videos
Our method learns to combine appearance and motion information to produce pixel level segmentation masks for all prominent objects in videos.
Predicting Foreground Object Ambiguity and Efficiently Crowdsourcing the Segmentation(s)
We propose the ambiguity problem for the foreground object segmentation task and motivate the importance of estimating and accounting for this ambiguity when designing vision systems.
Making 360$^{\circ}$ Video Watchable in 2D: Learning Videography for Click Free Viewing
360$^{\circ}$ video requires human viewers to actively control "where" to look while watching the video.
FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos
Our method learns to combine appearance and motion information to produce pixel level segmentation masks for all prominent objects in videos.
Ranked #21 on
Unsupervised Video Object Segmentation
on DAVIS 2016
Structured Prediction
Unsupervised Video Object Segmentation
+2
Pixel Objectness
We propose an end-to-end learning framework for generating foreground object segmentations.
Semantic Jitter: Dense Supervision for Visual Comparisons via Synthetic Images
Distinguishing subtle differences in attributes is valuable, yet learning to make visual comparisons remains non-trivial.
Pano2Vid: Automatic Cinematography for Watching 360$^{\circ}$ Videos
AutoCam leverages NFOV web video to discriminatively identify space-time "glimpses" of interest at each time instant, and then uses dynamic programming to select optimal human-like camera trajectories.
On-Demand Learning for Deep Image Restoration
While machine learning approaches to image restoration offer great promise, current methods risk training models fixated on performing well only for image corruption of a particular level of difficulty---such as a certain level of noise or blur.
Object-Centric Representation Learning from Unlabeled Videos
Compared to existing temporal coherence methods, our idea has the advantage of lightweight preprocessing of the unlabeled video (no tracking required) while still being able to extract object-level regions from which to learn invariances.
Crowdsourcing in Computer Vision
Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts.
Visual Question: Predicting If a Crowd Will Agree on the Answer
Visual question answering (VQA) systems are emerging from a desire to empower users to ask any natural language question about visual content and receive a valid answer in response.
Efficient Activity Detection in Untrimmed Video with Max-Subgraph Search
We show that this detection strategy permits an efficient branch-and-cut solution for the best-scoring---and possibly non-cubically shaped---portion of the video for a given activity classifier.
Click Carving: Segmenting Objects in Video with Point Clicks
We present a novel form of interactive video object segmentation where a few clicks by the user helps the system produce a full spatio-temporal segmentation of the object of interest.
Interactive Video Object Segmentation
Semantic Segmentation
+1
Pull the Plug? Predicting If Computers or Humans Should Segment Images
We propose a resource allocation framework for predicting how best to allocate a fixed budget of human annotation effort in order to collect higher quality segmentations for a given batch of images and automated methods.
Active Image Segmentation Propagation
We propose a semi-automatic method to obtain foreground object masks for a large set of related images.
Video Summarization with Long Short-term Memory
We propose a novel supervised learning technique for summarizing videos by automatically selecting keyframes or key subshots.
Look-ahead before you leap: end-to-end active recognition by forecasting the effect of motion
To verify this hypothesis, we attempt to induce this capacity in our active recognition pipeline, by simultaneously learning to forecast the effects of the agent's motions on its internal representation of the environment conditional on all past views.
Subjects and Their Objects: Localizing Interactees for a Person-Centric View of Importance
We propose to predict the "interactee" in novel images---that is, to localize the \emph{object} of a person's action.
Detangling People: Individuating Multiple Close People and Their Body Parts via Region Assembly
In addition, we demonstrate its impact on a proxemics recognition task, which demands a precise representation of "whose body part is where" in crowded images.
Detecting Engagement in Egocentric Video
In a wearable camera video, we see what the camera wearer sees.
Leaving Some Stones Unturned: Dynamic Feature Prioritization for Activity Detection in Streaming Video
Current approaches for activity recognition often ignore constraints on computational resources: 1) they rely on extensive feature computation to obtain rich descriptors on all frames, and 2) they assume batch-mode access to the entire test video at once.
Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Video
We propose to infer the "invisible pose" of a person behind the egocentric camera.
Summary Transfer: Exemplar-based Subset Selection for Video Summarization
Video summarization has unprecedented importance to help us digest, browse, and search today's ever-growing video collections.
Just Noticeable Differences in Visual Attributes
We develop a Bayesian local learning strategy to infer when images are indistinguishable for a given attribute.
Slow and steady feature analysis: higher order temporal coherence in video
While this standard approach captures the fact that high-level visual signals change slowly over time, it fails to capture *how* the visual content changes.
Predicting Important Objects for Egocentric Video Summarization
Our results on two egocentric video datasets show the method's promise relative to existing techniques for saliency and summarization.
WhittleSearch: Interactive Image Search with Relative Attribute Feedback
We propose a novel mode of feedback for image search, where a user describes which properties of exemplar images should be adjusted in order to more closely match his/her mental model of the image sought.
Discovering Attribute Shades of Meaning with the Crowd
We propose to discover shades of attribute meaning.
Learning image representations tied to ego-motion
Understanding how images of objects and scenes behave in response to specific ego-motions is a crucial aspect of proper visual development, yet existing visual learning methods are conspicuously disconnected from the physical source of their images.
Predicting Useful Neighborhoods for Lazy Local Learning
Lazy local learning methods train a classifier on the fly" at test time, using only a subset of the training instances that are most relevant to the novel test example.
Diverse Sequential Subset Selection for Supervised Video Summarization
Video summarization is a challenging problem with great application potential.
Zero-shot recognition with unreliable attributes
In principle, zero-shot learning makes it possible to train an object recognition model simply by specifying the category's attributes.
Large-Margin Determinantal Point Processes
Extensive empirical studies validate our contributions, including applications on challenging document and video summarization, where flexibility in modeling the kernel matrix and balancing different errors is indispensable.
Zero Shot Recognition with Unreliable Attributes
In principle, zero-shot learning makes it possible to train a recognition model simply by specifying the category's attributes.
Inferring Analogous Attributes
The appearance of an attribute can vary considerably from class to class (e. g., a "fluffy" dog vs. a "fluffy" towel), making standard class-independent attribute models break down.
Decorrelating Semantic Visual Attributes by Resisting the Urge to Share
Existing methods to learn visual attributes are prone to learning the wrong thing---namely, properties that are correlated with the attribute of interest among training samples.
Inferring Unseen Views of People
We pose unseen view synthesis as a probabilistic tensor completion problem.
Fine-Grained Visual Comparisons with Local Learning
Given two images, we want to predict which exhibits a particular visual attribute more than the other---even when the two images are quite similar.
Beyond Comparing Image Pairs: Setwise Active Learning for Relative Attributes
It is useful to automatically compare images based on their visual properties---to predict which image is brighter, more feminine, more blurry, etc.
Reshaping Visual Datasets for Domain Adaptation
By maximum distinctiveness, we require the underlying distributions of the identified domains to be different from each other; by maximum learnability, we ensure that a strong discriminative model can be learned from the domain.
Watching Unlabeled Video Helps Learn New Human Actions from Very Few Labeled Snapshots
We propose an approach to learn action categories from static images that leverages prior observations of generic human motion to augment its training process.
Story-Driven Summarization for Egocentric Video
We present a video summarization approach that discovers the story of an egocentric video.
Deformable Spatial Pyramid Matching for Fast Dense Correspondences
We introduce a fast deformable spatial pyramid (DSP) matching algorithm for computing dense pixel correspondences.
Semantic Kernel Forests from Multiple Taxonomies
When learning features for complex visual recognition problems, labeled image exemplars alone can be insufficient.
Learning a Tree of Metrics with Disjoint Visual Features
Given a hierarchical taxonomy that captures semantic similarity between the objects, we learn a corresponding tree of metrics (ToM).
Hashing Hyperplane Queries to Near Points with Applications to Large-Scale Active Learning
Our first approach maps the data to two-bit binary keys that are locality-sensitive for the angle between the hyperplane normal and a database point.
Multi-Level Active Prediction of Useful Image Annotations for Recognition
We introduce a framework for actively learning visual categories from a mixture of weakly and strongly labeled image examples.
Online Metric Learning and Fast Similarity Search
Metric learning algorithms can provide useful distance functions for a variety of domains, and recent work has shown good accuracy for problems where the learner can access all distance constraints at once.
