Search Results for author:
Found 152 papers, 44 papers with code
Proposal-based Video Completion
Video inpainting is an important technique for a wide variety of applications from video content editing to video restoration.
PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding
In this paper, we study building a Perception Language Model (PLM) in a fully open and reproducible framework for transparent research in image and video understanding.
Stitch-a-Recipe: Video Demonstration from Multistep Descriptions
When obtaining visual illustrations from text descriptions, today's methods take a description with-a single text context caption, or an action description-and retrieve or generate the matching visual context.
Switch-a-View: Few-Shot View Selection Learned from Edited Videos
We introduce Switch-a-View, a model that learns to automatically select the viewpoint to display at each timepoint when creating a how-to video.
Progress-Aware Video Frame Captioning
The results demonstrate that ProgressCaptioner significantly surpasses leading captioning models, producing precise captions that accurately capture action progression and set a new standard for temporal precision in video captioning.
FIction: 4D Future Interaction Prediction from Video
Anticipating how a person will interact with objects in an environment is essential for activity understanding, but existing methods are limited to the 2D space of video frames-capturing physically ungrounded predictions of 'what' and ignoring the 'where' and 'how'.
Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Videos
Given a multi-view video, which viewpoint is most informative for a human observer?
Human Action Anticipation: A Survey
Predicting future human behavior is an increasingly popular topic in computer vision, driven by the interest in applications such as autonomous vehicles, digital assistants and human-robot interactions.
ExpertAF: Expert Actionable Feedback from Video
Our method takes a video demonstration and its accompanying 3D body pose and generates (1) free-form expert commentary describing what the person is doing well and what they could improve, and (2) a visual expert demonstration that incorporates the required corrections.
Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos
We propose a novel ambient-aware audio generation model, AV-LDM.
HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness
We study the problem of precisely swapping objects in videos, with a focus on those interacted with by hands, given one user-provided reference object image.
Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Prediction
We propose the first treatment of sim2real for audio-visual navigation by disentangling it into acoustic field prediction (AFP) and waypoint navigation.
ActiveRIR: Active Audio-Visual Exploration for Acoustic Environment Modeling
An environment acoustic model represents how sound is transformed by the physical characteristics of an indoor environment, for any given source/receiver location.
SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos
We propose a novel self-supervised embedding to learn how actions sound from narrated in-the-wild egocentric videos.
Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videos
We investigate exocentric-to-egocentric cross-view translation, which aims to generate a first-person (egocentric) view of an actor based on a video recording that captures the actor from a third-person (exocentric) perspective.
Detours for Navigating Instructional Videos
We introduce the video detours problem for navigating instructional videos.
Learning Object State Changes in Videos: An Open-World Perspective
Object State Changes (OSCs) are pivotal for video understanding.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos
We propose a self-supervised method for learning representations based on spatial audio-visual correspondences in egocentric videos.
SpotEM: Efficient Video Search for Episodic Memory
The goal in episodic memory (EM) is to search a long egocentric video to answer a natural language query (e. g., "where did I leave my purse?").
Egocentric Video Task Translation @ Ego4D Challenge 2022
With no modification to the baseline architectures, our proposed approach achieves competitive performance on two Ego4D challenges, ranking the 1st in the talking to me challenge and the 3rd in the PNR keyframe localization challenge.
Novel-View Acoustic Synthesis
We introduce the novel-view acoustic synthesis (NVAS) task: given the sight and sound observed at a source viewpoint, can we synthesize the sound of that scene from an unseen target viewpoint?
A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems
Despite the advancement of machine learning techniques in recent years, state-of-the-art systems lack robustness to "real world" events, where the input distributions and tasks encountered by the deployed systems will not be limited to the original training context, and systems will instead need to adapt to novel distributions and tasks while deployed.
What You Say Is What You Show: Visual Narration Detection in Instructional Videos
Narrated ''how-to'' videos have emerged as a promising data source for a wide range of learning problems, from learning visual representations to training robot policies.
HierVL: Learning Hierarchical Video-Language Embeddings
Video-language embeddings are a promising avenue for injecting semantics into visual representations, but existing methods capture only short-term associations between seconds-long video clips and their accompanying text.
Ranked #3 on
Long Term Action Anticipation
on Ego4D
Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations
Can conversational videos captured from multiple egocentric viewpoints reveal the map of a scene in a cost-efficient way?
NaQ: Leveraging Narrations as Queries to Supervise Episodic Memory
Searching long egocentric videos with natural language queries (NLQ) has compelling applications in augmented reality and robotics, where a fluid index into everything that a person (agent) has seen before could augment human memory and surface relevant information on demand.
Egocentric Video Task Translation
Different video understanding tasks are typically treated in isolation, and even with distinct types of curated data (e. g., classifying sports in one dataset, tracking animals in another).
Few-View Object Reconstruction with Unknown Categories and Camera Poses
The reconstruction results under predicted poses are comparable to the ones using ground-truth poses.
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning
We introduce SoundSpaces 2. 0, a platform for on-the-fly geometry-based audio rendering for 3D environments.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Few-Shot Audio-Visual Learning of Environment Acoustics
Room impulse response (RIR) functions capture how the surrounding physical environment transforms the sounds heard by a listener, with implications for various applications in AR, VR, and robotics.
Visual Acoustic Matching
We introduce the visual acoustic matching task, in which an audio clip is transformed to sound like it was recorded in a target environment.
Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation
In reinforcement learning for visual navigation, it is common to develop a model for each new task, and train that model from scratch with task-specific interactions in 3D environments.
Active Audio-Visual Separation of Dynamic Sound Sources
We explore active audio-visual separation for dynamic sound sources, where an embodied agent moves intelligently in a 3D environment to continuously isolate the time-varying audio stream being emitted by an object of interest.
DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video
Dexterous multi-fingered robotic hands have a formidable action space, yet their morphological similarity to the human hand holds immense potential to accelerate robot learning.
Deep Reinforcement Learning
Human-Object Interaction Detection
+1
PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning
We propose Potential functions for ObjectGoal Navigation with Interaction-free learning (PONI), a modular approach that disentangles the skills of `where to look?'
Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Video
Binaural audio provides human listeners with an immersive spatial sound experience, but most existing videos lack binaural audio recordings.
Shaping embodied agent behavior with activity-context priors from egocentric video
For a given object, an activity-context prior represents the set of other compatible objects that are required for activities to succeed (e. g., a knife and cutting board brought together with a tomato are conducive to cutting).
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
Environment Predictive Coding for Visual Navigation
We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents.
Shapes as Product Differentiation: Neural Network Embedding in the Analysis of Markets for Fonts
We then study the causal effects of a merger on the merging firm's creative decisions using the constructed measures in a synthetic control method.
Learning Audio-Visual Dereverberation
We introduce Visually-Informed Dereverberation of Audio (VIDA), an end-to-end approach that learns to remove reverberation based on both the observed monaural sound and visual scene.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Anticipative Video Transformer
We propose Anticipative Video Transformer (AVT), an end-to-end attention-based video modeling architecture that attends to the previously observed video in order to anticipate future actions.
Ranked #2 on
Action Anticipation
on EPIC-KITCHENS-100 (test)
(using extra training data)
Egocentric Activity Recognition and Localization on a 3D Map
Given a video captured from a first person perspective and the environment context of where the video is recorded, can we recognize what the person is doing and identify where the action occurs in the 3D space?
Move2Hear: Active Audio-Visual Source Separation
We introduce the active audio-visual source separation problem, where an agent must move intelligently in order to better isolate the sounds coming from an object of interest in its environment.
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos
We introduce an approach for pre-training egocentric video models using large-scale third-person video datasets.
Multiview Pseudo-Labeling for Semi-supervised Learning from Video
We present a multiview pseudo-labeling approach to video learning, a novel framework that uses complementary views in the form of appearance and motion information for semi-supervised learning in video.
Environment Predictive Coding for Embodied Agents
We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents.
From Culture to Clothing: Discovering the World Events Behind A Century of Fashion Images
Fashion is intertwined with external cultural factors, but identifying these links remains a manual process limited to only the most salient phenomena.
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
Given a video, the goal is to extract the speech associated with a face in spite of simultaneous background sounds and/or other human speakers.
Audio-Visual Floorplan Reconstruction
Given only a few glimpses of an environment, how much can we infer about its entire floorplan?
Semantic Audio-Visual Navigation
We propose a transformer-based model to tackle this new semantic AudioGoal task, incorporating an inferred goal descriptor that captures both spatial and semantic properties of the target.
Discovering Underground Maps from Fashion
The fashion sense -- meaning the clothing styles people wear -- in a geographical region can reveal information about that region.
Modeling Fashion Influence from Photos
The discovered influence relationships reveal how both cities and brands exert and receive fashion influence for an array of visual styles inferred from the images.
Learning Dexterous Grasping with Object-Centric Visual Affordances
Our key idea is to embed an object-centric visual affordance model within a deep reinforcement learning loop to learn grasping policies that favor the same object regions favored by people.
Occupancy Anticipation for Efficient Exploration and Navigation
State-of-the-art navigation methods leverage a spatial memory to generalize to new environments, but their occupancy maps are limited to capturing the geometric structures directly observed by the agent.
Ranked #3 on
Robot Navigation
on Habitat 2020 Point Nav test-std
Learning to Set Waypoints for Audio-Visual Navigation
In audio-visual navigation, an agent intelligently travels through a complex, unmapped 3D environment using both sights and sounds to find a sound source (e. g., a phone ringing in another room).
Learning Affordance Landscapes for Interaction Exploration in 3D Environments
We introduce a reinforcement learning approach for exploration for interaction, whereby an embodied agent autonomously discovers the affordance landscape of a new unmapped 3D environment (such as an unfamiliar kitchen).
Learning Patterns of Tourist Movement and Photography from Geotagged Photos at Archaeological Heritage Sites in Cuzco, Peru
The popularity of media sharing platforms in recent decades has provided an abundance of open source data that remains underutilized by heritage scholars.
VisualEchoes: Spatial Image Representation Learning through Echolocation
Several animal species (e. g., bats, dolphins, and whales) and even visually impaired humans have the remarkable ability to perform echolocation: a biological sonar used to perceive spatial layout and locate objects in the world.
From Paris to Berlin: Discovering Fashion Style Influences Around the World
The evolution of clothing styles and their migration across the world is intriguing, yet difficult to describe quantitatively.
Audiovisual SlowFast Networks for Video Recognition
We present Audiovisual SlowFast Networks, an architecture for integrated audiovisual perception.
EGO-TOPO: Environment Affordances from Egocentric Video
We introduce a model for environment affordances that is learned directly from egocentric video.
Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias
Our key idea is to decorrelate feature representations of a category from its co-occurring context.
An Exploration of Embodied Visual Exploration
Embodied computer vision considers perception for robots in novel, unstructured environments.
SoundSpaces: Audio-Visual Navigation in 3D Environments
Moving around in the world is naturally a multisensory experience, but today's embodied agents are deaf---restricted to solely their visual perception of the environment.
ViBE: Dressing for Diverse Body Shapes
Body shape plays an important role in determining what garments will best suit a given person, yet today's clothing recommendation methods take a "one shape fits all" approach.
Listen to Look: Action Recognition by Previewing Audio
In the face of the video data deluge, today's expensive clip-level classifiers are increasingly impractical.
Ranked #8 on
Action Recognition
on ActivityNet
Emergence of Exploratory Look-Around Behaviors through Active Observation Completion
Standard computer vision systems assume access to intelligently captured inputs (e. g., photos from a human photographer), yet autonomously capturing good observations is a major challenge in itself.
Grounded Human-Object Interaction Hotspots from Video (Extended Abstract)
Learning how to interact with objects is an important step towards embodied visual intelligence, but existing techniques suffer from heavy supervision or sensing requirements.
Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback
We provide a detailed analysis of the characteristics of the Fashion IQ data, and present a transformer-based user simulator and interactive image retriever that can seamlessly integrate visual attributes with image features, user feedback, and dialog history, leading to improved performance over the state of the art in dialog-based image retrieval.
Predicting How to Distribute Work Between Algorithms and Humans to Segment an Image Batch
We propose a resource allocation framework for predicting how best to allocate a fixed budget of human annotation effort in order to collect higher quality segmentations for a given batch of images and automated methods.
You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions
The body pose of a person wearing a camera is of great interest for applications in augmented reality, healthcare, and robotics, yet much of the person's body is out of view for a typical wearable camera.
Fashion++: Minimal Edits for Outfit Improvement
We introduce Fashion++, an approach that proposes minimal adjustments to a full-body clothing outfit that will have maximal impact on its fashionability.
Co-Separating Sounds of Visual Objects
Learning how objects sound from video is challenging, since they often heavily overlap in a single audio channel.
Ranked #1 on
Audio Denoising
on AV-Bench - Wooden Horse
Next-Active-Object prediction from Egocentric Videos
Although First Person Vision systems can sense the environment from the user's perspective, they are generally unable to predict his intentions and goals.
Less is More: Learning Highlight Detection from Video Duration
Highlight detection has the potential to significantly ease video browsing, but existing methods often suffer from expensive supervision requirements, where human viewers must manually identify highlights in training videos.
Thinking Outside the Pool: Active Training Image Creation for Relative Attributes
Current wisdom suggests more labeled image data is always better, and obtaining labels is the bottleneck.
Extreme Relative Pose Estimation for RGB-D Scans via Scene Completion
In particular, instead of only performing scene completion from each individual scan, our approach alternates between relative pose estimation and scene completion.
2.5D Visual Sound
We devise a deep convolutional neural network that learns to decode the monaural (single-channel) soundtrack into its binaural counterpart by injecting visual information about object and scene configurations.
Grounded Human-Object Interaction Hotspots from Video
Learning how to interact with objects is an important step towards embodied visual intelligence, but existing techniques suffer from heavy supervision or sensing requirements.
Ranked #3 on
Video-to-image Affordance Grounding
on EPIC-Hotspot
Kernel Transformer Networks for Compact Spherical Convolution
KTNs efficiently transfer convolution kernels from perspective images to the equirectangular projection of 360{\deg} images.
SpotTune: Transfer Learning through Adaptive Fine-tuning
Transfer learning, which allows a source task to affect the inductive bias of the target task, is widely used in computer vision.
Retrospective Encoders for Video Summarization
The key idea is to complement the discriminative losses with another loss which measures if the predicted summary preserves the same information as in the original video.
Snap Angle Prediction for 360° Panoramas
360° panoramas are a rich medium, yet notoriously difficult to visualize in the 2D image plane.
Pixel Objectness: Learning to Segment Generic Objects Automatically in Images and Videos
We propose an end-to-end learning framework for segmenting generic objects in both images and videos.
Sidekick Policy Learning for Active Visual Exploration
We consider an active visual exploration scenario, where an agent must intelligently select its camera motions to efficiently reconstruct the full environment from only a limited set of narrow field-of-view glimpses.
Learning Compressible 360° Video Isomers
Standard video encoders developed for conventional narrow field-of-view video are widely applied to 360° video as well, with reasonable results.
Learning to Separate Object Sounds by Watching Unlabeled Video
Our work is the first to learn audio source separation from large-scale "in the wild" videos containing multiple audio sources per video.
Compare and Contrast: Learning Prominent Visual Differences
We collect instance-level annotations of most noticeable differences, and build a model trained on relative attribute features that predicts prominent differences for unseen pairs.
Snap Angle Prediction for 360$^{\circ}$ Panoramas
360$^{\circ}$ panoramas are a rich medium, yet notoriously difficult to visualize in the 2D image plane.
Attributes as Operators: Factorizing Unseen Attribute-Object Compositions
In addition, we show that not only can our model recognize unseen compositions robustly in an open-world setting, it can also generalize to compositions where objects themselves were unseen during training.
Ranked #5 on
Image Retrieval with Multi-Modal Query
on MIT-States
VizWiz Grand Challenge: Answering Visual Questions from Blind People
The study of algorithms to automatically answer visual questions currently is motivated by visual question answering (VQA) datasets constructed in artificial VQA settings.
Learning Compressible 360° Video Isomers
Standard video encoders developed for conventional narrow field-of-view video are widely applied to 360{\deg} video as well, with reasonable results.
Im2Flow: Motion Hallucination from Static Images for Action Recognition
Second, we show the power of hallucinated flow for recognition, successfully transferring the learned motion into a standard two-stream network for activity recognition.
Creating Capsule Wardrobes from Fashion Images
To permit efficient subset selection over the space of all outfit combinations, we develop submodular objective functions capturing the key ingredients of visual compatibility, versatility, and user-specific preference.
BlockDrop: Dynamic Inference Paths in Residual Networks
Very deep convolutional neural networks offer excellent recognition results, yet their computational expense limits their impact for many real-world applications.
Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks
It is common to implicitly assume access to intelligently captured inputs (e. g., photos from a human photographer), yet autonomously capturing good observations is itself a major challenge.
Learning Spherical Convolution for Fast Features from 360° Imagery
While 360{\deg} cameras offer tremendous new possibilities in vision, graphics, and augmented reality, the spherical images they produce make core feature extraction non-trivial.
Learning the Latent "Look": Unsupervised Discovery of a Style-Coherent Embedding from Fashion Images
Given a collection of unlabeled fashion images, our approach mines for the latent styles, then summarizes outfits by how they mix those styles.
Making 360deg Video Watchable in 2D: Learning Videography for Click Free Viewing
360deg video requires human viewers to actively control "where" to look while watching the video.
FusionSeg: Learning to Combine Motion and Appearance for Fully Automatic Segmentation of Generic Objects in Videos
Our method learns to combine appearance and motion information to produce pixel level segmentation masks for all prominent objects in videos.
Predicting Foreground Object Ambiguity and Efficiently Crowdsourcing the Segmentation(s)
We propose the ambiguity problem for the foreground object segmentation task and motivate the importance of estimating and accounting for this ambiguity when designing vision systems.
Making 360$^{\circ}$ Video Watchable in 2D: Learning Videography for Click Free Viewing
360$^{\circ}$ video requires human viewers to actively control "where" to look while watching the video.
FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos
Our method learns to combine appearance and motion information to produce pixel level segmentation masks for all prominent objects in videos.
Pixel Objectness
We propose an end-to-end learning framework for generating foreground object segmentations.
Semantic Jitter: Dense Supervision for Visual Comparisons via Synthetic Images
Distinguishing subtle differences in attributes is valuable, yet learning to make visual comparisons remains non-trivial.
Pano2Vid: Automatic Cinematography for Watching 360$^{\circ}$ Videos
AutoCam leverages NFOV web video to discriminatively identify space-time "glimpses" of interest at each time instant, and then uses dynamic programming to select optimal human-like camera trajectories.
On-Demand Learning for Deep Image Restoration
While machine learning approaches to image restoration offer great promise, current methods risk training models fixated on performing well only for image corruption of a particular level of difficulty---such as a certain level of noise or blur.
Object-Centric Representation Learning from Unlabeled Videos
Compared to existing temporal coherence methods, our idea has the advantage of lightweight preprocessing of the unlabeled video (no tracking required) while still being able to extract object-level regions from which to learn invariances.
Crowdsourcing in Computer Vision
Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts.
Visual Question: Predicting If a Crowd Will Agree on the Answer
Visual question answering (VQA) systems are emerging from a desire to empower users to ask any natural language question about visual content and receive a valid answer in response.
Efficient Activity Detection in Untrimmed Video with Max-Subgraph Search
We show that this detection strategy permits an efficient branch-and-cut solution for the best-scoring---and possibly non-cubically shaped---portion of the video for a given activity classifier.
Click Carving: Segmenting Objects in Video with Point Clicks
We present a novel form of interactive video object segmentation where a few clicks by the user helps the system produce a full spatio-temporal segmentation of the object of interest.
Pull the Plug? Predicting If Computers or Humans Should Segment Images
We propose a resource allocation framework for predicting how best to allocate a fixed budget of human annotation effort in order to collect higher quality segmentations for a given batch of images and automated methods.
Active Image Segmentation Propagation
We propose a semi-automatic method to obtain foreground object masks for a large set of related images.
Video Summarization with Long Short-term Memory
We propose a novel supervised learning technique for summarizing videos by automatically selecting keyframes or key subshots.
Look-ahead before you leap: end-to-end active recognition by forecasting the effect of motion
To verify this hypothesis, we attempt to induce this capacity in our active recognition pipeline, by simultaneously learning to forecast the effects of the agent's motions on its internal representation of the environment conditional on all past views.
Subjects and Their Objects: Localizing Interactees for a Person-Centric View of Importance
We propose to predict the "interactee" in novel images---that is, to localize the \emph{object} of a person's action.
Detangling People: Individuating Multiple Close People and Their Body Parts via Region Assembly
In addition, we demonstrate its impact on a proxemics recognition task, which demands a precise representation of "whose body part is where" in crowded images.
Detecting Engagement in Egocentric Video
In a wearable camera video, we see what the camera wearer sees.
Leaving Some Stones Unturned: Dynamic Feature Prioritization for Activity Detection in Streaming Video
Current approaches for activity recognition often ignore constraints on computational resources: 1) they rely on extensive feature computation to obtain rich descriptors on all frames, and 2) they assume batch-mode access to the entire test video at once.
Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Video
We propose to infer the "invisible pose" of a person behind the egocentric camera.
Summary Transfer: Exemplar-based Subset Selection for Video Summarization
Video summarization has unprecedented importance to help us digest, browse, and search today's ever-growing video collections.
Just Noticeable Differences in Visual Attributes
We develop a Bayesian local learning strategy to infer when images are indistinguishable for a given attribute.
Slow and steady feature analysis: higher order temporal coherence in video
While this standard approach captures the fact that high-level visual signals change slowly over time, it fails to capture *how* the visual content changes.
Predicting Important Objects for Egocentric Video Summarization
Our results on two egocentric video datasets show the method's promise relative to existing techniques for saliency and summarization.
Discovering Attribute Shades of Meaning with the Crowd
We propose to discover shades of attribute meaning.
WhittleSearch: Interactive Image Search with Relative Attribute Feedback
We propose a novel mode of feedback for image search, where a user describes which properties of exemplar images should be adjusted in order to more closely match his/her mental model of the image sought.
Learning image representations tied to ego-motion
Understanding how images of objects and scenes behave in response to specific ego-motions is a crucial aspect of proper visual development, yet existing visual learning methods are conspicuously disconnected from the physical source of their images.
Diverse Sequential Subset Selection for Supervised Video Summarization
Video summarization is a challenging problem with great application potential.
Predicting Useful Neighborhoods for Lazy Local Learning
Lazy local learning methods train a classifier on the fly" at test time, using only a subset of the training instances that are most relevant to the novel test example.
Zero-shot recognition with unreliable attributes
In principle, zero-shot learning makes it possible to train an object recognition model simply by specifying the category's attributes.
Large-Margin Determinantal Point Processes
Extensive empirical studies validate our contributions, including applications on challenging document and video summarization, where flexibility in modeling the kernel matrix and balancing different errors is indispensable.
Zero Shot Recognition with Unreliable Attributes
In principle, zero-shot learning makes it possible to train a recognition model simply by specifying the category's attributes.
Beyond Comparing Image Pairs: Setwise Active Learning for Relative Attributes
It is useful to automatically compare images based on their visual properties---to predict which image is brighter, more feminine, more blurry, etc.
Inferring Unseen Views of People
We pose unseen view synthesis as a probabilistic tensor completion problem.
Decorrelating Semantic Visual Attributes by Resisting the Urge to Share
Existing methods to learn visual attributes are prone to learning the wrong thing---namely, properties that are correlated with the attribute of interest among training samples.
Fine-Grained Visual Comparisons with Local Learning
Given two images, we want to predict which exhibits a particular visual attribute more than the other---even when the two images are quite similar.
Inferring Analogous Attributes
The appearance of an attribute can vary considerably from class to class (e. g., a "fluffy" dog vs. a "fluffy" towel), making standard class-independent attribute models break down.
Reshaping Visual Datasets for Domain Adaptation
By maximum distinctiveness, we require the underlying distributions of the identified domains to be different from each other; by maximum learnability, we ensure that a strong discriminative model can be learned from the domain.
Deformable Spatial Pyramid Matching for Fast Dense Correspondences
We introduce a fast deformable spatial pyramid (DSP) matching algorithm for computing dense pixel correspondences.
Story-Driven Summarization for Egocentric Video
We present a video summarization approach that discovers the story of an egocentric video.
Watching Unlabeled Video Helps Learn New Human Actions from Very Few Labeled Snapshots
We propose an approach to learn action categories from static images that leverages prior observations of generic human motion to augment its training process.
Semantic Kernel Forests from Multiple Taxonomies
When learning features for complex visual recognition problems, labeled image exemplars alone can be insufficient.
Learning a Tree of Metrics with Disjoint Visual Features
Given a hierarchical taxonomy that captures semantic similarity between the objects, we learn a corresponding tree of metrics (ToM).
Hashing Hyperplane Queries to Near Points with Applications to Large-Scale Active Learning
Our first approach maps the data to two-bit binary keys that are locality-sensitive for the angle between the hyperplane normal and a database point.
Online Metric Learning and Fast Similarity Search
Metric learning algorithms can provide useful distance functions for a variety of domains, and recent work has shown good accuracy for problems where the learner can access all distance constraints at once.
Multi-Level Active Prediction of Useful Image Annotations for Recognition
We introduce a framework for actively learning visual categories from a mixture of weakly and strongly labeled image examples.
