Search Results for author:
Found 310 papers, 170 papers with code
Unsupervised Learning of Visual Sense Models for Polysemous Words
Polysemy is a problem for methods that exploit image search engines to build object category models.
An Additive Latent Feature Model for Transparent Object Recognition
The appearance of a transparent patch is determined in part by the refraction of a background pattern through a transparent medium: the energy from the background usually dominates the patch appearance.
Learning to Hash with Binary Reconstructive Embeddings
Fast retrieval methods are increasingly critical for many large-scale analysis tasks, and there have been several recent methods that attempt to learn hash functions for fast and accurate nearest neighbor searches.
Filtering Abstract Senses From Image Search Results
When faced with the task of learning a visual model based only on the name of an object, a common approach is to find images on the web that are associated with the object name, and then train a visual classifier from the search result.
Factorized Latent Spaces with Structured Sparsity
Recent approaches to multi-view learning have shown that factorizing the information into parts that are shared across all views and parts that are private to each view could effectively account for the dependencies and independencies between the different input modalities.
Size Matters: Metric Visual Search Constraints from Monocular Metadata
Metric constraints are known to be highly discriminative for many objects, but if training is limited to data captured from a particular 3-D sensor the quantity of training data may be severly limited.
Heavy-tailed Distances for Gradient Based Image Descriptors
Many applications in computer vision measure the similarity between images or image patches based on some statistics such as oriented gradients.
Learning with Recursive Perceptual Representations
The use of random projections is key to our method, as we show in the experiments section, in which we observe a consistent improvement over previous --often more complicated-- methods on several vision and speech benchmarks.
Ranked #216 on
Image Classification
on CIFAR-10
Timely Object Recognition
On the timeliness measure, our method obtains at least $11\%$ better performance.
Efficient Learning of Domain-invariant Image Representations
We present an algorithm that learns representations which explicitly compensate for domain mismatch and which can be efficiently realized as linear classifiers.
Why Size Matters: Feature Coding as Nystrom Sampling
Recently, the computer vision and machine learning community has been in favor of feature extraction pipelines that rely on a coding step followed by a linear classifier, due to their overall simplicity, well understood properties of linear classifiers, and their computational efficiency.
Semi-supervised Domain Adaptation with Instance Constraints
Most successful object classification and detection methods rely on classifiers trained on large labeled datasets.
Towards Adapting ImageNet to Reality: Scalable Domain Adaptation with Implicit Low-rank Transformations
Images seen during test time are often not from the same distribution as images used for learning.
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
We evaluate whether features extracted from the activation of a deep convolutional network trained in a fully supervised fashion on a large, fixed set of object recognition tasks can be re-purposed to novel generic tasks.
Rich feature hierarchies for accurate object detection and semantic segmentation
We find that R-CNN outperforms OverFeat by a large margin on the 200-class ILSVRC2013 detection dataset.
Ranked #27 on
Object Detection
on PASCAL VOC 2007
(using extra training data)
Recognizing Image Style
The style of an image plays a significant role in how it is viewed, but style has received little attention in computer vision research.
PANDA: Pose Aligned Networks for Deep Attribute Modeling
We propose a method for inferring human attributes (such as gender, hair style, clothes style, expression, action) from images of people under large variation of viewpoint, pose, appearance, articulation and occlusion.
Ranked #7 on
Facial Attribute Classification
on LFWA
Modeling Radiometric Uncertainty for Vision with Tone-mapped Color Images
To produce images that are suitable for display, tone-mapping is widely used in digital cameras to map linear color measurements into narrow gamuts with limited dynamic range.
Deformable Part Descriptors for Fine-grained Recognition and Attribute Prediction
Recognizing objects in fine-grained domains can be extremely challenging due to the subtle differences between subcategories.
Ranked #25 on
Fine-Grained Image Classification
on CUB-200-2011
Visual Concept Learning: Combining Machine Vision and Bayesian Generalization on Concept Hierarchies
Learning a visual concept from a small number of positive examples is a significant challenge for machine learning algorithms.
One-Shot Adaptation of Supervised Deep Convolutional Models
In other words, are deep CNNs trained on large amounts of labeled data as susceptible to dataset bias as previous methods have been shown to be?
On learning to localize objects with minimal supervision
Learning to localize objects with minimal supervision is an important problem in computer vision, since large fully annotated datasets are extremely costly to obtain.
Ranked #35 on
Weakly Supervised Object Detection
on PASCAL VOC 2007
DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
Convolutional Neural Networks (CNNs) can provide accurate object classification.
Detection Bank: An Object Detection Based Video Representation for Multimedia Event Recognition
While low-level image features have proven to be effective representations for visual recognition tasks such as object recognition and scene classification, they are inadequate to capture complex semantic meaning required to solve high-level visual tasks such as multimedia event detection and recognition.
Anytime Recognition of Objects and Scenes
On suitable datasets, we can incorporate a semantic back-off strategy that gives maximally specific predictions for a desired level of accuracy; this provides a new view on the time course of human visual perception.
Continuous Manifold Based Adaptation for Evolving Visual Domains
The classic domain adaptation paradigm considers the world to be separated into stationary domains with clear boundaries between them.
Learning Scalable Discriminative Dictionary with Sample Relatedness
We use sample relatedness information to improve the generalization of the learned dictionary.
Caffe: Convolutional Architecture for Fast Feature Embedding
The framework is a BSD-licensed C++ library with Python and MATLAB bindings for training and deploying general-purpose convolutional neural networks and other deep models efficiently on commodity architectures.
Weakly-supervised Discovery of Visual Pattern Configurations
The increasing prominence of weakly labeled data nurtures a growing demand for object detection methods that can cope with minimal supervision.
Part-based R-CNNs for Fine-grained Category Detection
Semantic part localization can facilitate fine-grained categorization by explicitly isolating subtle appearance differences associated with specific object parts.
Ranked #63 on
Fine-Grained Image Classification
on CUB-200-2011
LSDA: Large Scale Detection Through Adaptation
A major challenge in scaling object detection is the difficulty of obtaining labeled images for large numbers of categories.
Deformable Part Models are Convolutional Neural Networks
Deformable part models (DPMs) and convolutional neural networks (CNNs) are two widely used tools for visual recognition.
Ranked #28 on
Object Detection
on PASCAL VOC 2007
DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks
Nearly one million Flickr images tagged with these ANPs are downloaded to train the classifiers of the concepts.
Do Convnets Learn Correspondence?
Convolutional neural nets (convnets) trained from massive labeled datasets have substantially improved the state-of-the-art in image classification and object detection.
Ranked #4 on
Keypoint Detection
on Pascal3D+
Fully Convolutional Networks for Semantic Segmentation
Convolutional networks are powerful visual models that yield hierarchies of features.
Ranked #2 on
Semantic Segmentation
on SkyScapes-Lane
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent, or "temporally deep", are effective for tasks involving sequences, visual and otherwise.
Ranked #3 on
Human Interaction Recognition
on BIT
Detector Discovery in the Wild: Joint Multiple Instance and Representation Learning
We develop methods for detector learning which exploit joint training over both weak and strong labels and which transfer learned perceptual representations from strongly-labeled auxiliary tasks.
Deep Domain Confusion: Maximizing for Domain Invariance
Recent reports suggest that a generic supervised deep CNN model trained on a large-scale dataset reduces, but does not remove, dataset bias on a standard benchmark.
Ranked #6 on
Domain Adaptation
on Office-Caltech
Fully Convolutional Multi-Class Multiple Instance Learning
We propose a novel MIL formulation of multi-class semantic segmentation learning by a fully convolutional network.
Learning Compact Convolutional Neural Networks with Nested Dropout
Recently, nested dropout was proposed as a method for ordering representation units in autoencoders by their information content, without diminishing reconstruction cost.
End-to-End Training of Deep Visuomotor Policies
Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control.
Sequence to Sequence -- Video to Text
Our LSTM model is trained on video-sentence pairs and learns to associate a sequence of video frames to a sequence of words in order to generate a description of the event in the video clip.
Constrained Convolutional Neural Networks for Weakly Supervised Segmentation
We propose Constrained CNN (CCNN), a method which uses a novel loss function to optimize for any set of linear constraints on the output space (i. e. predicted label distribution) of a CNN.
Deep Spatial Autoencoders for Visuomotor Learning
Our method uses a deep spatial autoencoder to acquire a set of feature points that describe the environment for the current task, such as the positions of objects, and then learns a motion skill with these feature points using an efficient reinforcement learning method based on local linear models.
Simultaneous Deep Transfer Across Domains and Tasks
Recent reports suggest that a generic supervised deep CNN model trained on a large-scale dataset reduces, but does not remove, dataset bias.
Spatial Semantic Regularisation for Large Scale Object Detection
Our approach proves to be especially useful in large scale settings with thousands of classes, where spatial and semantic interactions are very frequent and only weakly supervised detectors can be built due to a lack of bounding box annotations.
Quantification in-the-wild: data-sets and baselines
Quantification is the task of estimating the class-distribution of a data-set.
Neural Module Networks
Visual question answering is fundamentally compositional in nature---a question like "where is the dog?"
Ranked #6 on
Visual Question Answering (VQA)
on VQA v1 test-std
Grounding of Textual Phrases in Images by Reconstruction
We propose a novel approach which learns grounding by reconstructing a given phrase using an attention mechanism, which can be either latent or optimized directly.
Ranked #12 on
Phrase Grounding
on Flickr30k Entities Test
Natural Language Object Retrieval
In this paper, we address the task of natural language object retrieval, to localize a target object within a given image based on a natural language query of the object.
Ranked #12 on
Referring Expression Comprehension
on Talk2Car
Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
Current deep caption models can only describe objects contained in paired image-sentence corpora, despite the fact that they are pre-trained with large object recognition datasets, namely ImageNet.
Deep Learning for Tactile Understanding From Visual and Haptic Data
Robots which interact with the physical world will benefit from a fine-grained tactile understanding of objects and surfaces.
Compact Bilinear Pooling
Bilinear models has been shown to achieve impressive performance on a wide range of visual tasks, such as semantic segmentation, fine grained recognition and face recognition.
Data-dependent Initializations of Convolutional Neural Networks
Convolutional Neural Networks spread through computer vision like a wildfire, impacting almost all visual tasks imaginable.
Mapping Images to Sentiment Adjective Noun Pairs with Factorized Neural Nets
We consider the visual sentiment task of mapping an image to an adjective noun pair (ANP) such as "cute baby".
Fine-grained pose prediction, normalization, and recognition
Pose variation and subtle differences in appearance are key challenges to fine-grained classification.
Auxiliary Image Regularization for Deep CNNs with Noisy Labels
Precisely-labeled data sets with sufficient amount of samples are very important for training deep convolutional neural networks (CNNs).
Adapting Deep Visuomotor Representations with Weak Pairwise Constraints
We propose a novel, more powerful combination of both distribution and pairwise image alignment, and remove the requirement for expensive annotation by using weakly aligned pairs of images in the source and target domains.
Constrained Structured Regression with Convolutional Neural Networks
We present a regression framework which models the output distribution of neural networks.
Sequence to Sequence - Video to Text
Our LSTM model is trained on video-sentence pairs and learns to associate a sequence of video frames to a sequence of words in order to generate a description of the event in the video clip.
Learning The Structure of Deep Convolutional Networks
In this work, we develop a novel method for automatically learning aspects of the structure of a deep model, in order to improve its performance, especially when labeled training data are scarce.
Learning to Compose Neural Networks for Question Answering
We describe a question answering model that applies to both images and structured knowledge bases.
Segmentation from Natural Language Expressions
To produce pixelwise segmentation for the language expression, we propose an end-to-end trainable recurrent and convolutional network model that jointly learns to process visual and linguistic information.
Ranked #16 on
Referring Expression Segmentation
on J-HMDB
Generating Visual Explanations
Clearly explaining a rationale for a classification decision to an end-user can be as important as the decision itself.
Context Encoders: Feature Learning by Inpainting
In order to succeed at this task, context encoders need to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing part(s).
Fully Convolutional Networks for Semantic Segmentation
Convolutional networks are powerful visual models that yield hierarchies of features.
Ranked #2 on
Semantic Segmentation
on NYU Depth v2
(Mean Accuracy metric)
Adversarial Feature Learning
The ability of the Generative Adversarial Networks (GANs) framework to learn generative models mapping from simple latent distributions to arbitrarily complex data distributions has been demonstrated empirically, with compelling results showing that the latent space of such generators captures semantic variation in the data distribution.
Learning With Side Information Through Modality Hallucination
Thus, our method transfers information commonly extracted from depth training data to a network which can extract that information from the RGB counterpart.
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.
Captioning Images with Diverse Objects
We propose minimizing a joint objective which can learn from these diverse data sources and leverage distributional semantic embeddings, enabling the model to generalize and describe novel objects outside of image-caption datasets.
Clockwork Convnets for Video Semantic Segmentation
Recent years have seen tremendous progress in still-image segmentation; however the na\"ive application of these state-of-the-art algorithms to every video frame requires considerable computation and ignores the temporal continuity inherent in video.
Utilizing Large Scale Vision and Text Datasets for Image Segmentation from Referring Expressions
Image segmentation from referring expressions is a joint vision and language modeling task, where the input is an image and a textual expression describing a particular region in the image; and the goal is to localize and segment the specific image region based on the given expression.
Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer
Using deep reinforcement learning to train general purpose neural network policies alleviates some of the burden of manual representation engineering by using expressive policy classes, but exacerbates the challenge of data collection, since such methods tend to be less efficient than RL with low-dimensional, hand-designed representations.
Modeling Relationships in Referential Expressions with Compositional Modular Networks
In this paper we instead present a modular deep architecture capable of analyzing referential expressions into their component parts, identifying entities and relationships mentioned in the input expression and grounding them all in the scene.
Ranked #1 on
Visual Question Answering (VQA)
on Visual7W
End-to-end Learning of Driving Models from Large-scale Video Datasets
Robust perception-action models should be learned from training data with diverse visual appearances and realistic behaviors, yet current approaches to deep visuomotor policy learning have been generally limited to in-situ models learned from a single vehicle or a simulation environment.
FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation
In this paper, we introduce the first domain adaptive semantic segmentation method, proposing an unsupervised adversarial approach to pixel prediction problems.
Ranked #2 on
Image-to-Image Translation
on SYNTHIA Fall-to-Winter
Attentive Explanations: Justifying Decisions and Pointing to the Evidence
In contrast, humans can justify their decisions with natural language and point to the evidence in the visual world which led to their decisions.
Learning Features by Watching Objects Move
Given the extensive evidence that motion plays a key role in the development of the human visual system, we hope that this straightforward approach to unsupervised learning will be more effective than cleverly designed 'pretext' tasks studied in the literature.
Loss is its own Reward: Self-Supervision for Reinforcement Learning
Reinforcement learning optimizes policies for expected cumulative reward.
Visual Discovery at Pinterest
Over the past three years Pinterest has experimented with several visual search and recommendation services, including Related Pins (2014), Similar Looks (2015), Flashlight (2016) and Lens (2017).
Adversarial Discriminative Domain Adaptation
Adversarial learning methods are a promising approach to training robust deep networks, and can generate complex samples across diverse domains.
Learning Detection with Diverse Proposals
To predict a set of diverse and informative proposals with enriched representations, this paper introduces a differentiable Determinantal Point Process (DPP) layer that is able to augment the object detection architectures.
Learning to Reason: End-to-End Module Networks for Visual Question Answering
Natural language questions are inherently compositional, and many are most easily answered by reasoning about their decomposition into modular sub-problems.
Ranked #43 on
Visual Question Answering (VQA)
on VQA v2 test-dev
Generalized orderless pooling performs implicit salient matching
In this paper, we generalize average and bilinear pooling to "alpha-pooling", allowing for learning the pooling strategy during training.
Curiosity-driven Exploration by Self-supervised Prediction
In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether.
Deep Layer Aggregation
We augment standard architectures with deeper aggregation to better fuse information across layers.
Localizing Moments in Video with Natural Language
A key obstacle to training our MCN model is that current video datasets do not include pairs of localized video segments and referring expressions, or text descriptions which uniquely identify a corresponding moment.
Deep Object-Centric Representations for Generalizable Robot Learning
We devise an object-level attentional mechanism that can be used to determine relevant objects from a few trajectories or demonstrations, and then immediately incorporate those objects into a learned policy.
Fooling Vision and Language Models Despite Localization and Attention Mechanism
Our work sheds new light on understanding adversarial attacks on vision systems which have a language component and shows that attention, bounding box localization, and compositional internal structures are vulnerable to adversarial attacks.
Gradient-free Policy Architecture Search and Adaptation
We develop a method for policy architecture search and adaptation via gradient-free optimization which can learn to perform autonomous driving tasks.
CyCADA: Cycle-Consistent Adversarial Domain Adaptation
Domain adaptation is critical for success in new, unseen environments.
Attentive Explanations: Justifying Decisions and Pointing to the Evidence (Extended Abstract)
We also introduce a multimodal methodology for generating visual and textual explanations simultaneously.
Grounding Visual Explanations (Extended Abstract)
Existing models which generate textual explanations enforce task relevance through a discriminative term loss function, but such mechanisms only weakly constrain mentioned object parts to actually be present in the image.
SkipNet: Learning Dynamic Routing in Convolutional Networks
While deeper convolutional networks are needed to achieve maximum accuracy in visual perception tasks, for many inputs shallower networks are sufficient.
Learning to Segment Every Thing
Most methods for object instance segmentation require all training examples to be labeled with segmentation masks.
Toward Multimodal Image-to-Image Translation
Our proposed method encourages bijective consistency between the latent encoding and output modes.
Multi-Content GAN for Few-Shot Font Style Transfer
In this work, we focus on the challenge of taking partial observations of highly-stylized text and generalizing the observations to generate unobserved glyphs in the ornamented typeface.
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
Meta-learning allows an intelligent agent to leverage prior learning episodes as a basis for quickly improving performance on a novel task.
Reinforcement Learning from Imperfect Demonstrations
We propose a unified reinforcement learning algorithm, Normalized Actor-Critic (NAC), that effectively normalizes the Q-function, reducing the Q-values of actions unseen in the demonstration data.
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence
We propose a multimodal approach to explanation, and argue that the two modalities provide complementary explanatory strengths.
Women also Snowboard: Overcoming Bias in Captioning Models
We introduce a new Equalizer model that ensures equal gender probability when gender evidence is occluded in a scene and confident predictions when gender evidence is present.
Zero-Shot Visual Imitation
In our framework, the role of the expert is only to communicate the goals (i. e., what to imitate) during inference.
BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
Datasets drive vision progress, yet existing driving datasets are impoverished in terms of visual content and supported tasks to study multitask learning for autonomous driving.
Ranked #5 on
Multiple Object Tracking
on BDD100K test
Few-Shot Segmentation Propagation with Guided Networks
Learning-based methods for visual segmentation have made progress on particular types of segmentation tasks, but are limited by the necessary supervision, the narrow definitions of fixed tasks, and the lack of control during inference for correcting errors.
Deep Mixture of Experts via Shallow Embedding
Larger networks generally have greater representational power at the cost of increased computational complexity.
Speaker-Follower Models for Vision-and-Language Navigation
We use this speaker model to (1) synthesize new instructions for data augmentation and to (2) implement pragmatic reasoning, which evaluates how well candidate action sequences explain an instruction.
Learning Instance Segmentation by Interaction
The agent uses its current segmentation model to infer pixels that constitute objects and refines the segmentation model by interacting with these pixels.
Generating Counterfactual Explanations with Natural Language
We call such textual explanations counterfactual explanations, and propose an intuitive method to generate counterfactual explanations by inspecting which evidence in an input is missing, but might contribute to a different classification decision if present in the image.
Women also Snowboard: Overcoming Bias in Captioning Models (Extended Abstract)
Most machine learning methods are known to capture and exploit biases of the training data.
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
Model-based reinforcement learning (RL) is considered to be a promising approach to reduce the sample complexity that hinders model-free RL.
Compositional GAN: Learning Image-Conditional Binary Composition
Generative Adversarial Networks (GANs) can produce images of remarkable complexity and realism but are generally structured to sample from a single latent source ignoring the explicit spatial interaction between multiple entities that could be present in a scene.
Explainable Neural Computation via Stack Neural Module Networks
In complex inferential tasks like question answering, machine learning models must confront two challenges: the need to implement a compositional reasoning process, and, in many applications, the need for this reasoning process to be interpretable to assist users in both development and prediction.
Ranked #14 on
Referring Expression Comprehension
on Talk2Car
Grounding Visual Explanations
Our model improves the textual explanation quality of fine-grained classification decisions on the CUB dataset by mentioning phrases that are grounded in the image.
Textual Explanations for Self-Driving Vehicles
Finally, we explore a version of our model that generates rationalizations, and compare with introspective explanations on the same video segments.
Large-Scale Study of Curiosity-Driven Learning
However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent.
Ranked #14 on
Atari Games
on Atari 2600 Montezuma's Revenge
Localizing Moments in Video with Temporal Language
To benchmark whether our model, and other recent video localization models, can effectively reason about temporal language, we collect the novel TEMPOral reasoning in video and language (TEMPO) dataset.
Object Hallucination in Image Captioning
Despite continuously improving performance, contemporary image captioning models are prone to "hallucinating" objects that are not actually in a scene.
Uncertainty-guided Lifelong Learning in Bayesian Networks
Sequentially learning of tasks arriving in a continuous stream is a complex problem and becomes more challenging when the model has a fixed capacity.
Rethinking the Value of Network Pruning
Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm.
Discriminator Rejection Sampling
We propose a rejection sampling scheme using the discriminator of a GAN to approximately correct errors in the GAN generator distribution.
Modular Architecture for StarCraft II with Deep Reinforcement Learning
We present a novel modular architecture for StarCraft II AI.
Deep Object-Centric Policies for Autonomous Driving
While learning visuomotor skills in an end-to-end manner is appealing, deep neural networks are often uninterpretable and fail in surprising ways.
Joint Monocular 3D Vehicle Detection and Tracking
The framework can not only associate detections of vehicles in motion over time, but also estimate their complete 3D bounding box information from a sequence of 2D images captured on a moving platform.
Ranked #12 on
Multiple Object Tracking
on KITTI Tracking test
Disentangling Propagation and Generation for Video Prediction
A dynamic scene has two types of elements: those that move fluidly and can be predicted from previous frames, and those which are disoccluded (exposed) and cannot be extrapolated.
SPLAT: Semantic Pixel-Level Adaptation Transforms for Detection
Without dense labels, as is the case when only detection labels are available in the source, transformations are learned using CycleGAN alignment.
Spatio-Temporal Action Graph Networks
Events defined by the interaction of objects in a scene are often of critical importance; yet important events may have insufficient labeled examples to train a conventional deep model to generalize to future object appearance.
Few-shot Object Detection via Feature Reweighting
The feature learner extracts meta features that are generalizable to detect novel object classes, using training data from base classes with sufficient samples.
Ranked #21 on
Few-Shot Object Detection
on MS-COCO (30-shot)
Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders
Many approaches in generalized zero-shot learning rely on cross-modal mapping between the image feature space and the class embedding space.
Ranked #2 on
Generalized Few-Shot Learning
on AwA2
Adversarial Inference for Multi-Sentence Video Description
Among the main issues are the fluency and coherence of the generated descriptions, and their relevance to the video.
Hierarchical Discrete Distribution Decomposition for Match Density Estimation
Explicit representations of the global match distributions of pixel-wise correspondences between pairs of images are desirable for uncertainty estimation and downstream applications.
Ranked #13 on
Optical Flow Estimation
on KITTI 2015 (train)
Similarity R-C3D for Few-shot Temporal Activity Detection
In this paper, we present a conceptually simple and general yet novel framework for few-shot temporal activity detection which detects the start and end time of the few-shot input activities in an untrimmed video.
Robust Change Captioning
We present a novel Dual Dynamic Attention Model (DUDA) to perform robust Change Captioning.
Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity
We evaluate the performance of these dynamic and modular agents in simulated environments.
Efficient Receptive Field Learning by Dynamic Gaussian Structure
The visual world is vast and varied, but its variations divide into structured and unstructured factors.
Cross-Linked Variational Autoencoders for Generalized Zero-Shot Learning
While following the same direction, we also take artificial feature generation one step further and propose a model where a shared latent space of image features and class embeddings is learned by aligned variational autoencoders, for the purpose of generating latent features to train a softmax classifier.
Compositional GAN (Extended Abstract): Learning Image-Conditional Binary Composition
Generative Adversarial Networks (GANs) can produce images of surprising complexity and realism but are generally structured to sample from a single latent source ignoring the explicit spatial interaction between multiple entities that could be present in a scene.
Variational Adversarial Active Learning
Unlike conventional active learning algorithms, our approach is task agnostic, i. e., it does not depend on the performance of the task for which we are trying to acquire labeled data.
TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning
We show that TAFE-Net is highly effective in generalizing to new tasks or concepts and evaluate the TAFE-Net on a range of benchmarks in zero-shot and few-shot learning.
Ranked #1 on
Few-Shot Image Classification
on aPY - 0-Shot
Semi-supervised Domain Adaptation via Minimax Entropy
Contemporary domain adaptation methods are very effective at aligning feature distributions of source and target domains without any target supervision.
Blurring the Line Between Structure and Learning to Optimize and Adapt Receptive Fields
Adapting receptive fields by dynamic Gaussian structure further improves results, equaling the accuracy of free-form deformation while improving efficiency.
Meta-Learning to Guide Segmentation
To explore generalization, we analyze guidance as a bridge between different levels of supervision to segment classes as the union of instances.
Accurate Visual Localization for Automotive Applications
We propose a hybrid coarse-to-fine approach that leverages visual and GPS location cues.
Language-Conditioned Graph Networks for Relational Reasoning
E. g., conditioning on the "on" relationship to the plate, the object "mug" gathers messages from the object "plate" to update its representation to "mug on the plate", which can be easily consumed by a simple classifier for answer prediction.
Ranked #3 on
Referring Expression Comprehension
on CLEVR-Ref+
Monocular Plan View Networks for Autonomous Driving
Convolutions on monocular dash cam videos capture spatial invariances in the image plane but do not explicitly reason about distances and depth.
Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation
The actual grounding can connect language to the environment through multiple modalities, e. g. "stop at the door" might ground into visual objects, while "turn right" might rely only on the geometric structure of a route.
Uncertainty-guided Continual Learning with Bayesian Neural Networks
Continual learning aims to learn new tasks without forgetting previously learned ones.
Task-Aware Feature Generation for Zero-Shot Compositional Learning
In this work, we propose a task-aware feature generation (TFG) framework for compositional learning, which generates features of novel visual concepts by transferring knowledge from previously seen concepts.
Dynamic Scale Inference by Entropy Minimization
Given the variety of the visual world there is not one true scale for recognition: objects may appear at drastically different sizes across the visual field.
Weakly-Supervised Trajectory Segmentation for Learning Reusable Skills
Instead of direct manual supervision which is tedious and prone to bias, in this work, our goal is to extract reusable skills from a collection of human demonstrations collected directly for several end-tasks.
Blurring Structure and Learning to Optimize and Adapt Receptive Fields
Adapting receptive fields by dynamic Gaussian structure further improves results, equaling the accuracy of free-form deformation while improving efficiency.
Composable Semi-parametric Modelling for Long-range Motion Generation
Learning diverse and natural behaviors is one of the longstanding goal for creating intelligent characters in the animated world.
Scoring-Aggregating-Planning: Learning task-agnostic priors from interactions and sparse rewards for zero-shot generalization
In this paper, we propose Scoring-Aggregating-Planning (SAP), a framework that can learn task-agnostic semantics and dynamics priors from arbitrary quality interactions as well as the corresponding sparse rewards and then plan on unseen tasks in zero-shot condition.
Unsupervised Domain Adaptation through Self-Supervision
This paper addresses unsupervised domain adaptation, the setting where labeled training data is available on a source domain, but the goal is to have good performance on a target domain with only unlabeled data.
Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation
The agent is first presented with previous experiences in the training environment, along with task description in the form of trajectory-level sparse rewards.
Regularization Matters in Policy Optimization
In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks.
Exploring Simple and Transferable Recognition-Aware Image Processing
Interestingly, the processing model's ability to enhance recognition quality can transfer when evaluated on models of different architectures, recognized categories, tasks and training datasets.
Plan Arithmetic: Compositional Plan Vectors for Multi-Task Control
We show that CPVs can be learned within a one-shot imitation learning framework without any additional supervision or information about task hierarchy, and enable a demonstration-conditioned policy to generalize to tasks that sequence twice as many skills as the tasks seen during training.
Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA
Recent work has explored the TextVQA task that requires reading and understanding text in images to answer a question.
Semantic Bottleneck Scene Generation
For the former, we use an unconditional progressive segmentation generation network that captures the distribution of realistic semantic scene layouts.
Ranked #1 on
Image Generation
on Cityscapes-5K 256x512
Compositional Plan Vectors
We show that CPVs can be learned within a one-shot imitation learning framework without any additional supervision or information about task hierarchy, and enable a demonstration-conditioned policy to generalize to tasks that sequence twice as many skills as the tasks seen during training.
Learning Canonical Representations for Scene Graph to Image Generation
Generating realistic images of complex visual scenes becomes challenging when one wishes to control the structure of the generated images.
Ranked #3 on
Layout-to-Image Generation
on Visual Genome 256x256
Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks
Human action is naturally compositional: humans can easily recognize and perform actions with objects that are different from those used in training demonstrations.
Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
Learning robotic manipulation tasks using reinforcement learning with sparse rewards is currently impractical due to the outrageous data requirements.
Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning
The edge between these two lines of works has yet been underexplored, and the effectiveness of meta-learning in few-shot learning remains unclear.
Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning
This drawback hinders the model from learning subtle variance and fine-grained information.
Frustratingly Simple Few-Shot Object Detection
Such a simple approach outperforms the meta-learning methods by roughly 2~20 points on current benchmarks and sometimes even doubles the accuracy of the prior methods.
Ranked #17 on
Few-Shot Object Detection
on MS-COCO (30-shot)
Adversarial Continual Learning
We show that shared features are significantly less prone to forgetting and propose a novel hybrid continual learning framework that learns a disjoint representation for task-invariant and task-specific features required to solve a sequence of tasks.
Revisiting Few-shot Activity Detection with Class Similarity Control
In this paper, we present a conceptually simple and general yet novel framework for few-shot temporal activity detection based on proposal regression which detects the start and end time of the activities in untrimmed videos.
Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning
Weakly-supervised action localization requires training a model to localize the action segments in the video given only video level action label.
Ranked #9 on
Weakly Supervised Action Localization
on THUMOS’14
Spatio-Temporal Action Detection with Multi-Object Interaction
Spatio-temporal action detection in videos requires localizing the action both spatially and temporally in the form of an "action tube".
Contrastive Examples for Addressing the Tyranny of the Majority
Computer vision algorithms, e. g. for face recognition, favour groups of individuals that are better represented in the training data.
ParkPredict: Motion and Intent Prediction of Vehicles in Parking Lots
We investigate the problem of predicting driver behavior in parking lots, an environment which is less structured than typical road networks and features complex, interactive maneuvers in a compact space.
Rethinking preventing class-collapsing in metric learning with margin-based losses
We theoretically prove and empirically show that under reasonable noise assumptions, margin-based losses tend to project all samples of a class with various modes onto a single point in the embedding space, resulting in a class collapse that usually renders the space ill-sorted for classification or retrieval.
Quasi-Dense Similarity Learning for Multiple Object Tracking
Compared to methods with similar detectors, it boosts almost 10 points of MOTA and significantly decreases the number of ID switches on BDD100K and Waymo datasets.
Ranked #1 on
One-Shot Object Detection
on PASCAL VOC 2012 val
Tent: Fully Test-time Adaptation by Entropy Minimization
A model must adapt itself to generalize to new and different data during testing.
Compositional Video Synthesis with Action Graphs
Our generative model for this task (AG2Vid) disentangles motion and appearance features, and by incorporating a scheduling mechanism for actions facilitates a timely and coordinated video generation.
Video Prediction via Example Guidance
In video prediction tasks, one major challenge is to capture the multi-modal nature of future contents and dynamics.
Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation
We also demonstrate that reducing the task of room navigation to point navigation improves the performance further.
Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics
We demonstrate the efficacy of our method on hand gesture synthesis from body motion input, and as a strong body prior for single-view image-based 3D hand pose estimation.
What Should Not Be Contrastive in Contrastive Learning
Recent self-supervised contrastive methods have been able to produce impressive transferable visual representations by learning to be invariant to different data augmentations.
Identity-Aware Multi-Sentence Video Description
This auxiliary task allows us to propose a two-stage approach to Identity-Aware Video Description.
Hierarchical Style-based Networks for Motion Synthesis
Generating diverse and natural human motion is one of the long-standing goals for creating intelligent characters in the animated world.
ePointDA: An End-to-End Simulation-to-Real Domain Adaptation Framework for LiDAR Point Cloud Segmentation
They require prior knowledge of real-world statistics and ignore the pixel-level dropout noise gap and the spatial feature gap between different domains.
SelfAugment: Automatic Augmentation Policies for Self-Supervised Learning
A common practice in unsupervised representation learning is to use labeled data to evaluate the quality of the learned representations.
Reducing Class Collapse in Metric Learning with Easy Positive Sampling
We theoretically prove and empirically show that under reasonable noise assumptions, prevalent embedding losses in metric learning, e. g., triplet loss, tend to project all samples of a class with various modes onto a single point in the embedding space, resulting in a class collapse that usually renders the space ill-sorted for classification or retrieval.
Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting
The goal of continual learning (CL) is to learn a sequence of tasks without suffering from the phenomenon of catastrophic forgetting.
Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation
First, we provide a finite sample bound for both classification and regression problems under Semi-DA.
Auxiliary Task Reweighting for Minimum-data Learning
By adjusting the auxiliary task weights to minimize the divergence between the surrogate prior and the true prior of the main task, we obtain a more accurate prior estimation, achieving the goal of minimizing the required amount of training data for the main task and avoiding a costly grid search.
Modular Networks for Compositional Instruction Following
In our approach, subgoal modules each carry out natural language instructions for a specific subgoal type.
Fighting Copycat Agents in Behavioral Cloning from Observation Histories
Imitation learning trains policies to map from input observations to the actions that an expert would choose.
Temporal Action Detection with Multi-level Supervision
We extensively benchmark against the baselines for SSAD and OSAD on our created data splits in THUMOS14 and ActivityNet1. 2, and demonstrate the effectiveness of the proposed UFA and IB methods.
Minimax Active Learning
Active learning aims to develop label-efficient algorithms by querying the most representative samples to be labeled by a human annotator.
Contrastive Video Textures
By randomly traversing edges with high transition probabilities, we generate diverse temporally smooth videos with novel sequences and transitions.
Regularization Matters in Policy Optimization - An Empirical Study on Continuous Control
In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks.
Novelty Detection with Rotated Contrastive Predictive Coding
To this end, reconstruction-based learning is often used in which the normality of an observation is expressed in how well it can be reconstructed.
Discovering Autoregressive Orderings with Variational Inference
One strategy to recover this information is to decode both the content and location of tokens.
Unconditional Synthesis of Complex Scenes Using a Semantic Bottleneck
Coupling the high-fidelity generation capabilities of label-conditional image synthesis methods with the flexibility of unconditional generative models, we propose a semantic bottleneck GAN model for unconditional synthesis of complex scenes.
Instance-Aware Predictive Navigation in Multi-Agent Environments
To decide the action at each step, we seek the action sequence that can lead to safe future states based on the prediction module outputs by repeatedly sampling likely action sequences.
Monocular Quasi-Dense 3D Object Tracking
Experiments on our proposed simulation data and real-world benchmarks, including KITTI, nuScenes, and Waymo datasets, show that our tracking framework offers robust object association and tracking on urban-driving scenarios.
Ranked #7 on
Multiple Object Tracking
on KITTI Tracking test
Self-Supervised Pretraining Improves Self-Supervised Pretraining
Through experimentation on 16 diverse vision datasets, we show HPT converges up to 80x faster, improves accuracy across tasks, and improves the robustness of the self-supervised pretraining process to changes in the image augmentation policy or amount of pretraining data.
