Search Results for author:
Found 312 papers, 170 papers with code
A ConvNet for the 2020s
The "Roaring 20s" of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model.
Ranked #1 on
Classification
on InDL
Caffe: Convolutional Architecture for Fast Feature Embedding
The framework is a BSD-licensed C++ library with Python and MATLAB bindings for training and deploying general-purpose convolutional neural networks and other deep models efficiently on commodity architectures.
Curiosity-driven Exploration by Self-supervised Prediction
In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether.
Deep Layer Aggregation
We augment standard architectures with deeper aggregation to better fuse information across layers.
Learning to Segment Every Thing
Most methods for object instance segmentation require all training examples to be labeled with segmentation masks.
Toward Multimodal Image-to-Image Translation
Our proposed method encourages bijective consistency between the latent encoding and output modes.
Context Encoders: Feature Learning by Inpainting
In order to succeed at this task, context encoders need to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing part(s).
Fully Convolutional Networks for Semantic Segmentation
Convolutional networks are powerful visual models that yield hierarchies of features.
Ranked #2 on
Semantic Segmentation
on NYU Depth v2
(Mean Accuracy metric)
Adversarial Feature Learning
The ability of the Generative Adversarial Networks (GANs) framework to learn generative models mapping from simple latent distributions to arbitrarily complex data distributions has been demonstrated empirically, with compelling results showing that the latent space of such generators captures semantic variation in the data distribution.
CyCADA: Cycle-Consistent Adversarial Domain Adaptation
Domain adaptation is critical for success in new, unseen environments.
Adversarial Discriminative Domain Adaptation
Adversarial learning methods are a promising approach to training robust deep networks, and can generate complex samples across diverse domains.
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
We evaluate whether features extracted from the activation of a deep convolutional network trained in a fully supervised fashion on a large, fixed set of object recognition tasks can be re-purposed to novel generic tasks.
From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations
We present a framework for generating full-bodied photorealistic avatars that gesture according to the conversational dynamics of a dyadic interaction.
Rich feature hierarchies for accurate object detection and semantic segmentation
We find that R-CNN outperforms OverFeat by a large margin on the 200-class ILSVRC2013 detection dataset.
Ranked #27 on
Object Detection
on PASCAL VOC 2007
(using extra training data)
Sequential Modeling Enables Scalable Learning for Large Vision Models
We introduce a novel sequential modeling approach which enables learning a Large Vision Model (LVM) without making use of any linguistic data.
Rethinking the Value of Network Pruning
Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm.
Frustratingly Simple Few-Shot Object Detection
Such a simple approach outperforms the meta-learning methods by roughly 2~20 points on current benchmarks and sometimes even doubles the accuracy of the prior methods.
Ranked #17 on
Few-Shot Object Detection
on MS-COCO (30-shot)
VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation
Existing approaches to unsupervised video instance segmentation typically rely on motion estimates and experience difficulties tracking small or divergent motions.
Large-Scale Study of Curiosity-Driven Learning
However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent.
Ranked #14 on
Atari Games
on Atari 2600 Montezuma's Revenge
Neural Network Diffusion
The autoencoder extracts latent representations of a subset of the trained network parameters.
Modeling Relationships in Referential Expressions with Compositional Modular Networks
In this paper we instead present a modular deep architecture capable of analyzing referential expressions into their component parts, identifying entities and relationships mentioned in the input expression and grounding them all in the scene.
Ranked #1 on
Visual Question Answering (VQA)
on Visual7W
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.
Joint Monocular 3D Vehicle Detection and Tracking
The framework can not only associate detections of vehicles in motion over time, but also estimate their complete 3D bounding box information from a sequence of 2D images captured on a moving platform.
Ranked #12 on
Multiple Object Tracking
on KITTI Tracking test
Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning
The edge between these two lines of works has yet been underexplored, and the effectiveness of meta-learning in few-shot learning remains unclear.
Few-shot Object Detection via Feature Reweighting
The feature learner extracts meta features that are generalizable to detect novel object classes, using training data from base classes with sufficient samples.
Ranked #21 on
Few-Shot Object Detection
on MS-COCO (30-shot)
Monocular Quasi-Dense 3D Object Tracking
Experiments on our proposed simulation data and real-world benchmarks, including KITTI, nuScenes, and Waymo datasets, show that our tracking framework offers robust object association and tracking on urban-driving scenarios.
Ranked #7 on
Multiple Object Tracking
on KITTI Tracking test
PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editor
We propose PAIR Diffusion, a generic framework that can enable a diffusion model to control the structure and appearance properties of each object in the image.
Multi-Content GAN for Few-Shot Font Style Transfer
In this work, we focus on the challenge of taking partial observations of highly-stylized text and generalizing the observations to generate unobserved glyphs in the ornamented typeface.
Neural Module Networks
Visual question answering is fundamentally compositional in nature---a question like "where is the dog?"
Ranked #6 on
Visual Question Answering (VQA)
on VQA v1 test-std
Learning to Compose Neural Networks for Question Answering
We describe a question answering model that applies to both images and structured knowledge bases.
BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
Datasets drive vision progress, yet existing driving datasets are impoverished in terms of visual content and supported tasks to study multitask learning for autonomous driving.
Ranked #5 on
Multiple Object Tracking
on BDD100K test
Fully Convolutional Networks for Semantic Segmentation
Convolutional networks are powerful visual models that yield hierarchies of features.
Ranked #2 on
Semantic Segmentation
on SkyScapes-Lane
Quasi-Dense Similarity Learning for Multiple Object Tracking
Compared to methods with similar detectors, it boosts almost 10 points of MOTA and significantly decreases the number of ID switches on BDD100K and Waymo datasets.
Ranked #1 on
One-Shot Object Detection
on PASCAL VOC 2012 val
QDTrack: Quasi-Dense Similarity Learning for Appearance-Only Multiple Object Tracking
In this paper, we present Quasi-Dense Similarity Learning, which densely samples hundreds of object regions on a pair of images for contrastive learning.
Ranked #4 on
Multiple Object Tracking
on BDD100K test
K-LITE: Learning Transferable Visual Models with External Knowledge
We propose K-LITE, a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts.
LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models
Our method significantly outperforms the base diffusion model and several strong baselines in accurately generating images according to prompts that require various capabilities, doubling the generation accuracy across four tasks on average.
DETReg: Unsupervised Pretraining with Region Priors for Object Detection
Recent self-supervised pretraining methods for object detection largely focus on pretraining the backbone of the object detector, neglecting key parts of detection architecture.
Ranked #1 on
Few-Shot Object Detection
on COCO 2017
Tune it the Right Way: Unsupervised Validation of Domain Adaptation via Soft Neighborhood Density
Unsupervised domain adaptation (UDA) methods can dramatically improve generalization on unlabeled target domains.
InstanceDiffusion: Instance-level Control for Image Generation
Text-to-image diffusion models produce high quality images but do not offer control over individual instances in the image.
Ranked #3 on
Conditional Text-to-Image Synthesis
on COCO-MIG
Tent: Fully Test-time Adaptation by Entropy Minimization
A model must adapt itself to generalize to new and different data during testing.
Fighting Gradients with Gradients: Dynamic Defenses against Adversarial Attacks
Adversarial attacks optimize against models to defeat defenses.
Dropout Reduces Underfitting
Additionally, we explore a symmetric technique for regularizing overfitting models - late dropout, where dropout is not used in the early iterations and is only activated later in training.
Semi-supervised Domain Adaptation via Minimax Entropy
Contemporary domain adaptation methods are very effective at aligning feature distributions of source and target domains without any target supervision.
Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders
Many approaches in generalized zero-shot learning rely on cross-modal mapping between the image feature space and the class embedding space.
Ranked #2 on
Generalized Few-Shot Learning
on AwA2
Visual Prompting via Image Inpainting
How does one adapt a pre-trained visual model to novel downstream tasks without task-specific finetuning or any model modification?
Ranked #5 on
Personalized Segmentation
on PerSeg
Learning to Reason: End-to-End Module Networks for Visual Question Answering
Natural language questions are inherently compositional, and many are most easily answered by reasoning about their decomposition into modular sub-problems.
Ranked #44 on
Visual Question Answering (VQA)
on VQA v2 test-dev
Learning Features by Watching Objects Move
Given the extensive evidence that motion plays a key role in the development of the human visual system, we hope that this straightforward approach to unsupervised learning will be more effective than cleverly designed 'pretext' tasks studied in the literature.
Adversarial Continual Learning
We show that shared features are significantly less prone to forgetting and propose a novel hybrid continual learning framework that learns a disjoint representation for task-invariant and task-specific features required to solve a sequence of tasks.
Hierarchical Open-vocabulary Universal Image Segmentation
Open-vocabulary image segmentation aims to partition an image into semantic regions according to arbitrary text descriptions.
Ranked #1 on
Image Segmentation
on Pascal Panoptic Parts
SkipNet: Learning Dynamic Routing in Convolutional Networks
While deeper convolutional networks are needed to achieve maximum accuracy in visual perception tasks, for many inputs shallower networks are sufficient.
PyTouch: A Machine Learning Library for Touch Processing
With the increased availability of rich tactile sensors, there is an equally proportional need for open-source and integrated software capable of efficiently and effectively processing raw touch measurements into high-level signals that can be used for control and decision-making.
End-to-end Learning of Driving Models from Large-scale Video Datasets
Robust perception-action models should be learned from training data with diverse visual appearances and realistic behaviors, yet current approaches to deep visuomotor policy learning have been generally limited to in-situ models learned from a single vehicle or a simulation environment.
Grounding of Textual Phrases in Images by Reconstruction
We propose a novel approach which learns grounding by reconstructing a given phrase using an attention mechanism, which can be either latent or optimized directly.
Ranked #12 on
Phrase Grounding
on Flickr30k Entities Test
Compact Bilinear Pooling
Bilinear models has been shown to achieve impressive performance on a wide range of visual tasks, such as semantic segmentation, fine grained recognition and face recognition.
Variational Adversarial Active Learning
Unlike conventional active learning algorithms, our approach is task agnostic, i. e., it does not depend on the performance of the task for which we are trying to acquire labeled data.
When Do We Not Need Larger Vision Models?
Our results show that a multi-scale smaller model has comparable learning capacity to a larger model, and pre-training smaller models with S$^2$ can match or even exceed the advantage of larger models.
Zero-Shot Visual Imitation
In our framework, the role of the expert is only to communicate the goals (i. e., what to imitate) during inference.
Hierarchical Discrete Distribution Decomposition for Match Density Estimation
Explicit representations of the global match distributions of pixel-wise correspondences between pairs of images are desirable for uncertainty estimation and downstream applications.
Ranked #13 on
Optical Flow Estimation
on KITTI 2015 (train)
On-target Adaptation
Domain adaptation seeks to mitigate the shift between training on the \emph{source} domain and testing on the \emph{target} domain.
Masked Visual Pre-training for Motor Control
This paper shows that self-supervised visual pre-training from real-world images is effective for learning motor control tasks from pixels.
Real-World Robot Learning with Masked Visual Pre-training
Finally, we train a 307M parameter vision transformer on a massive collection of 4. 5M images from the Internet and egocentric videos, and demonstrate clearly the benefits of scaling visual pre-training for robot learning.
Localizing Moments in Video with Natural Language
A key obstacle to training our MCN model is that current video datasets do not include pairs of localized video segments and referring expressions, or text descriptions which uniquely identify a corresponding moment.
TOAST: Transfer Learning via Attention Steering
We introduce Top-Down Attention Steering (TOAST), a novel transfer learning algorithm that keeps the pre-trained backbone frozen, selects task-relevant features in the output, and feeds those features back to the model to steer the attention to the task-specific features.
Women also Snowboard: Overcoming Bias in Captioning Models
We introduce a new Equalizer model that ensures equal gender probability when gender evidence is occluded in a scene and confident predictions when gender evidence is present.
Top-Down Visual Attention from Analysis by Synthesis
In this paper, we consider top-down attention from a classic Analysis-by-Synthesis (AbS) perspective of vision.
Initializing Models with Larger Ones
Weight selection offers a new approach to leverage the power of pretrained models in resource-constrained settings, and we hope it can be a useful tool for training small models in the large-model era.
Few-Shot Segmentation Propagation with Guided Networks
Learning-based methods for visual segmentation have made progress on particular types of segmentation tasks, but are limited by the necessary supervision, the narrow definitions of fixed tasks, and the lack of control during inference for correcting errors.
Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning
This drawback hinders the model from learning subtle variance and fine-grained information.
Clockwork Convnets for Video Semantic Segmentation
Recent years have seen tremendous progress in still-image segmentation; however the na\"ive application of these state-of-the-art algorithms to every video frame requires considerable computation and ignores the temporal continuity inherent in video.
Data-dependent Initializations of Convolutional Neural Networks
Convolutional Neural Networks spread through computer vision like a wildfire, impacting almost all visual tasks imaginable.
Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks
Human action is naturally compositional: humans can easily recognize and perform actions with objects that are different from those used in training demonstrations.
Unsupervised Universal Image Segmentation
Several unsupervised image segmentation approaches have been proposed which eliminate the need for dense manually-annotated segmentation masks; current models separately handle either semantic segmentation (e. g., STEGO) or class-agnostic instance segmentation (e. g., CutLER), but not both (i. e., panoptic segmentation).
Ranked #1 on
Unsupervised Panoptic Segmentation
on COCO val2017
Deformable Part Models are Convolutional Neural Networks
Deformable part models (DPMs) and convolutional neural networks (CNNs) are two widely used tools for visual recognition.
Ranked #28 on
Object Detection
on PASCAL VOC 2007
Speaker-Follower Models for Vision-and-Language Navigation
We use this speaker model to (1) synthesize new instructions for data augmentation and to (2) implement pragmatic reasoning, which evaluates how well candidate action sequences explain an instruction.
Natural Language Object Retrieval
In this paper, we address the task of natural language object retrieval, to localize a target object within a given image based on a natural language query of the object.
Ranked #12 on
Referring Expression Comprehension
on Talk2Car
Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity
We evaluate the performance of these dynamic and modular agents in simulated environments.
Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics
We demonstrate the efficacy of our method on hand gesture synthesis from body motion input, and as a strong body prior for single-view image-based 3D hand pose estimation.
Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
Learning robotic manipulation tasks using reinforcement learning with sparse rewards is currently impractical due to the outrageous data requirements.
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent, or "temporally deep", are effective for tasks involving sequences, visual and otherwise.
Ranked #3 on
Human Interaction Recognition
on BIT
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
Model-based reinforcement learning (RL) is considered to be a promising approach to reduce the sample complexity that hinders model-free RL.
Prototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation
In this paper, we propose an end-to-end Prototypical Cross-domain Self-Supervised Learning (PCS) framework for Few-shot Unsupervised Domain Adaptation (FUDA).
Ranked #6 on
Semantic Segmentation
on DensePASS
Language-Conditioned Graph Networks for Relational Reasoning
E. g., conditioning on the "on" relationship to the plate, the object "mug" gathers messages from the object "plate" to update its representation to "mug on the plate", which can be easily consumed by a simple classifier for answer prediction.
Ranked #3 on
Referring Expression Comprehension
on CLEVR-Ref+
Segmentation from Natural Language Expressions
To produce pixelwise segmentation for the language expression, we propose an end-to-end trainable recurrent and convolutional network model that jointly learns to process visual and linguistic information.
Ranked #16 on
Referring Expression Segmentation
on J-HMDB
Region Similarity Representation Learning
We present Region Similarity Representation Learning (ReSim), a new approach to self-supervised representation learning for localization-based tasks such as object detection and segmentation.
Contrastive Test-Time Adaptation
We propose a novel way to leverage self-supervised contrastive learning to facilitate target feature learning, along with an online pseudo labeling scheme with refinement that significantly denoises pseudo labels.
Modular Visual Question Answering via Code Generation
We present a framework that formulates visual question answering as modular code generation.
Learning Instance Segmentation by Interaction
The agent uses its current segmentation model to infer pixels that constitute objects and refines the segmentation model by interacting with these pixels.
Unsupervised Domain Adaptation through Self-Supervision
This paper addresses unsupervised domain adaptation, the setting where labeled training data is available on a source domain, but the goal is to have good performance on a target domain with only unlabeled data.
Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
Current deep caption models can only describe objects contained in paired image-sentence corpora, despite the fact that they are pre-trained with large object recognition datasets, namely ImageNet.
ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension
Training a referring expression comprehension (ReC) model for a new visual domain requires collecting referring expressions, and potentially corresponding bounding boxes, for images in the domain.
Robust Object Detection via Instance-Level Temporal Cycle Confusion
Building reliable object detectors that are robust to domain shifts, such as various changes in context, viewpoint, and object appearances, is critical for real-world applications.
Uncertainty-guided Continual Learning with Bayesian Neural Networks
Continual learning aims to learn new tasks without forgetting previously learned ones.
Deep Domain Confusion: Maximizing for Domain Invariance
Recent reports suggest that a generic supervised deep CNN model trained on a large-scale dataset reduces, but does not remove, dataset bias on a standard benchmark.
Ranked #6 on
Domain Adaptation
on Office-Caltech
Explainable Neural Computation via Stack Neural Module Networks
In complex inferential tasks like question answering, machine learning models must confront two challenges: the need to implement a compositional reasoning process, and, in many applications, the need for this reasoning process to be interpretable to assist users in both development and prediction.
Ranked #14 on
Referring Expression Comprehension
on Talk2Car
Rethinking Patch Dependence for Masked Autoencoders
In this work, we re-examine inter-patch dependencies in the decoding mechanism of masked autoencoders (MAE).
FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation
In this paper, we introduce the first domain adaptive semantic segmentation method, proposing an unsupervised adversarial approach to pixel prediction problems.
Ranked #2 on
Image-to-Image Translation
on SYNTHIA Fall-to-Winter
Sequence to Sequence -- Video to Text
Our LSTM model is trained on video-sentence pairs and learns to associate a sequence of video frames to a sequence of words in order to generate a description of the event in the video clip.
Textual Explanations for Self-Driving Vehicles
Finally, we explore a version of our model that generates rationalizations, and compare with introspective explanations on the same video segments.
Constrained Convolutional Neural Networks for Weakly Supervised Segmentation
We propose Constrained CNN (CCNN), a method which uses a novel loss function to optimize for any set of linear constraints on the output space (i. e. predicted label distribution) of a CNN.
Compositional GAN: Learning Image-Conditional Binary Composition
Generative Adversarial Networks (GANs) can produce images of remarkable complexity and realism but are generally structured to sample from a single latent source ignoring the explicit spatial interaction between multiple entities that could be present in a scene.
Back to the Source: Diffusion-Driven Test-Time Adaptation
We instead update the target data, by projecting all test inputs toward the source domain with a generative diffusion model.
Self-Supervised Pretraining Improves Self-Supervised Pretraining
Through experimentation on 16 diverse vision datasets, we show HPT converges up to 80x faster, improves accuracy across tasks, and improves the robustness of the self-supervised pretraining process to changes in the image augmentation policy or amount of pretraining data.
Describing Differences in Image Sets with Natural Language
To aid in this discovery process, we explore the task of automatically describing the differences between two $\textbf{sets}$ of images, which we term Set Difference Captioning.
DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks
Nearly one million Flickr images tagged with these ANPs are downloaded to train the classifiers of the concepts.
Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation
As such, we explore how natural language descriptions of the domains seen in training data can be used with large vision models trained on diverse pretraining datasets to generate useful variations of the training data.
TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning
We show that TAFE-Net is highly effective in generalizing to new tasks or concepts and evaluate the TAFE-Net on a range of benchmarks in zero-shot and few-shot learning.
Ranked #1 on
Few-Shot Image Classification
on aPY - 0-Shot
Task-Aware Feature Generation for Zero-Shot Compositional Learning
In this work, we propose a task-aware feature generation (TFG) framework for compositional learning, which generates features of novel visual concepts by transferring knowledge from previously seen concepts.
Auto-Tuned Sim-to-Real Transfer
Policies trained in simulation often fail when transferred to the real world due to the `reality gap' where the simulator is unable to accurately capture the dynamics and visual properties of the real world.
Generalized orderless pooling performs implicit salient matching
In this paper, we generalize average and bilinear pooling to "alpha-pooling", allowing for learning the pooling strategy during training.
Multitask Vision-Language Prompt Tuning
Specifically, (i) we demonstrate the effectiveness of learning a single transferable prompt from multiple source tasks to initialize the prompt for each target task; (ii) we show many target tasks can benefit each other from sharing prompt vectors and thus can be jointly learned via multitask prompt tuning.
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence
We propose a multimodal approach to explanation, and argue that the two modalities provide complementary explanatory strengths.
LSDA: Large Scale Detection Through Adaptation
A major challenge in scaling object detection is the difficulty of obtaining labeled images for large numbers of categories.
Anytime Dense Prediction with Confidence Adaptivity
A cascade of "exits" is attached to the model to make multiple predictions.
Using Language to Extend to Unseen Domains
It is expensive to collect training data for every possible domain that a vision model may encounter when deployed.
Object Hallucination in Image Captioning
Despite continuously improving performance, contemporary image captioning models are prone to "hallucinating" objects that are not actually in a scene.
xT: Nested Tokenization for Larger Context in Large Images
Modern computer vision pipelines handle large images in one of two sub-optimal ways: down-sampling or cropping.
Early Convolutions Help Transformers See Better
To test whether this atypical design choice causes an issue, we analyze the optimization behavior of ViT models with their original patchify stem versus a simple counterpart where we replace the ViT stem by a small number of stacked stride-two 3*3 convolutions.
Robust Change Captioning
We present a novel Dual Dynamic Attention Model (DUDA) to perform robust Change Captioning.
Semantic Bottleneck Scene Generation
For the former, we use an unconditional progressive segmentation generation network that captures the distribution of realistic semantic scene layouts.
Ranked #1 on
Image Generation
on Cityscapes-5K 256x512
Object-Region Video Transformers
In this work, we present Object-Region Video Transformers (ORViT), an \emph{object-centric} approach that extends video transformer layers with a block that directly incorporates object representations.
Ranked #6 on
Action Recognition
on Diving-48
PANDA: Pose Aligned Networks for Deep Attribute Modeling
We propose a method for inferring human attributes (such as gender, hair style, clothes style, expression, action) from images of people under large variation of viewpoint, pose, appearance, articulation and occlusion.
Ranked #7 on
Facial Attribute Classification
on LFWA
Guiding Pretraining in Reinforcement Learning with Large Language Models
Reinforcement learning algorithms typically struggle in the absence of a dense, well-shaped reward function.
Localizing Moments in Video with Temporal Language
To benchmark whether our model, and other recent video localization models, can effectively reason about temporal language, we collect the novel TEMPOral reasoning in video and language (TEMPO) dataset.
Simultaneous Deep Transfer Across Domains and Tasks
Recent reports suggest that a generic supervised deep CNN model trained on a large-scale dataset reduces, but does not remove, dataset bias.
Adversarial Inference for Multi-Sentence Video Description
Among the main issues are the fluency and coherence of the generated descriptions, and their relevance to the video.
Exploring Simple and Transferable Recognition-Aware Image Processing
Interestingly, the processing model's ability to enhance recognition quality can transfer when evaluated on models of different architectures, recognized categories, tasks and training datasets.
Visual Attention Emerges from Recurrent Sparse Reconstruction
We present VARS, Visual Attention from Recurrent Sparse reconstruction, a new attention formulation built on two prominent features of the human visual attention mechanism: recurrency and sparsity.
Learning Detection with Diverse Proposals
To predict a set of diverse and informative proposals with enriched representations, this paper introduces a differentiable Determinantal Point Process (DPP) layer that is able to augment the object detection architectures.
Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting
The goal of continual learning (CL) is to learn a sequence of tasks without suffering from the phenomenon of catastrophic forgetting.
Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly
We first enable abstention capabilities for several VQA models, and analyze both their coverage, the portion of questions answered, and risk, the error on that portion.
Learning Canonical Representations for Scene Graph to Image Generation
Generating realistic images of complex visual scenes becomes challenging when one wishes to control the structure of the generated images.
Ranked #3 on
Layout-to-Image Generation
on Visual Genome 256x256
Compositional Video Synthesis with Action Graphs
Our generative model for this task (AG2Vid) disentangles motion and appearance features, and by incorporating a scheduling mechanism for actions facilitates a timely and coordinated video generation.
SelfAugment: Automatic Augmentation Policies for Self-Supervised Learning
A common practice in unsupervised representation learning is to use labeled data to evaluate the quality of the learned representations.
NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media
Online misinformation is a prevalent societal issue, with adversaries relying on tools ranging from cheap fakes to sophisticated deep fakes.
Voxel-informed Language Grounding
Natural language applied to natural 2D images describes a fundamentally 3D world.
Which One? Leveraging Context Between Objects and Multiple Views for Language Grounding
When connecting objects and their language referents in an embodied 3D environment, it is important to note that: (1) an object can be better characterized by leveraging comparative information between itself and other objects, and (2) an object's appearance can vary with camera position.
Discriminator Rejection Sampling
We propose a rejection sampling scheme using the discriminator of a GAN to approximately correct errors in the GAN generator distribution.
Regularization Matters in Policy Optimization
In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks.
Instance-Aware Predictive Navigation in Multi-Agent Environments
To decide the action at each step, we seek the action sequence that can lead to safe future states based on the prediction module outputs by repeatedly sampling likely action sequences.
Refine and Represent: Region-to-Object Representation Learning
Recent works in self-supervised learning have demonstrated strong performance on scene-level dense prediction tasks by pretraining with object-centric or region-based correspondence objectives.
Deep Object-Centric Representations for Generalizable Robot Learning
We devise an object-level attentional mechanism that can be used to determine relevant objects from a few trajectories or demonstrations, and then immediately incorporate those objects into a learned policy.
Teachable Reinforcement Learning via Advice Distillation
Training automated agents to complete complex tasks in interactive environments is challenging: reinforcement learning requires careful hand-engineering of reward functions, imitation learning requires specialized infrastructure and access to a human expert, and learning from intermediate forms of supervision (like binary preferences) is time-consuming and extracts little information from each human intervention.
Compositional Chain-of-Thought Prompting for Large Multimodal Models
The combination of strong visual backbones and Large Language Model (LLM) reasoning has led to Large Multimodal Models (LMMs) becoming the current standard for a wide range of vision and language (VL) tasks.
Ranked #30 on
Visual Reasoning
on Winoground
Identity-Aware Multi-Sentence Video Description
This auxiliary task allows us to propose a two-stage approach to Identity-Aware Video Description.
Finding Visual Task Vectors
In this work, we analyze the activations of MAE-VQGAN, a recent Visual Prompting model, and find task vectors, activations that encode task-specific information.
Accurate Visual Localization for Automotive Applications
We propose a hybrid coarse-to-fine approach that leverages visual and GPS location cues.
Video Prediction via Example Guidance
In video prediction tasks, one major challenge is to capture the multi-modal nature of future contents and dynamics.
Discovering Autoregressive Orderings with Variational Inference
One strategy to recover this information is to decode both the content and location of tokens.
Discovering Non-monotonic Autoregressive Orderings with Variational Inference
Permutations then serve as target generation orders for training an insertion-based Transformer language model.
G^3: Geolocation via Guidebook Grounding
We propose the task of Geolocation via Guidebook Grounding that uses a dataset of StreetView images from a diverse set of locations and an associated textual guidebook for GeoGuessr, a popular interactive geolocation game.
Deep Spatial Autoencoders for Visuomotor Learning
Our method uses a deep spatial autoencoder to acquire a set of feature points that describe the environment for the current task, such as the positions of objects, and then learns a motion skill with these feature points using an efficient reinforcement learning method based on local linear models.
Compositional Plan Vectors
We show that CPVs can be learned within a one-shot imitation learning framework without any additional supervision or information about task hierarchy, and enable a demonstration-conditioned policy to generalize to tasks that sequence twice as many skills as the tasks seen during training.
Captioning Images with Diverse Objects
We propose minimizing a joint objective which can learn from these diverse data sources and leverage distributional semantic embeddings, enabling the model to generalize and describe novel objects outside of image-caption datasets.
Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation
Detecting out-of-context media, such as "mis-captioned" images on Twitter, is a relevant problem, especially in domains of high public significance.
Spatio-Temporal Action Graph Networks
Events defined by the interaction of objects in a scene are often of critical importance; yet important events may have insufficient labeled examples to train a conventional deep model to generalize to future object appearance.
Fully Convolutional Multi-Class Multiple Instance Learning
We propose a novel MIL formulation of multi-class semantic segmentation learning by a fully convolutional network.
Explaining Reinforcement Learning Policies through Counterfactual Trajectories
In order for humans to confidently decide where to employ RL agents for real-world tasks, a human developer must validate that the agent will perform well at test-time.
Decentralized Vehicle Coordination: The Berkeley DeepDrive Drone Dataset
Decentralized multiagent planning has been an important field of research in robotics.
DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
Convolutional Neural Networks (CNNs) can provide accurate object classification.
Towards Learning to Play Piano with Dexterous Hands and Touch
The virtuoso plays the piano with passion, poetry and extraordinary technical ability.
A Coefficient Makes SVRG Effective
Stochastic Variance Reduced Gradient (SVRG), introduced by Johnson & Zhang (2013), is a theoretically compelling optimization method.
Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA
Recent work has explored the TextVQA task that requires reading and understanding text in images to answer a question.
Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning
Weakly-supervised action localization requires training a model to localize the action segments in the video given only video level action label.
Ranked #9 on
Weakly Supervised Action Localization
on THUMOS’14
Watch Those Words: Video Falsification Detection Using Word-Conditioned Facial Motion
In today's era of digital misinformation, we are increasingly faced with new threats posed by video falsification techniques.
Recognizing Image Style
The style of an image plays a significant role in how it is viewed, but style has received little attention in computer vision research.
Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation
The agent is first presented with previous experiences in the training environment, along with task description in the form of trajectory-level sparse rewards.
Regularization Matters in Policy Optimization - An Empirical Study on Continuous Control
In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks.
CLIP-It! Language-Guided Video Summarization
A generic video summary is an abridged version of a video that conveys the whole story and features the most important scenes.
Deep Mixture of Experts via Shallow Embedding
Larger networks generally have greater representational power at the cost of increased computational complexity.
Fooling Vision and Language Models Despite Localization and Attention Mechanism
Our work sheds new light on understanding adversarial attacks on vision systems which have a language component and shows that attention, bounding box localization, and compositional internal structures are vulnerable to adversarial attacks.
Reinforcement Learning from Imperfect Demonstrations
We propose a unified reinforcement learning algorithm, Normalized Actor-Critic (NAC), that effectively normalizes the Q-function, reducing the Q-values of actions unseen in the demonstration data.
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
Meta-learning allows an intelligent agent to leverage prior learning episodes as a basis for quickly improving performance on a novel task.
Attentive Explanations: Justifying Decisions and Pointing to the Evidence (Extended Abstract)
We also introduce a multimodal methodology for generating visual and textual explanations simultaneously.
Grounding Visual Explanations (Extended Abstract)
Existing models which generate textual explanations enforce task relevance through a discriminative term loss function, but such mechanisms only weakly constrain mentioned object parts to actually be present in the image.
Gradient-free Policy Architecture Search and Adaptation
We develop a method for policy architecture search and adaptation via gradient-free optimization which can learn to perform autonomous driving tasks.
Attentive Explanations: Justifying Decisions and Pointing to the Evidence
In contrast, humans can justify their decisions with natural language and point to the evidence in the visual world which led to their decisions.
Adapting Deep Visuomotor Representations with Weak Pairwise Constraints
We propose a novel, more powerful combination of both distribution and pairwise image alignment, and remove the requirement for expensive annotation by using weakly aligned pairs of images in the source and target domains.
Visual Discovery at Pinterest
Over the past three years Pinterest has experimented with several visual search and recommendation services, including Related Pins (2014), Similar Looks (2015), Flashlight (2016) and Lens (2017).
Loss is its own Reward: Self-Supervision for Reinforcement Learning
Reinforcement learning optimizes policies for expected cumulative reward.
Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer
Using deep reinforcement learning to train general purpose neural network policies alleviates some of the burden of manual representation engineering by using expressive policy classes, but exacerbates the challenge of data collection, since such methods tend to be less efficient than RL with low-dimensional, hand-designed representations.
Utilizing Large Scale Vision and Text Datasets for Image Segmentation from Referring Expressions
Image segmentation from referring expressions is a joint vision and language modeling task, where the input is an image and a textual expression describing a particular region in the image; and the goal is to localize and segment the specific image region based on the given expression.
End-to-End Training of Deep Visuomotor Policies
Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control.
Deep Learning for Tactile Understanding From Visual and Haptic Data
Robots which interact with the physical world will benefit from a fine-grained tactile understanding of objects and surfaces.
Generating Visual Explanations
Clearly explaining a rationale for a classification decision to an end-user can be as important as the decision itself.
Auxiliary Image Regularization for Deep CNNs with Noisy Labels
Precisely-labeled data sets with sufficient amount of samples are very important for training deep convolutional neural networks (CNNs).
Quantification in-the-wild: data-sets and baselines
Quantification is the task of estimating the class-distribution of a data-set.
Constrained Structured Regression with Convolutional Neural Networks
We present a regression framework which models the output distribution of neural networks.
Fine-grained pose prediction, normalization, and recognition
Pose variation and subtle differences in appearance are key challenges to fine-grained classification.
Mapping Images to Sentiment Adjective Noun Pairs with Factorized Neural Nets
We consider the visual sentiment task of mapping an image to an adjective noun pair (ANP) such as "cute baby".
Spatial Semantic Regularisation for Large Scale Object Detection
Our approach proves to be especially useful in large scale settings with thousands of classes, where spatial and semantic interactions are very frequent and only weakly supervised detectors can be built due to a lack of bounding box annotations.
Learning Compact Convolutional Neural Networks with Nested Dropout
Recently, nested dropout was proposed as a method for ordering representation units in autoencoders by their information content, without diminishing reconstruction cost.
Detector Discovery in the Wild: Joint Multiple Instance and Representation Learning
We develop methods for detector learning which exploit joint training over both weak and strong labels and which transfer learned perceptual representations from strongly-labeled auxiliary tasks.
Do Convnets Learn Correspondence?
Convolutional neural nets (convnets) trained from massive labeled datasets have substantially improved the state-of-the-art in image classification and object detection.
Ranked #4 on
Keypoint Detection
on Pascal3D+
Part-based R-CNNs for Fine-grained Category Detection
Semantic part localization can facilitate fine-grained categorization by explicitly isolating subtle appearance differences associated with specific object parts.
Ranked #64 on
Fine-Grained Image Classification
on CUB-200-2011
Weakly-supervised Discovery of Visual Pattern Configurations
The increasing prominence of weakly labeled data nurtures a growing demand for object detection methods that can cope with minimal supervision.
Detection Bank: An Object Detection Based Video Representation for Multimedia Event Recognition
While low-level image features have proven to be effective representations for visual recognition tasks such as object recognition and scene classification, they are inadequate to capture complex semantic meaning required to solve high-level visual tasks such as multimedia event detection and recognition.
On learning to localize objects with minimal supervision
Learning to localize objects with minimal supervision is an important problem in computer vision, since large fully annotated datasets are extremely costly to obtain.
Ranked #35 on
Weakly Supervised Object Detection
on PASCAL VOC 2007
Modeling Radiometric Uncertainty for Vision with Tone-mapped Color Images
To produce images that are suitable for display, tone-mapping is widely used in digital cameras to map linear color measurements into narrow gamuts with limited dynamic range.
One-Shot Adaptation of Supervised Deep Convolutional Models
In other words, are deep CNNs trained on large amounts of labeled data as susceptible to dataset bias as previous methods have been shown to be?
Towards Adapting ImageNet to Reality: Scalable Domain Adaptation with Implicit Low-rank Transformations
Images seen during test time are often not from the same distribution as images used for learning.
