Search Results for author:
Found 310 papers, 170 papers with code
Exposing the Limits of Video-Text Models through Contrast Sets
We test the robustness of recent methods on the proposed automatic contrast sets, and compare them to additionally collected human-generated counterparts, to assess their effectiveness.
EgoPet: Egomotion and Interaction Data from an Animal's Perspective
Animals perceive the world to plan their actions and interact with other agents to accomplish complex tasks, demonstrating capabilities that are still unmatched by AI systems.
Finding Visual Task Vectors
In this work, we analyze the activations of MAE-VQGAN, a recent Visual Prompting model, and find task vectors, activations that encode task-specific information.
ALOHa: A New Measure for Hallucination in Captioning Models
Despite recent advances in multimodal pre-training for visual description, state-of-the-art models still produce captions containing errors, such as hallucinating objects not present in a scene.
TraveLER: A Multi-LMM Agent Framework for Video Question-Answering
Specifically, we propose TraveLER, a model that can create a plan to "Traverse" through the video, ask questions about individual frames to "Locate" and store key information, and then "Evaluate" if there is enough information to answer the question.
When Do We Not Need Larger Vision Models?
Our results show that a multi-scale smaller model has comparable learning capacity to a larger model, and pre-training smaller models with S$^2$ can match or even exceed the advantage of larger models.
xT: Nested Tokenization for Larger Context in Large Images
Modern computer vision pipelines handle large images in one of two sub-optimal ways: down-sampling or cropping.
Humanoid Locomotion as Next Token Prediction
We cast real-world humanoid control as a next token prediction problem, akin to predicting the next word in language.
Neural Network Diffusion
The autoencoder extracts latent representations of a subset of the trained network parameters.
InstanceDiffusion: Instance-level Control for Image Generation
Text-to-image diffusion models produce high quality images but do not offer control over individual instances in the image.
Ranked #3 on
Conditional Text-to-Image Synthesis
on COCO-MIG
Rethinking Patch Dependence for Masked Autoencoders
In this work, we re-examine inter-patch dependencies in the decoding mechanism of masked autoencoders (MAE).
From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations
We present a framework for generating full-bodied photorealistic avatars that gesture according to the conversational dynamics of a dyadic interaction.
Unsupervised Universal Image Segmentation
Several unsupervised image segmentation approaches have been proposed which eliminate the need for dense manually-annotated segmentation masks; current models separately handle either semantic segmentation (e. g., STEGO) or class-agnostic instance segmentation (e. g., CutLER), but not both (i. e., panoptic segmentation).
Ranked #1 on
Unsupervised Panoptic Segmentation
on COCO val2017
See, Say, and Segment: Teaching LMMs to Overcome False Premises
Current open-source Large Multimodal Models (LMMs) excel at tasks such as open-vocabulary language grounding and segmentation but can suffer under false premises when queries imply the existence of something that is not actually present in the image.
Describing Differences in Image Sets with Natural Language
To aid in this discovery process, we explore the task of automatically describing the differences between two $\textbf{sets}$ of images, which we term Set Difference Captioning.
Recursive Visual Programming
Visual Programming (VP) has emerged as a powerful framework for Visual Question Answering (VQA).
Readout Guidance: Learning Control from Diffusion Features
We present Readout Guidance, a method for controlling text-to-image diffusion models with learned signals.
IMProv: Inpainting-based Multimodal Prompting for Computer Vision Tasks
Given a textual description of a visual task (e. g. "Left: input image, Right: foreground segmentation"), a few input-output visual examples, or both, the model in-context learns to solve it for a new test input.
Sequential Modeling Enables Scalable Learning for Large Vision Models
We introduce a novel sequential modeling approach which enables learning a Large Vision Model (LVM) without making use of any linguistic data.
Initializing Models with Larger Ones
Weight selection offers a new approach to leverage the power of pretrained models in resource-constrained settings, and we hope it can be a useful tool for training small models in the large-model era.
Object-based (yet Class-agnostic) Video Domain Adaptation
Existing video-based action recognition systems typically require dense annotation and struggle in environments when there is significant distribution shift relative to the training data.
Compositional Chain-of-Thought Prompting for Large Multimodal Models
The combination of strong visual backbones and Large Language Model (LLM) reasoning has led to Large Multimodal Models (LMMs) becoming the current standard for a wide range of vision and language (VL) tasks.
Ranked #30 on
Visual Reasoning
on Winoground
Self-correcting LLM-controlled Diffusion Models
Steered by an LLM controller, SLD turns text-to-image generation into an iterative closed-loop process, ensuring correctness in the resulting image.
From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation
Addressing the challenge of adapting pre-trained vision-language models for generating insightful explanations for visual reasoning tasks with limited annotations, we present ReVisE: a $\textbf{Re}$cursive $\textbf{Vis}$ual $\textbf{E}$xplanation algorithm.
Which One? Leveraging Context Between Objects and Multiple Views for Language Grounding
When connecting objects and their language referents in an embodied 3D environment, it is important to note that: (1) an object can be better characterized by leveraging comparative information between itself and other objects, and (2) an object's appearance can vary with camera position.
A Coefficient Makes SVRG Effective
Stochastic Variance Reduced Gradient (SVRG), introduced by Johnson & Zhang (2013), is a theoretically compelling optimization method.
Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game
Our benchmark results show that many models are vulnerable to the attack strategies in the Tensor Trust dataset.
CLAIR: Evaluating Image Captions with Large Language Models
The evaluation of machine-generated image captions poses an interesting yet persistent challenge.
LLM-grounded Video Diffusion Models
We show that LLMs are able to understand complex spatiotemporal dynamics from text alone and generate layouts that align closely with both the prompts and the object motion patterns typically observed in the real world.
Aligning Large Multimodal Models with Factually Augmented RLHF
Large Multimodal Models (LMM) are built across modalities and the misalignment between two modalities can result in "hallucination", generating textual outputs that are not grounded by the multimodal information in context.
VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation
Existing approaches to unsupervised video instance segmentation typically rely on motion estimates and experience difficulties tracking small or divergent motions.
Can Language Models Learn to Listen?
We present a framework for generating appropriate facial responses from a listener in dyadic social interactions based on the speaker's words.
Stochastic positional embeddings improve masked image modeling
Masked Image Modeling (MIM) is a promising self-supervised learning approach that enables learning from unlabeled images.
Hierarchical Open-vocabulary Universal Image Segmentation
Open-vocabulary image segmentation aims to partition an image into semantic regions according to arbitrary text descriptions.
Ranked #1 on
Image Segmentation
on Pascal Panoptic Parts
Hyperbolic Active Learning for Semantic Segmentation under Domain Shift
In HALO (Hyperbolic Active Learning Optimization), for the first time, we propose the use of epistemic uncertainty as a data acquisition strategy, following the intuition of selecting data points that are the least known.
Robot Learning with Sensorimotor Pre-training
We present a self-supervised sensorimotor pre-training approach for robotics.
Neural Relighting with Subsurface Scattering by Learning the Radiance Transfer Gradient
Reconstructing and relighting objects and scenes under varying lighting conditions is challenging: existing neural rendering methods often cannot handle the complex interactions between materials and light.
Modular Visual Question Answering via Code Generation
We present a framework that formulates visual question answering as modular code generation.
Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation
As such, we explore how natural language descriptions of the domains seen in training data can be used with large vision models trained on diverse pretraining datasets to generate useful variations of the training data.
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost.
TOAST: Transfer Learning via Attention Steering
We introduce Top-Down Attention Steering (TOAST), a novel transfer learning algorithm that keeps the pre-trained backbone frozen, selects task-relevant features in the output, and feeds those features back to the model to steer the attention to the task-specific features.
Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence
We propose Diffusion Hyperfeatures, a framework for consolidating multi-scale and multi-timestep feature maps into per-pixel feature descriptors that can be used for downstream tasks.
LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models
Our method significantly outperforms the base diffusion model and several strong baselines in accurately generating images according to prompts that require various capabilities, doubling the generation accuracy across four tasks on average.
Simple Token-Level Confidence Improves Caption Correctness
The ability to judge whether a caption correctly describes an image is a critical part of vision-language understanding.
Ranked #62 on
Visual Reasoning
on Winoground
Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs
For the visual side, we incorporate a special "SG Component" in the image transformer trained to predict SG information, while for the textual side, we utilize SGs to generate fine-grained captions that highlight different compositional aspects of the scene.
Ranked #24 on
Visual Reasoning
on Winoground
PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editor
We propose PAIR Diffusion, a generic framework that can enable a diffusion model to control the structure and appearance properties of each object in the image.
Top-Down Visual Attention from Analysis by Synthesis
In this paper, we consider top-down attention from a classic Analysis-by-Synthesis (AbS) perspective of vision.
Learning and Verification of Task Structure in Instructional Videos
We introduce a new pre-trained video model, VideoTaskformer, focused on representing the semantics and structure of instructional videos.
Scaling Vision-Language Models with Sparse Mixture of Experts
The field of natural language processing (NLP) has made significant strides in recent years, particularly in the development of large-scale vision-language models (VLMs).
Real-World Humanoid Locomotion with Reinforcement Learning
Humanoid robots that can autonomously operate in diverse environments have the potential to help address labour shortages in factories, assist elderly at homes, and colonize new planets.
Dropout Reduces Underfitting
Additionally, we explore a symmetric technique for regularizing overfitting models - late dropout, where dropout is not used in the early iterations and is only activated later in training.
Guiding Pretraining in Reinforcement Learning with Large Language Models
Reinforcement learning algorithms typically struggle in the absence of a dense, well-shaped reward function.
Back to the Source: Diffusion-Driven Adaptation To Test-Time Corruption
We update the target data instead, and project all test inputs toward the source domain with a generative diffusion model.
Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning
Large, pretrained models are commonly finetuned with imagery that is heavily augmented to mimic different conditions and scales, with the resulting models used for various tasks with imagery from a range of spatial scales.
Re-evaluating the Need for Multimodal Signals in Unsupervised Grammar Induction
Are multimodal inputs necessary for grammar induction?
PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data
In this work, we propose an approach to leverage synthetic scene data for improving video understanding.
Shape-Guided Diffusion with Inside-Outside Attention
We introduce precise object silhouette as a new form of user control in text-to-image diffusion models, which we dub Shape-Guided Diffusion.
G^3: Geolocation via Guidebook Grounding
We propose the task of Geolocation via Guidebook Grounding that uses a dataset of StreetView images from a diverse set of locations and an associated textual guidebook for GeoGuessr, a popular interactive geolocation game.
Multitask Vision-Language Prompt Tuning
Specifically, (i) we demonstrate the effectiveness of learning a single transferable prompt from multiple source tasks to initialize the prompt for each target task; (ii) we show many target tasks can benefit each other from sharing prompt vectors and thus can be jointly learned via multitask prompt tuning.
Using Language to Extend to Unseen Domains
It is expensive to collect training data for every possible domain that a vision model may encounter when deployed.
QDTrack: Quasi-Dense Similarity Learning for Appearance-Only Multiple Object Tracking
In this paper, we present Quasi-Dense Similarity Learning, which densely samples hundreds of object regions on a pair of images for contrastive learning.
Ranked #4 on
Multiple Object Tracking
on BDD100K test
Real-World Robot Learning with Masked Visual Pre-training
Finally, we train a 307M parameter vision transformer on a massive collection of 4. 5M images from the Internet and egocentric videos, and demonstrate clearly the benefits of scaling visual pre-training for robot learning.
Decentralized Vehicle Coordination: The Berkeley DeepDrive Drone Dataset
Decentralized multiagent planning has been an important field of research in robotics.
Prior Knowledge-Guided Attention in Self-Supervised Vision Transformers
SPAN operates by regularizing attention masks from separate transformer heads to follow various priors over semantic regions.
Studying Bias in GANs through the Lens of Race
In this work, we study how the performance and evaluation of generative image models are impacted by the racial composition of their training datasets.
Visual Prompting via Image Inpainting
How does one adapt a pre-trained visual model to novel downstream tasks without task-specific finetuning or any model modification?
Ranked #5 on
Personalized Segmentation
on PerSeg
Refine and Represent: Region-to-Object Representation Learning
Recent works in self-supervised learning have demonstrated strong performance on scene-level dense prediction tasks by pretraining with object-centric or region-based correspondence objectives.
TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliency
In this work, we focus on summarizing instructional videos, an under-explored area of video summarization.
Back to the Source: Diffusion-Driven Test-Time Adaptation
We instead update the target data, by projecting all test inputs toward the source domain with a generative diffusion model.
Disentangled Action Recognition with Knowledge Bases
In this paper, we aim to improve the generalization ability of the compositional action recognition model to novel verbs or novel nouns that are unseen during training time, by leveraging the power of knowledge graphs.
Structured Video Tokens @ Ego4D PNR Temporal Localization Challenge 2022
First, as both images and videos contain structured information, we enrich a transformer model with a set of \emph{object tokens} that can be used across images and videos.
Point- of-no-return (PNR) temporal localization
Temporal Localization
Bringing Image Scene Structure to Video via Frame-Clip Consistency of Object Tokens
We explore a particular instantiation of scene structure, namely a \emph{Hand-Object Graph}, consisting of hands and objects with their locations as nodes, and physical relations of contact/no-contact as edges.
Voxel-informed Language Grounding
Natural language applied to natural 2D images describes a fundamentally 3D world.
Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly
We first enable abstention capabilities for several VQA models, and analyze both their coverage, the portion of questions answered, and risk, the error on that portion.
Visual Attention Emerges from Recurrent Sparse Reconstruction
We present VARS, Visual Attention from Recurrent Sparse reconstruction, a new attention formulation built on two prominent features of the human visual attention mechanism: recurrency and sparsity.
Contrastive Test-Time Adaptation
We propose a novel way to leverage self-supervised contrastive learning to facilitate target feature learning, along with an online pseudo labeling scheme with refinement that significantly denoises pseudo labels.
K-LITE: Learning Transferable Visual Models with External Knowledge
We propose K-LITE, a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts.
Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion
We present a framework for modeling interactional communication in dyadic conversations: given multimodal inputs of a speaker, we autoregressively output multiple possibilities of corresponding listener motion.
ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension
Training a referring expression comprehension (ReC) model for a new visual domain requires collecting referring expressions, and potentially corresponding bounding boxes, for images in the domain.
Teachable Reinforcement Learning via Advice Distillation
Training automated agents to complete complex tasks in interactive environments is challenging: reinforcement learning requires careful hand-engineering of reward functions, imitation learning requires specialized infrastructure and access to a human expert, and learning from intermediate forms of supervision (like binary preferences) is time-consuming and extracts little information from each human intervention.
Masked Visual Pre-training for Motor Control
This paper shows that self-supervised visual pre-training from real-world images is effective for learning motor control tasks from pixels.
On Guiding Visual Attention with Language Specification
To do this, we ground task-relevant words or phrases with attention maps from a pretrained large-scale model.
Explaining Reinforcement Learning Policies through Counterfactual Trajectories
In order for humans to confidently decide where to employ RL agents for real-world tasks, a human developer must validate that the agent will perform well at test-time.
A ConvNet for the 2020s
The "Roaring 20s" of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model.
Ranked #1 on
Classification
on InDL
Watch Those Words: Video Falsification Detection Using Word-Conditioned Facial Motion
In today's era of digital misinformation, we are increasingly faced with new threats posed by video falsification techniques.
Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation
Detecting out-of-context media, such as "mis-captioned" images on Twitter, is a relevant problem, especially in domains of high public significance.
More Control for Free! Image Synthesis with Semantic Diffusion Guidance
We investigate fine-grained, continuous control of this model class, and introduce a novel unified framework for semantic diffusion guidance, which allows either language or image guidance, or both.
Learning to Detect Every Thing in an Open World
LDET leads to significant improvements on many datasets in the open-world instance segmentation task, outperforming baselines on cross-category generalization on COCO, as well as cross-dataset evaluation on UVO and Cityscapes.
Discovering Non-monotonic Autoregressive Orderings with Variational Inference
Permutations then serve as target generation orders for training an insertion-based Transformer language model.
Object-Region Video Transformers
In this work, we present Object-Region Video Transformers (ORViT), an \emph{object-centric} approach that extends video transformer layers with a block that directly incorporates object representations.
Ranked #6 on
Action Recognition
on Diving-48
Differentiable Gradient Sampling for Learning Implicit 3D Scene Reconstructions from a Single Image
Existing approaches for single object reconstruction impose supervision signals based on the loss of the signed distance value from all locations in a scene, posing difficulties when extending to real-world scenarios.
Pyramid Mini-Batching for Optimal Transport
Optimal transport theory provides a useful tool to measure the differences between two distributions.
Zero-Shot Reward Specification via Grounded Natural Language
Reward signals in reinforcement learning can be expensive signals in many tasks and often require access to direct state.
On-target Adaptation
Domain adaptation seeks to mitigate the shift between training on the \emph{source} domain and testing on the \emph{target} domain.
Tune it the Right Way: Unsupervised Validation of Domain Adaptation via Soft Neighborhood Density
Unsupervised domain adaptation (UDA) methods can dramatically improve generalization on unlabeled target domains.
Region-level Active Detector Learning
Active learning for object detection is conventionally achieved by applying techniques developed for classification in a way that aggregates individual detections into image-level selection criteria.
Predicting with Confidence on Unseen Distributions
Our work connects techniques from domain adaptation and predictive uncertainty literature, and allows us to predict model accuracy on challenging unseen distributions without access to labeled data.
CLIP-It! Language-Guided Video Summarization
A generic video summary is an abridged version of a video that conveys the whole story and features the most important scenes.
Early Convolutions Help Transformers See Better
To test whether this atypical design choice causes an issue, we analyze the optimization behavior of ViT models with their original patchify stem versus a simple counterpart where we replace the ViT stem by a small number of stacked stride-two 3*3 convolutions.
DETReg: Unsupervised Pretraining with Region Priors for Object Detection
Recent self-supervised pretraining methods for object detection largely focus on pretraining the backbone of the object detector, neglecting key parts of detection architecture.
Ranked #1 on
Few-Shot Object Detection
on COCO 2017
Towards Learning to Play Piano with Dexterous Hands and Touch
The virtuoso plays the piano with passion, poetry and extraordinary technical ability.
PyTouch: A Machine Learning Library for Touch Processing
With the increased availability of rich tactile sensors, there is an equally proportional need for open-source and integrated software capable of efficiently and effectively processing raw touch measurements into high-level signals that can be used for control and decision-making.
Fighting Gradients with Gradients: Dynamic Defenses against Adversarial Attacks
Adversarial attacks optimize against models to defeat defenses.
Robust Object Detection via Instance-Level Temporal Cycle Confusion
Building reliable object detectors that are robust to domain shifts, such as various changes in context, viewpoint, and object appearances, is critical for real-world applications.
Auto-Tuned Sim-to-Real Transfer
Policies trained in simulation often fail when transferred to the real world due to the `reality gap' where the simulator is unable to accurately capture the dynamics and visual properties of the real world.
NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media
Online misinformation is a prevalent societal issue, with adversaries relying on tools ranging from cheap fakes to sophisticated deep fakes.
Strumming to the Beat: Audio-Conditioned Contrastive Video Textures
We learn representations for video frames and frame-to-frame transition probabilities by fitting a video-specific model trained using contrastive learning.
Anytime Dense Prediction with Confidence Adaptivity
A cascade of "exits" is attached to the model to make multiple predictions.
Prototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation
In this paper, we propose an end-to-end Prototypical Cross-domain Self-Supervised Learning (PCS) framework for Few-shot Unsupervised Domain Adaptation (FUDA).
Ranked #6 on
Semantic Segmentation
on DensePASS
Region Similarity Representation Learning
We present Region Similarity Representation Learning (ReSim), a new approach to self-supervised representation learning for localization-based tasks such as object detection and segmentation.
Self-Supervised Pretraining Improves Self-Supervised Pretraining
Through experimentation on 16 diverse vision datasets, we show HPT converges up to 80x faster, improves accuracy across tasks, and improves the robustness of the self-supervised pretraining process to changes in the image augmentation policy or amount of pretraining data.
Monocular Quasi-Dense 3D Object Tracking
Experiments on our proposed simulation data and real-world benchmarks, including KITTI, nuScenes, and Waymo datasets, show that our tracking framework offers robust object association and tracking on urban-driving scenarios.
Ranked #7 on
Multiple Object Tracking
on KITTI Tracking test
Instance-Aware Predictive Navigation in Multi-Agent Environments
To decide the action at each step, we seek the action sequence that can lead to safe future states based on the prediction module outputs by repeatedly sampling likely action sequences.
Regularization Matters in Policy Optimization - An Empirical Study on Continuous Control
In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks.
Novelty Detection with Rotated Contrastive Predictive Coding
To this end, reconstruction-based learning is often used in which the normality of an observation is expressed in how well it can be reconstructed.
Unconditional Synthesis of Complex Scenes Using a Semantic Bottleneck
Coupling the high-fidelity generation capabilities of label-conditional image synthesis methods with the flexibility of unconditional generative models, we propose a semantic bottleneck GAN model for unconditional synthesis of complex scenes.
Discovering Autoregressive Orderings with Variational Inference
One strategy to recover this information is to decode both the content and location of tokens.
Contrastive Video Textures
By randomly traversing edges with high transition probabilities, we generate diverse temporally smooth videos with novel sequences and transitions.
Minimax Active Learning
Active learning aims to develop label-efficient algorithms by querying the most representative samples to be labeled by a human annotator.
Temporal Action Detection with Multi-level Supervision
We extensively benchmark against the baselines for SSAD and OSAD on our created data splits in THUMOS14 and ActivityNet1. 2, and demonstrate the effectiveness of the proposed UFA and IB methods.
Fighting Copycat Agents in Behavioral Cloning from Observation Histories
Imitation learning trains policies to map from input observations to the actions that an expert would choose.
Modular Networks for Compositional Instruction Following
In our approach, subgoal modules each carry out natural language instructions for a specific subgoal type.
Auxiliary Task Reweighting for Minimum-data Learning
By adjusting the auxiliary task weights to minimize the divergence between the surrogate prior and the true prior of the main task, we obtain a more accurate prior estimation, achieving the goal of minimizing the required amount of training data for the main task and avoiding a costly grid search.
Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation
First, we provide a finite sample bound for both classification and regression problems under Semi-DA.
Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting
The goal of continual learning (CL) is to learn a sequence of tasks without suffering from the phenomenon of catastrophic forgetting.
Reducing Class Collapse in Metric Learning with Easy Positive Sampling
We theoretically prove and empirically show that under reasonable noise assumptions, prevalent embedding losses in metric learning, e. g., triplet loss, tend to project all samples of a class with various modes onto a single point in the embedding space, resulting in a class collapse that usually renders the space ill-sorted for classification or retrieval.
SelfAugment: Automatic Augmentation Policies for Self-Supervised Learning
A common practice in unsupervised representation learning is to use labeled data to evaluate the quality of the learned representations.
ePointDA: An End-to-End Simulation-to-Real Domain Adaptation Framework for LiDAR Point Cloud Segmentation
They require prior knowledge of real-world statistics and ignore the pixel-level dropout noise gap and the spatial feature gap between different domains.
Hierarchical Style-based Networks for Motion Synthesis
Generating diverse and natural human motion is one of the long-standing goals for creating intelligent characters in the animated world.
Identity-Aware Multi-Sentence Video Description
This auxiliary task allows us to propose a two-stage approach to Identity-Aware Video Description.
What Should Not Be Contrastive in Contrastive Learning
Recent self-supervised contrastive methods have been able to produce impressive transferable visual representations by learning to be invariant to different data augmentations.
Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics
We demonstrate the efficacy of our method on hand gesture synthesis from body motion input, and as a strong body prior for single-view image-based 3D hand pose estimation.
Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation
We also demonstrate that reducing the task of room navigation to point navigation improves the performance further.
Video Prediction via Example Guidance
In video prediction tasks, one major challenge is to capture the multi-modal nature of future contents and dynamics.
Compositional Video Synthesis with Action Graphs
Our generative model for this task (AG2Vid) disentangles motion and appearance features, and by incorporating a scheduling mechanism for actions facilitates a timely and coordinated video generation.
Tent: Fully Test-time Adaptation by Entropy Minimization
A model must adapt itself to generalize to new and different data during testing.
Quasi-Dense Similarity Learning for Multiple Object Tracking
Compared to methods with similar detectors, it boosts almost 10 points of MOTA and significantly decreases the number of ID switches on BDD100K and Waymo datasets.
Ranked #1 on
One-Shot Object Detection
on PASCAL VOC 2012 val
Rethinking preventing class-collapsing in metric learning with margin-based losses
We theoretically prove and empirically show that under reasonable noise assumptions, margin-based losses tend to project all samples of a class with various modes onto a single point in the embedding space, resulting in a class collapse that usually renders the space ill-sorted for classification or retrieval.
ParkPredict: Motion and Intent Prediction of Vehicles in Parking Lots
We investigate the problem of predicting driver behavior in parking lots, an environment which is less structured than typical road networks and features complex, interactive maneuvers in a compact space.
Contrastive Examples for Addressing the Tyranny of the Majority
Computer vision algorithms, e. g. for face recognition, favour groups of individuals that are better represented in the training data.
Spatio-Temporal Action Detection with Multi-Object Interaction
Spatio-temporal action detection in videos requires localizing the action both spatially and temporally in the form of an "action tube".
Revisiting Few-shot Activity Detection with Class Similarity Control
In this paper, we present a conceptually simple and general yet novel framework for few-shot temporal activity detection based on proposal regression which detects the start and end time of the activities in untrimmed videos.
Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning
Weakly-supervised action localization requires training a model to localize the action segments in the video given only video level action label.
Ranked #9 on
Weakly Supervised Action Localization
on THUMOS’14
Adversarial Continual Learning
We show that shared features are significantly less prone to forgetting and propose a novel hybrid continual learning framework that learns a disjoint representation for task-invariant and task-specific features required to solve a sequence of tasks.
Frustratingly Simple Few-Shot Object Detection
Such a simple approach outperforms the meta-learning methods by roughly 2~20 points on current benchmarks and sometimes even doubles the accuracy of the prior methods.
Ranked #17 on
Few-Shot Object Detection
on MS-COCO (30-shot)
Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning
This drawback hinders the model from learning subtle variance and fine-grained information.
Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning
The edge between these two lines of works has yet been underexplored, and the effectiveness of meta-learning in few-shot learning remains unclear.
Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
Learning robotic manipulation tasks using reinforcement learning with sparse rewards is currently impractical due to the outrageous data requirements.
Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks
Human action is naturally compositional: humans can easily recognize and perform actions with objects that are different from those used in training demonstrations.
Learning Canonical Representations for Scene Graph to Image Generation
Generating realistic images of complex visual scenes becomes challenging when one wishes to control the structure of the generated images.
Ranked #3 on
Layout-to-Image Generation
on Visual Genome 256x256
Compositional Plan Vectors
We show that CPVs can be learned within a one-shot imitation learning framework without any additional supervision or information about task hierarchy, and enable a demonstration-conditioned policy to generalize to tasks that sequence twice as many skills as the tasks seen during training.
Semantic Bottleneck Scene Generation
For the former, we use an unconditional progressive segmentation generation network that captures the distribution of realistic semantic scene layouts.
Ranked #1 on
Image Generation
on Cityscapes-5K 256x512
Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA
Recent work has explored the TextVQA task that requires reading and understanding text in images to answer a question.
Plan Arithmetic: Compositional Plan Vectors for Multi-Task Control
We show that CPVs can be learned within a one-shot imitation learning framework without any additional supervision or information about task hierarchy, and enable a demonstration-conditioned policy to generalize to tasks that sequence twice as many skills as the tasks seen during training.
Regularization Matters in Policy Optimization
In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks.
Exploring Simple and Transferable Recognition-Aware Image Processing
Interestingly, the processing model's ability to enhance recognition quality can transfer when evaluated on models of different architectures, recognized categories, tasks and training datasets.
Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation
The agent is first presented with previous experiences in the training environment, along with task description in the form of trajectory-level sparse rewards.
Unsupervised Domain Adaptation through Self-Supervision
This paper addresses unsupervised domain adaptation, the setting where labeled training data is available on a source domain, but the goal is to have good performance on a target domain with only unlabeled data.
Blurring Structure and Learning to Optimize and Adapt Receptive Fields
Adapting receptive fields by dynamic Gaussian structure further improves results, equaling the accuracy of free-form deformation while improving efficiency.
Composable Semi-parametric Modelling for Long-range Motion Generation
Learning diverse and natural behaviors is one of the longstanding goal for creating intelligent characters in the animated world.
Scoring-Aggregating-Planning: Learning task-agnostic priors from interactions and sparse rewards for zero-shot generalization
In this paper, we propose Scoring-Aggregating-Planning (SAP), a framework that can learn task-agnostic semantics and dynamics priors from arbitrary quality interactions as well as the corresponding sparse rewards and then plan on unseen tasks in zero-shot condition.
Weakly-Supervised Trajectory Segmentation for Learning Reusable Skills
Instead of direct manual supervision which is tedious and prone to bias, in this work, our goal is to extract reusable skills from a collection of human demonstrations collected directly for several end-tasks.
Dynamic Scale Inference by Entropy Minimization
Given the variety of the visual world there is not one true scale for recognition: objects may appear at drastically different sizes across the visual field.
Task-Aware Feature Generation for Zero-Shot Compositional Learning
In this work, we propose a task-aware feature generation (TFG) framework for compositional learning, which generates features of novel visual concepts by transferring knowledge from previously seen concepts.
Uncertainty-guided Continual Learning with Bayesian Neural Networks
Continual learning aims to learn new tasks without forgetting previously learned ones.
Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation
The actual grounding can connect language to the environment through multiple modalities, e. g. "stop at the door" might ground into visual objects, while "turn right" might rely only on the geometric structure of a route.
Monocular Plan View Networks for Autonomous Driving
Convolutions on monocular dash cam videos capture spatial invariances in the image plane but do not explicitly reason about distances and depth.
Language-Conditioned Graph Networks for Relational Reasoning
E. g., conditioning on the "on" relationship to the plate, the object "mug" gathers messages from the object "plate" to update its representation to "mug on the plate", which can be easily consumed by a simple classifier for answer prediction.
Ranked #3 on
Referring Expression Comprehension
on CLEVR-Ref+
Accurate Visual Localization for Automotive Applications
We propose a hybrid coarse-to-fine approach that leverages visual and GPS location cues.
Meta-Learning to Guide Segmentation
To explore generalization, we analyze guidance as a bridge between different levels of supervision to segment classes as the union of instances.
Blurring the Line Between Structure and Learning to Optimize and Adapt Receptive Fields
Adapting receptive fields by dynamic Gaussian structure further improves results, equaling the accuracy of free-form deformation while improving efficiency.
Semi-supervised Domain Adaptation via Minimax Entropy
Contemporary domain adaptation methods are very effective at aligning feature distributions of source and target domains without any target supervision.
TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning
We show that TAFE-Net is highly effective in generalizing to new tasks or concepts and evaluate the TAFE-Net on a range of benchmarks in zero-shot and few-shot learning.
Ranked #1 on
Few-Shot Image Classification
on aPY - 0-Shot
Variational Adversarial Active Learning
Unlike conventional active learning algorithms, our approach is task agnostic, i. e., it does not depend on the performance of the task for which we are trying to acquire labeled data.
Compositional GAN (Extended Abstract): Learning Image-Conditional Binary Composition
Generative Adversarial Networks (GANs) can produce images of surprising complexity and realism but are generally structured to sample from a single latent source ignoring the explicit spatial interaction between multiple entities that could be present in a scene.
Cross-Linked Variational Autoencoders for Generalized Zero-Shot Learning
While following the same direction, we also take artificial feature generation one step further and propose a model where a shared latent space of image features and class embeddings is learned by aligned variational autoencoders, for the purpose of generating latent features to train a softmax classifier.
Efficient Receptive Field Learning by Dynamic Gaussian Structure
The visual world is vast and varied, but its variations divide into structured and unstructured factors.
Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity
We evaluate the performance of these dynamic and modular agents in simulated environments.
Robust Change Captioning
We present a novel Dual Dynamic Attention Model (DUDA) to perform robust Change Captioning.
Similarity R-C3D for Few-shot Temporal Activity Detection
In this paper, we present a conceptually simple and general yet novel framework for few-shot temporal activity detection which detects the start and end time of the few-shot input activities in an untrimmed video.
Hierarchical Discrete Distribution Decomposition for Match Density Estimation
Explicit representations of the global match distributions of pixel-wise correspondences between pairs of images are desirable for uncertainty estimation and downstream applications.
Ranked #13 on
Optical Flow Estimation
on KITTI 2015 (train)
Adversarial Inference for Multi-Sentence Video Description
Among the main issues are the fluency and coherence of the generated descriptions, and their relevance to the video.
Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders
Many approaches in generalized zero-shot learning rely on cross-modal mapping between the image feature space and the class embedding space.
Ranked #2 on
Generalized Few-Shot Learning
on AwA2
Few-shot Object Detection via Feature Reweighting
The feature learner extracts meta features that are generalizable to detect novel object classes, using training data from base classes with sufficient samples.
Ranked #21 on
Few-Shot Object Detection
on MS-COCO (30-shot)
Spatio-Temporal Action Graph Networks
Events defined by the interaction of objects in a scene are often of critical importance; yet important events may have insufficient labeled examples to train a conventional deep model to generalize to future object appearance.
SPLAT: Semantic Pixel-Level Adaptation Transforms for Detection
Without dense labels, as is the case when only detection labels are available in the source, transformations are learned using CycleGAN alignment.
Disentangling Propagation and Generation for Video Prediction
A dynamic scene has two types of elements: those that move fluidly and can be predicted from previous frames, and those which are disoccluded (exposed) and cannot be extrapolated.
Joint Monocular 3D Vehicle Detection and Tracking
The framework can not only associate detections of vehicles in motion over time, but also estimate their complete 3D bounding box information from a sequence of 2D images captured on a moving platform.
Ranked #12 on
Multiple Object Tracking
on KITTI Tracking test
Deep Object-Centric Policies for Autonomous Driving
While learning visuomotor skills in an end-to-end manner is appealing, deep neural networks are often uninterpretable and fail in surprising ways.
Modular Architecture for StarCraft II with Deep Reinforcement Learning
We present a novel modular architecture for StarCraft II AI.
Discriminator Rejection Sampling
We propose a rejection sampling scheme using the discriminator of a GAN to approximately correct errors in the GAN generator distribution.
Rethinking the Value of Network Pruning
Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm.
Uncertainty-guided Lifelong Learning in Bayesian Networks
Sequentially learning of tasks arriving in a continuous stream is a complex problem and becomes more challenging when the model has a fixed capacity.
Object Hallucination in Image Captioning
Despite continuously improving performance, contemporary image captioning models are prone to "hallucinating" objects that are not actually in a scene.
Localizing Moments in Video with Temporal Language
To benchmark whether our model, and other recent video localization models, can effectively reason about temporal language, we collect the novel TEMPOral reasoning in video and language (TEMPO) dataset.
Large-Scale Study of Curiosity-Driven Learning
However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent.
Ranked #14 on
Atari Games
on Atari 2600 Montezuma's Revenge
Textual Explanations for Self-Driving Vehicles
Finally, we explore a version of our model that generates rationalizations, and compare with introspective explanations on the same video segments.
Grounding Visual Explanations
Our model improves the textual explanation quality of fine-grained classification decisions on the CUB dataset by mentioning phrases that are grounded in the image.
Explainable Neural Computation via Stack Neural Module Networks
In complex inferential tasks like question answering, machine learning models must confront two challenges: the need to implement a compositional reasoning process, and, in many applications, the need for this reasoning process to be interpretable to assist users in both development and prediction.
Ranked #14 on
Referring Expression Comprehension
on Talk2Car
Compositional GAN: Learning Image-Conditional Binary Composition
Generative Adversarial Networks (GANs) can produce images of remarkable complexity and realism but are generally structured to sample from a single latent source ignoring the explicit spatial interaction between multiple entities that could be present in a scene.
