Search Results for author:
Found 187 papers, 95 papers with code
MetaPose: Fast 3D Pose from Multiple Views without 3D Supervision
In the era of deep learning, human pose estimation from multiple cameras with unknown calibration has received little attention to date.
Ranked #1 on
3D Human Pose Estimation
on SkiPose
MaskSketch: Unpaired Structure-guided Masked Image Generation
We show that intermediate self-attention maps of a masked generative transformer encode important structural information of the input image, such as scene layout and object shape, and we propose a novel sampling method based on this observation to enable structure-guided generation.
COCO-FUNIT: Few-Shot Unsupervised Image Translation with a Content Conditioned Style Encoder
Unsupervised image-to-image translation intends to learn a mapping of an image in a given domain to an analogous image in a different domain, without explicit supervision of the mapping.
CyCADA: Cycle-Consistent Adversarial Domain Adaptation
Domain adaptation is critical for success in new, unseen environments.
Adversarial Discriminative Domain Adaptation
Adversarial learning methods are a promising approach to training robust deep networks, and can generate complex samples across diverse domains.
Deep CORAL: Correlation Alignment for Deep Domain Adaptation
CORAL is a "frustratingly easy" unsupervised domain adaptation method that aligns the second-order statistics of the source and target distributions with a linear transformation.
Ranked #4 on
Domain Generalization
on NICO Animal
A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding
Webpages have been a rich, scalable resource for vision-language and language only tasks.
WikiWeb2M: A Page-Level Multimodal Wikipedia Dataset
Webpages have been a rich resource for language and vision-language tasks.
Modeling Relationships in Referential Expressions with Compositional Modular Networks
In this paper we instead present a modular deep architecture capable of analyzing referential expressions into their component parts, identifying entities and relationships mentioned in the input expression and grounding them all in the scene.
Ranked #1 on
Visual Question Answering (VQA)
on Visual7W
High precision grasp pose detection in dense clutter
Our focus in this paper is on improving the second step by using depth sensor scans from large online datasets to train a convolutional neural network.
Robotics
Grasp Pose Detection in Point Clouds
Many grasp detection methods achieve grasp success rates (grasp successes as a fraction of the total number of grasp attempts) between 75% and 95% for novel objects presented in isolation or in light clutter.
Robotics
RISE: Randomized Input Sampling for Explanation of Black-box Models
We compare our approach to state-of-the-art importance extraction methods using both an automatic deletion/insertion metric and a pointing metric based on human-annotated object segments.
Strong-Weak Distribution Alignment for Adaptive Object Detection
This motivates us to propose a novel method for detector adaptation based on strong local alignment and weak global alignment.
Ranked #2 on
Unsupervised Domain Adaptation
on SIM10K to BDD100K
Correlation Alignment for Unsupervised Domain Adaptation
In contrast to subspace manifold methods, it aligns the original feature distributions of the source and target domains, rather than the bases of lower-dimensional subspaces.
Ranked #8 on
Domain Adaptation
on Office-Caltech
Tune it the Right Way: Unsupervised Validation of Domain Adaptation via Soft Neighborhood Density
Unsupervised domain adaptation (UDA) methods can dramatically improve generalization on unlabeled target domains.
Semi-supervised Domain Adaptation via Minimax Entropy
Contemporary domain adaptation methods are very effective at aligning feature distributions of source and target domains without any target supervision.
Learning Multi-Level Hierarchies with Hindsight
Hierarchical agents have the potential to solve sequential decision making tasks with greater sample efficiency than their non-hierarchical counterparts because hierarchical agents can break down tasks into sets of subtasks that only require short sequences of decisions.
Learning to Reason: End-to-End Module Networks for Visual Question Answering
Natural language questions are inherently compositional, and many are most easily answered by reasoning about their decomposition into modular sub-problems.
Ranked #43 on
Visual Question Answering (VQA)
on VQA v2 test-dev
R-C3D: Region Convolutional 3D Network for Temporal Activity Detection
We address the problem of activity detection in continuous, untrimmed video streams.
Ranked #1 on
Action Recognition In Videos
on THUMOS’14
A Broader Study of Cross-Domain Few-Shot Learning
Extensive experiments on the proposed benchmark are performed to evaluate state-of-art meta-learning approaches, transfer learning approaches, and newer methods for cross-domain few-shot learning.
Ranked #3 on
Cross-Domain Few-Shot
on Plantae
cross-domain few-shot learning
Few-Shot Image Classification
+1
VisDA: The Visual Domain Adaptation Challenge
We present the 2017 Visual Domain Adaptation (VisDA) dataset and challenge, a large-scale testbed for unsupervised domain adaptation across visual domains.
Women also Snowboard: Overcoming Bias in Captioning Models
We introduce a new Equalizer model that ensures equal gender probability when gender evidence is occluded in a scene and confident predictions when gender evidence is present.
Domain Agnostic Learning with Disentangled Representations
Unsupervised model transfer has the potential to greatly improve the generalizability of deep models to novel domains.
Ranked #4 on
Multi-target Domain Adaptation
on DomainNet
Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval
Existing methods rely on supervised learning of CIR models using labeled triplets consisting of the query image, text specification, and the target image.
Ranked #1 on
Zero-shot Image Retrieval
on ImageNet-R
Speaker-Follower Models for Vision-and-Language Navigation
We use this speaker model to (1) synthesize new instructions for data augmentation and to (2) implement pragmatic reasoning, which evaluates how well candidate action sequences explain an instruction.
Universal Domain Adaptation through Self Supervision
While some methods address target settings with either partial or open-set categories, they assume that the particular setting is known a priori.
Natural Language Object Retrieval
In this paper, we address the task of natural language object retrieval, to localize a target object within a given image based on a natural language query of the object.
Ranked #12 on
Referring Expression Comprehension
on Talk2Car
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
Multi-task learning is an open and challenging problem in computer vision.
Ranked #104 on
Semantic Segmentation
on NYU Depth v2
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent, or "temporally deep", are effective for tasks involving sequences, visual and otherwise.
Ranked #3 on
Human Interaction Recognition
on BIT
Language-Conditioned Graph Networks for Relational Reasoning
E. g., conditioning on the "on" relationship to the plate, the object "mug" gathers messages from the object "plate" to update its representation to "mug on the plate", which can be easily consumed by a simple classifier for answer prediction.
Ranked #3 on
Referring Expression Comprehension
on CLEVR-Ref+
OVANet: One-vs-All Network for Universal Domain Adaptation
In this paper, we propose a method to learn the threshold using source samples and to adapt it to the target domain.
Ranked #6 on
Universal Domain Adaptation
on DomainNet
Many-to-many Splatting for Efficient Video Frame Interpolation
Motion-based video frame interpolation commonly relies on optical flow to warp pixels from the inputs to the desired interpolation instant.
Ranked #1 on
Video Frame Interpolation
on Xiph-4K (Crop)
Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
Current deep caption models can only describe objects contained in paired image-sentence corpora, despite the fact that they are pre-trained with large object recognition datasets, namely ImageNet.
Explainable Neural Computation via Stack Neural Module Networks
In complex inferential tasks like question answering, machine learning models must confront two challenges: the need to implement a compositional reasoning process, and, in many applications, the need for this reasoning process to be interpretable to assist users in both development and prediction.
Ranked #14 on
Referring Expression Comprehension
on Talk2Car
Deep Domain Confusion: Maximizing for Domain Invariance
Recent reports suggest that a generic supervised deep CNN model trained on a large-scale dataset reduces, but does not remove, dataset bias on a standard benchmark.
Ranked #6 on
Domain Adaptation
on Office-Caltech
Sequence to Sequence -- Video to Text
Our LSTM model is trained on video-sentence pairs and learns to associate a sequence of video frames to a sequence of words in order to generate a description of the event in the video clip.
Translating Videos to Natural Language Using Deep Recurrent Neural Networks
Solving the visual symbol grounding problem has long been a goal of artificial intelligence.
Black-box Explanation of Object Detectors via Saliency Maps
We propose D-RISE, a method for generating visual explanations for the predictions of object detectors.
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition
Specifically, given a video frame, a policy network is used to decide what input resolution should be used for processing by the action recognition model, with the goal of improving both accuracy and efficiency.
Extending the WILDS Benchmark for Unsupervised Adaptation
Unlabeled data can be a powerful point of leverage for mitigating these distribution shifts, as it is frequently much more available than labeled data and can often be obtained from distributions beyond the source distribution as well.
OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers
OpenMatch achieves state-of-the-art performance on three datasets, and even outperforms a fully supervised model in detecting outliers unseen in unlabeled data on CIFAR10.
OpenMatch: Open-Set Semi-supervised Learning with Open-set Consistency Regularization
\ours achieves state-of-the-art performance on three datasets, and even outperforms a fully supervised model in detecting outliers unseen in unlabeled data on CIFAR10.
Semi-Supervised Action Recognition with Temporal Contrastive Learning
We approach this problem by learning a two-pathway temporal contrastive model using unlabeled videos at two different speeds leveraging the fact that changing video speed does not change an action.
Real-time Semantic Segmentation with Fast Attention
The proposed architecture relies on our fast spatial attention, which is a simple yet efficient modification of the popular self-attention mechanism and captures the same rich spatial context at a small fraction of the computational cost, by changing the order of operations.
Ranked #32 on
Semantic Segmentation
on DensePASS
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition
Specifically, given a video segment, a multi-modal policy network is used to decide what modalities should be used for processing by the recognition model, with the goal of improving both accuracy and efficiency.
LSDA: Large Scale Detection Through Adaptation
A major challenge in scaling object detection is the difficulty of obtaining labeled images for large numbers of categories.
Fine-grained Angular Contrastive Learning with Coarse Labels
A very practical example of C2FS is when the target classes are sub-classes of the training classes.
Learning Similarity Conditions Without Explicit Supervision
Many real-world tasks require models to compare images along multiple similarity conditions (e. g. similarity in color, category or shape).
Multilevel Language and Vision Integration for Text-to-Clip Retrieval
To capture the inherent structures present in both text and video, we introduce a multilevel model that integrates vision and language features earlier and more tightly than prior work.
VisDA-2021 Competition Universal Domain Adaptation to Improve Performance on Out-of-Distribution Data
Progress in machine learning is typically measured by training and testing a model on the same distribution of data, i. e., the same domain.
DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations
Solving multi-label recognition (MLR) for images in the low-label regime is a challenging task with many real-world applications.
Object Hallucination in Image Captioning
Despite continuously improving performance, contemporary image captioning models are prone to "hallucinating" objects that are not actually in a scene.
Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments
In recent years, vision-language research has shifted to study tasks which require more complex reasoning, such as interactive question answering, visual common sense reasoning, and question-answer plausibility prediction.
A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility
To study VLN with unknown command feasibility, we introduce a new dataset Mobile app Tasks with Iterative Feedback (MoTIF), where the goal is to complete a natural language command in a mobile app.
ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes
Recyclable waste detection poses a unique computer vision challenge as it requires detection of highly deformable and often translucent objects in cluttered scenes without the kind of context information usually present in human-centric datasets.
Revisiting Image-Language Networks for Open-ended Phrase Detection
Most existing work that grounds natural language phrases in images starts with the assumption that the phrase in question is relevant to the image.
Active Domain Adaptation via Clustering Uncertainty-weighted Embeddings
Generalizing deep neural networks to new target domains is critical to their real-world utility.
Simultaneous Deep Transfer Across Domains and Tasks
Recent reports suggest that a generic supervised deep CNN model trained on a large-scale dataset reduces, but does not remove, dataset bias.
Unsupervised Domain Generalization by Learning a Bridge Across Domains
The ability to generalize learned representations across significantly different visual domains, such as between real photos, clipart, paintings, and sketches, is a fundamental capacity of the human visual system.
Return of Frustratingly Easy Domain Adaptation
Unlike human learning, machine learning often fails to handle changes between training (source) and test (target) input distributions.
Ranked #4 on
Domain Adaptation
on Synth Digits-to-SVHN
Separating Skills and Concepts for Novel Visual Question Answering
Generalization to out-of-distribution data has been a problem for Visual Question Answering (VQA) models.
Joint Event Detection and Description in Continuous Video Streams
In order to explicitly model temporal relationships between visual events and their captions in a single video, we also propose a two-level hierarchical captioning module that keeps track of context.
Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News
In addition to the valuable insights gleaned from our user study experiments, we provide a relatively effective approach based on detecting visual-semantic inconsistencies, which will serve as an effective first line of defense and a useful reference for future work in defending against machine-generated disinformation.
NewsStories: Illustrating articles with visual summaries
Thus, we explore a novel setting where the goal is to learn a self-supervised visual-language representation that is robust to varying text length and the number of images.
FETA: Towards Specializing Foundation Models for Expert Task Applications
However, as we show in this paper, FMs still have poor out-of-the-box performance on expert tasks (e. g. retrieval of car manuals technical illustrations from language queries), data for which is either unseen or belonging to a long-tail part of the data distribution of the huge datasets used for FM pre-training.
Ranked #1 on
Image-to-Text Retrieval
on FETA Car-Manuals
VisDA 2022 Challenge: Domain Adaptation for Industrial Waste Sorting
To test the abilities of computer vision models on this task, we present the VisDA 2022 Challenge on Domain Adaptation for Industrial Waste Sorting.
A Unified Framework for Domain Adaptive Pose Estimation
In this paper, we investigate the problem of domain adaptive 2D pose estimation that transfers knowledge learned on a synthetic source domain to a target domain without supervision.
Lasagna: Layered Score Distillation for Disentangled Object Relighting
Although generative editing methods now enable some forms of image editing, relighting is still beyond today's capabilities; existing methods struggle to keep other aspects of the image -- colors, shapes, and textures -- consistent after the edit.
Top-down Visual Saliency Guided by Captions
Neural image/video captioning models can generate accurate descriptions, but their internal process of mapping regions to words is a black box and therefore difficult to explain.
A Broad Study of Pre-training for Domain Generalization and Adaptation
In this paper, we provide a broad study and in-depth analysis of pre-training for domain adaptation and generalization, namely: network architectures, size, pre-training loss, and datasets.
Dynamic Network Quantization for Efficient Video Inference
Deep convolutional networks have recently achieved great success in video recognition, yet their practical realization remains a challenge due to the large amount of computational resources required to achieve robust recognition.
ERM++: An Improved Baseline for Domain Generalization
We also explore the relationship between DG performance and similarity to pre-training data, and find that similarity to pre-training data distributions is an important driver of performance, but that ERM++ with stronger initializations can deliver strong performance even on dissimilar datasets. Code is released at https://github. com/piotr-teterwak/erm_plusplus.
Bias Mimicking: A Simple Sampling Approach for Bias Mitigation
Using this notion, BM, through a novel training procedure, ensures that the model is exposed to the entire distribution per epoch without repeating samples.
Exploring Consistency in Cross-Domain Transformer for Domain Adaptive Semantic Segmentation
We propose to perform adaptation on attention maps with cross-domain attention layers that share features between the source and the target domains.
Adversarial Self-Defense for Cycle-Consistent GANs
The goal of unsupervised image-to-image translation is to map images from one domain to another without the ground truth correspondence between the two domains.
Unsupervised Video-to-Video Translation
Unsupervised image-to-image translation is a recently proposed task of translating an image to a different style or domain given only unpaired image examples at training time.
Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text
This paper investigates how linguistic knowledge mined from large text corpora can aid the generation of natural language descriptions of videos.
TwoStreamVAN: Improving Motion Modeling in Video Generation
Video generation is an inherently challenging task, as it requires modeling realistic temporal dynamics as well as spatial content.
Surprisingly Simple Semi-Supervised Domain Adaptation with Pretraining and Consistency
With our Pretraining and Consistency (PAC) approach, we achieve state of the art target accuracy on this semi-supervised domain adaptation task, surpassing multiple adversarial domain alignment methods, across multiple datasets.
Semi-supervised Domain Adaptation
Unsupervised Domain Adaptation
Captioning Images with Diverse Objects
We propose minimizing a joint objective which can learn from these diverse data sources and leverage distributional semantic embeddings, enabling the model to generalize and describe novel objects outside of image-caption datasets.
Neural Parameter Allocation Search
We introduce Neural Parameter Allocation Search (NPAS), a novel task where the goal is to train a neural network given an arbitrary, fixed parameter budget.
Moment Matching for Multi-Source Domain Adaptation
Conventional unsupervised domain adaptation (UDA) assumes that training data are sampled from a single domain.
Explaining Reinforcement Learning Policies through Counterfactual Trajectories
In order for humans to confidently decide where to employ RL agents for real-world tasks, a human developer must validate that the agent will perform well at test-time.
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
We propose a novel spatial attention architecture that aligns words with image patches in the first hop, and obtain improved results by adding a second attention hop which considers the whole question to choose visual evidence based on the results of the first hop.
Detector-Free Weakly Supervised Grounding by Separation
In this work, we focus on the task of Detector-Free WSG (DF-WSG) to solve WSG without relying on a pre-trained detector.
Ranked #1 on
Phrase Grounding
on Visual Genome
Mind the Backbone: Minimizing Backbone Distortion for Robust Object Detection
We propose to use Relative Gradient Norm (RGN) as a way to measure the vulnerability of a backbone to feature distortion, and show that high RGN is indeed correlated with lower OOD performance.
Evaluation of Correctness in Unsupervised Many-to-Many Image Translation
Given an input image from a source domain and a guidance image from a target domain, unsupervised many-to-many image-to-image (UMMI2I) translation methods seek to generate a plausible example from the target domain that preserves domain-invariant information of the input source image and inherits the domain-specific information from the guidance image.
Why do These Match? Explaining the Behavior of Image Similarity Models
Explaining a deep learning model can help users understand its behavior and allow researchers to discern its shortcomings.
Log-Likelihood Ratio Minimizing Flows: Towards Robust and Quantifiable Neural Distribution Alignment
We show that, under certain assumptions, this combination yields a deep neural likelihood-based minimization objective that attains a known lower bound upon convergence.
Domain2Vec: Domain Embedding for Unsupervised Domain Adaptation
Extensive experiments are conducted to demonstrate the power of our new datasets in benchmarking state-of-the-art multi-source domain adaptation methods, as well as the advantage of our proposed model.
The SwaNNFlight System: On-the-Fly Sim-to-Real Adaptation via Anchored Learning
And (iii) anchor critics to help stabilize the fine-tuning of agents during sim-to-real transfer, online learning from real data while retaining behavior optimized in simulation.
Learning to Compose SuperWeights for Neural Parameter Allocation Search
To address this, we generate layer weights by learning to compose sets of SuperWeights, which represent a group of trainable parameters.
Hierarchical Reinforcement Learning with Hindsight
Reinforcement Learning (RL) algorithms can suffer from poor sample efficiency when rewards are delayed and sparse.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
Adversarial Dropout Regularization
However, a drawback of this approach is that the critic simply labels the generated features as in-domain or not, without considering the boundaries between classes.
Ranked #2 on
Synthetic-to-Real Translation
on Syn2Real-C
Stable Distribution Alignment Using the Dual of the Adversarial Distance
Our empirical results suggest that using the dual formulation for the restricted family of linear discriminators results in a more stable convergence to a desirable solution when compared with the performance of a primal min-max GAN-like objective and an MMD objective under the same restrictions.
Contextual Multi-Scale Region Convolutional 3D Network for Activity Detection
In this paper, we propose the contextual multi-scale region convolutional 3D network (CMS-RC3D) for activity detection.
Learning a visuomotor controller for real world robotic grasping using simulated depth images
This paper proposes an approach to learning a closed-loop controller for robotic grasping that dynamically guides the gripper to the object.
Adapting Deep Visuomotor Representations with Weak Pairwise Constraints
We propose a novel, more powerful combination of both distribution and pairwise image alignment, and remove the requirement for expensive annotation by using weakly aligned pairs of images in the source and target domains.
Synthetic to Real Adaptation with Generative Correlation Alignment Networks
Experimentally, we show training off-the-shelf classifiers on the newly generated data can significantly boost performance when testing on the real image domains (PASCAL VOC 2007 benchmark and Office dataset), improving upon several existing methods.
Combining Texture and Shape Cues for Object Recognition With Minimal Supervision
We present a novel approach to object classification and detection which requires minimal supervision and which combines visual texture cues and shape information learned from freely available unlabeled web search results.
Fine-to-coarse Knowledge Transfer For Low-Res Image Classification
We address the difficult problem of distinguishing fine-grained object categories in low resolution images.
A Multi-scale Multiple Instance Video Description Network
Most state-of-the-art methods for solving this problem borrow existing deep convolutional neural network (CNN) architectures (AlexNet, GoogLeNet) to extract a visual representation of the input video.
Learning Deep Object Detectors from 3D Models
Crowdsourced 3D CAD models are becoming easily accessible online, and can potentially generate an infinite number of training images for almost any object category. We show that augmenting the training data of contemporary Deep Convolutional Neural Net (DCNN) models with such synthetic data can be effective, especially when real training data is limited or not well matched to the target domain.
Spatial Semantic Regularisation for Large Scale Object Detection
Our approach proves to be especially useful in large scale settings with thousands of classes, where spatial and semantic interactions are very frequent and only weakly supervised detectors can be built due to a lack of bounding box annotations.
What Do Deep CNNs Learn About Objects?
Deep convolutional neural networks learn extremely powerful image representations, yet most of that power is hidden in the millions of deep-layer parameters.
Detector Discovery in the Wild: Joint Multiple Instance and Representation Learning
We develop methods for detector learning which exploit joint training over both weak and strong labels and which transfer learned perceptual representations from strongly-labeled auxiliary tasks.
Modeling Radiometric Uncertainty for Vision with Tone-mapped Color Images
To produce images that are suitable for display, tone-mapping is widely used in digital cameras to map linear color measurements into narrow gamuts with limited dynamic range.
One-Shot Adaptation of Supervised Deep Convolutional Models
In other words, are deep CNNs trained on large amounts of labeled data as susceptible to dataset bias as previous methods have been shown to be?
Towards Adapting ImageNet to Reality: Scalable Domain Adaptation with Implicit Low-rank Transformations
Images seen during test time are often not from the same distribution as images used for learning.
Efficient Learning of Domain-invariant Image Representations
We present an algorithm that learns representations which explicitly compensate for domain mismatch and which can be efficiently realized as linear classifiers.
Syn2Real: A New Benchmark forSynthetic-to-Real Visual Domain Adaptation
In this paper, we present a new large-scale benchmark called Syn2Real, which consists of a synthetic domain rendered from 3D object models and two real-image domains containing the same object categories.
Women also Snowboard: Overcoming Bias in Captioning Models (Extended Abstract)
Most machine learning methods are known to capture and exploit biases of the training data.
Adapting control policies from simulation to reality using a pairwise loss
This paper proposes an approach to domain transfer based on a pairwise loss function that helps transfer control policies learned in simulation onto a real robot.
Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning
We present the Honda Research Institute Driving Dataset (HDD), a challenging dataset to enable research on learning driver behavior in real-life environments.
SPLAT: Semantic Pixel-Level Adaptation Transforms for Detection
Without dense labels, as is the case when only detection labels are available in the source, transformations are learned using CycleGAN alignment.
Guided Zoom: Questioning Network Evidence for Fine-grained Classification
We propose Guided Zoom, an approach that utilizes spatial grounding of a model's decision to make more informed predictions.
Size Matters: Metric Visual Search Constraints from Monocular Metadata
Metric constraints are known to be highly discriminative for many objects, but if training is limited to data captured from a particular 3-D sensor the quantity of training data may be severly limited.
Filtering Abstract Senses From Image Search Results
When faced with the task of learning a visual model based only on the name of an object, a common approach is to find images on the web that are associated with the object name, and then train a visual classifier from the search result.
Unsupervised Learning of Visual Sense Models for Polysemous Words
Polysemy is a problem for methods that exploit image search engines to build object category models.
Similarity R-C3D for Few-shot Temporal Activity Detection
In this paper, we present a conceptually simple and general yet novel framework for few-shot temporal activity detection which detects the start and end time of the few-shot input activities in an untrimmed video.
Semi-supervised Domain Adaptation with Instance Constraints
Most successful object classification and detection methods rely on classifiers trained on large labeled datasets.
Continuous Manifold Based Adaptation for Evolving Visual Domains
The classic domain adaptation paradigm considers the world to be separated into stationary domains with clear boundaries between them.
Confidence-Rated Multiple Instance Boosting for Object Detection
In this paper, we propose a new MIL method for object detection that is capable of handling the noisier automatically obtained annotations.
Sequence to Sequence - Video to Text
Our LSTM model is trained on video-sentence pairs and learns to associate a sequence of video frames to a sequence of words in order to generate a description of the event in the video clip.
Cross-Domain Image Manipulation by Demonstration
In this work we propose a model that can manipulate individual visual attributes of objects in a real scene using examples of how respective attribute manipulations affect the output of a simulation.
Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation
The actual grounding can connect language to the environment through multiple modalities, e. g. "stop at the door" might ground into visual objects, while "turn right" might rely only on the geometric structure of a route.
Two-Stream Region Convolutional 3D Network for Temporal Activity Detection
We address the problem of temporal activity detection in continuous, untrimmed video streams.
Ranked #4 on
Action Recognition
on THUMOS’14
Weakly-supervised Compositional FeatureAggregation for Few-shot Recognition
Learning from a few examples is a challenging task for machine learning.
Language Features Matter: Effective Language Representations for Vision-Language Tasks
Shouldn't language and vision features be treated equally in vision-language (VL) tasks?
MULE: Multimodal Universal Language Embedding
In this paper, we present a modular approach which can easily be incorporated into existing vision-language methods in order to support many languages.
LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval
However, while such approaches tend to focus on identifying relationships between elements of the video and language modalities, there is less emphasis on modeling relational context between video frames given the semantic context of the query.
Class-imbalanced Domain Adaptation: An Empirical Odyssey
Unsupervised domain adaptation is a promising way to generalize deep models to novel domains.
Federated Adversarial Domain Adaptation
In this work, we present a principled approach to the problem of federated domain adaptation, which aims to align the representations learned among the different nodes with the data distribution of the target node.
Explainable Deep Classification Models for Domain Generalization
Explanations are defined as regions of visual evidence upon which a deep classification network makes a decision.
Cross-domain Self-supervised Learning for Domain Adaptation with Few Source Labels
We show that when labeled source examples are limited, existing methods often fail to learn discriminative features applicable for both source and target domains.
Spatio-Temporal Action Detection with Multi-Object Interaction
Spatio-temporal action detection in videos requires localizing the action both spatially and temporally in the form of an "action tube".
Revisiting Few-shot Activity Detection with Class Similarity Control
In this paper, we present a conceptually simple and general yet novel framework for few-shot temporal activity detection based on proposal regression which detects the start and end time of the activities in untrimmed videos.
Learning to Scale Multilingual Representations for Vision-Language Tasks
Current multilingual vision-language models either require a large number of additional parameters for each supported language, or suffer performance degradation as languages are added.
Learning visual servo policies via planner cloning
Learning control policies for visual servoing in novel environments is an important problem.
Self-supervised Visual Attribute Learning for Fashion Compatibility
Our approach consists of three self-supervised tasks designed to capture different concepts that are neglected in prior work that we can select from depending on the needs of our downstream tasks.
Auxiliary Task Reweighting for Minimum-data Learning
By adjusting the auxiliary task weights to minimize the divergence between the surrogate prior and the true prior of the main task, we obtain a more accurate prior estimation, achieving the goal of minimizing the required amount of training data for the main task and avoiding a costly grid search.
Uncertainty-Aware Learning for Zero-Shot Semantic Segmentation
Recently, most ZSS methods focus on learning the visual-semantic correspondence to transfer knowledge from seen classes to unseen classes at the pixel level.
Temporal Action Detection with Multi-level Supervision
We extensively benchmark against the baselines for SSAD and OSAD on our created data splits in THUMOS14 and ActivityNet1. 2, and demonstrate the effectiveness of the proposed UFA and IB methods.
Learning from Lexical Perturbations for Consistent Visual Question Answering
Existing Visual Question Answering (VQA) models are often fragile and sensitive to input variations.
Select, Label, and Mix: Learning Discriminative Invariant Feature Representations for Partial Domain Adaptation
Partial domain adaptation which assumes that the unknown target label space is a subset of the source label space has attracted much attention in computer vision.
Ranked #1 on
Partial Domain Adaptation
on Office-31
Regularizing Action Policies for Smooth Control with Reinforcement Learning
A critical problem with the practical utility of controllers trained with deep Reinforcement Learning (RL) is the notable lack of smoothness in the actions learned by the RL policies.
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition
Temporal modelling is the key for efficient video action recognition.
VA-RED$^2$: Video Adaptive Redundancy Reduction
An inherent property of real-world videos is the high correlation of information across frames which can translate into redundancy in either temporal or spatial feature maps of the models, or both.
Honey, I Shrunk The Actor: A Case Study on Preserving Performance with Smaller Actors in Actor-Critic RL
Actors and critics in actor-critic reinforcement learning algorithms are functionally separate, yet they often use the same network architectures.
Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths
Second, to effectively transfer knowledge, we develop a dynamic block swapping method by randomly replacing the blocks in the lower-precision student network with the corresponding blocks in the higher-precision teacher network.
Learning Cross-modal Contrastive Features for Video Domain Adaptation
Learning transferable and domain adaptive feature representations from videos is important for video-relevant tasks such as action recognition.
CDS: Cross-Domain Self-Supervised Pre-Training
We present a two-stage pre-training approach that improves the generalization ability of standard single-domain pre-training.
MixtureEnsembles: Leveraging Parameter Sharing for Efficient Ensembles
We improve on these methods with MixtureEnsembles, which learns to factorize ensemble members with shared parameters by constructing each layer with a linear combination of templates.
Pyramid Mini-Batching for Optimal Transport
Optimal transport theory provides a useful tool to measure the differences between two distributions.
Multi-Critic Actor Learning: Teaching RL Policies to Act with Style
MultiCriticAL is tested in the context of multi-style learning, a special case of MTRL where agents are trained to behave with different distinct behavior styles, and yields up to 45% performance gains over the single-critic baselines and even successfully learns behavior styles in cases where single-critic approaches may simply fail to learn.
Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
Key to our approach is the ability to learn to spatially localize interactions with self-supervision on a large corpus of videos with accompanying transcribed narrations.
Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing
Unsupervised domain adaptation which aims to adapt models trained on a labeled source domain to a completely unlabeled target domain has attracted much attention in recent years.
Ranked #2 on
Unsupervised Domain Adaptation
on UCF-HMDB
Look at What I’m Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
Key to our approach is the ability to learn to spatially localize interactions with self-supervision on a large corpus of videos with accompanying transcribed narrations.
Exploiting Environmental Variation to Improve Policy Robustness in Reinforcement Learning
We propose a novel method to exploit this observation to develop robust actor policies, by automatically developing a sampling curriculum over environment settings to use in training.
wMAN: WEAKLY-SUPERVISED MOMENT ALIGNMENT NETWORK FOR TEXT-BASED VIDEO SEGMENT RETRIEVAL
Given a video and a sentence, the goal of weakly-supervised video moment retrieval is to locate the video segment which is described by the sentence without having access to temporal annotations during training.
Generalized Domain Adaptation with Covariate and Label Shift CO-ALignment
In this paper, we explore the task of Generalized Domain Adaptation (GDA): How to transfer knowledge across different domains in the presence of both covariate and label shift?
Disentangled Unsupervised Image Translation via Restricted Information Flow
Unsupervised image-to-image translation methods aim to map images from one domain into plausible examples from another domain while preserving structures shared across two domains.
Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Data
It is thus better to tailor synthetic pre-training data to a specific downstream task, for best performance.
Learning to Detect Every Thing in an Open World
LDET leads to significant improvements on many datasets in the open-world instance segmentation task, outperforming baselines on cross-category generalization on COCO, as well as cross-dataset evaluation on UVO and Cityscapes.
The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning
We present Sherlock, an annotated corpus of 103K images for testing machine capacity for abductive reasoning beyond literal image contents.
Temporal Relevance Analysis for Video Action Models
In this paper, we provide a deep analysis of temporal modeling for action recognition, an important but underexplored problem in the literature.
Prefix Conditioning Unifies Language and Label Supervision
In experiments, we show that this simple technique improves the performance in zero-shot image recognition accuracy and robustness to the image-level distribution shift.
Task2Sim: Towards Effective Pre-Training and Transfer From Synthetic Data
It is thus better to tailor synthetic pre-training data to a specific downstream task, for best performance.
Human Evaluation of Text-to-Image Models on a Multi-Task Benchmark
For example, asking a model to generate a varying number of the same object to measure its ability to count or providing a text prompt with several objects that each have a different attribute to identify its ability to match objects and attributes correctly.
Finding Differences Between Transformers and ConvNets Using Counterfactual Simulation Testing
One shortcoming of this is the fact that these deep neural networks cannot be easily evaluated for robustness issues with respect to specific scene variations.
Language-Guided Audio-Visual Source Separation via Trimodal Consistency
We propose a self-supervised approach for learning to perform audio source separation in videos based on natural language queries, using only unlabeled video and audio pairs as training data.
DIME-FM: DIstilling Multimodal and Efficient Foundation Models
We transfer the knowledge from the pre-trained CLIP-ViTL/14 model to a ViT-B/32 model, with only 40M public images and 28. 4M unpaired public sentences.
Hardwiring ViT Patch Selectivity into CNNs using Patch Mixing
We hypothesize that this power to ignore out-of-context information (which we name $\textit{patch selectivity}$), while integrating in-context information in a non-local manner in early layers, allows ViTs to more easily handle occlusion.
Multiscale Video Pretraining for Long-Term Activity Forecasting
To alleviate this issue, we propose Multiscale Video Pretraining (MVP), a novel self-supervised pretraining approach that learns robust representations for forecasting by learning to predict contextualized representations of future video clips over multiple timescales.
DualCoOp++: Fast and Effective Adaptation to Multi-Label Recognition with Limited Annotations
Previous works have focused on learning the alignment between textual and visual spaces to compensate for limited image labels, yet may suffer from reduced accuracy due to the scarcity of high-quality multi-label annotations.
From Fake to Real: Pretraining on Balanced Synthetic Images to Prevent Bias
By training on real and synthetic data separately, FFR avoids the issue of bias toward signals from the pair $(B, G)$.
Label Budget Allocation in Multi-Task Learning
How should the label budget (i. e. the amount of money spent on labeling) be allocated among different tasks to achieve optimal multi-task performance?
Socratis: Are large multimodal models emotionally aware?
We further see that current captioning metrics based on large vision-language models also fail to correlate with human preferences.
DIME-FM : DIstilling Multimodal and Efficient Foundation Models
In this paper, we introduce a new distillation mechanism (DIME-FM) that allows us to transfer the knowledge contained in large VLFMs to smaller, customized foundation models using a relatively small amount of inexpensive, unpaired images and sentences.
Video Frame Interpolation with Many-to-many Splatting and Spatial Selective Refinement
In this work, we first propose a fully differentiable Many-to-Many (M2M) splatting framework to interpolate frames efficiently.
CLAMP: Contrastive LAnguage Model Prompt-tuning
Our results show that LLMs can, indeed, achieve good image classification performance when adapted this way.
SynCDR : Training Cross Domain Retrieval Models with Synthetic Data
In cross-domain retrieval, a model is required to identify images from the same semantic category across two visual domains.
Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks
Furthermore, prior work's Typographic attacks against CLIP randomly sample a misleading class from a predefined set of categories.
Koala: Key frame-conditioned long video-LLM
Long video question answering is a challenging task that involves recognizing short-term activities and reasoning about their fine-grained relationships.
