Search Results for author:
Found 38 papers, 13 papers with code
Vision-Language Pretraining: Current Trends and the Future
In addition to the larger pretraining datasets, the transformer architecture (Vaswani et al., 2017) and in particular self-attention applied to two modalities are responsible for the impressive performance of the recent pretrained models on downstream tasks (Hendricks et al., 2021).
Neural Redshift: Random Networks are not Random Functions
Prevailing explanations are based on implicit biases of gradient descent (GD) but they cannot account for the capabilities of models from gradient-free methods nor the simplicity bias recently observed in untrained networks.
Zero-shot Retrieval: Augmenting Pre-trained Models with Search Engines
Large pre-trained models can dramatically reduce the amount of task-specific data required to solve a problem, but they often fail to capture domain-specific nuances out of the box.
Mitigating Biases with Diverse Ensembles and Diffusion Models
Spurious correlations in the data, where multiple cues are predictive of the target labels, often lead to a phenomenon known as shortcut learning, where a model relies on erroneous, easy-to-learn cues while ignoring reliable ones.
ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning
When personalized federated learning (FL) meets large foundation models, new challenges arise from various limitations in resources.
Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks
Spurious correlations in the data, where multiple cues are predictive of the target labels, often lead to shortcut learning phenomena, where a model may rely on erroneous, easy-to-learn, cues while ignoring reliable ones.
Learning Diverse Features in Vision Transformers for Improved Generalization
We observe improved out-of-distribution performance on diagnostic benchmarks (MNIST-CIFAR, Waterbirds) as a consequence of the enhanced diversity of features and the pruning of undesirable heads.
Selective Mixup Helps with Distribution Shifts, But Not (Only) because of Mixup
We have found a new equivalence between two successful methods: selective mixup and resampling.
Candidate Set Re-ranking for Composed Image Retrieval with Dual Multi-modal Encoder
An alternative approach is to allow interactions between the query and every possible candidate, i. e., reference-text-candidate triplets, and pick the best from the entire set.
Ranked #2 on
Image Retrieval
on CIRR
A Symbolic Framework for Evaluating Mathematical Reasoning and Generalisation with Transformers
This paper proposes a methodology for generating and perturbing detailed derivations of equations at scale, aided by a symbolic engine, to evaluate the generalisability of Transformers to out-of-distribution mathematical reasoning problems.
Bi-directional Training for Composed Image Retrieval via Text Prompt Learning
Composed image retrieval searches for a target image based on a multi-modal user query comprised of a reference image and modification text describing the desired changes.
Ranked #6 on
Image Retrieval
on Fashion IQ
SelecMix: Debiased Learning by Contradicting-pair Sampling
Experiments on standard benchmarks demonstrate the effectiveness of the method, in particular when label noise complicates the identification of bias-conflicting examples.
Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning
Underspecification refers to the existence of multiple models that are indistinguishable in their in-domain accuracy, even though they differ in other desirable properties such as out-of-distribution (OOD) performance.
EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering
In a simple setting similar to CLEVR, we find that CL representations also improve systematic generalization, and even match the performance of representations from a larger, supervised, ImageNet-pretrained model.
Active Learning by Feature Mixing
We identify unlabelled instances with sufficiently-distinct features by seeking inconsistencies in predictions resulting from interventions on their representations.
Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models
We demonstrate that with a relatively simple architecture, CIRPLANT outperforms existing methods on open-domain images, while matching state-of-the-art accuracy on the existing narrow datasets, such as fashion.
Ranked #10 on
Image Retrieval
on CIRR
Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalization
The method - the first to evade the simplicity bias - highlights the need for a better understanding and control of inductive biases in deep learning.
Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering
We use this new evaluation in a large-scale study of existing approaches for VQA.
Ranked #1 on
Visual Question Answering (VQA)
on VQA-CE
Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge
The technique brings clear benefits to knowledge-demanding question answering tasks (OK-VQA, FVQA) by capturing semantic and relational knowledge absent from existing models.
Unshuffling Data for Improved Generalization in Visual Question Answering
subsets treated as multiple training environments can guide the learning of models with better out-of-distribution generalization.
Counterfactual Vision-and-Language Navigation: Unravelling the Unseen
The task of vision-and-language navigation (VLN) requires an agent to follow text instructions to find its way through simulated household environments.
On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law
Out-of-distribution (OOD) testing is increasingly popular for evaluating a machine learning system's ability to generalize beyond the biases of a training set.
Visual Question Answering with Prior Class Semantics
We present a novel mechanism to embed prior knowledge in a model for visual question answering.
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision
One of the primary challenges limiting the applicability of deep learning is its susceptibility to learning spurious correlations rather than the underlying mechanisms of the task of interest.
Unshuffling Data for Improved Generalization
subsets treated as multiple training environments can guide the learning of models with better out-of-distribution generalization.
On Incorporating Semantic Prior Knowledge in Deep Learning Through Embedding-Space Constraints
We also show that incorporating this type of prior knowledge with our method brings consistent improvements, independently from the amount of supervised data used.
On Incorporating Semantic Prior Knowlegde in Deep Learning Through Embedding-Space Constraints
We also show that incorporating this type of prior knowledge with our method brings consistent improvements, independently from the amount of supervised data used.
V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matrices
One of the primary challenges faced by deep learning is the degree to which current methods exploit superficial statistics and dataset bias, rather than learning to generalise over the specific representations they have experienced.
Actively Seeking and Learning from Live Data
One of the key limitations of traditional machine learning methods is their requirement for training data that exemplifies all the information to be learned.
Visual Question Answering as a Meta Learning Task
At test time, the method is provided with a support set of example questions/answers, over which it reasons to resolve the given question.
Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments
This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering.
Ranked #10 on
Visual Navigation
on R2R
Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge
This paper presents a state-of-the-art model for visual question answering (VQA), which won the first place in the 2017 VQA Challenge.
Ranked #30 on
Visual Question Answering (VQA)
on VQA v2 test-std
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning.
Ranked #29 on
Visual Question Answering (VQA)
on VQA v2 test-std
Zero-Shot Visual Question Answering
Answering general questions about images requires methods capable of Zero-Shot VQA, that is, methods able to answer questions beyond the scope of the training questions.
Graph-Structured Representations for Visual Question Answering
This paper proposes to improve visual question answering (VQA) with structured representations of both scene contents and questions.
Visual Question Answering: A Survey of Methods and Datasets
Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities.
Learning to Extract Motion from Videos in Convolutional Neural Networks
Our contributions on network design and rotation invariance offer insights nonspecific to motion estimation.
Learning Similarity Metrics for Dynamic Scene Segmentation
This paper addresses the segmentation of videos with arbitrary motion, including dynamic textures, using novel motion features and a supervised learning approach.
