Search Results for author:
Found 26 papers, 21 papers with code
PaliGemma 2: A Family of Versatile VLMs for Transfer
PaliGemma 2 is an upgrade of the PaliGemma open Vision-Language Model (VLM) based on the Gemma 2 family of language models.
PaliGemma: A versatile 3B VLM for transfer
PaliGemma is an open Vision-Language Model (VLM) that is based on the SigLIP-So400m vision encoder and the Gemma-2B language model.
No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models
Third, we introduce the task of geo-localization as a novel evaluation metric to assess cultural diversity in VLMs.
Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratings
Human-rated prompt sets are generally small and the reliability of the ratings -- and thereby the prompt set used to compare models -- is not evaluated.
MuLan: A Study of Fact Mutability in Language Models
Facts are subject to contingencies and can be true or false in different circumstances.
Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models
We quantify bias amplification in pretraining and after fine-tuning on three families of vision-and-language models.
On the Interplay between Fairness and Explainability
In order to build reliable and trustworthy NLP applications, models need to be both fair across different demographics and explainable.
StoryBench: A Multifaceted Benchmark for Continuous Story Visualization
To fill this gap, we collect comprehensive human annotations on three existing datasets, and introduce StoryBench: a new, challenging multi-task benchmark to reliably evaluate forthcoming text-to-video models.
Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining
Recent work in vision-and-language pretraining has investigated supervised signals from object detection data to learn better, fine-grained multimodal representations.
Measuring Progress in Fine-grained Vision-and-Language Understanding
While pretraining on large-scale image-text data from the Web has facilitated rapid progress on many vision-and-language (V&L) tasks, recent work has demonstrated that pretrained models lack "fine-grained" understanding, such as the ability to recognise relationships, verbs, and numbers in images.
Ranked #14 on
Visual Reasoning
on Winoground
A Study of Autoregressive Decoders for Multi-Tasking in Computer Vision
A key finding is that a small decoder learned on top of a frozen pretrained encoder works surprisingly well.
Multilingual Multimodal Learning with Machine Translated Text
We call this framework TD-MML: Translated Data for Multilingual Multimodal Learning, and it can be applied to any multimodal dataset and model.
Zero-Shot Cross-Lingual Image-to-Text Retrieval
Zero-Shot Cross-Lingual Text-to-Image Retrieval
+3
Language Modelling with Pixels
We pretrain the 86M parameter PIXEL model on the same English data as BERT and evaluate on syntactic and semantic tasks in typologically diverse languages, including various non-Latin scripts.
Ranked #1 on
Named Entity Recognition
on MasakhaNER
Ancestor-to-Creole Transfer is Not a Walk in the Park
We aim to learn language models for Creole languages for which large volumes of data are not readily available, and therefore explore the potential transfer from ancestor languages (the 'Ancestry Transfer Hypothesis').
Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization
Vision-and-language (V&L) models pretrained on large-scale multimodal data have demonstrated strong performance on various tasks such as image captioning and visual question answering (VQA).
Mostra: A Flexible Balancing Framework to Trade-off User, Artist and Platform Objectives for Music Sequencing
We consider the task of sequencing tracks on music streaming platforms where the goal is to maximise not only user satisfaction, but also artist- and platform-centric objectives, needed to ensure long-term health and sustainability of the platform.
Challenges and Strategies in Cross-Cultural NLP
Various efforts in the Natural Language Processing (NLP) community have been made to accommodate linguistic diversity and serve speakers of many different languages.
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
Our benchmark enables the evaluation of multilingual multimodal models for transfer learning, not only in a zero-shot setting, but also in newly defined few-shot learning setups.
Visually Grounded Reasoning across Languages and Cultures
The design of widespread vision-and-language datasets and pre-trained encoders directly adopts, or draws inspiration from, the concepts and images of ImageNet.
Ranked #1 on
Zero-Shot Cross-Lingual Transfer
on MaRVL
On Language Models for Creoles
Creole languages such as Nigerian Pidgin English and Haitian Creole are under-resourced and largely ignored in the NLP literature.
Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers
Models that have learned to construct cross-modal representations using both modalities are expected to perform worse when inputs are missing from a modality.
The Role of Syntactic Planning in Compositional Image Captioning
Image captioning has focused on generalizing to images drawn from the same distribution as the training set, and not to the more challenging problem of generalizing to different distributions of images.
Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs
Large-scale pretraining and task-specific fine-tuning is now the standard methodology for many tasks in computer vision and natural language processing.
It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information
The performance of neural machine translation systems is commonly evaluated in terms of BLEU.
Enhancing Machine Translation with Dependency-Aware Self-Attention
Most neural machine translation models only rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism.
Matrix Completion in the Unit Hypercube via Structured Matrix Factorization
We tackle this challenge by using a two-fold approach: first, we transform this task into a constrained matrix completion problem with entries bounded in the unit interval [0, 1]; second, we propose two novel matrix factorization models that leverage our knowledge of the VFX environment.
