Search Results for author:
Found 51 papers, 32 papers with code
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding
Visually-situated language is ubiquitous -- sources range from textbooks with diagrams to web pages with images and tables, to mobile apps with buttons and forms.
Ranked #17 on
Visual Question Answering (VQA)
on InfographicVQA
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering
Visual language data such as plots, charts, and infographics are ubiquitous in the human world.
Ranked #1 on
Visual Question Answering
on PlotQA-D2
DePlot: One-shot visual language reasoning by plot-to-table translation
Compared with a SOTA model finetuned on more than >28k data points, DePlot+LLM with just one-shot prompting achieves a 24. 0% improvement over finetuned SOTA on human-written queries from the task of chart QA.
Chart Question Answering
Factual Inconsistency Detection in Chart Captioning
+3
TweetNLP: Cutting-Edge Natural Language Processing for Social Media
In this paper we present TweetNLP, an integrated platform for Natural Language Processing (NLP) in social media.
Language Models Can See: Plugging Visual Controls in Text Generation
MAGIC is a flexible framework and is theoretically compatible with any text generation tasks that incorporate image grounding.
Self-Alignment Pretraining for Biomedical Entity Representations
Despite the widespread success of self-supervised learning via masked language models (MLM), accurately capturing fine-grained semantic relationships in the biomedical domain remains a challenge.
Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linking
To this end, we propose and evaluate a series of cross-lingual transfer methods for the XL-BEL task, and demonstrate that general-domain bitext helps propagate the available English knowledge to languages with little to no in-domain data.
Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations
Predominantly, two formulations are used for sentence-pair tasks: bi-encoders and cross-encoders.
Ranked #7 on
Semantic Textual Similarity
on STS16
Visually Grounded Reasoning across Languages and Cultures
The design of widespread vision-and-language datasets and pre-trained encoders directly adopts, or draws inspiration from, the concepts and images of ImageNet.
Ranked #1 on
Zero-Shot Cross-Lingual Transfer
on MaRVL
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
Our benchmark enables the evaluation of multilingual multimodal models for transfer learning, not only in a zero-shot setting, but also in newly defined few-shot learning setups.
TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning
Masked language models (MLMs) such as BERT and RoBERTa have revolutionized the field of Natural Language Understanding in the past few years.
Visual Spatial Reasoning
Spatial relations are a basic part of human cognition.
Ranked #1 on
Visual Reasoning
on VSR
Visual Pivoting for (Unsupervised) Entity Alignment
This work studies the use of visual semantic representations to align entities in heterogeneous knowledge graphs (KGs).
Ranked #3 on
Multi-modal Entity Alignment
on UMVM-oea-d-w-v1
(using extra training data)
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders
In this work, we demonstrate that it is possible to turn MLMs into effective universal lexical and sentence encoders even without any additional data and without any supervision.
Ranked #15 on
Semantic Textual Similarity
on STS16
Contrastive Learning
Cross-Lingual Semantic Textual Similarity
+5
COMETA: A Corpus for Medical Entity Linking in the Social Media
Whilst there has been growing progress in Entity Linking (EL) for general language, existing datasets fail to address the complex nature of health terminology in layman's language.
Fine-Grained Controllable Text Generation Using Non-Residual Prompting
We propose a resource-efficient method for converting a pre-trained CLM into this architecture, and demonstrate its potential on various experiments, including the novel task of contextualized word inclusion.
Contrastive Out-of-Distribution Detection for Pretrained Transformers
Pretrained Transformers achieve remarkable performance when training and test data are from the same distribution.
HAL: Improved Text-Image Matching by Mitigating Visual Semantic Hubs
The hubness problem widely exists in high-dimensional embedding space and is a fundamental source of error for cross-modal matching tasks.
Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models
Our experiments validate the effectiveness of SparklesChat in understanding and reasoning across multiple images and dialogue turns.
Revisiting Parameter-Efficient Tuning: Are We Really There Yet?
Parameter-Efficient Tuning (PETuning) methods have been deemed by many as the new paradigm for using pretrained language models (PLMs).
Improving Word Translation via Two-Stage Contrastive Learning
At Stage C1, we propose to refine standard cross-lingual linear maps between static word embeddings (WEs) via a contrastive learning objective; we also show how to integrate it into the self-learning procedure for even more refined cross-lingual maps.
DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation
To train and validate the effectiveness of our DNN-based module, we propose a large-scale retinal disease image dataset.
Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT
Infusing factual knowledge into pre-trained models is fundamental for many knowledge-intensive tasks.
Auto-Classification of Retinal Diseases in the Limit of Sparse Data Using a Two-Streams Machine Learning Model
Automatic clinical diagnosis of retinal diseases has emerged as a promising approach to facilitate discovery in areas with limited access to specialists.
A Novel Hybrid Machine Learning Model for Auto-Classification of Retinal Diseases
Automatic clinical diagnosis of retinal diseases has emerged as a promising approach to facilitate discovery in areas with limited access to specialists.
Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Models
To catalyse the research in this direction, we release a well-curated biomedical knowledge probing benchmark, MedLAMA, which is constructed based on the Unified Medical Language System (UMLS) Metathesaurus.
Improving Bilingual Lexicon Induction with Cross-Encoder Reranking
This crucial step is done via 1) creating a word similarity dataset, comprising positive word pairs (i. e., true translations) and hard negative pairs induced from the original CLWE space, and then 2) fine-tuning an mPLM (e. g., mBERT or XLM-R) in a cross-encoder manner to predict the similarity scores.
Bilingual Lexicon Induction
Cross-Lingual Word Embeddings
+7
MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models
Recent work indicated that pretrained language models (PLMs) such as BERT and RoBERTa can be transformed into effective sentence and word encoders even via simple self-supervised techniques.
Prix-LM: Pretraining for Multilingual Knowledge Base Construction
To achieve this, it is crucial to represent multilingual knowledge in a shared/unified space.
Reranking Overgenerated Responses for End-to-End Task-Oriented Dialogue Systems
At training, the high-scoring partition comprises all generated responses whose similarity to the gold response is higher than the similarity of the greedy response to the gold response.
POSQA: Probe the World Models of LLMs with Size Comparisons
Embodied language comprehension emphasizes that language understanding is not solely a matter of mental processing in the brain but also involves interactions with the physical and social environment.
3DCNN-DQN-RNN: A Deep Reinforcement Learning Framework for Semantic Parsing of Large-scale 3D Point Clouds
Our method provides an automatic process that maps the raw data to the classification results.
3D Depthwise Convolution: Reducing Model Parameters in 3D Vision Tasks
Standard 3D convolution operations require much larger amounts of memory and computation cost than 2D convolution operations.
A Strong and Robust Baseline for Text-Image Matching
We review the current schemes of text-image matching models and propose improvements for both training and inference.
Upgrading the Newsroom: An Automated Image Selection System for News Articles
The system fuses multiple textual sources extracted from news articles and accepts multilingual inputs.
Integrating Transformers and Knowledge Graphs for Twitter Stance Detection
Stance detection (SD) entails classifying the sentiment of a text towards a given target, and is a relevant sub-task for opinion mining and social media analysis.
Sharpness-Aware Minimization with Dynamic Reweighting
Deep neural networks are often overparameterized and may not easily achieve model generalization.
Modality-Balanced Embedding for Video Retrieval
This modality imbalanceresults from a) modality gap: the relevance between a query and a video text is much easier to learn as the query is also a piece of text, with the same modality as the video text; b) data bias: most training samples can be solved solely by text matching.
Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders
In this work, we probe SEs for the amount of cross-lingual lexical knowledge stored in their parameters, and compare them against the original multilingual LMs.
On Reality and the Limits of Language Data: Aligning LLMs with Human Norms
Recent advancements in Large Language Models (LLMs) harness linguistic associations in vast natural language data for practical applications.
Do ever larger octopi still amplify reporting biases? Evidence from judgments of typical colour
Language models (LMs) trained on raw texts have no direct access to the physical world.
WinoDict: Probing language models for in-context word acquisition
We introduce a new in-context learning paradigm to measure Large Language Models' (LLMs) ability to learn novel words during inference.
How to tackle an emerging topic? Combining strong and weak labels for Covid news NER
Being able to train Named Entity Recognition (NER) models for emerging topics is crucial for many real-world applications especially in the medical domain where new topics are continuously evolving out of the scope of existing models and datasets.
On-the-fly Denoising for Data Augmentation in Natural Language Understanding
However, data augmentation may introduce noisy data that impairs training.
SAT: Size-Aware Transformer for 3D Point Cloud Semantic Segmentation
In this paper, we propose the Size-Aware Transformer (SAT) that can tailor effective receptive fields for objects of different sizes.
Ranked #10 on
Semantic Segmentation
on S3DIS Area5
Compositional Zero-Shot Domain Transfer with Text-to-Text Models
We evaluate DoT5 on the biomedical domain and the resource-lean subdomain of radiology, focusing on NLI, text summarisation and embedding learning.
Gemini: A Family of Highly Capable Multimodal Models
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding.
Ranked #1 on
Multi-task Language Understanding
on MMLU
(using extra training data)
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
In this report, we present the latest model of the Gemini family, Gemini 1. 5 Pro, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio.
Ranked #18 on
Code Generation
on HumanEval
Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
We propose a technique to transfer capabilities from LLMs to VLMs.
Ranked #1 on
Chart Question Answering
on ChartQA
(using extra training data)
Chart Question Answering
Optical Character Recognition (OCR)
LUQ: Long-text Uncertainty Quantification for LLMs
With \textsc{Luq} as the tool for UQ, we investigate behavior patterns of several popular LLMs' response confidence spectrum and how that interplays with the response' factuality.
Best Practices and Lessons Learned on Synthetic Data for Language Models
The success of AI models relies on the availability of large, diverse, and high-quality datasets, which can be challenging to obtain due to data scarcity, privacy concerns, and high costs.
