Variational Open-Domain Question Answering

23 Sep 2022  ·  Valentin Liévin, Andreas Geert Motzfeldt, Ida Riis Jensen, Ole Winther ·

Retrieval-augmented models have proven to be effective in natural language processing tasks, yet there remains a lack of research on their optimization using variational inference. We introduce the Variational Open-Domain (VOD) framework for end-to-end training and evaluation of retrieval-augmented models, focusing on open-domain question answering and language modelling. The VOD objective, a self-normalized estimate of the R\'enyi variational bound, approximates the task marginal likelihood and is evaluated under samples drawn from an auxiliary sampling distribution (cached retriever and/or approximate posterior). It remains tractable, even for retriever distributions defined on large corpora. We demonstrate VOD's versatility by training reader-retriever BERT-sized models on multiple-choice medical exam questions. On the MedMCQA dataset, we outperform the domain-tuned Med-PaLM by +5.3% despite using 2.500$\times$ fewer parameters. Our retrieval-augmented BioLinkBERT model scored 62.9% on the MedMCQA and 55.0% on the MedQA-USMLE. Last, we show the effectiveness of our learned retriever component in the context of medical semantic search.

PDF Abstract

Datasets


Introduced in the Paper:

FZ queries

Used in the Paper:

MMLU MedQA MedMCQA
Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Multiple Choice Question Answering (MCQA) MedMCQA VOD (BioLinkBERT) Dev Set (Acc-%) 0.583 # 3
Test Set (Acc-%) 0.629 # 4
Question Answering MedQA VOD (BioLinkBERT) Accuracy 55.0 # 12

Methods


No methods listed for this paper. Add relevant methods here