Multiple Choice Question Answering (MCQA)
34 papers with code • 31 benchmarks • 8 datasets
A multiple-choice question (MCQ) is composed of two parts: a stem that identifies the question or problem, and a set of alternatives or possible answers that contain a key that is the best answer to the question, and a number of distractors that are plausible but incorrect answers to the question.
In a k-way MCQA task, a model is provided with a question q, a set of candidate options O = {O1, . . . , Ok}, and a supporting context for each option C = {C1, . . . , Ck}. The model needs to predict the correct answer option that is best supported by the given contexts.
Benchmarks
These leaderboards are used to track progress in Multiple Choice Question Answering (MCQA)
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
MMLU
BIG-bench
MedMCQA
IndicGLUE
M3KE
FrenchMedMCQA
Most implemented papers
Llama 2: Open Foundation and Fine-Tuned Chat Models
In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters.
PaLM: Scaling Language Modeling with Pathways
lucidrains/CoCa-pytorch
•
•
Google Research 2022
To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM.
From Recognition to Cognition: Visual Commonsense Reasoning
rowanz/r2c
•
•
CVPR 2019
While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world.
Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answering
nijianmo/arc-etrr-code
•
•
NAACL 2019
In this paper we propose a retriever-reader model that learns to attend on essential terms during the question answering process.
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.
QuALITY: Question Answering with Long Input Texts, Yes!
nyu-mll/quality
•
•
NAACL 2022
To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5, 000 tokens, much longer than typical current models can process.
MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension
jind11/MMM-MCQA
•
•
Machine Reading Comprehension (MRC) for question answering (QA), which aims to answer a question given the relevant context passages, is an important way to test the ability of intelligence systems to understand human language.
Training Compute-Optimal Large Language Models
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget.
Variational Open-Domain Question Answering
VodLM/vod
•
•
Retrieval-augmented models have proven to be effective in natural language processing tasks, yet there remains a lack of research on their optimization using variational inference.
Counterfactual Variable Control for Robust and Interpretable Question Answering
We then conduct two novel CVC inference methods (on trained models) to capture the effect of comprehensive reasoning as the final prediction.
