Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation, but the quality bar for medical and clinical applications is high. Today, attempts to assess models' clinical knowledge typically rely on automated evaluations on limited benchmarks. There is no standard to evaluate model predictions and reasoning across a breadth of tasks. To address this, we present MultiMedQA, a benchmark combining six existing open question answering datasets spanning professional medical exams, research, and consumer queries; and HealthSearchQA, a new free-response dataset of medical questions searched online. We propose a framework for human evaluation of model answers along multiple axes including factuality, precision, possible harm, and bias. In addition, we evaluate PaLM (a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM, on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA, MedMCQA, PubMedQA, MMLU clinical topics), including 67.6% accuracy on MedQA (US Medical License Exam questions), surpassing prior state-of-the-art by over 17%. However, human evaluation reveals key gaps in Flan-PaLM responses. To resolve this we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, recall of knowledge, and medical reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal important limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLM models for clinical applications.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Multiple Choice Question Answering (MCQA) MedMCQA Flan-PaLM (540B, SC) Dev Set (Acc-%) 0.576 # 4
Multiple Choice Question Answering (MCQA) MedMCQA Flan-PaLM (540B, CoT) Dev Set (Acc-%) 0.536 # 7
Multiple Choice Question Answering (MCQA) MedMCQA Flan-PaLM (540B, Few-shot) Dev Set (Acc-%) 0.565 # 5
Multiple Choice Question Answering (MCQA) MedMCQA PaLM (540B, Few-shot) Dev Set (Acc-%) 0.545 # 6
Multiple Choice Question Answering (MCQA) MedMCQA Flan-PaLM (62B, Few-shot) Dev Set (Acc-%) 0.462 # 9
Multiple Choice Question Answering (MCQA) MedMCQA PaLM (62B, Few-shot) Dev Set (Acc-%) 0.434 # 10
Multiple Choice Question Answering (MCQA) MedMCQA Flan-PaLM (8B, Few-shot) Dev Set (Acc-%) 0.345 # 15
Multiple Choice Question Answering (MCQA) MedMCQA PaLM (8B, Few-shot) Dev Set (Acc-%) 0.267 # 18
Question Answering MedQA GPT-Neo (2.7 B) Accuracy 33.3 # 23
Question Answering MedQA PubMedGPT (2.7 B) Accuracy 50.3 # 15
Question Answering MedQA BioLinkBERT (340 M) Accuracy 45.1 # 17
Question Answering MedQA Flan-PaLM (540 B) Accuracy 67.6 # 9
Question Answering PubMedQA Flan-PaLM (62B, Few-shot) Accuracy 77.2 # 10
Question Answering PubMedQA PaLM (62B, Few-shot) Accuracy 57.8 # 26
Question Answering PubMedQA PaLM (8B, Few-shot) Accuracy 34 # 29
Question Answering PubMedQA Flan-PaLM (8B, Few-shot) Accuracy 67.6 # 24
Question Answering PubMedQA Flan-PaLM (540B, SC) Accuracy 75.2 # 15
Question Answering PubMedQA Flan-PaLM (540B, Few-shot) Accuracy 79 # 5
Question Answering PubMedQA PaLM (540B, Few-shot) Accuracy 55 # 28

Methods