REPLUG: Retrieval-Augmented Black-Box Language Models
We introduce REPLUG, a retrieval-augmented language modeling framework that treats the language model (LM) as a black box and augments it with a tuneable retrieval model. Unlike prior retrieval-augmented LMs that train language models with special cross attention mechanisms to encode the retrieved text, REPLUG simply prepends retrieved documents to the input for the frozen black-box LM. This simple design can be easily applied to any existing retrieval and language models. Furthermore, we show that the LM can be used to supervise the retrieval model, which can then find documents that help the LM make better predictions. Our experiments demonstrate that REPLUG with the tuned retriever significantly improves the performance of GPT-3 (175B) on language modeling by 6.3%, as well as the performance of Codex on five-shot MMLU by 5.1%.
PDF AbstractCode
Natural Questions
WikiText-2
MMLU
TriviaQA
WikiText-103
The Pile
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Uses Extra Training Data |
Benchmark |
|---|---|---|---|---|---|---|---|
| Multi-task Language Understanding | MMLU | code-davinci-002 175B + REPLUG LSR (5-shot) | Average (%) | 71.8 | # 24 | ||
| Multi-task Language Understanding | MMLU | code-davinci-002 175B (5-shot) | Average (%) | 68.3 | # 36 | ||
| Question Answering | Natural Questions | code-davinci-002 175B + REPLUG (few-shot) | EM | 44.7 | # 12 | ||
| Question Answering | Natural Questions | code-davinci-002 175B + REPLUG LSR (few-shot) | EM | 45.5 | # 9 | ||
| Question Answering | TriviaQA | code-davinci-002 175B + REPLUG (Few-Shot) | EM | 76.8 | # 12 | ||
| Question Answering | TriviaQA | code-davinci-002 175B + REPLUG LSR (Few-Shot) | EM | 77.3 | # 10 |
