Adaptable and Interpretable Neural MemoryOver Symbolic Knowledge
Past research has demonstrated that large neural language models (LMs) encode surprising amounts of factual information: however, augmenting or modifying this information requires modifying a corpus and retraining, which is computationally expensive. To address this problem, we develop a neural LM that includes an interpretable neuro-symbolic KB in the form of a {``}fact memory{''}. Each element of the fact memory is formed from a triple of vectors, where each vector corresponds to a KB entity or relation. Our LM improves performance on knowledge-intensive question-answering tasks, sometimes dramatically, including a 27 point increase in one setting of WebQuestionsSP over a state-of-the-art open-book model, despite using 5{\%} of the parameters. Most interestingly, we demonstrate that the model can be modified, without \textit{any} re-training, by updating the fact memory.
PDF Abstract
WebQuestions
