The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarks

1 code implementation • 26 Oct 2023 • Kaiser Sun, Adina Williams, Dieuwke Hupkes

NLP models have progressed drastically in recent years, according to numerous datasets proposed to evaluate performance.

Tokenization Consistency Matters for Generative Models on Extractive NLP Tasks

1 code implementation • 19 Dec 2022 • Kaiser Sun, Peng Qi, Yuhao Zhang, Lan Liu, William Yang Wang, Zhiheng Huang

We show that, with consistent tokenization, the model performs better in both in-domain and out-of-domain datasets, with a notable average of +1. 7 F2 gain when a BART model is trained on SQuAD and evaluated on 8 QA datasets.

Extractive Question-Answering Hallucination +1

State-of-the-art generalisation research in NLP: A taxonomy and review

no code implementations • 6 Oct 2022 • Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, Zhijing Jin

We present a taxonomy for characterising and understanding generalisation research in NLP.

Effective Attention Sheds Light On Interpretability

1 code implementation • Findings (ACL) 2021 • Kaiser Sun, Ana Marasović

An attention matrix of a transformer self-attention sublayer can provably be decomposed into two components and only one of them (effective attention) contributes to the model output.

Language Modelling