Search Results for author: Jonathan Berant

Found 116 papers, 76 papers with code

What’s in Your Head? Emergent Behaviour in Multi-Task Transformer Models

no code implementations EMNLP 2021 Mor Geva, Uri Katz, Aviv Ben-Arie, Jonathan Berant

In this work, we examine the behaviour of non-target heads, that is, the output of heads when given input that belongs to a different task than the one they were trained for.

Language Modelling Question Answering

Memory-efficient Transformers via Top-k Attention

1 code implementation EMNLP (sustainlp) 2021 Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, Jonathan Berant

Following the success of dot-product attention in Transformers, numerous approximations have been recently proposed to address its quadratic complexity with respect to the input length.

Large Language Models for Psycholinguistic Plausibility Pretesting

no code implementations8 Feb 2024 Samuel Joseph Amouyal, Aya Meltzer-Asscher, Jonathan Berant

In this work, we investigate whether Language Models (LMs) can be used to generate these plausibility judgements.

Transforming and Combining Rewards for Aligning Large Language Models

no code implementations1 Feb 2024 ZiHao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D'Amour, Sanmi Koyejo, Victor Veitch

A common approach for aligning language models to human preferences is to first learn a reward model from preference data, and then use this reward model to update the language model.

Language Modelling

Theoretical guarantees on the best-of-n alignment policy

no code implementations3 Jan 2024 Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D'Amour, Jacob Eisenstein, Chirag Nagpal, Ananda Theertha Suresh

A commonly used analytical expression in the literature claims that the KL divergence between the best-of-$n$ policy and the base policy is equal to $\log (n) - (n-1)/n.$ We disprove the validity of this claim, and show that it is an upper bound on the actual KL divergence.

Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

no code implementations14 Dec 2023 Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D'Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, Jonathan Berant

However, even pretrain reward ensembles do not eliminate reward hacking: we show several qualitative reward hacking phenomena that are not mitigated by ensembling because all reward models in the ensemble exhibit similar error patterns.

Language Modelling

SEMQA: Semi-Extractive Multi-Source Question Answering

1 code implementation8 Nov 2023 Tal Schuster, Adam D. Lelkes, Haitian Sun, Jai Gupta, Jonathan Berant, William W. Cohen, Donald Metzler

Experimenting with several LLMs in various settings, we find this task to be surprisingly challenging, demonstrating the importance of QuoteSum for developing and studying such consolidation capabilities.

Attribute Long Form Question Answering +1

Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors

1 code implementation4 Oct 2023 Ido Amos, Jonathan Berant, Ankit Gupta

Modeling long-range dependencies across sequences is a longstanding goal in machine learning and has led to architectures, such as state space models, that dramatically outperform Transformers on long sequences.

Denoising

Making Retrieval-Augmented Language Models Robust to Irrelevant Context

1 code implementation2 Oct 2023 Ori Yoran, Tomer Wolfson, Ori Ram, Jonathan Berant

An important desideratum of RALMs, is that retrieved information helps model performance when it is relevant, and does not harm performance when it is not.

Language Modelling Natural Language Inference +2

Long-range Language Modeling with Self-retrieval

no code implementations23 Jun 2023 Ohad Rubin, Jonathan Berant

We train the retriever component with a semantic objective, where the goal is to retrieve chunks that increase the probability of the next chunk, according to a reference LM.

Language Modelling Retrieval

From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces

1 code implementation NeurIPS 2023 Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, Kristina Toutanova

Much of the previous work towards digital agents for graphical user interfaces (GUIs) has relied on text-based representations (derived from HTML or other structured data sources), which are not always readily available.

Instruction Following

ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding

1 code implementation23 May 2023 Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, Omer Levy

We introduce ZeroSCROLLS, a zero-shot benchmark for natural language understanding over long texts, which contains only test and small validation sets, without training data.

Natural Language Understanding

Answering Questions by Meta-Reasoning over Multiple Chains of Thought

1 code implementation25 Apr 2023 Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, Jonathan Berant

Modern systems for multi-hop question answering (QA) typically break questions into a sequence of reasoning steps, termed chain-of-thought (CoT), before arriving at a final answer.

Multi-hop Question Answering Question Answering

Crawling the Internal Knowledge-Base of Language Models

no code implementations30 Jan 2023 Roi Cohen, Mor Geva, Jonathan Berant, Amir Globerson

Here, we propose to address this goal by extracting a knowledge-graph of facts from a given language model.

Language Modelling

What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary

1 code implementation20 Dec 2022 Ori Ram, Liat Bezalel, Adi Zicher, Yonatan Belinkov, Jonathan Berant, Amir Globerson

We leverage this insight and propose a simple way to enrich query and passage representations with lexical information at inference time, and show that this significantly improves performance compared to the original model in zero-shot settings, and specifically on the BEIR benchmark.

Retrieval

Diverse Demonstrations Improve In-context Compositional Generalization

1 code implementation13 Dec 2022 Itay Levy, Ben Bogin, Jonathan Berant

In-context learning has shown great success in i. i. d semantic parsing splits, where the training and test sets are drawn from the same distribution.

In-Context Learning Semantic Parsing

Simplifying and Understanding State Space Models with Diagonal Linear RNNs

1 code implementation1 Dec 2022 Ankit Gupta, Harsh Mehta, Jonathan Berant

Sequence models based on linear state spaces (SSMs) have recently emerged as a promising choice of architecture for modeling long range dependencies across various modalities.

Training Vision-Language Models with Less Bimodal Supervision

1 code implementation1 Nov 2022 Elad Segal, Ben Bogin, Jonathan Berant

We experiment with a high-performing vision-language model, and analyze the effect of bimodal supervision on three vision-language tasks.

Language Modelling

Analyzing Transformers in Embedding Space

1 code implementation6 Sep 2022 Guy Dar, Mor Geva, Ankit Gupta, Jonathan Berant

In this work, we present a theoretical analysis where all parameters of a trained Transformer are interpreted by projecting them into the embedding space, that is, the space of vocabulary items they operate on.

Efficient Long-Text Understanding with Short-Text Models

1 code implementation1 Aug 2022 Maor Ivgi, Uri Shaham, Jonathan Berant

Transformer-based pretrained language models (LMs) are ubiquitous across natural language understanding, but cannot be applied to long sequences such as stories, scientific articles and long documents, due to their quadratic complexity.

Long-range modeling Natural Language Understanding

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

3 code implementations9 Jun 2022 Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, ZiRui Wang, Ziyi Wu

BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.

Common Sense Reasoning Math +1

Inferring Implicit Relations in Complex Questions with Language Models

1 code implementation28 Apr 2022 Uri Katz, Mor Geva, Jonathan Berant

A prominent challenge for modern language understanding systems is the ability to answer implicit reasoning questions, where the required reasoning steps for answering the question are not mentioned in the text explicitly.

Implicit Relations Question Answering +1

Diagonal State Spaces are as Effective as Structured State Spaces

2 code implementations27 Mar 2022 Ankit Gupta, Albert Gu, Jonathan Berant

Modeling long range dependencies in sequential data is a fundamental step towards attaining human-level performance in many modalities such as text, vision, audio and video.

Long-range modeling

Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments

no code implementations13 Feb 2022 Maor Ivgi, Yair Carmon, Jonathan Berant

Neural scaling laws define a predictable relationship between a model's parameter count and its performance after training in the form of a power law.

Model Selection

Unobserved Local Structures Make Compositional Generalization Hard

1 code implementation15 Jan 2022 Ben Bogin, Shivanshu Gupta, Jonathan Berant

While recent work has convincingly showed that sequence-to-sequence models struggle to generalize to new compositions (termed compositional generalization), little is known on what makes compositional generalization hard on a particular test instance.

Semantic Parsing

CommonsenseQA 2.0: Exposing the Limits of AI through Gamification

no code implementations14 Jan 2022 Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, Jonathan Berant

Constructing benchmarks that test the abilities of modern natural language understanding models is difficult - pre-trained language models exploit artifacts in benchmarks to achieve human parity, but still fail on adversarial examples and make errors that demonstrate a lack of common sense.

Common Sense Reasoning Natural Language Understanding

SCROLLS: Standardized CompaRison Over Long Language Sequences

2 code implementations10 Jan 2022 Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, Omer Levy

NLP benchmarks have largely focused on short texts, such as sentences and paragraphs, even though long texts comprise a considerable amount of natural language in the wild.

Long-range modeling Natural Language Inference +1

Learning To Retrieve Prompts for In-Context Learning

2 code implementations NAACL 2022 Ohad Rubin, Jonathan Herzig, Jonathan Berant

In-context learning is a recent paradigm in natural language understanding, where a large pre-trained language model (LM) observes a test instance and a few training examples as its input, and directly decodes the output without any update to its parameters.

In-Context Learning Language Modelling +1

Learning to Retrieve Passages without Supervision

1 code implementation NAACL 2022 Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, Amir Globerson

Dense retrievers for open-domain question answering (ODQA) have been shown to achieve impressive performance by training on large datasets of question-passage pairs.

Contrastive Learning Open-Domain Question Answering +1

Weakly Supervised Text-to-SQL Parsing through Question Decomposition

1 code implementation Findings (NAACL) 2022 Tomer Wolfson, Daniel Deutch, Jonathan Berant

Given questions, their QDMR structures (annotated by non-experts or automatically predicted), and the answers, we are able to automatically synthesize SQL queries that are used to train text-to-SQL models.

SQL Parsing Text-To-SQL

COVR: A test-bed for Visually Grounded Compositional Generalization with real images

1 code implementation EMNLP 2021 Ben Bogin, Shivanshu Gupta, Matt Gardner, Jonathan Berant

Due to the automatic generation process, COVR facilitates the creation of compositional splits, where models at test time need to generalize to new concepts and compositions in a zero- or few-shot setting.

Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalization

1 code implementation EMNLP 2021 Inbar Oren, Jonathan Herzig, Jonathan Berant

We evaluate our approach on a new split of the schema2QA dataset, and show that it leads to dramatic improvements in compositional generalization as well as moderate improvements in the traditional i. i. d setup.

Semantic Parsing

Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition

1 code implementation29 Jul 2021 Mor Geva, Tomer Wolfson, Jonathan Berant

We evaluate a range of RC models on our evaluation sets, which reveals large performance gaps on generated examples compared to the original data.

Natural Language Understanding Reading Comprehension

Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills

1 code implementation ACL 2022 Ori Yoran, Alon Talmor, Jonathan Berant

Models pre-trained with a language modeling objective possess ample world knowledge and language skills, but are known to struggle in tasks that require reasoning.

Language Modelling Reading Comprehension +1

Memory-efficient Transformers via Top-$k$ Attention

1 code implementation13 Jun 2021 Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, Jonathan Berant

Following the success of dot-product attention in Transformers, numerous approximations have been recently proposed to address its quadratic complexity with respect to the input length.

Question Decomposition with Dependency Graphs

1 code implementation AKBC 2021 Matan Hasson, Jonathan Berant

In this work, we present a QDMR parser that is based on dependency graphs (DGs), where nodes in the graph are words and edges describe logical relations that correspond to the different computation steps.

What's in your Head? Emergent Behaviour in Multi-Task Transformer Models

no code implementations13 Apr 2021 Mor Geva, Uri Katz, Aviv Ben-Arie, Jonathan Berant

In this work, we examine the behaviour of non-target heads, that is, the output of heads when given input that belongs to a different task than the one they were trained for.

Language Modelling Question Answering

Achieving Model Robustness through Discrete Adversarial Training

1 code implementation EMNLP 2021 Maor Ivgi, Jonathan Berant

In this work, we address this gap and leverage discrete attacks for online augmentation, where adversarial examples are generated at every training step, adapting to the changing nature of the model.

Value-aware Approximate Attention

1 code implementation EMNLP 2021 Ankit Gupta, Jonathan Berant

Following the success of dot-product attention in Transformers, numerous approximations have been recently proposed to address its quadratic complexity with respect to the input length.

Language Modelling

BERTese: Learning to Speak to BERT

no code implementations EACL 2021 Adi Haviv, Jonathan Berant, Amir Globerson

In this work, we propose a method for automatically rewriting queries into "BERTese", a paraphrase query that is directly optimized towards better knowledge extraction.

Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

1 code implementation6 Jan 2021 Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, Jonathan Berant

A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly.

Question Answering StrategyQA

Few-Shot Question Answering by Pretraining Span Selection

4 code implementations ACL 2021 Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, Omer Levy

Given a passage with multiple sets of recurring spans, we mask in each set all recurring spans but one, and ask the model to select the correct span in the passage for each masked span.

Question Answering

Transformer Feed-Forward Layers Are Key-Value Memories

1 code implementation EMNLP 2021 Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy

Feed-forward layers constitute two-thirds of a transformer model's parameters, yet their role in the network remains under-explored.

SmBoP: Semi-autoregressive Bottom-up Semantic Parsing

1 code implementation ACL (spnlp) 2021 Ohad Rubin, Jonathan Berant

We apply SmBoP on Spider, a challenging zero-shot semantic parsing benchmark, and show that SmBoP leads to a 2. 2x speed-up in decoding time and a $\sim$5x speed-up in training time, compared to a semantic parser that uses autoregressive decoding.

Semantic Parsing

Evaluating NLP Models via Contrast Sets

no code implementations1 Oct 2020 Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, A. Zhang, Ben Zhou

Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.

Reading Comprehension Sentiment Analysis

Learning Object Detection from Captions via Textual Scene Attributes

no code implementations30 Sep 2020 Achiya Jerbi, Roei Herzig, Jonathan Berant, Gal Chechik, Amir Globerson

In this work, we argue that captions contain much richer information about the image, including attributes of objects and their relations.

Image Captioning Object +2

Scene Graph to Image Generation with Contextualized Object Layout Refinement

no code implementations23 Sep 2020 Maor Ivgi, Yaniv Benny, Avichai Ben-David, Jonathan Berant, Lior Wolf

We empirically show on the COCO-STUFF dataset that our approach improves the quality of both the intermediate layout and the final image.

Image Generation Object

Span-based Semantic Parsing for Compositional Generalization

1 code implementation ACL 2021 Jonathan Herzig, Jonathan Berant

Despite the success of sequence-to-sequence (seq2seq) models in semantic parsing, recent work has shown that they fail in compositional generalization, i. e., the ability to generalize to new structures built of components observed during training.

Semantic Parsing

A Simple Global Neural Discourse Parser

no code implementations2 Sep 2020 Yichu Zhou, Omri Koshorek, Vivek Srikumar, Jonathan Berant

Discourse parsing is largely dominated by greedy parsers with manually-designed features, while global parsing is rare due to its computational expense.

Discourse Parsing

Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering

1 code implementation1 Jul 2020 Ben Bogin, Sanjay Subramanian, Matt Gardner, Jonathan Berant

However, state-of-the-art models in grounded question answering often do not explicitly perform decomposition, leading to difficulties in generalization to out-of-distribution examples.

Inductive Bias Question Answering +1

Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge

1 code implementation NeurIPS 2020 Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, Jonathan Berant

In this work, we provide a first demonstration that LMs can be trained to reliably perform systematic reasoning combining both implicit, pre-trained knowledge and explicit natural language statements.

World Knowledge

GMAT: Global Memory Augmentation for Transformers

1 code implementation5 Jun 2020 Ankit Gupta, Jonathan Berant

Moreover, global memory can also be used for sequence compression, by representing a long input sequence with the memory representations only.

Language Modelling Masked Language Modeling +1

Obtaining Faithful Interpretations from Compositional Neural Networks

1 code implementation ACL 2020 Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, Matt Gardner

Neural module networks (NMNs) are a popular approach for modeling compositionality: they achieve high accuracy when applied to problems in language and vision, while reflecting the compositional structure of the problem in the network architecture.

Explaining Question Answering Models through Text Generation

1 code implementation12 Apr 2020 Veronica Latcinnik, Jonathan Berant

Large pre-trained language models (LMs) have been shown to perform surprisingly well when fine-tuned on tasks that require commonsense and world knowledge.

Question Answering Text Generation +1

Injecting Numerical Reasoning Skills into Language Models

2 code implementations ACL 2020 Mor Geva, Ankit Gupta, Jonathan Berant

In this work, we show that numerical reasoning is amenable to automatic data generation, and thus one can inject this skill into pre-trained LMs, by generating large amounts of data, and training in a multi-task setup.

Data Augmentation Language Modelling +2

Evaluating the Evaluation of Diversity in Natural Language Generation

1 code implementation EACL 2021 Guy Tevet, Jonathan Berant

Despite growing interest in natural language generation (NLG) models that produce diverse outputs, there is currently no principled method for evaluating the diversity of an NLG system.

Text Generation

oLMpics -- On what Language Model Pre-training Captures

1 code implementation31 Dec 2019 Alon Talmor, Yanai Elazar, Yoav Goldberg, Jonathan Berant

A fundamental challenge is to understand whether the performance of a LM on a task should be attributed to the pre-trained representations or to the process of fine-tuning on the task data.

Language Modelling

On Making Reading Comprehension More Comprehensive

no code implementations WS 2019 Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, Sewon Min

In this work, we justify a question answering approach to reading comprehension and describe the various kinds of questions one might use to more fully test a system{'}s comprehension of a passage, moving beyond questions that only probe local predicate-argument structures.

Machine Reading Comprehension Question Answering

A Simple and Effective Model for Answering Multi-span Questions

4 code implementations EMNLP 2020 Elad Segal, Avia Efrat, Mor Shoham, Amir Globerson, Jonathan Berant

Models for reading comprehension (RC) commonly restrict their output space to the set of all single contiguous spans from the input, in order to alleviate the learning problem and avoid the need for a model that generates text explicitly.

Question Answering Reading Comprehension

Question Answering is a Format; When is it Useful?

no code implementations25 Sep 2019 Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, Sewon Min

In this opinion piece, we argue that question answering should be considered a format which is sometimes useful for studying particular phenomena, not a phenomenon or task in itself.

Machine Translation Question Answering +4

Don't paraphrase, detect! Rapid and Effective Data Collection for Semantic Parsing

1 code implementation IJCNLP 2019 Jonathan Herzig, Jonathan Berant

Assuming access to unlabeled utterances from the true distribution, we combine crowdsourcing with a paraphrase model to detect correct logical forms for the unlabeled utterances.

Semantic Parsing

MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension

1 code implementation ACL 2019 Alon Talmor, Jonathan Berant

A large number of reading comprehension (RC) datasets has been created recently, but little analysis has been done on whether they generalize to one another, and the extent to which existing datasets can be leveraged for improving performance on new ones.

Reading Comprehension

Grammar-based Neural Text-to-SQL Generation

no code implementations30 May 2019 Kevin Lin, Ben Bogin, Mark Neumann, Jonathan Berant, Matt Gardner

The sequence-to-sequence paradigm employed by neural text-to-SQL models typically performs token-level decoding and does not consider generating SQL hierarchically from a grammar.

Semantic Parsing Text-To-SQL

Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing

1 code implementation ACL 2019 Ben Bogin, Matt Gardner, Jonathan Berant

Research on parsing language to SQL has largely ignored the structure of the database (DB) schema, either because the DB was very simple, or because it was observed at both training and test time.

SQL Parsing Text-To-SQL

White-to-Black: Efficient Distillation of Black-Box Adversarial Attacks

1 code implementation NAACL 2019 Yotam Gil, Yoav Chai, Or Gorodissky, Jonathan Berant

Adversarial examples are important for understanding the behavior of neural models, and can improve their robustness through adversarial training.

Efficient Neural Network

DiscoFuse: A Large-Scale Dataset for Discourse-Based Sentence Fusion

2 code implementations NAACL 2019 Mor Geva, Eric Malmi, Idan Szpektor, Jonathan Berant

We author a set of rules for identifying a diverse set of discourse phenomena in raw text, and decomposing the text into two independent sentences.

Sentence Sentence Fusion +2

Differentiable Scene Graphs

1 code implementation26 Feb 2019 Moshiko Raboh, Roei Herzig, Gal Chechik, Jonathan Berant, Amir Globerson

In many domains, it is preferable to train systems jointly in an end-to-end manner, but SGs are not commonly used as intermediate components in visual reasoning systems because being discrete and sparse, scene-graph representations are non-differentiable and difficult to optimize.

Visual Reasoning

Neural network gradient-based learning of black-box function interfaces

no code implementations ICLR 2019 Alon Jacovi, Guy Hadash, Einat Kermany, Boaz Carmeli, Ofer Lavi, George Kour, Jonathan Berant

We propose a method for end-to-end training of a base neural network that integrates calls to existing black-box functions.

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

3 code implementations NAACL 2019 Alon Talmor, Jonathan Herzig, Nicholas Lourie, Jonathan Berant

To investigate question answering with prior knowledge, we present CommonsenseQA: a challenging new dataset for commonsense question answering.

Ranked #30 on Common Sense Reasoning on CommonsenseQA (using extra training data)

Common Sense Reasoning Multiple-choice +2

Value-based Search in Execution Space for Mapping Instructions to Programs

1 code implementation NAACL 2019 Dor Muhlgay, Jonathan Herzig, Jonathan Berant

Training models to map natural language instructions to programs given target world supervision only requires searching for good programs at training time.

Evaluating Text GANs as Language Models

1 code implementation NAACL 2019 Guy Tevet, Gavriel Habib, Vered Shwartz, Jonathan Berant

Generative Adversarial Networks (GANs) are a promising approach for text generation that, unlike traditional language models (LM), does not suffer from the problem of ``exposure bias''.

Text Generation

Emergence of Communication in an Interactive World with Consistent Speakers

1 code implementation3 Sep 2018 Ben Bogin, Mor Geva, Jonathan Berant

Training agents to communicate with one another given task-based supervision only has attracted considerable attention recently, due to the growing interest in developing models for human-agent interaction.

Explaining Queries over Web Tables to Non-Experts

no code implementations14 Aug 2018 Jonathan Berant, Daniel Deutch, Amir Globerson, Tova Milo, Tomer Wolfson

Designing a reliable natural language (NL) interface for querying tables has been a longtime goal of researchers in both the data management and natural language processing (NLP) communities.

Management Translation

Repartitioning of the ComplexWebQuestions Dataset

1 code implementation25 Jul 2018 Alon Talmor, Jonathan Berant

Recently, Talmor and Berant (2018) introduced ComplexWebQuestions - a dataset focused on answering complex questions by decomposing them into a sequence of simpler questions and extracting the answer from retrieved web snippets.

Reading Comprehension

Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing

4 code implementations NeurIPS 2018 Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc Le, Ni Lao

We present Memory Augmented Policy Optimization (MAPO), a simple and novel way to leverage a memory buffer of promising trajectories to reduce the variance of policy gradient estimate.

Combinatorial Optimization Program Synthesis +2

Weakly Supervised Semantic Parsing with Abstract Examples

no code implementations ACL 2018 Omer Goldman, Veronica Latcinnik, Ehud Nave, Amir Globerson, Jonathan Berant

Training semantic parsers from weak supervision (denotations) rather than strong supervision (programs) complicates training in two ways.

Semantic Parsing Visual Reasoning

Learning to Search in Long Documents Using Document Structure

1 code implementation COLING 2018 Mor Geva, Jonathan Berant

Reading comprehension models are based on recurrent neural networks that sequentially process the document tokens.

Information Retrieval Question Answering +2

Decoupling Structure and Lexicon for Zero-Shot Semantic Parsing

1 code implementation EMNLP 2018 Jonathan Herzig, Jonathan Berant

Building a semantic parser quickly in a new domain is a fundamental challenge for conversational interfaces, as current semantic parsers require expensive supervision and lack the ability to generalize to new domains.

Semantic Parsing

Text Segmentation as a Supervised Learning Task

2 code implementations NAACL 2018 Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, Jonathan Berant

Text segmentation, the task of dividing a document into contiguous segments based on its semantic structure, is a longstanding challenge in language understanding.

Clustering Segmentation +1

Polyglot Semantic Parsing in APIs

2 code implementations NAACL 2018 Kyle Richardson, Jonathan Berant, Jonas Kuhn

Traditional approaches to semantic parsing (SP) work by training individual models for each available parallel dataset of text-meaning pairs.

Semantic Parsing Translation

The Web as a Knowledge-base for Answering Complex Questions

2 code implementations NAACL 2018 Alon Talmor, Jonathan Berant

In this paper, we present a novel framework for answering broad and complex questions, assuming answering simple questions is possible using a search engine and a reading comprehension model.

Reading Comprehension

Contextualized Word Representations for Reading Comprehension

1 code implementation NAACL 2018 Shimi Salant, Jonathan Berant

Reading a document and extracting an answer to a question about its content has attracted substantial attention recently.

Language Modelling Question Answering +1

Weakly-supervised Semantic Parsing with Abstract Examples

1 code implementation14 Nov 2017 Omer Goldman, Veronica Latcinnik, Udi Naveh, Amir Globerson, Jonathan Berant

Training semantic parsers from weak supervision (denotations) rather than strong supervision (programs) complicates training in two ways.

Semantic Parsing Visual Reasoning

Inducing Regular Grammars Using Recurrent Neural Networks

1 code implementation28 Oct 2017 Mor Cohen, Avi Caciularu, Idan Rejwan, Jonathan Berant

Grammar induction is the task of learning a grammar from a set of examples.

Evaluating Semantic Parsing against a Simple Web-based Question Answering Model

1 code implementation SEMEVAL 2017 Alon Talmor, Mor Geva, Jonathan Berant

Semantic parsing shines at analyzing complex natural language that involves composition and computation over multiple pieces of evidence.

Question Answering Semantic Parsing

Coarse-to-Fine Question Answering for Long Documents

no code implementations ACL 2017 Eunsol Choi, Daniel Hewlett, Jakob Uszkoreit, Illia Polosukhin, Alex Lacoste, re, Jonathan Berant

We present a framework for question answering that can efficiently scale to longer documents while maintaining or even improving performance of state-of-the-art models.

Question Answering Reading Comprehension +1

Neural Semantic Parsing over Multiple Knowledge-bases

1 code implementation ACL 2017 Jonathan Herzig, Jonathan Berant

A fundamental challenge in developing semantic parsers is the paucity of strong supervision in the form of language utterances annotated with logical form.

Semantic Parsing

Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision (Short Version)

no code implementations4 Dec 2016 Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. Forbus, Ni Lao

In this work, we propose the Manager-Programmer-Computer framework, which integrates neural networks with non-differentiable memory to support abstract, scalable and precise operations through a friendly neural computer interface.

Feature Engineering Natural Language Understanding +2

Hierarchical Question Answering for Long Documents

no code implementations6 Nov 2016 Eunsol Choi, Daniel Hewlett, Alexandre Lacoste, Illia Polosukhin, Jakob Uszkoreit, Jonathan Berant

We present a framework for question answering that can efficiently scale to longer documents while maintaining or even improving performance of state-of-the-art models.

Question Answering Reading Comprehension +1

Learning Recurrent Span Representations for Extractive Question Answering

1 code implementation4 Nov 2016 Kenton Lee, Shimi Salant, Tom Kwiatkowski, Ankur Parikh, Dipanjan Das, Jonathan Berant

In this paper, we focus on this answer extraction task, presenting a novel model architecture that efficiently builds fixed length representations of all spans in the evidence document with a recurrent network.

Answer Selection Extractive Question-Answering +2

Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision

2 code implementations ACL 2017 Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. Forbus, Ni Lao

Harnessing the statistical power of neural networks to perform language understanding and symbolic reasoning is difficult, when it requires executing efficient discrete operations against a large knowledge-base.

Feature Engineering Structured Prediction

Imitation Learning of Agenda-based Semantic Parsers

1 code implementation TACL 2015 Jonathan Berant, Percy Liang

Semantic parsers conventionally construct logical forms bottom-up in a fixed order, resulting in the generation of many extraneous partial logical forms.

Imitation Learning Question Answering +1

Cannot find the paper you are looking for? You can Submit a new open access paper.