Search Results for author: Ashish Sabharwal

Found 77 papers, 42 papers with code

The Illusion of State in State-Space Models

no code implementations12 Apr 2024 William Merrill, Jackson Petty, Ashish Sabharwal

Our analysis reveals that the expressive power of SSMs is limited very similarly to transformers: SSMs cannot express computation outside the complexity class $\mathsf{TC}^0$.

Transformers as Transducers

no code implementations2 Apr 2024 Lena Strobl, Dana Angluin, David Chiang, Jonathan Rawski, Ashish Sabharwal

We study the sequence-to-sequence mapping capacity of transformers by relating them to finite transducers, and find that they can express surprisingly large classes of transductions.

Hard Attention POS

Data-driven Discovery with Large Generative Models

no code implementations21 Feb 2024 Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Sanchaita Hazra, Ashish Sabharwal, Peter Clark

With the accumulation of data at an unprecedented rate, its potential to fuel scientific discovery is growing exponentially.

Leveraging Code to Improve In-context Learning for Semantic Parsing

1 code implementation16 Nov 2023 Ben Bogin, Shivanshu Gupta, Peter Clark, Ashish Sabharwal

In-context learning (ICL) is an appealing approach for semantic parsing due to its few-shot nature and improved generalization.

In-Context Learning Semantic Parsing

ADaPT: As-Needed Decomposition and Planning with Language Models

1 code implementation8 Nov 2023 Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, Tushar Khot

Large Language Models (LLMs) are increasingly being used for interactive decision-making tasks requiring planning and adapting to the environment.

Decision Making

Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs

1 code implementation8 Nov 2023 Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, Tushar Khot

Our experiments with ChatGPT-3. 5 show that this bias is ubiquitous - 80% of our personas demonstrate bias; it is significant - some datasets show performance drops of 70%+; and can be especially harmful for certain groups - some personas suffer statistically significant drops on 80%+ of the datasets.

Fairness Math

QualEval: Qualitative Evaluation for Model Improvement

no code implementations6 Nov 2023 Vishvak Murahari, Ameet Deshpande, Peter Clark, Tanmay Rajpurohit, Ashish Sabharwal, Karthik Narasimhan, Ashwin Kalyan

In this work, we address the shortcomings of quantitative metrics by proposing QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement.

The Expressive Power of Transformers with Chain of Thought

no code implementations11 Oct 2023 William Merrill, Ashish Sabharwal

Motivated by this, we ask: Does such intermediate generation fundamentally extend the computational power of a decoder-only transformer?

Closing the Curious Case of Neural Text Degeneration

1 code implementation2 Oct 2023 Matthew Finlayson, John Hewitt, Alexander Koller, Swabha Swayamdipta, Ashish Sabharwal

We provide a theoretical explanation for the effectiveness of the truncation sampling by proving that truncation methods that discard tokens below some probability threshold (the most common type of truncation) can guarantee that all sampled tokens have nonzero true probability.

Text Generation

Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy

1 code implementation24 May 2023 Sarah Wiegreffe, Matthew Finlayson, Oyvind Tafjord, Peter Clark, Ashish Sabharwal

For example, both normalization and prompting methods for reducing SFC can be ineffective or even detrimental to task performance for some LMs.

In-Context Learning Multiple-choice +1

IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions

no code implementations23 May 2023 Wenhao Yu, Meng Jiang, Peter Clark, Ashish Sabharwal

Although counterfactual reasoning is a fundamental aspect of intelligence, the lack of large-scale counterfactual open-domain question-answering (QA) benchmarks makes it difficult to evaluate and improve models on this ability.

counterfactual Counterfactual Reasoning +2

Improving Language Models via Plug-and-Play Retrieval Feedback

no code implementations23 May 2023 Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, Ashish Sabharwal

ReFeed first generates initial outputs, then utilizes a retrieval model to acquire relevant information from large document collections, and finally incorporates the retrieved information into the in-context demonstration for output refinement, thereby addressing the limitations of LLMs in a more efficient and cost-effective manner.

Retrieval

Language Models with Rationality

no code implementations23 May 2023 Nora Kassner, Oyvind Tafjord, Ashish Sabharwal, Kyle Richardson, Hinrich Schuetze, Peter Clark

To address this, our goals are to make model beliefs and their inferential relationships explicit, and to resolve inconsistencies that may exist, so that answers are supported by interpretable chains of reasoning drawn from a consistent network of beliefs.

Question Answering

Specializing Smaller Language Models towards Multi-Step Reasoning

2 code implementations30 Jan 2023 Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, Tushar Khot

by paying the price of decreased generic ability, we can clearly lift up the scaling curve of models smaller than 10B towards a specialized multi-step math reasoning ability.

Math Model Selection

Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions

2 code implementations20 Dec 2022 Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal

While using the question to retrieve relevant text from an external knowledge source helps LLMs, we observe that this one-step retrieve-and-read approach is insufficient for multi-step QA.

Hallucination Question Answering +1

Decomposed Prompting: A Modular Approach for Solving Complex Tasks

1 code implementation5 Oct 2022 Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, Ashish Sabharwal

On symbolic reasoning tasks, we can further decompose sub-tasks that are hard for LLMs into even simpler solvable sub-tasks.

Information Retrieval Retrieval

Complexity-Based Prompting for Multi-Step Reasoning

no code implementations3 Oct 2022 Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, Tushar Khot

In this work, we propose complexity-based prompting, a simple and effective example selection scheme for multi-step reasoning.

Date Understanding GSM8K +2

The Parallelism Tradeoff: Limitations of Log-Precision Transformers

no code implementations2 Jul 2022 William Merrill, Ashish Sabharwal

Despite their omnipresence in modern NLP, characterizing the computational power of transformer neural nets remains an interesting open question.

Open-Ended Question Answering

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

4 code implementations9 Jun 2022 Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, ZiRui Wang, Ziyi Wu

BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.

Common Sense Reasoning Math +1

Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts

1 code implementation25 May 2022 Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal

We show how to use question decompositions to teach language models these broad reasoning skills in a robust fashion.

Question Answering

Better Retrieval May Not Lead to Better Question Answering

no code implementations7 May 2022 Zhengzhong Liang, Tushar Khot, Steven Bethard, Mihai Surdeanu, Ashish Sabharwal

Considerable progress has been made recently in open-domain question answering (QA) problems, which require Information Retrieval (IR) and Reading Comprehension (RC).

Information Retrieval Open-Domain Question Answering +3

What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment

1 code implementation19 Apr 2022 Matthew Finlayson, Kyle Richardson, Ashish Sabharwal, Peter Clark

We propose Hard RegSet as a challenging instruction learning task, and a controlled environment for studying instruction learning.

Out-of-Distribution Generalization

Pushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiability

no code implementations16 Dec 2021 Kyle Richardson, Ashish Sabharwal

Our results, however, reveal important limitations too: a careful sampling of training data is crucial for building models that generalize to larger problems, and transformer models' limited scale-invariance suggests they are far from learning robust deductive reasoning algorithms.

Hey AI, Can You Solve Complex Tasks by Talking to Agents?

1 code implementation Findings (ACL) 2022 Tushar Khot, Kyle Richardson, Daniel Khashabi, Ashish Sabharwal

To help develop models that can leverage existing systems, we propose a new challenge: Learning to solve complex tasks by communicating with existing agents (or models) in natural language.

MuSiQue: Multihop Questions via Single-hop Question Composition

2 code implementations2 Aug 2021 Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal

Multihop reasoning remains an elusive goal as existing multihop benchmarks are known to be largely solvable via shortcuts.

Multi-hop Question Answering Question Answering

Saturated Transformers are Constant-Depth Threshold Circuits

no code implementations30 Jun 2021 William Merrill, Ashish Sabharwal, Noah A. Smith

Transformers have become a standard neural network architecture for many NLP problems, motivating theoretical analysis of their power in terms of formal languages.

Hard Attention

Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?

1 code implementation Findings (ACL) 2021 Jieyu Zhao, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Kai-Wei Chang

We investigate the effectiveness of natural language interventions for reading-comprehension systems, studying this in the context of social stereotypes.

Ethics Few-Shot Learning +2

Multi-Modal Answer Validation for Knowledge-Based VQA

1 code implementation23 Mar 2021 Jialin Wu, Jiasen Lu, Ashish Sabharwal, Roozbeh Mottaghi

Instead of searching for the answer in a vast collection of often irrelevant facts as most existing approaches do, MAVEx aims to learn how to extract relevant knowledge from noisy sources, which knowledge source to trust for each answer candidate, and how to validate the candidate using that source.

Question Answering Retrieval +1

ReadOnce Transformers: Reusable Representations of Text for Transformers

no code implementations ACL 2021 Shih-ting Lin, Ashish Sabharwal, Tushar Khot

We present ReadOnce Transformers, an approach to convert a transformer-based model into one that can build an information-capturing, task-independent, and compressed representation of text.

Document Summarization

Temporal Reasoning on Implicit Events from Distant Supervision

no code implementations NAACL 2021 Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, Dan Roth

We propose TRACIE, a novel temporal reasoning dataset that evaluates the degree to which systems understand implicit events -- events that are not mentioned explicitly in natural language text but can be inferred from it.

Natural Language Inference

UnQovering Stereotyping Biases via Underspecified Questions

1 code implementation Findings of the Association for Computational Linguistics 2020 Tao Li, Tushar Khot, Daniel Khashabi, Ashish Sabharwal, Vivek Srikumar

Our broad study reveals that (1) all these models, with and without fine-tuning, have notable stereotyping biases in these classes; (2) larger models often have higher bias; and (3) the effect of fine-tuning on bias varies strongly with the dataset and the model size.

Question Answering

Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models

1 code implementation NAACL 2021 Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, Ashish Sabharwal

We propose a general framework called Text Modular Networks(TMNs) for building interpretable systems that learn to solve complex tasks by decomposing them into simpler ones solvable by existing models.

Question Answering

Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning

1 code implementation EMNLP 2020 Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal

For a recent large-scale model (XLNet), we show that only 18 points out of its answer F1 score of 72 on HotpotQA are obtained through multifact reasoning, roughly the same as that of a simpler RNN baseline.

Multi-hop Question Answering Question Answering +1

A Simple Yet Strong Pipeline for HotpotQA

no code implementations EMNLP 2020 Dirk Groeneveld, Tushar Khot, Mausam, Ashish Sabharwal

State-of-the-art models for multi-hop question answering typically augment large-scale language models like BERT with additional, intuitively useful capabilities such as named entity recognition, graph-based reasoning, and question decomposition.

Multi-hop Question Answering named-entity-recognition +4

More Bang for Your Buck: Natural Perturbation for Robust Question Answering

no code implementations EMNLP 2020 Daniel Khashabi, Tushar Khot, Ashish Sabharwal

While recent models have achieved human-level scores on many NLP datasets, we observe that they are considerably sensitive to small changes in input.

Question Answering

Adversarial Filters of Dataset Biases

1 code implementation ICML 2020 Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E. Peters, Ashish Sabharwal, Yejin Choi

Large neural models have demonstrated human-level performance on language and vision benchmarks, while their performance degrades considerably on adversarial or out-of-distribution samples.

Natural Language Inference

What Does My QA Model Know? Devising Controlled Probes using Expert Knowledge

2 code implementations31 Dec 2019 Kyle Richardson, Ashish Sabharwal

Open-domain question answering (QA) is known to involve several underlying knowledge and reasoning challenges, but are models actually learning such knowledge when trained on benchmark tasks?

General Knowledge Knowledge Graphs +1

Approximating the Permanent by Sampling from Adaptive Partitions

1 code implementation NeurIPS 2019 Jonathan Kuck, Tri Dao, Hamid Rezatofighi, Ashish Sabharwal, Stefano Ermon

Computing the permanent of a non-negative matrix is a core problem with practical applications ranging from target tracking to statistical thermodynamics.

Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses

1 code implementation ACL 2020 Erfan Sadeqi Azer, Daniel Khashabi, Ashish Sabharwal, Dan Roth

Empirical research in Natural Language Processing (NLP) has adopted a narrow set of principles for assessing hypotheses, relying mainly on p-value computation, which suffers from several known issues.

Bayesian Inference Misconceptions

Towards Efficient Discrete Integration via Adaptive Quantile Queries

no code implementations13 Oct 2019 Fan Ding, Hanjing Wang, Ashish Sabharwal, Yexiang Xue

On a suite of UAI inference challenge benchmarks, it saves 81. 5% of WISH queries while retaining the quality of results.

Probing Natural Language Inference Models through Semantic Fragments

3 code implementations16 Sep 2019 Kyle Richardson, Hai Hu, Lawrence S. Moss, Ashish Sabharwal

Our experiments, using a library of 8 such semantic fragments, reveal two remarkable findings: (a) State-of-the-art models, including BERT, that are pre-trained on existing NLI benchmark datasets perform poorly on these new fragments, even though the phenomena probed here are central to the NLI task.

Natural Language Inference

From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project

no code implementations4 Sep 2019 Peter Clark, Oren Etzioni, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, Sumithra Bhakthavatsalam, Dirk Groeneveld, Michal Guerquin, Michael Schmitz

This paper reports unprecedented success on the Grade 8 New York Regents Science Exam, where for the first time a system scores more than 90% on the exam's non-diagram, multiple choice (NDMC) questions.

Multiple-choice Question Answering

Question Answering as Global Reasoning over Semantic Abstractions

1 code implementation9 Jun 2019 Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Dan Roth

We propose a novel method for exploiting the semantic structure of text to answer multiple-choice questions.

Information Retrieval Multiple-choice +2

On the Possibilities and Limitations of Multi-hop Reasoning Under Linguistic Imperfections

no code implementations8 Jan 2019 Daniel Khashabi, Erfan Sadeqi Azer, Tushar Khot, Ashish Sabharwal, Dan Roth

The idea is to consider two interrelated spaces: a conceptual meaning space that is unambiguous and complete but hidden, and a linguistic space that captures a noisy grounding of the meaning space in the words of a language---the level at which all systems, whether neural or symbolic, operate.

Expanding Holographic Embeddings for Knowledge Completion

no code implementations NeurIPS 2018 Yexiang Xue, Yang Yuan, Zhitian Xu, Ashish Sabharwal

Neural models operating over structured spaces such as knowledge graphs require a continuous embedding of the discrete elements of this space (such as entities) as well as the relationships between them.

Knowledge Graphs

QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships

no code implementations20 Nov 2018 Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, Ashish Sabharwal

Many natural language questions require recognizing and reasoning with qualitative relationships (e. g., in science, economics, and medicine), but are challenging to answer with corpus-based methods.

Friction Semantic Parsing

Exploiting Explicit Paths for Multi-hop Reading Comprehension

1 code implementation ACL 2019 Souvik Kundu, Tushar Khot, Ashish Sabharwal, Peter Clark

To capture additional context, PathNet also composes the passage representations along each path to compute a passage-based representation.

Implicit Relations Knowledge Graphs +1

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

1 code implementation EMNLP 2018 Todor Mihaylov, Peter Clark, Tushar Khot, Ashish Sabharwal

Our oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts.

Question Answering Retrieval

Bridging Knowledge Gaps in Neural Entailment via Symbolic Models

no code implementations EMNLP 2018 Dongyeop Kang, Tushar Khot, Ashish Sabharwal, Peter Clark

We focus on filling these knowledge gaps in the Science Entailment task, by leveraging an external structured knowledge base (KB) of science facts.

Natural Language Inference

AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples

1 code implementation ACL 2018 Dongyeop Kang, Tushar Khot, Ashish Sabharwal, Eduard Hovy

We consider the problem of learning textual entailment models with limited supervision (5K-10K training examples), and present two complementary approaches for it.

Natural Language Inference Negation

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

1 code implementation14 Mar 2018 Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord

We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering.

Question Answering Retrieval

Approximate Inference via Weighted Rademacher Complexity

1 code implementation27 Jan 2018 Jonathan Kuck, Ashish Sabharwal, Stefano Ermon

Rademacher complexity is often used to characterize the learnability of a hypothesis class and is known to be related to the class size.

LEMMA

Learning What is Essential in Questions

1 code implementation CONLL 2017 Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Dan Roth

Question answering (QA) systems are easily distracted by irrelevant or redundant words in questions, especially when faced with long or multi-sentence questions in difficult domains.

Information Retrieval Question Answering +2

Answering Complex Questions Using Open Information Extraction

1 code implementation ACL 2017 Tushar Khot, Ashish Sabharwal, Peter Clark

While there has been substantial progress in factoid question-answering (QA), answering complex questions remains challenging, typically requiring both a large body of knowledge and inference techniques.

Open Information Extraction Question Answering +1

Knowledge Completion for Generics using Guided Tensor Factorization

no code implementations TACL 2018 Hanie Sedghi, Ashish Sabharwal

Given a knowledge base or KB containing (noisy) facts about common nouns or generics, such as "all trees produce oxygen" or "some animals live in forests", we consider the problem of inferring additional such facts at a precision similar to that of the starting KB.

Active Learning General Knowledge +1

Adaptive Concentration Inequalities for Sequential Decision Problems

no code implementations NeurIPS 2016 Shengjia Zhao, Enze Zhou, Ashish Sabharwal, Stefano Ermon

A key challenge in sequential decision problems is to determine how many samples are needed for an agent to make reliable decisions with good probabilistic guarantees.

Two-sample testing

Question Answering via Integer Programming over Semi-Structured Knowledge

no code implementations20 Apr 2016 Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Peter Clark, Oren Etzioni, Dan Roth

We propose a structured inference system for this task, formulated as an Integer Linear Program (ILP), that answers natural language questions using a semi-structured knowledge base derived from text, including questions requiring multi-step inference and a combination of multiple facts.

Information Retrieval Question Answering +1

Selecting Near-Optimal Learners via Incremental Data Allocation

no code implementations31 Dec 2015 Ashish Sabharwal, Horst Samulowitz, Gerald Tesauro

We study a novel machine learning (ML) problem setting of sequentially allocating small subsets of training data amongst a large set of classifiers.

Parsing Algebraic Word Problems into Equations

no code implementations TACL 2015 Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, Siena Dumas Ang

This paper formalizes the problem of solving multi-sentence algebraic word problems as that of generating and scoring equation trees.

Coreference Resolution Sentence

Embed and Project: Discrete Sampling with Universal Hashing

no code implementations NeurIPS 2013 Stefano Ermon, Carla P. Gomes, Ashish Sabharwal, Bart Selman

We consider the problem of sampling from a probability distribution defined over a high-dimensional discrete set, specified for instance by a graphical model.

Combinatorial Optimization

Density Propagation and Improved Bounds on the Partition Function

no code implementations NeurIPS 2012 Stefano Ermon, Ashish Sabharwal, Bart Selman, Carla P. Gomes

Given a probabilistic graphical model, its density of states is a function that, for any likelihood value, gives the number of configurations with that probability.

Tree Decomposition

Cannot find the paper you are looking for? You can Submit a new open access paper.