Search Results for author: Benjamin Van Durme

Found 206 papers, 72 papers with code

Improved Induction of Narrative Chains via Cross-Document Relations

1 code implementation *SEM (NAACL) 2022 Andrew Blair-Stanek, Benjamin Van Durme

The standard approach for inducing narrative chains considers statistics gathered per individual document.

RE-Adapt: Reverse Engineered Adaptation of Large Language Models

no code implementations23 May 2024 William Fleshman, Benjamin Van Durme

We introduce RE-Adapt, an approach to fine-tuning large language models on new domains without degrading any pre-existing instruction-tuning.

Instruction Following Retrieval

AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guarantees

no code implementations12 Apr 2024 William Fleshman, Aleem Khan, Marc Marone, Benjamin Van Durme

Large language models (LLMs) are increasingly capable of completing knowledge intensive tasks by recalling information from a static pretraining corpus.

Continual Learning

Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data

no code implementations5 Apr 2024 Jingyu Zhang, Marc Marone, Tianjian Li, Benjamin Van Durme, Daniel Khashabi

To address these limitations, we tackle the verifiability goal with a different philosophy: we trivialize the verification process by developing models that quote verbatim statements from trusted sources in pre-training data.


FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions

1 code implementation22 Mar 2024 Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, Luca Soldaini

First, we introduce our dataset FollowIR, which contains a rigorous instruction evaluation benchmark as well as a training set for helping IR models learn to better follow real-world instructions.

Information Retrieval Retrieval +1

Dated Data: Tracing Knowledge Cutoffs in Large Language Models

no code implementations19 Mar 2024 Jeffrey Cheng, Marc Marone, Orion Weller, Dawn Lawrie, Daniel Khashabi, Benjamin Van Durme

Using this analysis, we find that effective cutoffs often differ from reported cutoffs.

A Closer Look at Claim Decomposition

no code implementations18 Mar 2024 Miriam Wanner, Seth Ebner, Zhengping Jiang, Mark Dredze, Benjamin Van Durme

We investigate how various methods of claim decomposition -- especially LLM-based methods -- affect the result of an evaluation approach such as the recently proposed FActScore, finding that it is sensitive to the decomposition method used.


Tur[k]ingBench: A Challenge Benchmark for Web Agents

no code implementations18 Mar 2024 Kevin Xu, Yeganeh Kordi, Kate Sanders, Yizhong Wang, Adam Byerly, Jack Zhang, Benjamin Van Durme, Daniel Khashabi

We evaluate the performance of state-of-the-art models, including language-only, vision-only, and layout-only models, and their combinations, on this benchmark.

LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error

1 code implementation7 Mar 2024 Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, Yu Su

We find that existing LLMs, including GPT-4 and open-source LLMs specifically fine-tuned for tool use, only reach a correctness rate in the range of 30% to 60%, far from reliable use in practice.

Continual Learning In-Context Learning

TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning

no code implementations29 Feb 2024 Kate Sanders, Nathaniel Weir, Benjamin Van Durme

It is challenging to perform question-answering over complex, multimodal content such as television clips.

Question Answering Video Understanding

RORA: Robust Free-Text Rationale Evaluation

no code implementations28 Feb 2024 Zhengping Jiang, Yining Lu, Hanjie Chen, Daniel Khashabi, Benjamin Van Durme, Anqi Liu

This is achieved by assessing the conditional V-information \citep{hewitt-etal-2021-conditional} with a predictive family robust against leaky features that can be exploited by a small model.

Decision Making

Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic

no code implementations22 Feb 2024 Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark, Benjamin Van Durme

Contemporary language models enable new opportunities for structured reasoning with text, such as the construction and evaluation of intuitive, proof-like textual entailment trees without relying on brittle formal logic.

Formal Logic Knowledge Distillation +2

Streaming Sequence Transduction through Dynamic Compression

1 code implementation2 Feb 2024 Weiting Tan, Yunmo Chen, Tongfei Chen, Guanghui Qin, Haoran Xu, Heidi C. Zhang, Benjamin Van Durme, Philipp Koehn

We introduce STAR (Stream Transduction with Anchor Representations), a novel Transformer-based model designed for efficient sequence-to-sequence transduction over streams.

Automatic Speech Recognition Automatic Speech Recognition (ASR) +1

MultiMUC: Multilingual Template Filling on MUC-4

1 code implementation29 Jan 2024 William Gantt, Shabnam Behzad, Hannah Youngeun An, Yunmo Chen, Aaron Steven White, Benjamin Van Durme, Mahsa Yarmohammadi

We introduce MultiMUC, the first multilingual parallel corpus for template filling, comprising translations of the classic MUC-4 template filling benchmark into five languages: Arabic, Chinese, Farsi, Korean, and Russian.

Machine Translation Translation

Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation

1 code implementation16 Jan 2024 Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, Young Jin Kim

However, even the top-performing 13B LLM-based translation models, like ALMA, does not match the performance of state-of-the-art conventional encoder-decoder translation models or larger-scale LLMs such as GPT-4.

Decoder Machine Translation +1

Reframing Tax Law Entailment as Analogical Reasoning

no code implementations12 Jan 2024 Xinrui Zou, Ming Zhang, Nathaniel Weir, Benjamin Van Durme, Nils Holzenberger

We re-frame statutory reasoning as an analogy task, where each instance of the analogy task involves a combination of two instances of statutory reasoning.


Do Androids Know They're Only Dreaming of Electric Sheep?

no code implementations28 Dec 2023 Sky CH-Wang, Benjamin Van Durme, Jason Eisner, Chris Kedzie

We design probes trained on the internal representations of a transformer language model to predict its hallucinatory behavior on three grounded generation tasks.

Hallucination Hallucination Evaluation +1

BLT: Can Large Language Models Handle Basic Legal Text?

1 code implementation16 Nov 2023 Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme

We find that the best publicly available LLMs like GPT-4, Claude, and {PaLM 2} currently perform poorly at basic legal text handling.

Interpreting User Requests in the Context of Natural Language Standing Instructions

1 code implementation16 Nov 2023 Nikita Moghe, Patrick Xia, Jacob Andreas, Jason Eisner, Benjamin Van Durme, Harsh Jhamtani

Users of natural language interfaces, generally powered by Large Language Models (LLMs), often must repeat their preferences each time they make a similar request.

Toucan: Token-Aware Character Level Language Modeling

no code implementations15 Nov 2023 William Fleshman, Benjamin Van Durme

Character-level language models obviate the need for separately trained tokenizers, but efficiency suffers from longer sequence lengths.

Language Modelling

FAMuS: Frames Across Multiple Sources

2 code implementations9 Nov 2023 Siddharth Vashishtha, Alexander Martin, William Gantt, Benjamin Van Durme, Aaron Steven White

Understanding event descriptions is a central aspect of language processing, but current approaches focus overwhelmingly on single sentences or documents.

Sentence valid

Narrowing the Gap between Zero- and Few-shot Machine Translation by Matching Styles

no code implementations4 Nov 2023 Weiting Tan, Haoran Xu, Lingfeng Shen, Shuyue Stella Li, Kenton Murray, Philipp Koehn, Benjamin Van Durme, Yunmo Chen

Large language models trained primarily in a monolingual setting have demonstrated their ability to generalize to machine translation using zero- and few-shot examples with in-context learning.

In-Context Learning Machine Translation +1

InstructExcel: A Benchmark for Natural Language Instruction in Excel

no code implementations23 Oct 2023 Justin Payan, Swaroop Mishra, Mukul Singh, Carina Negreanu, Christian Poelitz, Chitta Baral, Subhro Roy, Rasika Chakravarthy, Benjamin Van Durme, Elnaz Nouri

With the evolution of Large Language Models (LLMs) we can solve increasingly more complex NLP tasks across various domains, including spreadsheets.

A Unified View of Evaluation Metrics for Structured Prediction

1 code implementation20 Oct 2023 Yunmo Chen, William Gantt, Tongfei Chen, Aaron Steven White, Benjamin Van Durme

We present a conceptual framework that unifies a variety of evaluation metrics for different structured prediction tasks (e. g. event and relation extraction, syntactic and semantic parsing).

Relation Extraction Semantic Parsing +1

Nugget: Neural Agglomerative Embeddings of Text

no code implementations3 Oct 2023 Guanghui Qin, Benjamin Van Durme

This is problematic, as the amount of information contained in text often varies with the length of the input.

Language Modelling Machine Translation +1

Dodo: Dynamic Contextual Compression for Decoder-only LMs

no code implementations3 Oct 2023 Guanghui Qin, Corby Rosset, Ethan C. Chau, Nikhil Rao, Benjamin Van Durme

For example, in the autoencoding task, Dodo shrinks context at a 20x compression ratio with a BLEU score of 98% for reconstruction, achieving nearly lossless encoding.

Decoder Language Modelling +1

OpenAI Cribbed Our Tax Example, But Can GPT-4 Really Do Tax?

no code implementations15 Sep 2023 Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme

The authors explain where OpenAI got the tax law example in its livestream demonstration of GPT-4, why GPT-4 got the wrong answer, and how it fails to reliably calculate taxes.

MegaWika: Millions of reports and their sources across 50 diverse languages

no code implementations13 Jul 2023 Samuel Barham, Orion Weller, Michelle Yuan, Kenton Murray, Mahsa Yarmohammadi, Zhengping Jiang, Siddharth Vashishtha, Alexander Martin, Anqi Liu, Aaron Steven White, Jordan Boyd-Graber, Benjamin Van Durme

To foster the development of new models for collaborative AI-assisted report generation, we introduce MegaWika, consisting of 13 million Wikipedia articles in 50 diverse languages, along with their 71 million referenced source materials.

Cross-Lingual Question Answering Retrieval +1

MultiVENT: Multilingual Videos of Events with Aligned Natural Text

no code implementations6 Jul 2023 Kate Sanders, David Etter, Reno Kriz, Benjamin Van Durme

Everyday news coverage has shifted from traditional broadcasts towards a wide range of presentation formats such as first-hand, unedited video footage.

Information Retrieval Retrieval +1

Evaluating Paraphrastic Robustness in Textual Entailment Models

no code implementations29 Jun 2023 Dhruv Verma, Yash Kumar Lal, Shreyashee Sinha, Benjamin Van Durme, Adam Poliak

We present PaRTE, a collection of 1, 126 pairs of Recognizing Textual Entailment (RTE) examples to evaluate whether models are robust to paraphrasing.

Natural Language Inference RTE

Zero and Few-shot Semantic Parsing with Ambiguous Inputs

1 code implementation1 Jun 2023 Elias Stengel-Eskin, Kyle Rawlins, Benjamin Van Durme

We attempt to address this shortcoming by introducing AmP, a framework, dataset, and challenge for translating ambiguous natural language to formal representations like logic and code.

Semantic Parsing

"According to ...": Prompting Language Models Improves Quoting from Pre-Training Data

no code implementations22 May 2023 Orion Weller, Marc Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, Benjamin Van Durme

Large Language Models (LLMs) may hallucinate and generate fake information, despite pre-training on factual data.

NevIR: Negation in Neural Information Retrieval

1 code implementation12 May 2023 Orion Weller, Dawn Lawrie, Benjamin Van Durme

Although the Information Retrieval (IR) community has adopted LMs as the backbone of modern IR architectures, there has been little to no research in understanding how negation impacts neural IR.

Information Retrieval Negation +1

Did You Mean...? Confidence-based Trade-offs in Semantic Parsing

no code implementations29 Mar 2023 Elias Stengel-Eskin, Benjamin Van Durme

We then examine how confidence scores can help optimize the trade-off between usability and safety.

Semantic Parsing

Can GPT-3 Perform Statutory Reasoning?

1 code implementation13 Feb 2023 Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme

Statutory reasoning is the task of reasoning with facts and statutes, which are rules written in natural language by a legislature.

Defending Against Disinformation Attacks in Open-Domain Question Answering

no code implementations20 Dec 2022 Orion Weller, Aleem Khan, Nathaniel Weir, Dawn Lawrie, Benjamin Van Durme

Recent work in open-domain question answering (ODQA) has shown that adversarial poisoning of the search collection can cause large drops in accuracy for production systems.

Data Poisoning Misinformation +1

When Do Decompositions Help for Machine Reading?

no code implementations20 Dec 2022 Kangda Wei, Dawn Lawrie, Benjamin Van Durme, Yunmo Chen, Orion Weller

Answering complex questions often requires multi-step reasoning in order to obtain the final answer.

Reading Comprehension Retrieval

Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models

no code implementations1 Dec 2022 Zhuowan Li, Cihang Xie, Benjamin Van Durme, Alan Yuille

Despite the impressive advancements achieved through vision-and-language pretraining, it remains unclear whether this joint learning paradigm can help understand each individual modality.

Attribute Representation Learning

Calibrated Interpretation: Confidence Estimation in Semantic Parsing

2 code implementations14 Nov 2022 Elias Stengel-Eskin, Benjamin Van Durme

Sequence generation models are increasingly being used to translate natural language into programs, i. e. to perform executable semantic parsing.

Semantic Parsing

Automatic Document Selection for Efficient Encoder Pretraining

no code implementations20 Oct 2022 Yukun Feng, Patrick Xia, Benjamin Van Durme, João Sedoc

Building pretrained language models is considered expensive and data-intensive, but must we increase dataset size to achieve better performance?


An Empirical Study on Finding Spans

no code implementations13 Oct 2022 Weiwei Gu, Boyuan Zheng, Yunmo Chen, Tongfei Chen, Benjamin Van Durme

We present an empirical study on methods for span finding, the selection of consecutive tokens in text for some downstream tasks.

Iterative Document-level Information Extraction via Imitation Learning

2 code implementations12 Oct 2022 Yunmo Chen, William Gantt, Weiwei Gu, Tongfei Chen, Aaron Steven White, Benjamin Van Durme

We present a novel iterative extraction model, IterX, for extracting complex relations, or templates (i. e., N-tuples representing a mapping from named slots to spans of text) within a document.

4-ary Relation Extraction Imitation Learning

Ambiguous Images With Human Judgments for Robust Visual Event Classification

no code implementations6 Oct 2022 Kate Sanders, Reno Kriz, Anqi Liu, Benjamin Van Durme

However, humans are frequently presented with visual data that they cannot classify with 100% certainty, and models trained on standard vision benchmarks achieve low performance when evaluated on this data.

NELLIE: A Neuro-Symbolic Inference Engine for Grounded, Compositional, and Explainable Reasoning

no code implementations16 Sep 2022 Nathaniel Weir, Peter Clark, Benjamin Van Durme

Our goal is a modern approach to answering questions via systematic reasoning where answers are supported by human interpretable proof trees grounded in an NL corpus of authoritative facts.

Hallucination Language Modelling +1

Multilingual Coreference Resolution in Multiparty Dialogue

1 code implementation2 Aug 2022 Boyuan Zheng, Patrick Xia, Mahsa Yarmohammadi, Benjamin Van Durme

Existing multiparty dialogue datasets for entity coreference resolution are nascent, and many challenges are still unaddressed.

coreference-resolution Data Augmentation

Zero-shot Cross-lingual Transfer is Under-specified Optimization

1 code implementation RepL4NLP (ACL) 2022 Shijie Wu, Benjamin Van Durme, Mark Dredze

Pretrained multilingual encoders enable zero-shot cross-lingual transfer, but often produce unreliable models that exhibit high performance variance on the target language.

Zero-Shot Cross-Lingual Transfer

BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsing

1 code implementation NeurIPS 2023 Subhro Roy, Sam Thomson, Tongfei Chen, Richard Shin, Adam Pauls, Jason Eisner, Benjamin Van Durme

We introduce BenchCLAMP, a Benchmark to evaluate Constrained LAnguage Model Parsing, that includes context-free grammars for seven semantic parsing datasets and two syntactic parsing datasets with varied output representations, as well as a constrained decoding interface to generate only valid outputs covered by these grammars.

Decoder Language Modelling +3

Asking the Right Questions in Low Resource Template Extraction

no code implementations25 May 2022 Nils Holzenberger, Yunmo Chen, Benjamin Van Durme

Information Extraction (IE) researchers are mapping tasks to Question Answering (QA) in order to leverage existing large QA resources, and thereby improve data efficiency.

Question Answering

The Curious Case of Control

1 code implementation24 May 2022 Elias Stengel-Eskin, Benjamin Van Durme

Given the advanced fluency of large generative language models, we ask whether model outputs are consistent with these heuristics, and to what degree different models are consistent with each other.

When More Data Hurts: A Troubling Quirk in Developing Broad-Coverage Natural Language Understanding Systems

1 code implementation24 May 2022 Elias Stengel-Eskin, Emmanouil Antonios Platanios, Adam Pauls, Sam Thomson, Hao Fang, Benjamin Van Durme, Jason Eisner, Yu Su

Rejecting class imbalance as the sole culprit, we reveal that the trend is closely associated with an effect we call source signal dilution, where strong lexical cues for the new symbol become diluted as the training dataset grows.

Intent Recognition Natural Language Understanding +1

Addressing Resource and Privacy Constraints in Semantic Parsing Through Data Augmentation

no code implementations Findings (ACL) 2022 Kevin Yang, Olivia Deng, Charles Chen, Richard Shin, Subhro Roy, Benjamin Van Durme

We introduce a novel setup for low-resource task-oriented semantic parsing which incorporates several constraints that may arise in real-world scenarios: (1) lack of similar datasets/models from a related domain, (2) inability to sample useful logical forms directly from a grammar, and (3) privacy requirements for unlabeled natural utterances.

Data Augmentation Semantic Parsing

Visual Commonsense in Pretrained Unimodal and Multimodal Models

1 code implementation NAACL 2022 Chenyu Zhang, Benjamin Van Durme, Zhuowan Li, Elias Stengel-Eskin

Our commonsense knowledge about objects includes their typical visual attributes; we know that bananas are typically yellow or green, and not purple.

Attribute Visual Commonsense Tests +1

The NLP Task Effectiveness of Long-Range Transformers

no code implementations16 Feb 2022 Guanghui Qin, Yukun Feng, Benjamin Van Durme

Transformer models cannot easily scale to long sequences due to their O(N^2) time and space complexity.

Few-Shot Semantic Parsing with Language Models Trained On Code

no code implementations NAACL 2022 Richard Shin, Benjamin Van Durme

Intuitively, such models can more easily output canonical utterances as they are closer to the natural language used for pre-training.

Semantic Parsing

BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation

2 code implementations EMNLP 2021 Haoran Xu, Benjamin Van Durme, Kenton Murray

The success of bidirectional encoders using masked language models, such as BERT, on numerous natural language processing tasks has prompted researchers to attempt to incorporate these pre-trained models into neural machine translation (NMT) systems.

de-en Language Modelling +3

Guided Generation of Cause and Effect

no code implementations21 Jul 2021 Zhongyang Li, Xiao Ding, Ting Liu, J. Edward Hu, Benjamin Van Durme

We present a conditional text generation framework that posits sentential expressions of possible causes and effects.

Conditional Text Generation Knowledge Graphs

Factoring Statutory Reasoning as Language Understanding Challenges

1 code implementation ACL 2021 Nils Holzenberger, Benjamin Van Durme

Statutory reasoning is the task of determining whether a legal statute, stated in natural language, applies to the text description of a case.

Natural Language Inference

Human Schema Curation via Causal Association Rule Mining

1 code implementation LREC (LAW) 2022 Noah Weber, Anton Belyy, Nils Holzenberger, Rachel Rudinger, Benjamin Van Durme

Event schemas are structured knowledge sources defining typical real-world scenarios (e. g., going to an airport).

Moving on from OntoNotes: Coreference Resolution Model Transfer

2 code implementations EMNLP 2021 Patrick Xia, Benjamin Van Durme

Academic neural models for coreference resolution (coref) are typically trained on a single dataset, OntoNotes, and model improvements are benchmarked on that same dataset.


Joint Universal Syntactic and Semantic Parsing

1 code implementation12 Apr 2021 Elias Stengel-Eskin, Kenton Murray, Sheng Zhang, Aaron Steven White, Benjamin Van Durme

While numerous attempts have been made to jointly parse syntax and semantics, high performance in one domain typically comes at the price of performance in the other.

Semantic Parsing

InFillmore: Frame-Guided Language Generation with Bidirectional Context

no code implementations Joint Conference on Lexical and Computational Semantics 2021 Jiefu Ou, Nathaniel Weir, Anton Belyy, Felix Yu, Benjamin Van Durme

We propose a structured extension to bidirectional-context conditional language generation, or "infilling," inspired by Frame Semantic theory (Fillmore, 1976).

Text Infilling

Gradual Fine-Tuning for Low-Resource Domain Adaptation

2 code implementations EACL (AdaptNLP) 2021 Haoran Xu, Seth Ebner, Mahsa Yarmohammadi, Aaron Steven White, Benjamin Van Durme, Kenton Murray

Fine-tuning is known to improve NLP models by adapting an initial model trained on more plentiful but less domain-salient examples to data in a target domain.

Domain Adaptation

Joint Modeling of Arguments for Event Understanding

1 code implementation20 Nov 2020 Yunmo Chen, Tongfei Chen, Benjamin Van Durme

We recognize the task of event argument linking in documents as similar to that of intent slot resolution in dialogue, providing a Transformer-based model that extends from a recently proposed solution to resolve references to slots.


COD3S: Diverse Generation with Discrete Semantic Signatures

1 code implementation EMNLP 2020 Nathaniel Weir, João Sedoc, Benjamin Van Durme

We present COD3S, a novel method for generating semantically diverse sentences using neural sequence-to-sequence (seq2seq) models.

Semantic Textual Similarity Sentence

Iterative Paraphrastic Augmentation with Discriminative Span Alignment

no code implementations1 Jul 2020 Ryan Culkin, J. Edward Hu, Elias Stengel-Eskin, Guanghui Qin, Benjamin Van Durme

We introduce a novel paraphrastic augmentation strategy based on sentence-level lexically constrained paraphrasing and discriminative span alignment.


Script Induction as Association Rule Mining

no code implementations WS 2020 Anton Belyy, Benjamin Van Durme

We show that the count-based Script Induction models of Chambers and Jurafsky (2008) and Jans et al. (2012) can be unified in a general framework of narrative chain likelihood maximization.

Cloze Test

Incremental Neural Coreference Resolution in Constant Memory

1 code implementation EMNLP 2020 Patrick Xia, João Sedoc, Benjamin Van Durme

We investigate modeling coreference resolution under a fixed memory constraint by extending an incremental clustering algorithm to utilize contextualized encoders and neural components.

Clustering coreference-resolution +1

Complementing Lexical Retrieval with Semantic Residual Embedding

no code implementations29 Apr 2020 Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, Jamie Callan

This paper presents CLEAR, a retrieval model that seeks to complement classical lexical exact-match models such as BM25 with semantic matching signals from a neural embedding matching model.

Information Retrieval Retrieval

Probing Neural Language Models for Human Tacit Assumptions

no code implementations10 Apr 2020 Nathaniel Weir, Adam Poliak, Benjamin Van Durme

Our prompts are based on human responses in a psychological study of conceptual associations.

Hierarchical Entity Typing via Multi-level Learning to Rank

1 code implementation ACL 2020 Tongfei Chen, Yunmo Chen, Benjamin Van Durme

We propose a novel method for hierarchical entity classification that embraces ontological structure at both training and during prediction.

Decoder Entity Typing +1

Causal Inference of Script Knowledge

no code implementations EMNLP 2020 Noah Weber, Rachel Rudinger, Benjamin Van Durme

When does a sequence of events define an everyday scenario and how can this knowledge be induced from text?

Causal Inference

Reading the Manual: Event Extraction as Definition Comprehension

no code implementations EMNLP (spnlp) 2020 Yunmo Chen, Tongfei Chen, Seth Ebner, Aaron Steven White, Benjamin Van Durme

We ask whether text understanding has progressed to where we may extract event information through incremental refinement of bleached statements derived from annotation manuals.

Event Extraction

Multi-Sentence Argument Linking

no code implementations ACL 2020 Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, Benjamin Van Durme

We present a novel document-level model for finding argument spans that fill an event's roles, connecting related ideas in sentence-level semantic role labeling and coreference resolution.

coreference-resolution Semantic Role Labeling +2

Interactive Refinement of Cross-Lingual Word Embeddings

1 code implementation EMNLP 2020 Michelle Yuan, Mozhi Zhang, Benjamin Van Durme, Leah Findlater, Jordan Boyd-Graber

Cross-lingual word embeddings transfer knowledge between languages: models trained on high-resource languages can predict in low-resource languages.

Active Learning Cross-Lingual Word Embeddings +3

Bag-of-Words Transfer: Non-Contextual Techniques for Multi-Task Learning

no code implementations WS 2019 Seth Ebner, Felicity Wang, Benjamin Van Durme

Many architectures for multi-task learning (MTL) have been proposed to take advantage of transfer among tasks, often involving complex models and training procedures.

Multi-Task Learning Sentence +1

Universal Decompositional Semantic Parsing

no code implementations ACL 2020 Elias Stengel-Eskin, Aaron Steven White, Sheng Zhang, Benjamin Van Durme

We introduce a transductive model for parsing into Universal Decompositional Semantics (UDS) representations, which jointly learns to map natural language utterances into UDS graph structures and annotate the graph with decompositional semantic attribute scores.

Attribute Semantic Parsing

Exact and/or Fast Nearest Neighbors

1 code implementation6 Oct 2019 Matthew Francis-Landau, Benjamin Van Durme

Prior methods for retrieval of nearest neighbors in high dimensions are fast and approximate--providing probabilistic guarantees of returning the correct answer--or slow and exact performing an exhaustive search.

Data Structures and Algorithms

Uncertain Natural Language Inference

no code implementations ACL 2020 Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, Benjamin Van Durme

We introduce Uncertain Natural Language Inference (UNLI), a refinement of Natural Language Inference (NLI) that shifts away from categorical labels, targeting instead the direct prediction of subjective probability assessments.

Learning-To-Rank Natural Language Inference +1

Broad-Coverage Semantic Parsing as Transduction

no code implementations IJCNLP 2019 Sheng Zhang, Xutai Ma, Kevin Duh, Benjamin Van Durme

We unify different broad-coverage semantic parsing tasks under a transduction paradigm, and propose an attention-based neural framework that incrementally builds a meaning representation via a sequence of semantic relations.

AMR Parsing UCCA Parsing

A Discriminative Neural Model for Cross-Lingual Word Alignment

no code implementations IJCNLP 2019 Elias Stengel-Eskin, Tzu-Ray Su, Matt Post, Benjamin Van Durme

We introduce a novel discriminative word alignment model, which we integrate into a Transformer-based machine translation model.

Machine Translation NER +2

Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference

1 code implementation ACL 2019 Yonatan Belinkov, Adam Poliak, Stuart M. Shieber, Benjamin Van Durme, Alexander M. Rush

In contrast to standard approaches to NLI, our methods predict the probability of a premise given a hypothesis and NLI label, discouraging models from ignoring the premise.

Natural Language Inference

Learning to Rank for Plausible Plausibility

no code implementations ACL 2019 Zhongyang Li, Tongfei Chen, Benjamin Van Durme

Researchers illustrate improvements in contextual encoding strategies via resultant performance on a battery of shared Natural Language Understanding (NLU) tasks.

Learning-To-Rank Natural Language Understanding

Improved Lexically Constrained Decoding for Translation and Monolingual Rewriting

1 code implementation NAACL 2019 J. Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, Benjamin Van Durme

Lexically-constrained sequence decoding allows for explicit positive or negative phrase-based constraints to be placed on target output strings in generation tasks such as machine translation or monolingual text rewriting.

Data Augmentation Machine Translation +3

AMR Parsing as Sequence-to-Graph Transduction

1 code implementation ACL 2019 Sheng Zhang, Xutai Ma, Kevin Duh, Benjamin Van Durme

Our experimental results outperform all previously reported SMATCH scores, on both AMR 2. 0 (76. 3% F1 on LDC2017T10) and AMR 1. 0 (70. 2% F1 on LDC2014T12).

AMR Parsing

Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling

no code implementations ICLR 2019 Samuel R. Bowman, Ellie Pavlick, Edouard Grave, Benjamin Van Durme, Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen

Work on the problem of contextualized word representation—the development of reusable neural network components for sentence understanding—has recently seen a surge of progress centered on the unsupervised pretraining task of language modeling with methods like ELMo (Peters et al., 2018).

Language Modelling Sentence

Probing What Different NLP Tasks Teach Machines about Function Word Comprehension

no code implementations SEMEVAL 2019 Najoung Kim, Roma Patel, Adam Poliak, Alex Wang, Patrick Xia, R. Thomas McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, Ellie Pavlick

Our results show that pretraining on language modeling performs the best on average across our probing tasks, supporting its widespread use for pretraining state-of-the-art NLP models, and CCG supertagging and NLI pretraining perform comparably.

CCG Supertagging Language Modelling +3

Fine-Grained Temporal Relation Extraction

no code implementations ACL 2019 Siddharth Vashishtha, Benjamin Van Durme, Aaron Steven White

We present a novel semantic framework for modeling temporal relations and event durations that maps pairs of events to real-valued scales.

Relation Temporal Relation Extraction +1

Decomposing Generalization: Models of Generic, Habitual, and Episodic Statements

no code implementations TACL 2019 Venkata Subrahmanyan Govindarajan, Benjamin Van Durme, Aaron Steven White

We present a novel semantic framework for modeling linguistic expressions of generalization---generic, habitual, and episodic statements---as combinations of simple, real-valued referential properties of predicates and their arguments.

Word Embeddings

Cross-lingual Decompositional Semantic Parsing

no code implementations EMNLP 2018 Sheng Zhang, Xutai Ma, Rachel Rudinger, Kevin Duh, Benjamin Van Durme

We introduce the task of cross-lingual decompositional semantic parsing: mapping content provided in a source language into a decompositional semantic analysis based on a target language.

Semantic Parsing

Lexicosyntactic Inference in Neural Models

no code implementations EMNLP 2018 Aaron Steven White, Rachel Rudinger, Kyle Rawlins, Benjamin Van Durme

We use this dataset, which we make publicly available, to probe the behavior of current state-of-the-art neural systems, showing that these systems make certain systematic errors that are clearly visible through the lens of factuality prediction.

Efficient Online Scalar Annotation with Bounded Support

no code implementations ACL 2018 Keisuke Sakaguchi, Benjamin Van Durme

We describe a novel method for efficiently eliciting scalar annotations for dataset construction and system quality estimation by human judgments.

Halo: Learning Semantics-Aware Representations for Cross-Lingual Information Extraction

no code implementations SEMEVAL 2018 Hongyuan Mei, Sheng Zhang, Kevin Duh, Benjamin Van Durme

Cross-lingual information extraction (CLIE) is an important and challenging task, especially in low resource scenarios.


On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference

1 code implementation NAACL 2018 Adam Poliak, Yonatan Belinkov, James Glass, Benjamin Van Durme

We propose a process for investigating the extent to which sentence representations arising from neural machine translation (NMT) systems encode distinct semantic phenomena.

Machine Translation Natural Language Inference +4

Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation

no code implementations EMNLP (ACL) 2018 Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, Benjamin Van Durme

We present a large-scale collection of diverse natural language inference (NLI) datasets that help provide insight into how well a sentence representation captures distinct types of reasoning.

Natural Language Inference Sentence

Cross-lingual Semantic Parsing

no code implementations21 Apr 2018 Sheng Zhang, Kevin Duh, Benjamin Van Durme

We introduce the task of cross-lingual semantic parsing: mapping content provided in a source language into a meaning representation based on a target language.

Semantic Parsing

Neural-Davidsonian Semantic Proto-role Labeling

1 code implementation EMNLP 2018 Rachel Rudinger, Adam Teichert, Ryan Culkin, Sheng Zhang, Benjamin Van Durme

We present a model for semantic proto-role labeling (SPRL) using an adapted bidirectional LSTM encoding strategy that we call "Neural-Davidsonian": predicate-argument structure is represented as pairs of hidden states corresponding to predicate and argument head tokens of the input sequence.


Neural models of factuality

1 code implementation NAACL 2018 Rachel Rudinger, Aaron Steven White, Benjamin Van Durme

We present two neural models for event factuality prediction, which yield significant performance gains over previous models on three event factuality datasets: FactBank, UW, and MEANTIME.

Selective Decoding for Cross-lingual Open Information Extraction

no code implementations IJCNLP 2017 Sheng Zhang, Kevin Duh, Benjamin Van Durme

Cross-lingual open information extraction is the task of distilling facts from the source language into representations in the target language.

Decoder Machine Translation +1

Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework

no code implementations IJCNLP 2017 Aaron Steven White, Pushpendre Rastogi, Kevin Duh, Benjamin Van Durme

We propose to unify a variety of existing semantic classification tasks, such as semantic role labeling, anaphora resolution, and paraphrase detection, under the heading of Recognizing Textual Entailment (RTE).

General Classification Image Captioning +4

Pocket Knowledge Base Population

no code implementations ACL 2017 Travis Wolfe, Mark Dredze, Benjamin Van Durme

Existing Knowledge Base Population methods extract relations from a closed relational schema with limited coverage leading to sparse KBs.

Knowledge Base Population Open Information Extraction +1

Error-repair Dependency Parsing for Ungrammatical Texts

1 code implementation ACL 2017 Keisuke Sakaguchi, Matt Post, Benjamin Van Durme

We propose a new dependency parsing scheme which jointly parses a sentence and repairs grammatical errors by extending the non-directional transition-based formalism of Goldberg and Elhadad (2010) with three additional actions: SUBSTITUTE, DELETE, INSERT.

Dependency Parsing Sentence

Streaming Word Embeddings with the Space-Saving Algorithm

2 code implementations24 Apr 2017 Chandler May, Kevin Duh, Benjamin Van Durme, Ashwin Lall

We develop a streaming (one-pass, bounded-memory) word embedding algorithm based on the canonical skip-gram with negative sampling algorithm implemented in word2vec.

Word Embeddings

Social Bias in Elicited Natural Language Inferences

1 code implementation WS 2017 Rachel Rudinger, Ch May, ler, Benjamin Van Durme

We analyze the Stanford Natural Language Inference (SNLI) corpus in an investigation of bias and stereotyping in NLP data.

Language Modelling Natural Language Inference +1

Efficient, Compositional, Order-sensitive n-gram Embeddings

1 code implementation EACL 2017 Adam Poliak, Pushpendre Rastogi, M. Patrick Martin, Benjamin Van Durme

We propose ECO: a new way to generate embeddings for phrases that is Efficient, Compositional, and Order-sensitive.

Word Embeddings

Discriminative Information Retrieval for Question Answering Sentence Selection

1 code implementation EACL 2017 Tongfei Chen, Benjamin Van Durme

We propose a framework for discriminative IR atop linguistic features, trained to improve the recall of answer candidate passage retrieval, the initial step in text-based question answering.

Passage Retrieval Question Answering +2

The Semantic Proto-Role Linking Model

no code implementations EACL 2017 Aaron Steven White, Kyle Rawlins, Benjamin Van Durme

We propose the semantic proto-role linking model, which jointly induces both predicate-specific semantic roles and predicate-general semantic proto-roles based on semantic proto-role property likelihood judgments.

Semantic Role Labeling

Feature Generation for Robust Semantic Role Labeling

no code implementations22 Feb 2017 Travis Wolfe, Mark Dredze, Benjamin Van Durme

Hand-engineered feature sets are a well understood method for creating robust NLP models, but they require a lot of expertise and effort to create.

Semantic Role Labeling

Ordinal Common-sense Inference

no code implementations TACL 2017 Sheng Zhang, Rachel Rudinger, Kevin Duh, Benjamin Van Durme

Humans have the capacity to draw common-sense inferences from natural language: various things that are likely but not certain to hold based on established discourse, and are rarely stated explicitly.

Common Sense Reasoning Natural Language Inference

Computational linking theory

no code implementations8 Oct 2016 Aaron Steven White, Drew Reisinger, Rachel Rudinger, Kyle Rawlins, Benjamin Van Durme

A linking theory explains how verbs' semantic arguments are mapped to their syntactic arguments---the inverse of the Semantic Role Labeling task from the shallow semantic parsing literature.

Semantic Parsing Semantic Role Labeling

Robsut Wrod Reocginiton via semi-Character Recurrent Neural Network

1 code implementation7 Aug 2016 Keisuke Sakaguchi, Kevin Duh, Matt Post, Benjamin Van Durme

Inspired by the findings from the Cmabrigde Uinervtisy effect, we propose a word recognition model based on a semi-character level recurrent neural network (scRNN).

Spelling Correction

A Critical Examination of RESCAL for Completion of Knowledge Bases with Transitive Relations

no code implementations16 May 2016 Pushpendre Rastogi, Benjamin Van Durme

Link prediction in large knowledge graphs has received a lot of attention recently because of its importance for inferring missing relations and for completing and improving noisily extracted knowledge graphs.

Knowledge Graphs Link Prediction

Sublinear Partition Estimation

2 code implementations7 Aug 2015 Pushpendre Rastogi, Benjamin Van Durme

The output scores of a neural network classifier are converted to probabilities via normalizing over the scores of all competing categories.

Language Modelling Object Recognition