Search Results for author:
Found 81 papers, 55 papers with code
How Much Do Modifications to Transformer Language Models Affect Their Ability to Learn Linguistic Knowledge?
Recent progress in large pretrained language models (LMs) has led to a growth of analyses examining what kinds of linguistic knowledge are encoded by these models.
IGA: An Intent-Guided Authoring Assistant
While large-scale pretrained language models have significantly improved writing assistance functionalities such as autocomplete, more complex and controllable writing assistants have yet to be explored.
Unsupervised Parsing with S-DIORA: Single Tree Encoding for Deep Inside-Outside Recursive Autoencoders
The deep inside-outside recursive autoencoder (DIORA; Drozdov et al. 2019) is a self-supervised neural model that learns to induce syntactic tree structures for input sentences *without access to labeled training data*.
FABLES: Evaluating faithfulness and content selection in book-length summarization
While LLM-based auto-raters have proven reliable for factuality and coherence in other settings, we implement several LLM raters of faithfulness and find that none correlates strongly with human annotations, especially with regard to detecting unfaithful claims.
GEE! Grammar Error Explanation with Large Language Models
To address this gap, we propose the task of grammar error explanation, where a system needs to provide one-sentence explanations for each grammatical error in a pair of erroneous and corrected sentences.
Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generation
In this paper, we present the discovery that a student model distilled from a few-shot prompted LLM can commonly generalize better than its teacher to unseen examples on such tasks.
TopicGPT: A Prompt-based Topic Modeling Framework
Topic modeling is a well-established technique for exploring text corpora.
PaRaDe: Passage Ranking using Demonstrations with Large Language Models
Recent studies show that large language models (LLMs) can be instructed to effectively perform zero-shot passage re-ranking, in which the results of a first stage retrieval method, such as BM25, are rated and reordered to improve relevance.
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Specifically, we introduce FreshQA, a novel dynamic QA benchmark encompassing a diverse range of question and answer types, including questions that require fast-changing world knowledge as well as questions with false premises that need to be debunked.
BooookScore: A systematic exploration of book-length summarization in the era of LLMs
We find that closed-source LLMs such as GPT-4 and Claude 2 produce summaries with higher BooookScore than those generated by open-source models.
Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF
During the last stage of RLHF, a large language model is aligned to human intents via PPO training, a process that generally requires large-scale computational resources.
A Critical Evaluation of Evaluations for Long-form Question Answering
We present a careful analysis of experts' evaluation, which focuses on new aspects such as the comprehensiveness of the answer.
KNN-LM Does Not Improve Open-ended Text Generation
Digging deeper, we find that interpolating with a retrieval distribution actually increases perplexity compared to a baseline Transformer LM for the majority of tokens in the WikiText-103 test set, even though the overall perplexity is lower due to a smaller number of tokens for which perplexity dramatically decreases after interpolation.
PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents
PEARL outperforms zero-shot and chain-of-thought prompting on this dataset, and ablation experiments show that each stage of PEARL is critical to its performance.
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Evaluating the factuality of long-form text generated by large language models (LMs) is non-trivial because (1) generations often contain a mixture of supported and unsupported pieces of information, making binary judgments of quality inadequate, and (2) human evaluation is time-consuming and costly.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Large language models (LLMs) are competitive with the state of the art on a wide range of sentence-level translation datasets.
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
To increase the robustness of AI-generated text detection to paraphrase attacks, we introduce a simple defense that relies on retrieving semantically-similar generations and must be maintained by a language model API provider.
Stealing the Decoding Algorithms of Language Models
A key component of generating text from modern language models (LM) is the selection and tuning of decoding algorithms.
How Does In-Context Learning Help Prompt Tuning?
This motivates the use of parameter-efficient adaptation methods such as prompt tuning (PT), which adds a small number of tunable embeddings to an otherwise frozen model, and in-context learning (ICL), in which demonstrations of the task are provided to the model in natural language without any additional training.
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Motivated by our survey, we present LongEval, a set of guidelines for human evaluation of faithfulness in long-form summaries that addresses the following challenges: (1) How can we achieve high inter-annotator agreement on faithfulness scores?
You can't pick your neighbors, or can you? When and how to rely on retrieval in the $k$NN-LM
Retrieval-enhanced language models (LMs), which condition their predictions on text retrieved from large external datastores, have recently shown significant perplexity improvements compared to standard LMs.
Ranked #9 on
Language Modelling
on WikiText-103
DEMETR: Diagnosing Evaluation Metrics for Translation
While machine translation evaluation metrics based on string overlap (e. g., BLEU) have their limitations, their computations are transparent: the BLEU score assigned to a particular candidate translation can be traced back to the presence or absence of certain words.
Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature
Using Par3, we discover that expert literary translators prefer reference human translations over machine-translated paragraphs at a rate of 84%, while state-of-the-art automatic MT metrics do not correlate with those preferences.
SLING: Sino Linguistic Evaluation of Large Language Models
To understand what kinds of linguistic knowledge are encoded by pretrained Chinese language models (LMs), we introduce the benchmark of Sino LINGuistics (SLING), which consists of 38K minimal sentence pairs in Mandarin Chinese grouped into 9 high-level linguistic phenomena.
ezCoref: Towards Unifying Annotation Guidelines for Coreference Resolution
Large-scale, high-quality corpora are critical for advancing research in coreference resolution.
Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation
In this paper, we explore the challenging problem of performing a generative task in a target language when labeled data is only available in English, using summarization as a case study.
RankGen: Improving Text Generation with Large Ranking Models
Given an input sequence (or prefix), modern language models often assign high probabilities to output sequences that are repetitive, incoherent, or irrelevant to the prefix; as such, model-generated text also contains such artifacts.
Modeling Exemplification in Long-form Question Answering via Retrieval
We show that not only do state-of-the-art LFQA models struggle to generate relevant examples, but also that standard evaluation metrics such as ROUGE are insufficient to judge exemplification quality.
ChapterBreak: A Challenge Dataset for Long-Range Language Models
While numerous architectures for long-range language models (LRLMs) have recently been proposed, a meaningful evaluation of their discourse-level language understanding capabilities has not yet followed.
RELIC: Retrieving Evidence for Literary Claims
Humanities scholars commonly provide evidence for claims that they make about a work of literature (e. g., a novel) in the form of quotations from the work.
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation
Recent text generation research has increasingly focused on open-ended domains such as story and poetry generation.
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning
Despite their recent successes in tackling many NLP tasks, large-scale pre-trained language models do not perform as well in few-shot settings where only a handful of training examples are available.
Ranked #1 on
Few-Shot NLI
on SNLI (8 training examples per class)
Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration
Phrase representations derived from BERT often do not exhibit complex phrasal compositionality, as the model relies instead on lexical similarity to determine semantic relatedness.
Improved Latent Tree Induction with Distant Supervision via Span Constraints
For over thirty years, researchers have developed and analyzed methods for latent tree induction as an approach for unsupervised syntactic parsing.
TABBIE: Pretrained Representations of Tabular Data
Existing work on tabular representation learning jointly models tables and associated text using self-supervised objective functions derived from pretrained language models such as BERT.
Ranked #1 on
Column Type Annotation
on VizNet-Sato-Full
(Weighted-F1 metric)
IGA : An Intent-Guided Authoring Assistant
While large-scale pretrained language models have significantly improved writing assistance functionalities such as autocomplete, more complex and controllable writing assistants have yet to be explored.
Revisiting Simple Neural Probabilistic Language Models
Recent progress in language modeling has been driven not only by advances in neural architectures, but also through hardware and optimization improvements.
Ranked #59 on
Language Modelling
on WikiText-103
Changing the Mind of Transformers for Topically-Controllable Language Generation
Our framework consists of two components: (1) a method that produces a set of candidate topics by predicting the centers of word clusters in the possible continuations, and (2) a text generation model whose output adheres to the chosen topics.
Hurdles to Progress in Long-form Question Answering
The task of long-form question answering (LFQA) involves retrieving documents relevant to a given question and using them to generate a paragraph-length answer.
Ranked #3 on
Question Answering
on KILT: ELI5
Weakly-Supervised Open-Retrieval Conversational Question Answering
Our method is more flexible as it can handle both span answers and freeform answers.
Analyzing Gender Bias within Narrative Tropes
Popular media reflects and reinforces societal biases through the use of tropes, which are narrative elements, such as archetypal characters and plot arcs, that occur frequently across media.
Reformulating Unsupervised Style Transfer as Paraphrase Generation
Modern NLP defines the task of style transfer as modifying the style of a given sentence without appreciably changing its semantics, which implies that the outputs of style transfer systems should be paraphrases of their inputs.
STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation
Systems for story generation are asked to produce plausible and enjoyable stories given an input context.
Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models
To benefit from this observation, we train an energy-based model to mimic the behavior of the task measure (i. e., the energy-based model assigns lower energy to samples with higher BLEU score), which is resulted in a re-ranking algorithm based on the samples drawn from NMT: energy-based re-ranking (EBR).
Open-Retrieval Conversational Question Answering
We build an end-to-end system for ORConvQA, featuring a retriever, a reranker, and a reader that are all based on Transformers.
Hard-Coded Gaussian Attention for Neural Machine Translation
Recent work has questioned the importance of the Transformer's multi-headed attention for achieving high translation quality.
Exploring and Predicting Transferability across NLP Tasks
We also develop task embeddings that can be used to predict the most transferable source tasks for a given target task, and we validate their effectiveness in experiments controlled for source and target data size.
Which Evaluations Uncover Sense Representations that Actually Make Sense?
Text representations are critical for modern natural language processing.
Unsupervised Labeled Parsing with Deep Inside-Outside Recursive Autoencoders
Understanding text often requires identifying meaningful constituent spans such as noun phrases and verb phrases.
Thieves on Sesame Street! Model Extraction of BERT-based APIs
We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model.
Investigating Sports Commentator Bias within a Large Corpus of American Football Broadcasts
Sports broadcasters inject drama into play-by-play commentary by building team and player narratives through subjective analyses and anecdotes.
Attentive History Selection for Conversational Question Answering
First, we propose a positional history answer embedding method to encode conversation history with position information using BERT in a natural way.
Encouraging Paragraph Embeddings to Remember Sentence Identity Improves Classification
While paragraph embedding models are remarkably effective for downstream classification tasks, what they learn and encode into a single vector remains opaque.
Generating Question-Answer Hierarchies
The process of knowledge acquisition can be viewed as a question-answer game between a student and a teacher in which the student typically starts by asking broad, open-ended questions before drilling down into specifics (Hintikka, 1981; Hakkarainen and Sintonen, 2002).
Syntactically Supervised Transformers for Faster Neural Machine Translation
Standard decoders for neural machine translation autoregressively generate a single target token per time step, which slows inference especially for long outputs.
Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Auto-Encoders
We introduce the deep inside-outside recursive autoencoder (DIORA), a fully-unsupervised method for discovering syntax that simultaneously learns representations for constituents within the induced tree.
BERT with History Answer Embedding for Conversational Question Answering
One of the major challenges to multi-turn conversational search is to model the conversation history to answer the current question.
Casting Light on Invisible Cities: Computationally Engaging with Literary Criticism
Literary critics often attempt to uncover meaning in a single work of literature through careful reading and analysis.
Quizbowl: The Case for Incremental Question Answering
Throughout this paper, we show that collaborations with the vibrant trivia community have contributed to the quality of our dataset, spawned new research directions, and doubled as an exciting way to engage the public with research in machine learning and natural language processing.
Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoders
We introduce deep inside-outside recursive autoencoders (DIORA), a fully-unsupervised method for discovering syntax that simultaneously learns representations for constituents within the induced tree.
QuAC: Question Answering in Context
We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total).
A Differentiable Self-disambiguated Sense Embedding Model via Scaled Gumbel Softmax
We present a differentiable multi-prototype word representation model that disentangles senses of polysemous words and produces meaningful sense-specific embeddings without external resources.
Revisiting the Importance of Encoding Logic Rules in Sentiment Classification
We analyze the performance of different sentiment classification models on syntactically complex inputs like A-but-B sentences.
QuAC : Question Answering in Context
We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total).
Inducing and Embedding Senses with Scaled Gumbel Softmax
Methods for learning word sense embeddings represent a single word with multiple sense-specific vectors.
Pathologies of Neural Models Make Interpretations Difficult
In existing interpretation methods for NLP, a word's importance is determined by either input perturbation---measuring the decrease in model confidence when that word is removed---or by the gradient with respect to that word.
Learning to Color from Language
Automatic colorization is the process of adding color to greyscale images.
Adversarial Example Generation with Syntactically Controlled Paraphrase Networks
We propose syntactically controlled paraphrase networks (SCPNs) and use them to generate adversarial examples.
Deep contextualized word representations
We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e. g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i. e., to model polysemy).
Ranked #3 on
Only Connect Walls Dataset Task 1 (Grouping)
on OCW
(Wasserstein Distance (WD) metric, using extra
training data)
Citation Intent Classification
Conversational Response Selection
+8
Search-based Neural Structured Learning for Sequential Question Answering
Recent work in semantic parsing for question answering has focused on long and complicated questions, many of which would seem unnatural if asked in a normal conversation between two humans.
The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
While computers can now describe what is explicitly depicted in natural images, in this paper we examine whether they can understand the closure-driven narratives conveyed by stylized artwork and dialogue in comic book panels.
Answering Complicated Question Intents Expressed in Decomposed Question Sequences
Recent work in semantic parsing for question answering has focused on long and complicated questions, many of which would seem unnatural if asked in a normal conversation between two humans.
Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
Most tasks in natural language processing can be cast into question answering (QA) problems over language input.
Ranked #66 on
Sentiment Analysis
on SST-2 Binary classification
