Search Results for author: Jacob Eisenstein

Found 89 papers, 28 papers with code

On Writing a Textbook on Natural Language Processing

no code implementations NAACL (TeachingNLP) 2021 Jacob Eisenstein

There are thousands of papers about natural language processing and computational linguistics, but very few textbooks.

Transforming and Combining Rewards for Aligning Large Language Models

no code implementations1 Feb 2024 ZiHao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D'Amour, Sanmi Koyejo, Victor Veitch

A common approach for aligning language models to human preferences is to first learn a reward model from preference data, and then use this reward model to update the language model.

Language Modelling

Theoretical guarantees on the best-of-n alignment policy

no code implementations3 Jan 2024 Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D'Amour, Jacob Eisenstein, Chirag Nagpal, Ananda Theertha Suresh

A commonly used analytical expression in the literature claims that the KL divergence between the best-of-$n$ policy and the base policy is equal to $\log (n) - (n-1)/n.$ We disprove the validity of this claim, and show that it is an upper bound on the actual KL divergence.

Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

no code implementations14 Dec 2023 Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D'Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, Jonathan Berant

However, even pretrain reward ensembles do not eliminate reward hacking: we show several qualitative reward hacking phenomena that are not mitigated by ensembling because all reward models in the ensemble exhibit similar error patterns.

Language Modelling

Selectively Answering Ambiguous Questions

no code implementations24 May 2023 Jeremy R. Cole, Michael J. Q. Zhang, Daniel Gillick, Julian Martin Eisenschlos, Bhuwan Dhingra, Jacob Eisenstein

We investigate question answering from this perspective, focusing on answering a subset of questions with a high degree of accuracy, from a set of questions in which many are inherently ambiguous.

Question Answering

MD3: The Multi-Dialect Dataset of Dialogues

no code implementations19 May 2023 Jacob Eisenstein, Vinodkumar Prabhakaran, Clara Rivera, Dorottya Demszky, Devyani Sharma

We introduce a new dataset of conversational speech representing English from India, Nigeria, and the United States.

Dialect-robust Evaluation of Generated Text

no code implementations2 Nov 2022 Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Timothy Dozat, Dan Garrette, Aditya Siddhant, Jacob Eisenstein, Sebastian Gehrmann

Evaluation metrics that are not robust to dialect variation make it impossible to tell how well systems perform for many groups of users, and can even penalize systems for producing text in lower-resource dialects.

nlg evaluation

Predicting Long-Term Citations from Short-Term Linguistic Influence

1 code implementation24 Oct 2022 Sandeep Soni, David Bamman, Jacob Eisenstein

A standard measure of the influence of a research paper is the number of times it is cited.

Informativeness and Invariance: Two Perspectives on Spurious Correlations in Natural Language

no code implementations NAACL 2022 Jacob Eisenstein

Spurious correlations are a threat to the trustworthiness of natural language processing systems, motivating research into methods for identifying and eliminating them.

Informativeness Vocal Bursts Valence Prediction

Learning to Look Inside: Augmenting Token-Based Encoders with Character-Level Information

no code implementations1 Aug 2021 Yuval Pinter, Amanda Stent, Mark Dredze, Jacob Eisenstein

Commonly-used transformer language models depend on a tokenization schema which sets an unchangeable subword vocabulary prior to pre-training, destined to be applied to all downstream tasks regardless of domain shift, novel word formations, or other sources of vocabulary mismatch.

Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer

no code implementations30 Jun 2021 Iulia Turc, Kenton Lee, Jacob Eisenstein, Ming-Wei Chang, Kristina Toutanova

Zero-shot cross-lingual transfer is emerging as a practical solution: pre-trained models later fine-tuned on one transfer language exhibit surprising performance when tested on many target languages.

Question Answering Zero-Shot Cross-Lingual Transfer

Time-Aware Language Models as Temporal Knowledge Bases

no code implementations29 Jun 2021 Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, William W. Cohen

We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum -- those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data.


Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests

no code implementations NeurIPS 2021 Victor Veitch, Alexander D'Amour, Steve Yadlowsky, Jacob Eisenstein

We introduce counterfactual invariance as a formalization of the requirement that changing irrelevant parts of the input shouldn't change model predictions.

Causal Inference counterfactual +2

Counterfactual Invariance to Spurious Correlations in Text Classification

no code implementations NeurIPS 2021 Victor Veitch, Alexander D'Amour, Steve Yadlowsky, Jacob Eisenstein

We introduce counterfactual invariance as a formalization of the requirement that changing irrelevant parts of the input shouldn't change model predictions.

Causal Inference counterfactual +2

Abolitionist Networks: Modeling Language Change in Nineteenth-Century Activist Newspapers

1 code implementation12 Mar 2021 Sandeep Soni, Lauren Klein, Jacob Eisenstein

This paper supplements recent qualitative work on the role of women in abolition's vanguard, as well as the role of the Black press, with a quantitative text modeling approach.

Diachronic Word Embeddings Word Embeddings

Tuiteamos o pongamos un tuit? Investigating the Social Constraints of Loanword Integration in Spanish Social Media

no code implementations SCiL 2021 Ian Stewart, Diyi Yang, Jacob Eisenstein

In social media, we find that speaker background and expectations of formality explain loanword and native word integration, such that authors who use more Spanish and who write to a wider audience tend to use integrated verb forms more often.

Learning to Recognize Dialect Features

no code implementations NAACL 2021 Dorottya Demszky, Devyani Sharma, Jonathan H. Clark, Vinodkumar Prabhakaran, Jacob Eisenstein

Evaluation on a test set of 22 dialect features of Indian English demonstrates that these models learn to recognize many features with high accuracy, and that a few minimal pairs can be as effective for training as thousands of labeled examples.

Will it Unblend?

1 code implementation SCiL 2021 Yuval Pinter, Cassandra L. Jacobs, Jacob Eisenstein

Natural language processing systems often struggle with out-of-vocabulary (OOV) terms, which do not appear in training data.

Sparse, Dense, and Attentional Representations for Text Retrieval

1 code implementation1 May 2020 Yi Luan, Jacob Eisenstein, Kristina Toutanova, Michael Collins

Dual encoders perform retrieval by encoding documents and queries into dense lowdimensional vectors, scoring each document by its inner product with the query.

Open-Domain Question Answering Retrieval +1

Characterizing Collective Attention via Descriptor Context: A Case Study of Public Discussions of Crisis Events

1 code implementation19 Sep 2019 Ian Stewart, Diyi Yang, Jacob Eisenstein

But according to rationalist models of natural language communication, the collective salience of each entity will be expressed not only in how often it is mentioned, but in the form that those mentions take.

Follow the Leader: Documents on the Leading Edge of Semantic Change Get More Citations

1 code implementation9 Sep 2019 Sandeep Soni, Kristina Lerman, Jacob Eisenstein

However, simply knowing that a word has changed in meaning is insufficient to identify the instances of word usage that convey the historical or the newer meaning.

Diachronic Word Embeddings Word Embeddings

How we do things with words: Analyzing text as social and cultural data

no code implementations2 Jul 2019 Dong Nguyen, Maria Liakata, Simon DeDeo, Jacob Eisenstein, David Mimno, Rebekah Tromble, Jane Winters

Second, we hope to provide a set of best practices for working with thick social and cultural concepts.

Measuring and Modeling Language Change

no code implementations NAACL 2019 Jacob Eisenstein

Such questions are fundamental to the social sciences and humanities, and scholars in these disciplines are increasingly turning to computational techniques for answers.

Causal Inference Two-sample testing +1

Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling

1 code implementation IJCNLP 2019 Xiaochuang Han, Jacob Eisenstein

To address this scenario, we propose domain-adaptive fine-tuning, in which the contextualized embeddings are adapted by masked language modeling on text from the target domain.

Language Modelling Masked Language Modeling +3

Character Eyes: Seeing Language through Character-Level Taggers

1 code implementation WS 2019 Yuval Pinter, Marc Marone, Jacob Eisenstein

Character-level models have been used extensively in recent years in NLP tasks as both supplements and replacements for closed-vocabulary token-level word representations.


Interactional Stancetaking in Online Forums

no code implementations CL 2018 Scott F. Kiesling, Umashanthi Pavalanathan, Jim Fitzpatrick, Xiaochuang Han, Jacob Eisenstein

Theories of interactional stancetaking have been put forward as holistic accounts, but until now, these theories have been applied only through detailed qualitative analysis of (portions of) a few individual conversations.

Predicting Semantic Relations using Global Graph Properties

1 code implementation EMNLP 2018 Yuval Pinter, Jacob Eisenstein

Semantic graphs, such as WordNet, are resources which curate natural language on two distinguishable layers.

Link Prediction

Si O No, Que Penses? Catalonian Independence and Linguistic Identity on Social Media

no code implementations NAACL 2018 Ian Stewart, Yuval Pinter, Jacob Eisenstein

We also find that Catalan is used more often in referendum-related discourse than in other contexts, contrary to prior findings on language variation.

Stylistic Variation in Social Media Part-of-Speech Tagging

no code implementations WS 2018 Murali Raghu Babu Balusu, Taha Merghani, Jacob Eisenstein

While prior work found that similar approaches yield performance improvements in sentiment analysis and entity linking, we were unable to obtain performance improvements in part-of-speech tagging, despite strong evidence for the link between part-of-speech error rates and social network structure.

Entity Linking Part-Of-Speech Tagging +1

Sí o no, què penses? Catalonian Independence and Linguistic Identity on Social Media

1 code implementation13 Apr 2018 Ian Stewart, Yuval Pinter, Jacob Eisenstein

We also find that Catalan is used more often in referendum-related discourse than in other contexts, contrary to prior findings on language variation.

Detecting Social Influence in Event Cascades by Comparing Discriminative Rankers

1 code implementation16 Feb 2018 Sandeep Soni, Shawn Ling Ramirez, Jacob Eisenstein

However, detecting social influence from observational data is challenging due to confounds like homophily and practical issues like missing data.

Explainable Prediction of Medical Codes from Clinical Text

3 code implementations NAACL 2018 James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, Jacob Eisenstein

Our method aggregates information across the document using a convolutional neural network, and uses an attention mechanism to select the most relevant segments for each of the thousands of possible codes.

Medical Code Prediction

#anorexia, #anarexia, #anarexyia: Characterizing Online Community Practices with Orthographic Variation

no code implementations4 Dec 2017 Ian Stewart, Stevie Chancellor, Munmun De Choudhury, Jacob Eisenstein

We also demonstrate the utility of orthographic variation as a new lens to study sociolinguistic change in online communities, particularly when the change results from an exogenous force such as a content ban.

Making "fetch" happen: The influence of social and linguistic context on nonstandard word growth and decline

1 code implementation1 Sep 2017 Ian Stewart, Jacob Eisenstein

In an online community, new words come and go: today's "haha" may be replaced by tomorrow's "lol."

Mimicking Word Embeddings using Subword RNNs

2 code implementations EMNLP 2017 Yuval Pinter, Robert Guthrie, Jacob Eisenstein

In this paper, we present MIMICK, an approach to generating OOV word embeddings compositionally, by learning a function from spellings to distributional embeddings.

Word Embeddings

A Multidimensional Lexicon for Interpersonal Stancetaking

no code implementations ACL 2017 Umashanthi Pavalanathan, Jim Fitzpatrick, Scott Kiesling, Jacob Eisenstein

The sociolinguistic construct of stancetaking describes the activities through which discourse participants create and signal relationships to their interlocutors, to the topic of discussion, and to the talk itself.

Word Embeddings

Unsupervised Learning for Lexicon-Based Classification

1 code implementation21 Nov 2016 Jacob Eisenstein

In lexicon-based classification, documents are assigned labels by comparing the number of words that appear from two opposed lexicons, such as positive and negative sentiment.

Classification General Classification

The Social Dynamics of Language Change in Online Networks

no code implementations7 Sep 2016 Rahul Goel, Sandeep Soni, Naman Goyal, John Paparrizos, Hanna Wallach, Fernando Diaz, Jacob Eisenstein

Language change is a complex social phenomenon, revealing pathways of communication and sociocultural influence.

Shallow Discourse Parsing Using Distributed Argument Representations and Bayesian Optimization

no code implementations14 Jun 2016 Akanksha, Jacob Eisenstein

This paper describes the Georgia Tech team's approach to the CoNLL-2016 supplementary evaluation on discourse relation sense classification.

Bayesian Optimization Discourse Parsing +2

Part-of-Speech Tagging for Historical English

no code implementations NAACL 2016 Yi Yang, Jacob Eisenstein

We evaluate several domain adaptation methods on the task of tagging Early Modern English and Modern British English texts in the Penn Corpora of Historical English.

Part-Of-Speech Tagging Unsupervised Domain Adaptation +1

A Latent Variable Recurrent Neural Network for Discourse Relation Language Models

1 code implementation7 Mar 2016 Yangfeng Ji, Gholamreza Haffari, Jacob Eisenstein

This paper presents a novel latent variable recurrent neural network architecture for jointly modeling sequences of words and (possibly latent) discourse relations between adjacent sentences.

Classification Dialog Act Classification +4

A Kernel Independence Test for Geographical Language Variation

1 code implementation CL 2017 Dong Nguyen, Jacob Eisenstein

Quantifying the degree of spatial dependence for linguistic variables is a key task for analyzing dialectal variation.

Nonparametric Bayesian Storyline Detection from Microtexts

no code implementations WS 2016 Vinodh Krishnan, Jacob Eisenstein

News events and social media are composed of evolving storylines, which capture public attention for a limited period of time.

Clustering Retrieval

Overcoming Language Variation in Sentiment Analysis with Social Attention

1 code implementation TACL 2017 Yi Yang, Jacob Eisenstein

Variation in language is ubiquitous, particularly in newer forms of writing such as social media.

Sentiment Analysis

Document Context Language Models

1 code implementation12 Nov 2015 Yangfeng Ji, Trevor Cohn, Lingpeng Kong, Chris Dyer, Jacob Eisenstein

Text documents are structured on multiple levels of detail: individual words are related by syntax, but larger units of text are related by discourse structure.


Emoticons vs. Emojis on Twitter: A Causal Inference Approach

no code implementations28 Oct 2015 Umashanthi Pavalanathan, Jacob Eisenstein

Online writing lacks the non-verbal cues present in face-to-face communication, which provide additional contextual information about the utterance, such as the speaker's intention or affective state.

Causal Inference

Confounds and Consequences in Geotagged Twitter Data

no code implementations EMNLP 2015 Umashanthi Pavalanathan, Jacob Eisenstein

Twitter is often used in quantitative studies that identify geographically-preferred topics, writing styles, and entities.

One Vector is Not Enough: Entity-Augmented Distributed Semantics for Discourse Relations

no code implementations TACL 2015 Yangfeng Ji, Jacob Eisenstein

A more subtle challenge is that it is not enough to represent the meaning of each argument of a discourse relation, because the relation may depend on links between lowerlevel components, such as entity mentions.

Question Answering Relation +1

Entity-Augmented Distributional Semantics for Discourse Relations

no code implementations17 Dec 2014 Yangfeng Ji, Jacob Eisenstein

A more subtle challenge is that it is not enough to represent the meaning of each sentence of a discourse relation, because the relation may depend on links between lower-level elements, such as entity mentions.

Relation Sentence

Unsupervised Domain Adaptation with Feature Embeddings

1 code implementation14 Dec 2014 Yi Yang, Jacob Eisenstein

Representation learning is the dominant technique for unsupervised domain adaptation, but existing approaches often require the specification of "pivot features" that generalize across domains, which are selected by task-specific heuristics.

Representation Learning Unsupervised Domain Adaptation

Learning Document-Level Semantic Properties from Free-Text Annotations

no code implementations15 Jan 2014 S. R. K. Branavan, Harr Chen, Jacob Eisenstein, Regina Barzilay

The paraphrase structure is linked with a latent topic model of the review texts, enabling the system to predict the properties of unannotated documents and to effectively aggregate the semantic properties of multiple reviews.


Diffusion of Lexical Change in Social Media

no code implementations18 Oct 2012 Jacob Eisenstein, Brendan O'Connor, Noah A. Smith, Eric P. Xing

Computer-mediated communication is driving fundamental changes in the nature of written language.

Gender identity and lexical variation in social media

1 code implementation16 Oct 2012 David Bamman, Jacob Eisenstein, Tyler Schnoebelen

Examining individuals whose language does not match the classifier's model for their gender, we find that they have social networks that include significantly fewer same-gender social connections and that, in general, social network homophily is correlated with the use of same-gender language markers.


Cannot find the paper you are looking for? You can Submit a new open access paper.