Search Results for author: Tatsunori Hashimoto

Found 45 papers, 27 papers with code

Removing RLHF Protections in GPT-4 via Fine-Tuning

no code implementations9 Nov 2023 Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, Daniel Kang

In tandem, LLM vendors have been increasingly enabling fine-tuning of their most powerful models.

On the Fairness ROAD: Robust Optimization for Adversarial Debiasing

1 code implementation27 Oct 2023 Vincent Grari, Thibault Laugel, Tatsunori Hashimoto, Sylvain Lamprier, Marcin Detyniecki

In the field of algorithmic fairness, significant attention has been put on group fairness criteria, such as Demographic Parity and Equalized Odds.


Learning to (Learn at Test Time)

1 code implementation20 Oct 2023 Yu Sun, Xinhao Li, Karan Dalal, Chloe Hsu, Sanmi Koyejo, Carlos Guestrin, Xiaolong Wang, Tatsunori Hashimoto, Xinlei Chen

Our inner loop turns out to be equivalent to linear attention when the inner-loop learner is only a linear model, and to self-attention when it is a kernel estimator.


Benchmarking and Improving Generator-Validator Consistency of Language Models

no code implementations3 Oct 2023 Xiang Lisa Li, Vaishnavi Shrivastava, Siyan Li, Tatsunori Hashimoto, Percy Liang

To improve the consistency of LMs, we propose to finetune on the filtered generator and validator responses that are GV-consistent, and call this approach consistency fine-tuning.

Benchmarking Instruction Following

Identifying the Risks of LM Agents with an LM-Emulated Sandbox

1 code implementation25 Sep 2023 Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, Tatsunori Hashimoto

Alongside the emulator, we develop an LM-based automatic safety evaluator that examines agent failures and quantifies associated risks.

Language Modelling Test +1

Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions

1 code implementation14 Sep 2023 Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, James Zou

Training large language models to follow instructions makes them perform better on a wide range of tasks, generally becoming more helpful.

Where's the Liability in Harmful AI Speech?

no code implementations9 Aug 2023 Peter Henderson, Tatsunori Hashimoto, Mark Lemley

We argue that AI should not be categorically immune from liability in these scenarios and that as courts grapple with the already fine-grained complexities of platform algorithms, the technical details of generative AI loom above with thornier questions.

Robust Distortion-free Watermarks for Language Models

2 code implementations28 Jul 2023 Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, Percy Liang

We generate watermarked text by mapping a sequence of random numbers -- which we compute using a randomized watermark key -- to a sample from the language model.

Language Modelling

Whose Opinions Do Language Models Reflect?

1 code implementation30 Mar 2023 Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, Tatsunori Hashimoto

Language models (LMs) are increasingly being used in open-ended contexts, where the opinions reflected by LMs in response to subjective queries can have a profound impact, both on user satisfaction, as well as shaping the views of society at large.

Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models

no code implementations26 Feb 2023 Kaitlyn Zhou, Dan Jurafsky, Tatsunori Hashimoto

The increased deployment of LMs for real-world tasks involving knowledge and facts makes it important to understand model epistemology: what LMs think they know, and how their attitudes toward that knowledge are affected by language use in their inputs.

Decision Making Question Answering +1

Out-of-Domain Robustness via Targeted Augmentations

1 code implementation23 Feb 2023 Irena Gao, Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, Percy Liang

Models trained on one set of domains often suffer performance drops on unseen domains, e. g., when wildlife monitoring models are deployed in new camera locations.

Evaluating Self-Supervised Learning via Risk Decomposition

1 code implementation6 Feb 2023 Yann Dubois, Tatsunori Hashimoto, Percy Liang

Our decomposition consists of four error components: approximation, representation usability, probe generalization, and encoder generalization.

Representation Learning Self-Supervised Learning

Contrastive Error Attribution for Finetuned Language Models

no code implementations21 Dec 2022 Faisal Ladhak, Esin Durmus, Tatsunori Hashimoto

We show that existing approaches for error tracing, such as gradient-based influence measures, do not perform reliably for detecting faithfulness errors in NLG datasets.

Text Generation Text Summarization

Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale

1 code implementation7 Nov 2022 Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, Aylin Caliskan

For example, we find cases of prompting for basic traits or social roles resulting in images reinforcing whiteness as ideal, prompting for occupations resulting in amplification of racial and gender disparities, and prompting for objects resulting in reification of American norms.

Contrastive Decoding: Open-ended Text Generation as Optimization

2 code implementations27 Oct 2022 Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, Mike Lewis

We propose contrastive decoding (CD), a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint.

Language Modelling Text Generation

Scaling up Trustless DNN Inference with Zero-Knowledge Proofs

no code implementations17 Oct 2022 Daniel Kang, Tatsunori Hashimoto, Ion Stoica, Yi Sun

In this work, we present the first practical ImageNet-scale method to verify ML model inference non-interactively, i. e., after the inference has been done.

Retrieval SNARKS +1

A Closer Look at the Calibration of Differentially Private Learners

no code implementations15 Oct 2022 HANLIN ZHANG, Xuechen Li, Prithviraj Sen, Salim Roukos, Tatsunori Hashimoto

Across 7 tasks, temperature scaling and Platt scaling with DP-SGD result in an average 3. 1-fold reduction in the in-domain expected calibration error and only incur at most a minor percent drop in accuracy.

Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning

no code implementations15 Jul 2022 Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, Tatsunori Hashimoto

The development of CLIP [Radford et al., 2021] has sparked a debate on whether language supervision can result in vision models with more transferable representations than traditional image-only methods.

Descriptive Representation Learning

Undersampling is a Minimax Optimal Robustness Intervention in Nonparametric Classification

1 code implementation26 May 2022 Niladri S. Chatterji, Saminul Haque, Tatsunori Hashimoto

While a broad range of techniques have been proposed to tackle distribution shift, the simple baseline of training on an $\textit{undersampled}$ balanced dataset often achieves close to state-of-the-art-accuracy across several popular benchmarks.

Binary Classification

Spurious Correlations in Reference-Free Evaluation of Text Generation

1 code implementation ACL 2022 Esin Durmus, Faisal Ladhak, Tatsunori Hashimoto

Model-based, reference-free evaluation metrics have been proposed as a fast and cost-effective approach to evaluate Natural Language Generation (NLG) systems.

Abstractive Text Summarization Text Generation

Distributionally Robust Models with Parametric Likelihood Ratios

1 code implementation ICLR 2022 Paul Michel, Tatsunori Hashimoto, Graham Neubig

As machine learning models are deployed ever more broadly, it becomes increasingly important that they are not only able to perform well on their training distribution, but also yield accurate predictions when confronted with distribution shift.

text-classification Text Classification

Language modeling via stochastic processes

1 code implementation ICLR 2022 Rose E Wang, Esin Durmus, Noah Goodman, Tatsunori Hashimoto

Recent work in self-supervised learning suggests that models can learn good latent representations via contrastive learning, which can be effective for discriminative tasks.

Contrastive Learning Language Modelling +2

Jury Learning: Integrating Dissenting Voices into Machine Learning Models

no code implementations7 Feb 2022 Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeffrey T. Hancock, Tatsunori Hashimoto, Michael S. Bernstein

We introduce jury learning, a supervised ML approach that resolves these disagreements explicitly through the metaphor of a jury: defining which people or groups, in what proportion, determine the classifier's prediction.

BIG-bench Machine Learning Medical Diagnosis +1

Is Importance Weighting Incompatible with Interpolating Classifiers?

1 code implementation ICLR 2022 Ke Alexander Wang, Niladri S. Chatterji, Saminul Haque, Tatsunori Hashimoto

As a remedy, we show that polynomially-tailed losses restore the effects of importance reweighting in correcting distribution shift in overparameterized models.

Extending the WILDS Benchmark for Unsupervised Adaptation

1 code implementation ICLR 2022 Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, Percy Liang

Unlabeled data can be a powerful point of leverage for mitigating these distribution shifts, as it is frequently much more available than labeled data and can often be obtained from distributions beyond the source distribution as well.

Large Language Models Can Be Strong Differentially Private Learners

4 code implementations ICLR 2022 Xuechen Li, Florian Tramèr, Percy Liang, Tatsunori Hashimoto

Differentially Private (DP) learning has seen limited success for building large deep learning models of text, and straightforward attempts at applying Differentially Private Stochastic Gradient Descent (DP-SGD) to NLP tasks have resulted in large performance drops and high computational overhead.

On the Opportunities and Risks of Foundation Models

3 code implementations16 Aug 2021 Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, aditi raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, Percy Liang

AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.

Transfer Learning

Proof: Accelerating Approximate Aggregation Queries with Expensive Predicates

no code implementations27 Jul 2021 Daniel Kang, John Guibas, Peter Bailis, Tatsunori Hashimoto, Yi Sun, Matei Zaharia

Given a dataset $\mathcal{D}$, we are interested in computing the mean of a subset of $\mathcal{D}$ which matches a predicate.

Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions

1 code implementation ACL 2021 Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, Tatsunori Hashimoto

In conversation, uptake happens when a speaker builds on the contribution of their interlocutor by, for example, acknowledging, repeating or reformulating what they have said.

Question Answering

On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies

1 code implementation NAACL 2021 Tianyi Zhang, Tatsunori Hashimoto

We study how masking and predicting tokens in an unsupervised fashion can give rise to linguistic structures and downstream performance gains.

Inductive Bias Language Modelling +1

Modeling the Second Player in Distributionally Robust Optimization

1 code implementation ICLR 2021 Paul Michel, Tatsunori Hashimoto, Graham Neubig

Distributionally robust optimization (DRO) provides a framework for training machine learning models that are able to perform well on a collection of related data distributions (the "uncertainty set").

Model Selection

Predicting the impact of dataset composition on model performance

no code implementations1 Jan 2021 Tatsunori Hashimoto

Real-world machine learning systems are often are trained using a mix of data sources with varying cost and quality.

Experimental Design Machine Translation +2

Distributionally Robust Losses for Latent Covariate Mixtures

1 code implementation28 Jul 2020 John Duchi, Tatsunori Hashimoto, Hongseok Namkoong

While modern large-scale datasets often consist of heterogeneous subpopulations -- for example, multiple demographic groups or multiple text corpora -- the standard practice of minimizing average loss fails to guarantee uniformly low losses across all subpopulations.

Robustness to Spurious Correlations via Human Annotations

1 code implementation ICML 2020 Megha Srivastava, Tatsunori Hashimoto, Percy Liang

The reliability of machine learning systems critically assumes that the associations between features and labels remain similar between training and test distributions.

Common Sense Reasoning Test

Improved Natural Language Generation via Loss Truncation

no code implementations ACL 2020 Daniel Kang, Tatsunori Hashimoto

In this work, we show that the distinguishability of the models and reference serves as a principled and robust alternative for handling invalid references.

Text Generation

Inferring Multidimensional Rates of Aging from Cross-Sectional Data

1 code implementation12 Jul 2018 Emma Pierson, Pang Wei Koh, Tatsunori Hashimoto, Daphne Koller, Jure Leskovec, Nicholas Eriksson, Percy Liang

Motivated by the study of human aging, we present an interpretable latent-variable model that learns temporal dynamics from cross-sectional data.

Human Aging Time Series +1

Cannot find the paper you are looking for? You can Submit a new open access paper.