Search Results for author: Keisuke Sakaguchi

Found 48 papers, 29 papers with code

proScript: Partially Ordered Scripts Generation

no code implementations Findings (EMNLP) 2021 Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras, Niket Tandon, Peter Clark, Yejin Choi

Scripts – prototypical event sequences describing everyday activities – have been shown to help understand narratives by providing expectations, resolving ambiguity, and filling in unstated information.

Text Generation valid

Test-time Augmentation for Factual Probing

1 code implementation26 Oct 2023 Go Kamoda, Benjamin Heinzerling, Keisuke Sakaguchi, Kentaro Inui

Factual probing is a method that uses prompts to test if a language model "knows" certain world knowledge facts.

Language Modelling Test +1

Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations

1 code implementation31 Mar 2023 Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, Dragomir Radev

In this work, we evaluate LLM APIs (ChatGPT, GPT-3, and GPT-4) on the Japanese national medical licensing examinations from the past five years, including the current year.

Causal schema induction for knowledge discovery

1 code implementation27 Mar 2023 Michael Regan, Jena D. Hwang, Keisuke Sakaguchi, James Pustejovsky

In this work, we investigate how to apply schema induction models to the task of knowledge discovery for enhanced search of English-language news texts.

Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction

no code implementations25 Mar 2023 Steven Coyne, Keisuke Sakaguchi, Diana Galvan-Sosa, Michael Zock, Kentaro Inui

GPT-3 and GPT-4 models are powerful, achieving high performance on a variety of Natural Language Processing tasks.

Grammatical Error Correction

I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation

no code implementations19 Dec 2022 Chandra Bhagavatula, Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Lianhui Qin, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West, Yejin Choi

Here, we investigate an alternative that a priori seems impossible: can smaller language models (e. g., GPT-2) win over models that are orders of magnitude larger and better (e. g., GPT-3), if powered with novel commonsense distillation algorithms?

Imitation Learning Knowledge Distillation

Towards Automated Document Revision: Grammatical Error Correction, Fluency Edits, and Beyond

1 code implementation23 May 2022 Masato Mita, Keisuke Sakaguchi, Masato Hagiwara, Tomoya Mizumoto, Jun Suzuki, Kentaro Inui

Natural language processing technology has rapidly improved automated grammatical error correction tasks, and the community begins to explore document-level revision as one of the next challenges.

Grammatical Error Correction Language Modelling

Twist Decoding: Diverse Generators Guide Each Other

1 code implementation19 May 2022 Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Hao Peng, Ximing Lu, Dragomir Radev, Yejin Choi, Noah A. Smith

Our extensive evaluations on machine translation and scientific paper summarization demonstrate that Twist decoding substantially outperforms each model decoded in isolation over various scenarios, including cases where domain-specific and general-purpose models are both available.

Machine Translation Text Generation +1

Interscript: A dataset for interactive learning of scripts through error feedback

1 code implementation15 Dec 2021 Niket Tandon, Aman Madaan, Peter Clark, Keisuke Sakaguchi, Yiming Yang

We present a new dataset, Interscript, containing user feedback on a deployed model that generates complex everyday tasks.

Structured Prediction

Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand

2 code implementations NAACL 2022 Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander R. Fabbri, Yejin Choi, Noah A. Smith

We therefore propose a generalization of leaderboards, bidimensional leaderboards (Billboards), that simultaneously tracks progress in language generation models and metrics for their evaluation.

Image Captioning Machine Translation +1

Improving Neural Model Performance through Natural Language Feedback on Their Explanations

no code implementations18 Apr 2021 Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Yiming Yang, Peter Clark, Keisuke Sakaguchi, Ed Hovy

A class of explainable NLP models for reasoning tasks support their decisions by generating free-form or structured explanations, but what happens when these supporting structures contain errors?

proScript: Partially Ordered Scripts Generation via Pre-trained Language Models

no code implementations16 Apr 2021 Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras, Niket Tandon, Peter Clark, Yejin Choi

Scripts - standardized event sequences describing typical everyday activities - have been shown to help understand narratives by providing expectations, resolving ambiguity, and filling in unstated information.

Text Generation valid

GrammarTagger: A Multilingual, Minimally-Supervised Grammar Profiler for Language Education

1 code implementation7 Apr 2021 Masato Hagiwara, Joshua Tanner, Keisuke Sakaguchi

We present GrammarTagger, an open-source grammar profiler which, given an input text, identifies grammatical features useful for language education.

A Dataset for Tracking Entities in Open Domain Procedural Text

no code implementations EMNLP 2020 Niket Tandon, Keisuke Sakaguchi, Bhavana Dalvi Mishra, Dheeraj Rajagopal, Peter Clark, Michal Guerquin, Kyle Richardson, Eduard Hovy

Our solution is a new task formulation where given just a procedural text as input, the task is to generate a set of state change tuples(entity, at-tribute, before-state, after-state)for each step, where the entity, attribute, and state values must be predicted from an open vocabulary.

COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs

3 code implementations12 Oct 2020 Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, Yejin Choi

Next, we show that ATOMIC 2020 is better suited for training knowledge models that can generate accurate, representative knowledge for new, unseen entities and events.

Knowledge Graphs Natural Language Understanding

WIQA: A dataset for "What if..." reasoning over procedural text

1 code implementation10 Sep 2019 Niket Tandon, Bhavana Dalvi Mishra, Keisuke Sakaguchi, Antoine Bosselut, Peter Clark

We introduce WIQA, the first large-scale dataset of "What if..." questions over procedural text.


Uncertain Natural Language Inference

no code implementations ACL 2020 Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, Benjamin Van Durme

We introduce Uncertain Natural Language Inference (UNLI), a refinement of Natural Language Inference (NLI) that shifts away from categorical labels, targeting instead the direct prediction of subjective probability assessments.

Learning-To-Rank Natural Language Inference +1

WinoGrande: An Adversarial Winograd Schema Challenge at Scale

3 code implementations24 Jul 2019 Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi

The key steps of the dataset construction consist of (1) a carefully designed crowdsourcing procedure, followed by (2) systematic bias reduction using a novel AfLite algorithm that generalizes human-detectable word associations to machine-detectable embedding associations.

Transfer Learning

Efficient Online Scalar Annotation with Bounded Support

no code implementations ACL 2018 Keisuke Sakaguchi, Benjamin Van Durme

We describe a novel method for efficiently eliciting scalar annotations for dataset construction and system quality estimation by human judgments.

GEC into the future: Where are we going and how do we get there?

no code implementations WS 2017 Keisuke Sakaguchi, Courtney Napoles, Joel Tetreault

The field of grammatical error correction (GEC) has made tremendous bounds in the last ten years, but new questions and obstacles are revealing themselves.

Grammatical Error Correction Machine Translation

Error-repair Dependency Parsing for Ungrammatical Texts

1 code implementation ACL 2017 Keisuke Sakaguchi, Matt Post, Benjamin Van Durme

We propose a new dependency parsing scheme which jointly parses a sentence and repairs grammatical errors by extending the non-directional transition-based formalism of Goldberg and Elhadad (2010) with three additional actions: SUBSTITUTE, DELETE, INSERT.

Dependency Parsing

JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction

1 code implementation EACL 2017 Courtney Napoles, Keisuke Sakaguchi, Joel Tetreault

We present a new parallel corpus, JHU FLuency-Extended GUG corpus (JFLEG) for developing and evaluating grammatical error correction (GEC).

Grammatical Error Correction

There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction

1 code implementation EMNLP 2016 Courtney Napoles, Keisuke Sakaguchi, Joel Tetreault

We show that reference-less grammaticality metrics correlate very strongly with human judgments and are competitive with the leading reference-based evaluation metrics.

Benchmarking Grammatical Error Correction

Robsut Wrod Reocginiton via semi-Character Recurrent Neural Network

1 code implementation7 Aug 2016 Keisuke Sakaguchi, Kevin Duh, Matt Post, Benjamin Van Durme

Inspired by the findings from the Cmabrigde Uinervtisy effect, we propose a word recognition model based on a semi-character level recurrent neural network (scRNN).

Spelling Correction

GLEU Without Tuning

1 code implementation9 May 2016 Courtney Napoles, Keisuke Sakaguchi, Matt Post, Joel Tetreault

The GLEU metric was proposed for evaluating grammatical error corrections using n-gram overlap with a set of reference sentences, as opposed to precision/recall of specific annotated errors (Napoles et al., 2015).

Reassessing the Goals of Grammatical Error Correction: Fluency Instead of Grammaticality

1 code implementation TACL 2016 Keisuke Sakaguchi, Courtney Napoles, Matt Post, Joel Tetreault

The field of grammatical error correction (GEC) has grown substantially in recent years, with research directed at both evaluation metrics and improved system performance against those metrics.

Grammatical Error Correction

Cannot find the paper you are looking for? You can Submit a new open access paper.