Search Results for author: Masahiro Kaneko

Found 55 papers, 21 papers with code

IMPARA: Impact-Based Metric for GEC Using Parallel Data

1 code implementation COLING 2022 Koki Maeda, Masahiro Kaneko, Naoaki Okazaki

Correlations between IMPARA and human scores indicate that IMPARA is comparable or better than existing evaluation methods.

Grammatical Error Correction

Debiasing Isn’t Enough! – on the Effectiveness of Debiasing MLMs and Their Social Biases in Downstream Tasks

no code implementations COLING 2022 Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki

We study the relationship between task-agnostic intrinsic and task-specific extrinsic social bias evaluation measures for MLMs, and find that there exists only a weak correlation between these two types of evaluation measures.

ProQE: Proficiency-wise Quality Estimation dataset for Grammatical Error Correction

no code implementations LREC 2022 Yujin Takahashi, Masahiro Kaneko, Masato Mita, Mamoru Komachi

This study investigates how supervised quality estimation (QE) models of grammatical error correction (GEC) are affected by the learners’ proficiency with the data.

Grammatical Error Correction

Studying The Impact Of Document-level Context On Simultaneous Neural Machine Translation

no code implementations MTSummit 2021 Raj Dabre, Aizhan Imankulova, Masahiro Kaneko

To this end and in this paper and we propose wait-k simultaneous document-level NMT where we keep the context encoder as it is and replace the source sentence encoder and target language decoder with their wait-k equivalents.

Decoder Machine Translation +3

Sampling-based Pseudo-Likelihood for Membership Inference Attacks

1 code implementation17 Apr 2024 Masahiro Kaneko, Youmi Ma, Yuki Wata, Naoaki Okazaki

In this study, we propose a Sampling-based Pseudo-Likelihood (\textbf{SPL}) method for MIA (\textbf{SaMIA}) that calculates SPL using only the text generated by an LLM to detect leaks.

A Little Leak Will Sink a Great Ship: Survey of Transparency for Large Language Models from Start to Finish

no code implementations24 Mar 2024 Masahiro Kaneko, Timothy Baldwin

In this paper, we conduct an experimental survey to elucidate the relationship between the leakage rate and both the output rate and detection rate for personal information, copyrighted texts, and benchmark data.

Few-Shot Learning

Eagle: Ethical Dataset Given from Real Interactions

no code implementations22 Feb 2024 Masahiro Kaneko, Danushka Bollegala, Timothy Baldwin

The existing evaluation metrics and methods to address these ethical challenges use datasets intentionally created by instructing humans to create instances including ethical problems.

The Gaps between Pre-train and Downstream Settings in Bias Evaluation and Debiasing

no code implementations16 Jan 2024 Masahiro Kaneko, Danushka Bollegala, Timothy Baldwin

Moreover, the performance degradation due to debiasing is also lower in the ICL case compared to that in the FT case.

In-Context Learning

How You Prompt Matters! Even Task-Oriented Constraints in Instructions Affect LLM-Generated Text Detection

1 code implementation14 Nov 2023 Ryuto Koike, Masahiro Kaneko, Naoaki Okazaki

Furthermore, our analysis indicates that the high instruction-following ability of LLMs fosters the large impact of such constraints on detection performance.

Instruction Following Large Language Model +4

SAIE Framework: Support Alone Isn't Enough -- Advancing LLM Training with Adversarial Remarks

no code implementations14 Nov 2023 Mengsay Loem, Masahiro Kaneko, Naoaki Okazaki

Large Language Models (LLMs) can justify or critique their predictions through discussions with other models or humans, thereby enriching their intrinsic understanding of instances.

GSM8K Math

Controlled Generation with Prompt Insertion for Natural Language Explanations in Grammatical Error Correction

1 code implementation20 Sep 2023 Masahiro Kaneko, Naoaki Okazaki

Generating explanations for GEC corrections involves aligning input and output tokens, identifying correction points, and presenting corresponding explanations consistently.

Grammatical Error Correction

Evaluating Gender Bias of Pre-trained Language Models in Natural Language Inference by Considering All Labels

1 code implementation18 Sep 2023 Panatchakorn Anantaprayoon, Masahiro Kaneko, Naoaki Okazaki

In Natural Language Inference (NLI), existing bias evaluation methods have focused on the prediction results of one specific label out of three labels, such as neutral.

Natural Language Inference

The Impact of Debiasing on the Performance of Language Models in Downstream Tasks is Underestimated

no code implementations16 Sep 2023 Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki

In this study, we compare the impact of debiasing on performance across multiple downstream tasks using a wide-range of benchmark datasets that containing female, male, and stereotypical words.

In-Contextual Gender Bias Suppression for Large Language Models

1 code implementation13 Sep 2023 Daisuke Oba, Masahiro Kaneko, Danushka Bollegala

We show that, using CrowsPairs dataset, our textual preambles covering counterfactual statements can suppress gender biases in English LLMs such as LLaMA2.

counterfactual Data Augmentation +1

OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examples

1 code implementation21 Jul 2023 Ryuto Koike, Masahiro Kaneko, Naoaki Okazaki

Experiments in the domain of student essays show that the proposed detector improves the detection performance on the attacker-generated texts by up to +41. 3 points F1-score.

Adversarial Attack Detection DeepFake Detection +5

Reducing Sequence Length by Predicting Edit Operations with Large Language Models

no code implementations19 May 2023 Masahiro Kaneko, Naoaki Okazaki

Experiments show that the proposed method achieves comparable performance to the baseline in four tasks, paraphrasing, formality style transfer, GEC, and text simplification, despite reducing the length of the target text by as small as 21%.

Formality Style Transfer Grammatical Error Correction +2

Solving NLP Problems through Human-System Collaboration: A Discussion-based Approach

1 code implementation19 May 2023 Masahiro Kaneko, Graham Neubig, Naoaki Okazaki

Humans work together to solve common problems by having discussions, explaining, and agreeing or disagreeing with each other.

Natural Language Inference

Comparing Intrinsic Gender Bias Evaluation Measures without using Human Annotated Examples

no code implementations28 Jan 2023 Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki

Prior works have relied on human annotated examples to compare existing intrinsic bias evaluation measures.

Debiasing isn't enough! -- On the Effectiveness of Debiasing MLMs and their Social Biases in Downstream Tasks

no code implementations6 Oct 2022 Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki

We study the relationship between task-agnostic intrinsic and task-specific extrinsic social bias evaluation measures for Masked Language Models (MLMs), and find that there exists only a weak correlation between these two types of evaluation measures.

Are Neighbors Enough? Multi-Head Neural n-gram can be Alternative to Self-attention

no code implementations27 Jul 2022 Mengsay Loem, Sho Takase, Masahiro Kaneko, Naoaki Okazaki

Impressive performance of Transformer has been attributed to self-attention, where dependencies between entire input in a sequence are considered at every position.

Position

Gender Bias in Meta-Embeddings

no code implementations19 May 2022 Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki

Different methods have been proposed to develop meta-embeddings from a given set of source embeddings.

Gender Bias in Masked Language Models for Multiple Languages

1 code implementation NAACL 2022 Masahiro Kaneko, Aizhan Imankulova, Danushka Bollegala, Naoaki Okazaki

Unfortunately, it was reported that MLMs also learn discriminative biases regarding attributes such as gender and race.

Attribute Sentence

Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddings

1 code implementation14 Mar 2022 Yi Zhou, Masahiro Kaneko, Danushka Bollegala

Sense embedding learning methods learn different embeddings for the different senses of an ambiguous word.

Word Embeddings

Interpretability for Language Learners Using Example-Based Grammatical Error Correction

1 code implementation ACL 2022 Masahiro Kaneko, Sho Takase, Ayana Niwa, Naoaki Okazaki

In this study, we introduce an Example-Based GEC (EB-GEC) that presents examples to language learners as a basis for a correction result.

Grammatical Error Correction

Proficiency Matters Quality Estimation in Grammatical Error Correction

no code implementations17 Jan 2022 Yujin Takahashi, Masahiro Kaneko, Masato Mita, Mamoru Komachi

This study investigates how supervised quality estimation (QE) models of grammatical error correction (GEC) are affected by the learners' proficiency with the data.

Grammatical Error Correction

ExtraPhrase: Efficient Data Augmentation for Abstractive Summarization

no code implementations NAACL (ACL) 2022 Mengsay Loem, Sho Takase, Masahiro Kaneko, Naoaki Okazaki

Through experiments, we show that ExtraPhrase improves the performance of abstractive summarization tasks by more than 0. 50 points in ROUGE scores compared to the setting without data augmentation.

Abstractive Text Summarization Data Augmentation +1

Comparison of Grammatical Error Correction Using Back-Translation Models

no code implementations NAACL 2021 Aomi Koyama, Kengo Hotate, Masahiro Kaneko, Mamoru Komachi

Therefore, GEC studies have developed various methods to generate pseudo data, which comprise pairs of grammatical and artificially produced ungrammatical sentences.

Grammatical Error Correction Translation

Simultaneous Multi-Pivot Neural Machine Translation

no code implementations15 Apr 2021 Raj Dabre, Aizhan Imankulova, Masahiro Kaneko, Abhisek Chakrabarty

Parallel corpora are indispensable for training neural machine translation (NMT) models, and parallel corpora for most language pairs do not exist or are scarce.

Machine Translation NMT +1

Unmasking the Mask -- Evaluating Social Biases in Masked Language Models

1 code implementation15 Apr 2021 Masahiro Kaneko, Danushka Bollegala

To overcome the above-mentioned disfluencies, we propose All Unmasked Likelihood (AUL), a bias evaluation measure that predicts all tokens in a test case given the MLM embedding of the unmasked input.

Selection bias Sentence

Dictionary-based Debiasing of Pre-trained Word Embeddings

1 code implementation EACL 2021 Masahiro Kaneko, Danushka Bollegala

Word embeddings trained on large corpora have shown to encode high levels of unfair discriminatory gender, racial, religious and ethnic biases.

Word Embeddings

Debiasing Pre-trained Contextualised Embeddings

2 code implementations EACL 2021 Masahiro Kaneko, Danushka Bollegala

In comparison to the numerous debiasing methods proposed for the static non-contextualised word embeddings, the discriminative biases in contextualised embeddings have received relatively little attention.

Sentence Word Embeddings

Generating Diverse Corrections with Local Beam Search for Grammatical Error Correction

no code implementations COLING 2020 Kengo Hotate, Masahiro Kaneko, Mamoru Komachi

In this study, we propose a beam search method to obtain diverse outputs in a local sequence transduction task where most of the tokens in the source and target sentences overlap, such as in grammatical error correction (GEC).

Grammatical Error Correction Sentence

Cross-lingual Transfer Learning for Grammatical Error Correction

no code implementations COLING 2020 Ikumi Yamashita, Satoru Katsumata, Masahiro Kaneko, Aizhan Imankulova, Mamoru Komachi

Cross-lingual transfer learning from high-resource languages (the source models) is effective for training models of low-resource languages (the target models) for various tasks.

Cross-Lingual Transfer Grammatical Error Correction +1

Autoencoding Improves Pre-trained Word Embeddings

no code implementations COLING 2020 Masahiro Kaneko, Danushka Bollegala

Prior work investigating the geometry of pre-trained word embeddings have shown that word embeddings to be distributed in a narrow cone and by centering and projecting using principal component vectors one can increase the accuracy of a given set of pre-trained word embeddings.

Word Embeddings

English-to-Japanese Diverse Translation by Combining Forward and Backward Outputs

no code implementations WS 2020 Masahiro Kaneko, Aizhan Imankulova, Tosho Hirasawa, Mamoru Komachi

We introduce our TMU system that is submitted to The 4th Workshop on Neural Generation and Translation (WNGT2020) to English-to-Japanese (En→Ja) track on Simultaneous Translation And Paraphrase for Language Education (STAPLE) shared task.

Machine Translation NMT +2

Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction

1 code implementation ACL 2020 Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, Kentaro Inui

The answer to this question is not as straightforward as one might expect because the previous common methods for incorporating a MLM into an EncDec model have potential drawbacks when applied to GEC.

Decoder Grammatical Error Correction +1

Towards Multimodal Simultaneous Neural Machine Translation

1 code implementation WMT (EMNLP) 2020 Aizhan Imankulova, Masahiro Kaneko, Tosho Hirasawa, Mamoru Komachi

Simultaneous translation involves translating a sentence before the speaker's utterance is completed in order to realize real-time understanding in multiple languages.

Machine Translation Sentence +1

Japanese-Russian TMU Neural Machine Translation System using Multilingual Model for WAT 2019

no code implementations WS 2019 Aizhan Imankulova, Masahiro Kaneko, Mamoru Komachi

We introduce our system that is submitted to the News Commentary task (Japanese{\textless}-{\textgreater}Russian) of the 6th Workshop on Asian Translation.

Machine Translation NMT +1

TMU Transformer System Using BERT for Re-ranking at BEA 2019 Grammatical Error Correction on Restricted Track

no code implementations WS 2019 Masahiro Kaneko, Kengo Hotate, Satoru Katsumata, Mamoru Komachi

Thus, it is not straightforward to utilize language representations trained from a large corpus, such as Bidirectional Encoder Representations from Transformers (BERT), in a form suitable for the learner{'}s grammatical errors.

Grammatical Error Correction Re-Ranking +1

Gender-preserving Debiasing for Pre-trained Word Embeddings

1 code implementation ACL 2019 Masahiro Kaneko, Danushka Bollegala

Word embeddings learnt from massive text collections have demonstrated significant levels of discriminative biases such as gender, racial or ethnic biases, which in turn bias the down-stream NLP applications that use those word embeddings.

Word Embeddings

Multi-Head Multi-Layer Attention to Deep Language Representations for Grammatical Error Detection

no code implementations15 Apr 2019 Masahiro Kaneko, Mamoru Komachi

In this work, we investigate the effect of utilizing information not only from the final layer but also from intermediate layers of a pre-trained language representation model to detect grammatical errors.

Grammatical Error Detection Sentence

TMU System for SLAM-2018

1 code implementation WS 2018 Masahiro Kaneko, Tomoyuki Kajiwara, Mamoru Komachi

We introduce the TMU systems for the second language acquisition modeling shared task 2018 (Settles et al., 2018).

Language Acquisition

Cannot find the paper you are looking for? You can Submit a new open access paper.