Search Results for author: Mengzhou Xia

Found 18 papers, 12 papers with code

Detecting Pretraining Data from Large Language Models

no code implementations25 Oct 2023 Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, Luke Zettlemoyer

Min-K% Prob can be applied without any knowledge about the pretraining corpus or any additional training, departing from previous detection methods that require training a reference model on data that is similar to the pretraining data.

Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

1 code implementation10 Oct 2023 Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, Danqi Chen

In this work, we study structured pruning as an effective means to develop smaller LLMs from pre-trained, larger models.

Language Modelling Question Answering +1

Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation

1 code implementation10 Oct 2023 Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, Danqi Chen

Finally, we propose an effective alignment method that explores diverse generation strategies, which can reasonably reduce the misalignment rate under our attack.

Trainable Transformer in Transformer

1 code implementation3 Jul 2023 Abhishek Panigrahi, Sadhika Malladi, Mengzhou Xia, Sanjeev Arora

In this work, we propose an efficient construction, Transformer in Transformer (in short, TinT), that allows a transformer to simulate and fine-tune complex models internally during inference (e. g., pre-trained language models).

Language Modelling

InstructEval: Systematic Evaluation of Instruction Selection Methods

no code implementations1 Jul 2023 Anirudh Ajith, Chris Pan, Mengzhou Xia, Ameet Deshpande, Karthik Narasimhan

In-context learning (ICL) performs tasks by prompting a large language model (LLM) using an instruction and a small set of annotated examples called demonstrations.

Benchmarking Language Modelling +1

MABEL: Attenuating Gender Bias using Textual Entailment Data

2 code implementations26 Oct 2022 Jacqueline He, Mengzhou Xia, Christiane Fellbaum, Danqi Chen

To this end, we propose MABEL (a Method for Attenuating Gender Bias using Entailment Labels), an intermediate pre-training approach for mitigating gender bias in contextualized representations.

Contrastive Learning Fairness +1

Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models

no code implementations26 Oct 2022 Mozes van de Kar, Mengzhou Xia, Danqi Chen, Mikel Artetxe

Our results suggest that the success of prompting can partly be explained by the model being exposed to similar examples during pretraining, which can be directly retrieved through regular expressions.

Text Classification Text Infilling +2

Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models

1 code implementation30 May 2022 Mengzhou Xia, Mikel Artetxe, Jingfei Du, Danqi Chen, Ves Stoyanov

In this work, we adapt prompt-based few-shot learning to ELECTRA and show that it outperforms masked language models in a wide range of tasks.

Few-Shot Learning Text Infilling

Structured Pruning Learns Compact and Accurate Models

2 code implementations ACL 2022 Mengzhou Xia, Zexuan Zhong, Danqi Chen

The growing size of neural language models has led to increased attention in model compression.

Model Compression

Non-Parametric Few-Shot Learning for Word Sense Disambiguation

1 code implementation NAACL 2021 Howard Chen, Mengzhou Xia, Danqi Chen

One significant challenge in supervised all-words WSD is to classify among senses for a majority of words that lie in the long-tail distribution.

Few-Shot Learning Word Sense Disambiguation

MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning

2 code implementations NAACL 2021 Mengzhou Xia, Guoqing Zheng, Subhabrata Mukherjee, Milad Shokouhi, Graham Neubig, Ahmed Hassan Awadallah

Extensive experiments on real-world low-resource languages - without access to large-scale monolingual corpora or large amounts of labeled data - for tasks like cross-lingual sentiment analysis and named entity recognition show the effectiveness of our approach.

Cross-Lingual Transfer Meta-Learning +5

Demoting Racial Bias in Hate Speech Detection

no code implementations WS 2020 Mengzhou Xia, Anjalie Field, Yulia Tsvetkov

In current hate speech datasets, there exists a high correlation between annotators' perceptions of toxicity and signals of African American English (AAE).

Hate Speech Detection

Predicting Performance for Natural Language Processing Tasks

1 code implementation ACL 2020 Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, Graham Neubig

Given the complexity of combinations of tasks, languages, and domains in natural language processing (NLP) research, it is computationally prohibitive to exhaustively test newly proposed models on each possible experimental setting.

Generalized Data Augmentation for Low-Resource Translation

no code implementations ACL 2019 Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos, Graham Neubig

Translation to or from low-resource languages LRLs poses challenges for machine translation in terms of both adequacy and fluency.

Data Augmentation Translation +1

Choosing Transfer Languages for Cross-Lingual Learning

1 code implementation ACL 2019 Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, Graham Neubig

Cross-lingual transfer, where a high-resource transfer language is used to improve the accuracy of a low-resource task language, is now an invaluable tool for improving performance of natural language processing (NLP) on low-resource languages.

Cross-Lingual Transfer

Cannot find the paper you are looking for? You can Submit a new open access paper.