LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning

29 May 2023  ·  Amirhossein Abaskohi, Sascha Rothe, Yadollah Yaghoobzadeh ·

In recent years, there has been significant progress in developing pre-trained language models for NLP. However, these models often struggle when fine-tuned on small datasets. To address this issue, researchers have proposed various adaptation approaches. Prompt-based tuning is arguably the most common way, especially for larger models. Previous research shows that adding contrastive learning to prompt-based fine-tuning is effective as it helps the model generate embeddings that are more distinguishable between classes, and it can also be more sample-efficient as the model learns from positive and negative examples simultaneously. One of the most important components of contrastive learning is data augmentation, but unlike computer vision, effective data augmentation for NLP is still challenging. This paper proposes LM-CPPF, Contrastive Paraphrasing-guided Prompt-based Fine-tuning of Language Models, which leverages prompt-based few-shot paraphrasing using generative language models, especially large language models such as GPT-3 and OPT-175B, for data augmentation. Our experiments on multiple text classification benchmarks show that this augmentation method outperforms other methods, such as easy data augmentation, back translation, and multiple templates.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Linguistic Acceptability CoLA LM-CPPF RoBERTa-base Accuracy 14.1% # 42
Sentiment Analysis CR LM-CPPF RoBERTa-base Accuracy 93.3 # 2
Natural Language Inference MultiNLI LM-CPPF RoBERTa-base Accuracy 68.4 # 4
Natural Language Inference QNLI LM-CPPF RoBERTa-base Accuracy 70.2% # 41
Sentiment Analysis SST-2 Binary classification LM-CPPF RoBERTa-base Accuracy 93.2 # 40
Sentiment Analysis SST-5 Fine-grained classification LM-CPPF RoBERTa-base Accuracy 54.9 # 6