Search Results for author: Kai-Wei Chang

Found 208 papers, 110 papers with code

On the Transferability of Adversarial Attacks against Neural Text Classifier

no code implementations EMNLP 2021 Liping Yuan, Xiaoqing Zheng, Yi Zhou, Cho-Jui Hsieh, Kai-Wei Chang

Based on these studies, we propose a genetic algorithm to find an ensemble of models that can be used to induce adversarial examples to fool almost all existing models.

text-classification Text Classification

Robustness and Adversarial Examples in Natural Language Processing

no code implementations EMNLP (ACL) 2021 Kai-Wei Chang, He He, Robin Jia, Sameer Singh

In particular, we will review recent studies on analyzing the weakness of NLP systems when facing adversarial inputs and data with a distribution shift.

Using Item Response Theory to Measure Gender and Racial Bias of a BERT-based Automated English Speech Assessment System

no code implementations NAACL (BEA) 2022 Alexander Kwako, Yixin Wan, Jieyu Zhao, Kai-Wei Chang, Li Cai, Mark Hansen

This study addresses the need to examine potential biases of transformer-based models in the context of automated English speech assessment.

Towards Adversarially Robust Text Classifiers by Learning to Reweight Clean Examples

no code implementations Findings (ACL) 2022 Jianhan Xu, Cenyuan Zhang, Xiaoqing Zheng, Linyang Li, Cho-Jui Hsieh, Kai-Wei Chang, Xuanjing Huang

Most of the existing defense methods improve the adversarial robustness by making the models adapt to the training set augmented with some adversarial examples.

Adversarial Robustness

Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertainty

1 code implementation2 Dec 2023 Cheng-Fu Yang, Haoyang Xu, Te-Lin Wu, Xiaofeng Gao, Kai-Wei Chang, Feng Gao

In this paper, we aim to tackle this problem with a unified framework consisting of an end-to-end trainable method and a planning algorithm.

Denoising Vision-Language Navigation

A Reevaluation of Event Extraction: Past, Present, and Future Challenges

1 code implementation16 Nov 2023 Kuan-Hao Huang, I-Hung Hsu, Tanmay Parekh, Zhiyu Xie, Zixuan Zhang, Premkumar Natarajan, Kai-Wei Chang, Nanyun Peng, Heng Ji

In this work, we first identify and discuss these evaluation challenges, including the unfair comparisons resulting from different assumptions about data or different data preprocessing steps, the incompleteness of the current evaluation framework leading to potential dataset bias or data split bias, and low reproducibility of prior studies.

Event Extraction

JAB: Joint Adversarial Prompting and Belief Augmentation

no code implementations16 Nov 2023 Ninareh Mehrabi, Palash Goyal, Anil Ramakrishna, Jwala Dhamala, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta

With the recent surge of language models in different applications, attention to safety and robustness of these models has gained significant importance.

VideoCon: Robust Video-Language Alignment via Contrast Captions

1 code implementation15 Nov 2023 Hritik Bansal, Yonatan Bitton, Idan Szpektor, Kai-Wei Chang, Aditya Grover

Despite being (pre)trained on a massive amount of data, state-of-the-art video-language alignment models are not robust to semantically-plausible contrastive changes in the video captions.

Language Modelling Large Language Model +5

Lumos: Learning Agents with Unified Data, Modular Design, and Open-Source LLMs

1 code implementation9 Nov 2023 Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, Bill Yuchen Lin

Leveraging this unified data and modular design, Lumos not only achieves comparable or superior performance to current, state-of-the-art agents, but also exhibits several key advantages: (1) Lumos surpasses GPT-4/3. 5-based agents in complex question answering and web tasks, while equalling the performance of significantly larger LLM agents on math tasks; (2) Lumos outperforms open-source agents created through conventional training methods and those using chain-of-thoughts training; and (3) Lumos is capable of effectively generalizing to unseen interactive tasks, outperforming larger LLM-based agents and even exceeding performance of specialized agents.

Math Question Answering

Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks

1 code implementation1 Nov 2023 Po-Nien Kung, Fan Yin, Di wu, Kai-Wei Chang, Nanyun Peng

Instruction tuning (IT) achieves impressive zero-shot generalization results by training large language models (LLMs) on a massive amount of diverse tasks with instructions.

Informativeness Out-of-Distribution Generalization

What's "up" with vision-language models? Investigating their struggle with spatial reasoning

no code implementations30 Oct 2023 Amita Kamath, Jack Hessel, Kai-Wei Chang

Recent vision-language (VL) models are powerful, but can they reliably distinguish "right" from "left"?

An Exploration of In-Context Learning for Speech Language Model

no code implementations19 Oct 2023 Ming-Hao Hsu, Kai-Wei Chang, Shang-Wen Li, Hung-Yi Lee

Despite the success of ICL in NLP, little work is exploring the possibility of ICL in speech processing.

Few-Shot Learning Language Modelling

LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following

1 code implementation18 Oct 2023 Cheng-Fu Yang, Yen-Chun Chen, Jianwei Yang, Xiyang Dai, Lu Yuan, Yu-Chiang Frank Wang, Kai-Wei Chang

Additional analysis shows that the contrastive objective and meta-actions are complementary in achieving the best results, and the resulting agent better aligns its states with corresponding instructions, making it more suitable for real-world embodied agents.

Contrastive Learning Instruction Following

Will the Prince Get True Love's Kiss? On the Model Sensitivity to Gender Perturbation over Fairytale Texts

no code implementations16 Oct 2023 Christina Chance, Da Yin, Dakuo Wang, Kai-Wei Chang

Using counterfactual data augmentation to the FairytaleQA dataset, we evaluate model robustness against swapped gender character information, and then mitigate learned biases by introducing counterfactual gender stereotypes during training time.

counterfactual Data Augmentation +1

Mitigating Bias for Question Answering Models by Tracking Bias Influence

no code implementations13 Oct 2023 Mingyu Derek Ma, Jiun-Yu Kao, Arpit Gupta, Yu-Hsiang Lin, Wenbo Zhao, Tagyoung Chung, Wei Wang, Kai-Wei Chang, Nanyun Peng

Based on the intuition that a model would lean to be more biased if it learns from a biased example, we measure the bias level of a query instance by observing its influence on another instance.

Multiple-choice Multi-Task Learning +1

"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters

1 code implementation13 Oct 2023 Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, Nanyun Peng

Through benchmarking evaluation on 2 popular LLMs- ChatGPT and Alpaca, we reveal significant gender biases in LLM-generated recommendation letters.

Benchmarking Fairness

Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systems

1 code implementation8 Oct 2023 Yixin Wan, Jieyu Zhao, Aman Chadha, Nanyun Peng, Kai-Wei Chang

Recent advancements in Large Language Models empower them to follow freeform instructions, including imitating generic or specific demographic personas in conversations.


Towards General-Purpose Text-Instruction-Guided Voice Conversion

no code implementations25 Sep 2023 Chun-Yi Kuan, Chen An Li, Tsu-Yuan Hsu, Tse-Yang Lin, Ho-Lam Chung, Kai-Wei Chang, Shuo-Yiin Chang, Hung-Yi Lee

This paper introduces a novel voice conversion (VC) model, guided by text instructions such as "articulate slowly with a deep tone" or "speak in a cheerful boyish voice".

Language Modelling Specificity +1

Self-Augmentation Improves Zero-Shot Cross-Lingual Transfer

no code implementations19 Sep 2023 Fei Wang, Kuan-Hao Huang, Kai-Wei Chang, Muhao Chen

In this paper, we propose a simple yet effective method, SALT, to improve the zero-shot cross-lingual transfer of the multilingual pretrained language models without the help of such external data.

Multilingual NLP Zero-Shot Cross-Lingual Transfer

Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech

1 code implementation18 Sep 2023 Chien-yu Huang, Ke-Han Lu, Shih-Heng Wang, Chi-Yuan Hsiao, Chun-Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, Roshan Sharma, Shinji Watanabe, Bhiksha Ramakrishnan, Shady Shehata, Hung-Yi Lee

To achieve comprehensive coverage of diverse speech tasks and harness instruction tuning, we invite the community to collaborate and contribute, facilitating the dynamic growth of the benchmark.

Contextual Label Projection for Cross-Lingual Structure Extraction

no code implementations16 Sep 2023 Tanmay Parekh, I-Hung Hsu, Kuan-Hao Huang, Kai-Wei Chang, Nanyun Peng

In this paper, we introduce CLAP, which first translates text to the target language and performs contextual translation on the labels using the translated text as the context, ensuring better accuracy for the translated labels.

Cross-Lingual Transfer Event Argument Extraction +1

FLIRT: Feedback Loop In-context Red Teaming

no code implementations8 Aug 2023 Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta

Here we propose an automatic red teaming framework that evaluates a given model and exposes its vulnerabilities against unsafe and inappropriate content generation.

Response Generation

UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding

1 code implementation3 Jul 2023 Rui Sun, Zhecan Wang, Haoxuan You, Noel Codella, Kai-Wei Chang, Shih-Fu Chang

However, we find visual and textual fine-grained information, e. g., keywords in the sentence and objects in the image, can be fairly informative for semantics understanding.

Image-text matching Text Matching +1

SpeechGen: Unlocking the Generative Power of Speech Language Models with Prompts

no code implementations3 Jun 2023 Haibin Wu, Kai-Wei Chang, Yuan-Kuei Wu, Hung-Yi Lee

In this paper, we present pioneering research that explores the application of prompt tuning to stimulate speech LMs for various generation tasks, within a unified framework called SpeechGen, with around 10M trainable parameters.

Open-Ended Question Answering

Red Teaming Language Model Detectors with Language Models

2 code implementations31 May 2023 Zhouxing Shi, Yihan Wang, Fan Yin, Xiangning Chen, Kai-Wei Chang, Cho-Jui Hsieh

The prevalence and strong capability of large language models (LLMs) present significant safety and ethical risks if exploited by malicious users.

Adversarial Robustness Language Modelling +2

MiniSUPERB: Lightweight Benchmark for Self-supervised Speech Models

no code implementations30 May 2023 Yu-Hsiang Wang, Huang-Yu Chen, Kai-Wei Chang, Winston Hsu, Hung-Yi Lee

In this paper, we introduce MiniSUPERB, a lightweight benchmark that efficiently evaluates SSL speech models with comparable results to SUPERB but lower computational costs significantly.

Self-Supervised Learning

ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppression

no code implementations26 May 2023 Yixin Wan, Yuan Zhou, Xiulian Peng, Kai-Wei Chang, Yan Lu

To begin with, we are among the first to comprehensively investigate mainstream KD techniques on DNS models to resolve the two challenges.

Knowledge Distillation

PIP: Parse-Instructed Prefix for Syntactically Controlled Paraphrase Generation

no code implementations26 May 2023 Yixin Wan, Kuan-Hao Huang, Kai-Wei Chang

Existing fine-tuning methods for this task are costly as all the parameters of the model need to be updated during the training process.

Paraphrase Generation

IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models

1 code implementation24 May 2023 Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad A. Ayyubi, Kai-Wei Chang, Shih-Fu Chang

Specifically, IdealGPT utilizes an LLM to generate sub-questions, a VLM to provide corresponding sub-answers, and another LLM to reason to achieve the final answer.

Text encoders bottleneck compositionality in contrastive vision-language models

1 code implementation24 May 2023 Amita Kamath, Jack Hessel, Kai-Wei Chang

We first curate CompPrompts, a set of increasingly compositional image captions that VL models should be able to capture (e. g., single object, to object+property, to multiple interacting objects).

Image Captioning

Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysis

no code implementations23 May 2023 Oscar Chew, Hsuan-Tien Lin, Kai-Wei Chang, Kuan-Hao Huang

Recent research has revealed that deep learning models have a tendency to leverage spurious correlations that exist in the training set but may not hold true in general circumstances.

text-classification Text Classification

Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation

1 code implementation23 May 2023 Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, Kai-Wei Chang

Instruction tuning has emerged to enhance the capabilities of large language models (LLMs) to comprehend instructions and generate appropriate responses.

Continual Learning

KPEval: Towards Fine-grained Semantic-based Evaluation of Keyphrase Extraction and Generation Systems

1 code implementation27 Mar 2023 Di wu, Da Yin, Kai-Wei Chang

Despite the significant advancements in keyphrase extraction and keyphrase generation methods, the predominant approach for evaluation only relies on exact matching with human references and disregards reference-free attributes.

Keyphrase Extraction Keyphrase Generation

Factoring the Matrix of Domination: A Critical Review and Reimagination of Intersectionality in AI Fairness

no code implementations16 Mar 2023 Anaelia Ovalle, Arjun Subramonian, Vagrant Gautam, Gilbert Gee, Kai-Wei Chang

Through a critical review of how intersectionality is discussed in 30 papers from the AI fairness literature, we deductively and inductively: 1) map how intersectionality tenets operate within the AI fairness paradigm and 2) uncover gaps between the conceptualization and operationalization of intersectionality.


CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning

1 code implementation ICCV 2023 Hritik Bansal, Nishad Singhi, Yu Yang, Fan Yin, Aditya Grover, Kai-Wei Chang

Multimodal contrastive pretraining has been used to train multimodal representation models, such as CLIP, on large amounts of paired image-text data.

Backdoor Attack Contrastive Learning +1

SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks

no code implementations1 Mar 2023 Kai-Wei Chang, Yu-Kai Wang, Hua Shen, Iu-thing Kang, Wei-Cheng Tseng, Shang-Wen Li, Hung-Yi Lee

For speech processing, SpeechPrompt shows its high parameter efficiency and competitive performance on a few speech classification tasks.

Ranked #16 on Spoken Language Understanding on Fluent Speech Commands (using extra training data)

Classification Language Modelling +1

Semantic Strengthening of Neuro-Symbolic Learning

no code implementations28 Feb 2023 Kareem Ahmed, Kai-Wei Chang, Guy Van Den Broeck

Numerous neuro-symbolic approaches have recently been proposed typically with the goal of adding symbolic knowledge to the output layer of a neural network.

Ensemble knowledge distillation of self-supervised speech models

no code implementations24 Feb 2023 Kuan-Po Huang, Tzu-hsun Feng, Yu-Kuan Fu, Tsu-Yuan Hsu, Po-Chieh Yen, Wei-Cheng Tseng, Kai-Wei Chang, Hung-Yi Lee

We tried two different aggregation techniques, layerwise-average and layerwise-concatenation, to the representations of different teacher models and found that the former was more effective.

Automatic Speech Recognition Automatic Speech Recognition (ASR) +4

GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methods

no code implementations CVPR 2023 Da Yin, Feng Gao, Govind Thattai, Michael Johnston, Kai-Wei Chang

A key goal for the advancement of AI is to develop technologies that serve the needs not just of one group but of all communities regardless of their geographical region.

PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English

1 code implementation20 Dec 2022 Jianfeng Chi, Wasi Uddin Ahmad, Yuan Tian, Kai-Wei Chang

Privacy policies provide individuals with information about their rights and how their personal information is handled.

Language Modelling Natural Language Understanding

Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study

1 code implementation20 Dec 2022 Di wu, Wasi Uddin Ahmad, Kai-Wei Chang

However, there lacks a systematic study of how the two types of approaches compare and how different design choices can affect the performance of PLM-based models.

Keyphrase Extraction Keyphrase Generation

A Survey of Deep Learning for Mathematical Reasoning

2 code implementations20 Dec 2022 Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, Kai-Wei Chang

Mathematical reasoning is a fundamental aspect of human intelligence and is applicable in various fields, including science, engineering, finance, and everyday life.

Math Mathematical Reasoning

Find Someone Who: Visual Commonsense Understanding in Human-Centric Grounding

no code implementations14 Dec 2022 Haoxuan You, Rui Sun, Zhecan Wang, Kai-Wei Chang, Shih-Fu Chang

We present a new commonsense task, Human-centric Commonsense Grounding, that tests the models' ability to ground individuals given the context descriptions about what happened before, and their mental/physical states or intentions.

Auditing Algorithmic Fairness in Machine Learning for Health with Severity-Based LOGAN

no code implementations16 Nov 2022 Anaelia Ovalle, Sunipa Dev, Jieyu Zhao, Majid Sarrafzadeh, Kai-Wei Chang

Therefore, ML auditing tools must be (1) better aligned with ML4H auditing principles and (2) able to illuminate and characterize communities vulnerable to the most harm.

Bias Detection Clustering +1

Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense

no code implementations10 Nov 2022 Zhecan Wang, Haoxuan You, Yicheng He, Wenhao Li, Kai-Wei Chang, Shih-Fu Chang

Visual commonsense understanding requires Vision Language (VL) models to not only understand image and text but also cross-reference in-between to fully integrate and achieve comprehension of the visual scene described.

Unsupervised Syntactically Controlled Paraphrase Generation with Abstract Meaning Representations

no code implementations2 Nov 2022 Kuan-Hao Huang, Varun Iyer, Anoop Kumar, Sriram Venkatapathy, Kai-Wei Chang, Aram Galstyan

In this paper, we demonstrate that leveraging Abstract Meaning Representations (AMR) can greatly improve the performance of unsupervised syntactically controlled paraphrase generation.

Data Augmentation Paraphrase Generation

Investigating Ensemble Methods for Model Robustness Improvement of Text Classifiers

no code implementations28 Oct 2022 Jieyu Zhao, Xuezhi Wang, Yao Qin, Jilin Chen, Kai-Wei Chang

Large pre-trained language models have shown remarkable performance over the past few years.

How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions?

1 code implementation27 Oct 2022 Hritik Bansal, Da Yin, Masoud Monajatipoor, Kai-Wei Chang

To this end, we introduce an Ethical NaTural Language Interventions in Text-to-Image GENeration (ENTIGEN) benchmark dataset to evaluate the change in image generations conditional on ethical interventions across three social axes -- gender, skin color, and culture.

Cultural Vocal Bursts Intensity Prediction

ADDMU: Detection of Far-Boundary Adversarial Examples with Data and Model Uncertainty Estimation

1 code implementation22 Oct 2022 Fan Yin, Yao Li, Cho-Jui Hsieh, Kai-Wei Chang

Finally, our analysis shows that the two types of uncertainty provided by \textbf{ADDMU} can be leveraged to characterize adversarial examples and identify the ones that contribute most to model's robustness in adversarial training.

The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks

1 code implementation18 Oct 2022 Nikil Roashan Selvam, Sunipa Dev, Daniel Khashabi, Tushar Khot, Kai-Wei Chang

How reliably can we trust the scores obtained from social bias benchmarks as faithful indicators of problematic social biases in a given language model?

Language Modelling

Watermarking Pre-trained Language Models with Backdooring

no code implementations14 Oct 2022 Chenxi Gu, Chengsong Huang, Xiaoqing Zheng, Kai-Wei Chang, Cho-Jui Hsieh

Large pre-trained language models (PLMs) have proven to be a crucial component of modern natural language processing systems.

Multi-Task Learning

An Analysis of the Effects of Decoding Algorithms on Fairness in Open-Ended Language Generation

no code implementations7 Oct 2022 Jwala Dhamala, Varun Kumar, Rahul Gupta, Kai-Wei Chang, Aram Galstyan

We present a systematic analysis of the impact of decoding algorithms on LM fairness, and analyze the trade-off between fairness, diversity and quality.

Fairness Text Generation

Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning

2 code implementations29 Sep 2022 Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, Ashwin Kalyan

However, it is unknown if the models can handle more complex problems that involve math reasoning over heterogeneous information, such as tabular data.

Logical Reasoning Math +1

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

1 code implementation20 Sep 2022 Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, Ashwin Kalyan

We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering ScienceQA questions.

Multimodal Deep Learning Multiple-choice +4

Improving the Adversarial Robustness of NLP Models by Information Bottleneck

1 code implementation Findings (ACL) 2022 Cenyuan Zhang, Xiang Zhou, Yixin Wan, Xiaoqing Zheng, Kai-Wei Chang, Cho-Jui Hsieh

Existing studies have demonstrated that adversarial examples can be directly attributed to the presence of non-robust features, which are highly predictive, but can be easily manipulated by adversaries to fool NLP models.

Adversarial Robustness SST-2

Semantic Probabilistic Layers for Neuro-Symbolic Learning

1 code implementation1 Jun 2022 Kareem Ahmed, Stefano Teso, Kai-Wei Chang, Guy Van Den Broeck, Antonio Vergari

We design a predictive layer for structured-output prediction (SOP) that can be plugged into any neural network guaranteeing its predictions are consistent with a set of predefined symbolic constraints.

Hierarchical Multi-label Classification Logical Reasoning

Controllable Text Generation with Neurally-Decomposed Oracle

1 code implementation27 May 2022 Tao Meng, Sidi Lu, Nanyun Peng, Kai-Wei Chang

We propose a general and efficient framework to control auto-regressive generation models with NeurAlly-Decomposed Oracle (NADO).

Language Modelling Machine Translation +1

GENEVA: Benchmarking Generalizability for Event Argument Extraction with Hundreds of Event Types and Argument Roles

1 code implementation25 May 2022 Tanmay Parekh, I-Hung Hsu, Kuan-Hao Huang, Kai-Wei Chang, Nanyun Peng

We utilize this ontology to further introduce GENEVA, a diverse generalizability benchmarking dataset comprising four test suites, aimed at evaluating models' ability to handle limited data and unseen event type generalization.

Benchmarking Event Argument Extraction +1

TAGPRIME: A Unified Framework for Relational Structure Extraction

1 code implementation25 May 2022 I-Hung Hsu, Kuan-Hao Huang, Shuning Zhang, Wenxin Cheng, Premkumar Natarajan, Kai-Wei Chang, Nanyun Peng

In this work, we propose to take a unified view of all these tasks and introduce TAGPRIME to address relational structure extraction problems.

Event Argument Extraction Language Modelling +2

GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models

1 code implementation24 May 2022 Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, Kai-Wei Chang

In this paper, we introduce a benchmark dataset, Geo-Diverse Commonsense Multilingual Language Models Analysis (GeoMLAMA), for probing the diversity of the relational knowledge in multilingual PLMs.

Language Modelling

Conditional Supervised Contrastive Learning for Fair Text Classification

1 code implementation23 May 2022 Jianfeng Chi, William Shand, Yaodong Yu, Kai-Wei Chang, Han Zhao, Yuan Tian

Contrastive representation learning has gained much attention due to its superior performance in learning representations from both image and sequential data.

Contrastive Learning Fairness +3

Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks

no code implementations22 Apr 2022 Zhecan Wang, Noel Codella, Yen-Chun Chen, Luowei Zhou, Xiyang Dai, Bin Xiao, Jianwei Yang, Haoxuan You, Kai-Wei Chang, Shih-Fu Chang, Lu Yuan

Experiments demonstrate that MAD leads to consistent gains in the low-shot, domain-shifted, and fully-supervised conditions on VCR, SNLI-VE, and VQA, achieving SOTA performance on VCR compared to other single models pretrained with image-text data.

Question Answering Visual Commonsense Reasoning +2

Retrieval Enhanced Data Augmentation for Question Answering on Privacy Policies

no code implementations19 Apr 2022 Md Rizwan Parvez, Jianfeng Chi, Wasi Uddin Ahmad, Yuan Tian, Kai-Wei Chang

Prior studies in privacy policies frame the question answering (QA) task as identifying the most relevant text segment or a list of sentences from a policy document given a user query.

Data Augmentation Question Answering +1

SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks

1 code implementation31 Mar 2022 Kai-Wei Chang, Wei-Cheng Tseng, Shang-Wen Li, Hung-Yi Lee

We report in this paper the first exploration of the prompt tuning paradigm for speech processing tasks based on Generative Spoken Language Model (GSLM).

Language Modelling Self-Supervised Learning

Measuring Fairness of Text Classifiers via Prediction Sensitivity

no code implementations ACL 2022 Satyapriya Krishna, Rahul Gupta, Apurv Verma, Jwala Dhamala, Yada Pruksachatkun, Kai-Wei Chang

With the rapid growth in language processing applications, fairness has emerged as an important consideration in data-driven solutions.

counterfactual Fairness +2

Representation Learning for Resource-Constrained Keyphrase Generation

1 code implementation15 Mar 2022 Di wu, Wasi Uddin Ahmad, Sunipa Dev, Kai-Wei Chang

State-of-the-art keyphrase generation methods generally depend on large annotated datasets, limiting their performance in domains with limited annotated data.

Denoising Domain Adaptation +4

A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models

no code implementations17 Feb 2022 Da Yin, Li Dong, Hao Cheng, Xiaodong Liu, Kai-Wei Chang, Furu Wei, Jianfeng Gao

With the increasing of model capacity brought by pre-trained language models, there emerges boosting needs for more knowledgeable natural language processing (NLP) models with advanced functionalities including providing and making flexible use of encyclopedic and commonsense knowledge.

Language Modelling

Neuro-Symbolic Entropy Regularization

no code implementations25 Jan 2022 Kareem Ahmed, Eric Wang, Kai-Wei Chang, Guy Van Den Broeck

We propose a loss, neuro-symbolic entropy regularization, that encourages the model to confidently predict a valid object.

Structured Prediction valid

SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning

no code implementations16 Dec 2021 Zhecan Wang, Haoxuan You, Liunian Harold Li, Alireza Zareian, Suji Park, Yiqing Liang, Kai-Wei Chang, Shih-Fu Chang

As for pre-training, a scene-graph-aware pre-training method is proposed to leverage structure knowledge extracted in the visual scene graph.

Visual Commonsense Reasoning

Grounded Language-Image Pre-training

1 code implementation CVPR 2022 Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, Jianfeng Gao

The unification brings two benefits: 1) it allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model; 2) GLIP can leverage massive image-text pairs by generating grounding boxes in a self-training fashion, making the learned representation semantic-rich.

Few-Shot Object Detection object-detection +2

Toward Degradation-Robust Voice Conversion

no code implementations14 Oct 2021 Chien-yu Huang, Kai-Wei Chang, Hung-Yi Lee

However, in real-world scenarios, it is difficult to collect clean utterances of a speaker, and they are usually degraded by noises or reverberations.

Denoising Speech Enhancement +1

Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitution

1 code implementation EMNLP 2021 Zongyi Li, Jianhan Xu, Jiehang Zeng, Linyang Li, Xiaoqing Zheng, Qi Zhang, Kai-Wei Chang, Cho-Jui Hsieh

Recent studies have shown that deep neural networks are vulnerable to intentionally crafted adversarial examples, and various methods have been proposed to defend against adversarial word-substitution attacks for neural NLP models.


DEGREE: A Data-Efficient Generation-Based Event Extraction Model

2 code implementations NAACL 2022 I-Hung Hsu, Kuan-Hao Huang, Elizabeth Boschee, Scott Miller, Prem Natarajan, Kai-Wei Chang, Nanyun Peng

Given a passage and a manually designed prompt, DEGREE learns to summarize the events mentioned in the passage into a natural sentence that follows a predefined pattern.

Event Extraction Structured Prediction +1

Retrieval Augmented Code Generation and Summarization

1 code implementation Findings (EMNLP) 2021 Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang

To mimic developers' code or summary generation behavior, we propose a retrieval augmented framework, REDCODER, that retrieves relevant code or summaries from a retrieval database and provides them as a supplement to code generation or summarization models.

 Ranked #1 on Code Generation on CodeXGLUE - CodeSearchNet (using extra training data)

Code Generation Code Summarization +1

AVATAR: A Parallel Corpus for Java-Python Program Translation

1 code implementation26 Aug 2021 Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, Kai-Wei Chang

Automating program translation is of paramount importance in software migration, and recently researchers explored unsupervised approaches due to the unavailability of parallel corpora.


On Measures of Biases and Harms in NLP

no code implementations7 Aug 2021 Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Akihiro Nishi, Nanyun Peng, Kai-Wei Chang

Recent studies show that Natural Language Processing (NLP) technologies propagate societal biases about demographic groups associated with attributes such as gender, race, and nationality.

Defense against Synonym Substitution-based Adversarial Attacks via Dirichlet Neighborhood Ensemble

1 code implementation ACL 2021 Yi Zhou, Xiaoqing Zheng, Cho-Jui Hsieh, Kai-Wei Chang, Xuanjing Huang

Although deep neural networks have achieved prominent performance on many NLP tasks, they are vulnerable to adversarial examples.

How Much Can CLIP Benefit Vision-and-Language Tasks?

4 code implementations13 Jul 2021 Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, Kurt Keutzer

Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world.

Ranked #4 on Vision and Language Navigation on RxR (using extra training data)

Question Answering Vision and Language Navigation +2

Clinical Named Entity Recognition using Contextualized Token Representations

no code implementations23 Jun 2021 Yichao Zhou, Chelsea Ju, J. Harry Caufield, Kevin Shih, Calvin Chen, Yizhou Sun, Kai-Wei Chang, Peipei Ping, Wei Wang

To facilitate various downstream applications using clinical case reports (CCRs), we pre-train two deep contextualized language models, Clinical Embeddings from Language Model (C-ELMo) and Clinical Contextual String Embeddings (C-Flair) using the clinical-related corpus from the PubMed Central.

Language Modelling named-entity-recognition +3

Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classification

no code implementations Findings (ACL) 2021 Yada Pruksachatkun, Satyapriya Krishna, Jwala Dhamala, Rahul Gupta, Kai-Wei Chang

Existing bias mitigation methods to reduce disparities in model outcomes across cohorts have focused on data augmentation, debiasing model embeddings, or adding fairness-based optimization objectives during training.

Data Augmentation Fairness +2

Syntax-augmented Multilingual BERT for Cross-lingual Transfer

1 code implementation ACL 2021 Wasi Uddin Ahmad, Haoran Li, Kai-Wei Chang, Yashar Mehdad

In recent years, we have seen a colossal effort in pre-training multilingual text encoders using large-scale corpora in many languages to facilitate cross-lingual transfer learning.

Cross-Lingual Transfer named-entity-recognition +7

Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?

1 code implementation Findings (ACL) 2021 Jieyu Zhao, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Kai-Wei Chang

We investigate the effectiveness of natural language interventions for reading-comprehension systems, studying this in the context of social stereotypes.

Ethics Few-Shot Learning +2

``Nice Try, Kiddo'': Investigating Ad Hominems in Dialogue Responses

no code implementations NAACL 2021 Emily Sheng, Kai-Wei Chang, Prem Natarajan, Nanyun Peng

Ad hominem attacks are those that target some feature of a person{'}s character instead of the position the person is maintaining.

Abusive Language

Societal Biases in Language Generation: Progress and Challenges

1 code implementation ACL 2021 Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng

Technology for language generation has advanced rapidly, spurred by advancements in pre-training large models on massive amounts of data and the need for intelligent agents to communicate in a natural manner.

Fairness Text Generation

Evaluating the Values of Sources in Transfer Learning

1 code implementation NAACL 2021 Md Rizwan Parvez, Kai-Wei Chang

Transfer learning that adapts a model trained on data-rich sources to low-resource targets has been widely applied in natural language processing (NLP).

Cross-Lingual POS Tagging Transfer Learning

Revealing Persona Biases in Dialogue Systems

1 code implementation18 Apr 2021 Emily Sheng, Josh Arnold, Zhou Yu, Kai-Wei Chang, Nanyun Peng

Dialogue systems in the form of chatbots and personal assistants are being increasingly integrated into people's lives.

On the Sensitivity and Stability of Model Interpretations in NLP

1 code implementation ACL 2022 Fan Yin, Zhouxing Shi, Cho-Jui Hsieh, Kai-Wei Chang

We propose two new criteria, sensitivity and stability, that provide complementary notions of faithfulness to the existed removal-based criteria.

Adversarial Robustness Dependency Parsing +2

Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training

1 code implementation EMNLP 2021 Kuan-Hao Huang, Wasi Uddin Ahmad, Nanyun Peng, Kai-Wei Chang

Pre-trained multilingual language encoders, such as multilingual BERT and XLM-R, show great potential for zero-shot cross-lingual transfer.

text-classification Text Classification +3

Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models

1 code implementation NAACL 2021 James Y. Huang, Kuan-Hao Huang, Kai-Wei Chang

In this work, we present ParaBART, a semantic sentence embedding model that learns to disentangle semantics and syntax in sentence embeddings obtained by pre-trained language models.

Semantic Similarity Semantic Textual Similarity +2

Unified Pre-training for Program Understanding and Generation

1 code implementation NAACL 2021 Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang

Experiments on code summarization in the English language, code generation, and code translation in seven programming languages show that PLBART outperforms or rivals state-of-the-art models.

Clone Detection Code Summarization +6

CREATe: Clinical Report Extraction and Annotation Technology

no code implementations28 Feb 2021 Yichao Zhou, Wei-Ting Chen, BoWen Zhang, David Lee, J. Harry Caufield, Kai-Wei Chang, Yizhou Sun, Peipei Ping, Wei Wang

Clinical case reports are written descriptions of the unique aspects of a particular clinical case, playing an essential role in sharing clinical experiences about atypical disease phenotypes and new therapies.

BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation

1 code implementation27 Jan 2021 Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta

To systematically study and benchmark social biases in open-ended language generation, we introduce the Bias in Open-Ended Language Generation Dataset (BOLD), a large-scale dataset that consists of 23, 679 English text generation prompts for bias benchmarking across five domains: profession, gender, race, religion, and political ideology.

Benchmarking Text Generation

Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs

1 code implementation EACL 2021 Kuan-Hao Huang, Kai-Wei Chang

We also demonstrate that the performance of SynPG is competitive or even better than supervised models when the unannotated data is large.

Data Augmentation Disentanglement +1

Intent Classification and Slot Filling for Privacy Policies

1 code implementation ACL 2021 Wasi Uddin Ahmad, Jianfeng Chi, Tu Le, Thomas Norton, Yuan Tian, Kai-Wei Chang

We refer to predicting the privacy practice explained in a sentence as intent classification and identifying the text spans sharing specific information as slot filling.

General Classification intent-classification +2

Generating Sports News from Live Commentary: A Chinese Dataset for Sports Game Summarization

1 code implementation Asian Chapter of the Association for Computational Linguistics 2020 Kuan-Hao Huang, Chen Li, Kai-Wei Chang

To deeply study this task, we present SportsSum, a Chinese sports game summarization dataset which contains 5, 428 soccer games of live commentaries and the corresponding news articles.

On the Transferability of Adversarial Attacksagainst Neural Text Classifier

no code implementations17 Nov 2020 Liping Yuan, Xiaoqing Zheng, Yi Zhou, Cho-Jui Hsieh, Kai-Wei Chang

Based on these studies, we propose a genetic algorithm to find an ensemble of models that can be used to induce adversarial examples to fool almost all existing models.

text-classification Text Classification

Cross-Lingual Dependency Parsing by POS-Guided Word Reordering

no code implementations Findings of the Association for Computational Linguistics 2020 Lu Liu, Yi Zhou, Jianhan Xu, Xiaoqing Zheng, Kai-Wei Chang, Xuanjing Huang

The words in each sentence of a source language corpus are rearranged to meet the word order in a target language under the guidance of a part-of-speech based language model (LM).

Dependency Parsing Language Modelling +1

"Nice Try, Kiddo": Investigating Ad Hominems in Dialogue Responses

1 code implementation24 Oct 2020 Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng

Ad hominem attacks are those that target some feature of a person's character instead of the position the person is maintaining.

Abusive Language

GATE: Graph Attention Transformer Encoder for Cross-lingual Relation and Event Extraction

1 code implementation6 Oct 2020 Wasi Uddin Ahmad, Nanyun Peng, Kai-Wei Chang

Recent progress in cross-lingual relation and event extraction use graph convolutional networks (GCNs) with universal dependency parses to learn language-agnostic sentence representations such that models trained on one language can be applied to other languages.

Event Extraction Graph Attention

PolicyQA: A Reading Comprehension Dataset for Privacy Policies

1 code implementation Findings of the Association for Computational Linguistics 2020 Wasi Uddin Ahmad, Jianfeng Chi, Yuan Tian, Kai-Wei Chang

Prior studies in this domain frame the QA task as retrieving the most relevant text segment or a list of sentences from the policy document given a question.

Question Answering Reading Comprehension

What Does BERT with Vision Look At?

no code implementations ACL 2020 Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, Kai-Wei Chang

Pre-trained visually grounded language models such as ViLBERT, LXMERT, and UNITER have achieved significant performance improvement on vision-and-language tasks but what they learn during pre-training remains unclear.

Language Modelling

``The Boating Store Had Its Best Sail Ever'': Pronunciation-attentive Contextualized Pun Recognition

no code implementations ACL 2020 Yichao Zhou, Jyun-Yu Jiang, Jieyu Zhao, Kai-Wei Chang, Wei Wang

In this paper, we propose Pronunciation-attentive Contextualized Pun Recognition (PCPR) to perceive human humor, detect if a sentence contains puns and locate them in the sentence.

GPT-GNN: Generative Pre-Training of Graph Neural Networks

2 code implementations27 Jun 2020 Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, Yizhou Sun

Graph neural networks (GNNs) have been demonstrated to be powerful in modeling graph-structured data.

Graph Generation

Defense against Adversarial Attacks in NLP via Dirichlet Neighborhood Ensemble

1 code implementation20 Jun 2020 Yi Zhou, Xiaoqing Zheng, Cho-Jui Hsieh, Kai-Wei Chang, Xuanjing Huang

Despite neural networks have achieved prominent performance on many natural language processing (NLP) tasks, they are vulnerable to adversarial examples.

An Integer Linear Programming Framework for Mining Constraints from Data

1 code implementation18 Jun 2020 Tao Meng, Kai-Wei Chang

This raises a question -- \emph{can we mine constraints and rules from data based on a learning algorithm?}

Multi-class Classification Multi-Label Classification

Mitigating Gender Bias Amplification in Distribution by Posterior Regularization

1 code implementation ACL 2020 Shengyu Jia, Tao Meng, Jieyu Zhao, Kai-Wei Chang

With little performance loss, our method can almost remove the bias amplification in the distribution.

On the Robustness of Language Encoders against Grammatical Errors

1 code implementation ACL 2020 Fan Yin, Quanyu Long, Tao Meng, Kai-Wei Chang

We conduct a thorough study to diagnose the behaviors of pre-trained language encoders (ELMo, BERT, and RoBERTa) when confronted with natural grammatical errors.

Cloze Test Linguistic Acceptability

"The Boating Store Had Its Best Sail Ever": Pronunciation-attentive Contextualized Pun Recognition

1 code implementation29 Apr 2020 Yichao Zhou, Jyun-Yu Jiang, Jieyu Zhao, Kai-Wei Chang, Wei Wang

In this paper, we propose Pronunciation-attentive Contextualized Pun Recognition (PCPR) to perceive human humor, detect if a sentence contains puns and locate them in the sentence.

Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond

5 code implementations NeurIPS 2020 Kaidi Xu, Zhouxing Shi, huan zhang, Yihan Wang, Kai-Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, Cho-Jui Hsieh

Linear relaxation based perturbation analysis (LiRPA) for neural networks, which computes provable linear bounds of output neurons given a certain amount of input perturbation, has become a core component in robustness verification and certified defense.


Robustness Verification for Transformers

1 code implementation ICLR 2020 Zhouxing Shi, huan zhang, Kai-Wei Chang, Minlie Huang, Cho-Jui Hsieh

Robustness verification that aims to formally certify the prediction behavior of neural networks has become an important tool for understanding model behavior and obtaining safety guarantees.

Sentiment Analysis

Towards Understanding Gender Bias in Relation Extraction

1 code implementation ACL 2020 Andrew Gaut, Tony Sun, Shirlyn Tang, Yuxin Huang, Jing Qian, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, William Yang Wang

We use WikiGenderBias to evaluate systems for bias and find that NRE systems exhibit gender biased predictions and lay groundwork for future evaluation of bias in NRE.

counterfactual Data Augmentation +3

Cross-lingual Dependency Parsing with Unlabeled Auxiliary Languages

1 code implementation CONLL 2019 Wasi Uddin Ahmad, Zhisong Zhang, Xuezhe Ma, Kai-Wei Chang, Nanyun Peng

We conduct experiments on cross-lingual dependency parsing where we train a dependency parser on a source language and transfer it to a wide range of target languages.

Cross-Lingual Transfer Dependency Parsing +2

Visualizing Trends of Key Roles in News Articles

1 code implementation IJCNLP 2019 Chen Xia, Haoxiang Zhang, Jacob Moghtader, Allen Wu, Kai-Wei Chang

There are tons of news articles generated every day reflecting the activities of key roles such as people, organizations and political parties.

BOSH: An Efficient Meta Algorithm for Decision-based Attacks

no code implementations10 Sep 2019 Zhenxin Xiao, Puyudi Yang, Yuchen Jiang, Kai-Wei Chang, Cho-Jui Hsieh

Adversarial example generation becomes a viable method for evaluating the robustness of a machine learning model.

Adversarial Attack Bayesian Optimization

Learning to Discriminate Perturbations for Blocking Adversarial Attacks in Text Classification

1 code implementation IJCNLP 2019 Yichao Zhou, Jyun-Yu Jiang, Kai-Wei Chang, Wei Wang

To identify adversarial attacks, a perturbation discriminator validates how likely a token in the text is perturbed and provides a set of potential perturbations.

Blocking General Classification +3