Search Results for author: Lifu Huang

Found 75 papers, 28 papers with code

Semi-supervised New Event Type Induction and Event Detection

no code implementations EMNLP 2020 Lifu Huang, Heng Ji

We design a Semi-Supervised Vector Quantized Variational Autoencoder framework to automatically learn a discrete latent type representation for each seen and unseen type and optimize them using seen type event annotations.

Event Detection Event Extraction +1

New Frontiers of Information Extraction

no code implementations NAACL (ACL) 2022 Muhao Chen, Lifu Huang, Manling Li, Ben Zhou, Heng Ji, Dan Roth

This tutorial targets researchers and practitioners who are interested in AI and ML technologies for structural information extraction (IE) from unstructured textual sources.

Towards Automatic Curation of Antibiotic Resistance Genes via Statement Extraction from Scientific Papers: A Benchmark Dataset and Models

1 code implementation BioNLP (ACL) 2022 Sidhant Chandak, Liqing Zhang, Connor Brown, Lifu Huang

Antibiotic resistance has become a growing worldwide concern as new resistance mechanisms are emerging and spreading globally, and thus detecting and collecting the cause – Antibiotic Resistance Genes (ARGs), have been more critical than ever.

Lateralization LoRA: Interleaved Instruction Tuning with Modality-Specialized Adaptations

no code implementations4 Jul 2024 Zhiyang Xu, Minqian Liu, Ying Shen, Joy Rimchala, Jiaxin Zhang, Qifan Wang, Yu Cheng, Lifu Huang

Lateralization LoRA employs a hybrid approach, combining the traditional linear LoRA and a Convolutional LoRA for generating text and images, enabling the generation of high-quality text and images by leveraging modality-specific structures and parameter sets.

Attribute Image Generation

Debate as Optimization: Adaptive Conformal Prediction and Diverse Retrieval for Event Extraction

no code implementations18 Jun 2024 Sijia Wang, Lifu Huang

We propose a multi-agent debate as optimization (DAO) system for event extraction, where the primary objective is to iteratively refine the large language models (LLMs) outputs through debating without parameter tuning.

Conformal Prediction Event Detection +3

InternalInspector $I^2$: Robust Confidence Estimation in LLMs through Internal States

no code implementations17 Jun 2024 Mohammad Beigi, Ying Shen, Runing Yang, Zihao Lin, Qifan Wang, Ankith Mohan, Jianfeng He, Ming Jin, Chang-Tien Lu, Lifu Huang

Despite their vast capabilities, Large Language Models (LLMs) often struggle with generating reliable outputs, frequently producing high-confidence inaccuracies known as hallucinations.

Benchmarking Contrastive Learning +4

Targeted Augmentation for Low-Resource Event Extraction

no code implementations14 May 2024 Sijia Wang, Lifu Huang

Addressing the challenge of low-resource information extraction remains an ongoing issue due to the inherent information scarcity within limited training examples.

Data Augmentation Diversity +1

Many-to-many Image Generation with Auto-regressive Diffusion Models

no code implementations3 Apr 2024 Ying Shen, Yizhe Zhang, Shuangfei Zhai, Lifu Huang, Joshua M. Susskind, Jiatao Gu

This paper introduces a domain-general framework for many-to-many image generation, capable of producing interrelated image series from a given set of images, offering a scalable solution that obviates the need for task-specific solutions across different multi-image scenarios.

Image Generation Novel View Synthesis

X-Shot: A Unified System to Handle Frequent, Few-shot and Zero-shot Learning Simultaneously in Classification

1 code implementation6 Mar 2024 Hanzi Xu, Muhao Chen, Lifu Huang, Slobodan Vucetic, Wenpeng Yin

In recent years, few-shot and zero-shot learning, which learn to predict labels with limited annotated instances, have garnered significant attention.

Domain Generalization Instruction Following +1

Multimodal Instruction Tuning with Conditional Mixture of LoRA

no code implementations24 Feb 2024 Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, Lifu Huang

Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in diverse tasks across different domains, with an increasing focus on improving their zero-shot generalization capabilities for unseen multimodal tasks.

Zero-shot Generalization

Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning

no code implementations18 Feb 2024 Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, Lifu Huang

Despite vision-language models' (VLMs) remarkable capabilities as versatile visual assistants, two substantial challenges persist within the existing VLM frameworks: (1) lacking task diversity in pretraining and visual instruction tuning, and (2) annotation error and bias in GPT-4 synthesized instruction tuning data.

Hallucination Visual Question Answering

Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?

1 code implementation23 Jan 2024 Cheng Han, Qifan Wang, Yiming Cui, Wenguan Wang, Lifu Huang, Siyuan Qi, Dongfang Liu

As the scale of vision models continues to grow, the emergence of Visual Prompt Tuning (VPT) as a parameter-efficient transfer learning technique has gained attention due to its superior performance compared to traditional full-finetuning.

Transfer Learning Visual Prompt Tuning

RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training

1 code implementation7 Dec 2023 Jaehyung Kim, Yuning Mao, Rui Hou, Hanchao Yu, Davis Liang, Pascale Fung, Qifan Wang, Fuli Feng, Lifu Huang, Madian Khabsa

Under a unified evaluation of fine-tuned LMs by incorporating four representative perspectives of model robustness, we demonstrate the effectiveness of RoAST compared to state-of-the-art fine-tuning methods on six different types of LMs, which indicates its usefulness in practice.

Adversarial Robustness

X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects

no code implementations15 Nov 2023 Minqian Liu, Ying Shen, Zhiyang Xu, Yixin Cao, Eunah Cho, Vaibhav Kumar, Reza Ghanadan, Lifu Huang

Natural Language Generation (NLG) typically involves evaluating the generated text in various aspects (e. g., consistency and naturalness) to obtain a comprehensive assessment.

Dialogue Generation Language Modelling +2

MULTISCRIPT: Multimodal Script Learning for Supporting Open Domain Everyday Tasks

1 code implementation8 Oct 2023 Jingyuan Qi, Minqian Liu, Ying Shen, Zhiyang Xu, Lifu Huang

Automatically generating scripts (i. e. sequences of key steps described in text) from video demonstrations and reasoning about the subsequent steps are crucial to the modern AI virtual assistants to guide humans to complete everyday tasks, especially unfamiliar ones.

Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models

no code implementations4 Oct 2023 Zihao Lin, Yan Sun, Yifan Shi, Xueqian Wang, Lifu Huang, Li Shen, DaCheng Tao

With the blowout development of pre-trained models (PTMs), the efficient tuning of these models for diverse downstream applications has emerged as a pivotal research concern.

A Survey of Document-Level Information Extraction

no code implementations23 Sep 2023 Hanwen Zheng, Sijia Wang, Lifu Huang

Document-level information extraction (IE) is a crucial task in natural language processing (NLP).

coreference-resolution

Teamwork Is Not Always Good: An Empirical Study of Classifier Drift in Class-incremental Information Extraction

1 code implementation26 May 2023 Minqian Liu, Lifu Huang

Class-incremental learning (CIL) aims to develop a learning system that can continually learn new classes from a data stream without forgetting previously learned classes.

Class Incremental Learning Incremental Learning

RE$^2$: Region-Aware Relation Extraction from Visually Rich Documents

1 code implementation24 May 2023 Pritika Ramu, Sijia Wang, Lalla Mouatadid, Joy Rimchala, Lifu Huang

Current research in form understanding predominantly relies on large pre-trained language models, necessitating extensive data for pre-training.

Graph Attention Relation +1

Iteratively Improving Biomedical Entity Linking and Event Extraction via Hard Expectation-Maximization

no code implementations24 May 2023 Xiaochu Li, Minqian Liu, Zhiyang Xu, Lifu Huang

To solve these challenges, we propose joint biomedical entity linking and event extraction by regarding the event structures and entity references in knowledge bases as latent variables and updating the two task-specific models in a hard Expectation-Maximization (EM) fashion: (1) predicting the missing variables for each partially annotated dataset based on the current two task-specific models, and (2) updating the parameters of each model on the corresponding pseudo completed dataset.

Entity Linking Event Extraction +1

AMELI: Enhancing Multimodal Entity Linking with Fine-Grained Attributes

no code implementations24 May 2023 Barry Menglong Yao, Yu Chen, Qifan Wang, Sijia Wang, Minqian Liu, Zhiyang Xu, Licheng Yu, Lifu Huang

We propose attribute-aware multimodal entity linking, where the input is a mention described with a text and image, and the goal is to predict the corresponding target entity from a multimodal knowledge base (KB) where each entity is also described with a text description, a visual image and a set of attributes and values.

Attribute Entity Linking

The Art of SOCRATIC QUESTIONING: Recursive Thinking with Large Language Models

1 code implementation24 May 2023 Jingyuan Qi, Zhiyang Xu, Ying Shen, Minqian Liu, Di Jin, Qifan Wang, Lifu Huang

Chain-of-Thought (CoT) prompting enables large language models to solve complex reasoning problems by generating intermediate steps.

Language Modelling Math +2

Understand the Dynamic World: An End-to-End Knowledge Informed Framework for Open Domain Entity State Tracking

no code implementations26 Apr 2023 Mingchen Li, Lifu Huang

Open domain entity state tracking aims to predict reasonable state changes of entities (i. e., [attribute] of [entity] was [before_state] and [after_state] afterwards) given the action descriptions.

Attribute Decoder

ExClaim: Explainable Neural Claim Verification Using Rationalization

1 code implementation21 Jan 2023 Sai Gurrapu, Lifu Huang, Feras A. Batarseh

We introduce a novel claim verification approach, namely: ExClaim, that attempts to provide an explainable claim verification system with foundational evidence.

Claim Verification Decision Making +3

Rationalization for Explainable NLP: A Survey

no code implementations21 Jan 2023 Sai Gurrapu, Ajay Kulkarni, Lifu Huang, Ismini Lourentzou, Laura Freeman, Feras A. Batarseh

Recent improvements in natural language generation have made rationalization an attractive technique because it is intuitive, human-comprehensible, and accessible to non-technical users.

Explainable Artificial Intelligence (XAI) Question Answering +3

MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning

1 code implementation21 Dec 2022 Zhiyang Xu, Ying Shen, Lifu Huang

Our results indicate that fine-tuning the model on a diverse set of tasks and instructions leads to a reduced sensitivity to variations in instructions for each task.

Transfer Learning Zero-Shot Learning

Multimedia Generative Script Learning for Task Planning

1 code implementation25 Aug 2022 Qingyun Wang, Manling Li, Hou Pong Chan, Lifu Huang, Julia Hockenmaier, Girish Chowdhary, Heng Ji

Goal-oriented generative script learning aims to generate subsequent steps to reach a particular goal, which is an essential task to assist robots or humans in performing stereotypical activities.

Contrastive Learning Decoder +5

End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models

1 code implementation25 May 2022 Barry Menglong Yao, Aditya Shah, Lichao Sun, Jin-Hee Cho, Lifu Huang

We propose end-to-end multimodal fact-checking and explanation generation, where the input is a claim and a large collection of web sources, including articles, images, videos, and tweets, and the goal is to assess the truthfulness of the claim by retrieving relevant evidence and predicting a truthfulness label (e. g., support, refute or not enough information), and to generate a statement to summarize and explain the reasoning and ruling process.

Claim Verification Explanation Generation +2

Probing Script Knowledge from Pre-Trained Models

no code implementations16 Apr 2022 Zijian Jin, Xingyu Zhang, Mo Yu, Lifu Huang

Script knowledge is critical for humans to understand the broad daily tasks and routine activities in the world.

Story Generation

Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detection

1 code implementation COLING 2022 Minqian Liu, Shiyu Chang, Lifu Huang

Lifelong event detection aims to incrementally update a model with new event types and data while retaining the capability on previously learned old types.

Event Detection

Identifying and Measuring Token-Level Sentiment Bias in Pre-trained Language Models with Prompts

no code implementations15 Apr 2022 Apoorv Garg, Deval Srivastava, Zhiyang Xu, Lifu Huang

Due to the superior performance, large-scale pre-trained language models (PLMs) have been widely adopted in many aspects of human society.

The Art of Prompting: Event Detection based on Type Specific Prompts

no code implementations14 Apr 2022 Sijia Wang, Mo Yu, Lifu Huang

We compare various forms of prompts to represent event types and develop a unified framework to incorporate the event type specific prompts for supervised, few-shot, and zero-shot event detection.

Event Detection Vocal Bursts Type Prediction

Efficient Federated Learning on Knowledge Graphs via Privacy-preserving Relation Embedding Aggregation

1 code implementation17 Mar 2022 Kai Zhang, Yu Wang, Hongyi Wang, Lifu Huang, Carl Yang, Xun Chen, Lichao Sun

Furthermore, we propose a Federated learning paradigm with privacy-preserving Relation embedding aggregation (FedR) to tackle the privacy issue in FedE.

Entity Embeddings Federated Learning +4

PLANET: Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generation

no code implementations ACL 2022 Zhe Hu, Hou Pong Chan, Jiachen Liu, Xinyan Xiao, Hua Wu, Lifu Huang

Despite recent progress of pre-trained language models on generating fluent text, existing methods still suffer from incoherence problems in long-form text generation tasks that require proper content control and planning to form a coherent high-level logical flow.

Contrastive Learning Decoder +2

Efficient Long Sequence Encoding via Synchronization

no code implementations15 Mar 2022 Xiangyang Mou, Mo Yu, Bingsheng Yao, Lifu Huang

Pre-trained Transformer models have achieved successes in a wide range of NLP tasks, but are inefficient when dealing with long input sequences.

What Makes the Story Forward? Inferring Commonsense Explanations as Prompts for Future Event Generation

no code implementations18 Jan 2022 Li Lin, Yixin Cao, Lifu Huang, Shu'ang Li, Xuming Hu, Lijie Wen, Jianmin Wang

To alleviate the knowledge forgetting issue, we design two modules, Im and Gm, for each type of knowledge, which are combined via prompt tuning.

Information Retrieval Retrieval +1

MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding

2 code implementations20 Dec 2021 Revanth Gangi Reddy, Xilin Rui, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avirup Sil, Shih-Fu Chang, Alexander Schwing, Heng Ji

Specifically, the task involves multi-hop questions that require reasoning over image-caption pairs to identify the grounded visual object being referred to and then predicting a span from the news body text to answer the question.

Answer Generation Data Augmentation +2

Query and Extract: Refining Event Extraction as Type-oriented Binary Decoding

no code implementations Findings (ACL) 2022 Sijia Wang, Mo Yu, Shiyu Chang, Lichao Sun, Lifu Huang

Event extraction is typically modeled as a multi-class classification problem where event types and argument roles are treated as atomic symbols.

Multi-class Classification Natural Language Queries +2

How Knowledge Graph and Attention Help? A Qualitative Analysis into Bag-level Relation Extraction

1 code implementation ACL 2021 Zikun Hu, Yixin Cao, Lifu Huang, Tat-Seng Chua

In this paper, we contribute a dataset and propose a paradigm to quantitatively evaluate the effect of attention and KG on bag-level relation extraction (RE).

Relation Relation Extraction

How Knowledge Graph and Attention Help? A Quantitative Analysis into Bag-level Relation Extraction

1 code implementation26 Jul 2021 Zikun Hu, Yixin Cao, Lifu Huang, Tat-Seng Chua

In this paper, we contribute a dataset and propose a paradigm to quantitatively evaluate the effect of attention and KG on bag-level relation extraction (RE).

Relation Relation Extraction

Membership Inference Attacks on Knowledge Graphs

no code implementations16 Apr 2021 Yu Wang, Lifu Huang, Philip S. Yu, Lichao Sun

Membership inference attacks (MIAs) infer whether a specific data record is used for target model training.

Inference Attack Knowledge Graph Embedding +3

The Future is not One-dimensional: Complex Event Schema Induction by Graph Modeling for Event Prediction

1 code implementation EMNLP 2021 Manling Li, Sha Li, Zhenhailong Wang, Lifu Huang, Kyunghyun Cho, Heng Ji, Jiawei Han, Clare Voss

We introduce a new concept of Temporal Complex Event Schema: a graph-based schema representation that encompasses events, arguments, temporal connections and argument relations.

Global Attention for Name Tagging

no code implementations CONLL 2018 Boliang Zhang, Spencer Whitehead, Lifu Huang, Heng Ji

Many name tagging approaches use local contextual information with much success, but fail when the local context is ambiguous or limited.

ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis

1 code implementation INLG (ACL) 2020 Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, Nazneen Fatema Rajani

To assist human review process, we build a novel ReviewRobot to automatically assign a review score and write comments for multiple categories such as novelty and meaningful comparison.

Review Generation valid

Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning

no code implementations IJCNLP 2019 Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi

In this paper, we introduce Cosmos QA, a large-scale dataset of 35, 600 problems that require commonsense-based reading comprehension, formulated as multiple-choice questions.

Machine Reading Comprehension Multiple-choice

Biomedical Event Extraction based on Knowledge-driven Tree-LSTM

no code implementations NAACL 2019 Diya Li, Lifu Huang, Heng Ji, Jiawei Han

Event extraction for the biomedical domain is more challenging than that in the general news domain since it requires broader acquisition of domain-specific knowledge and deeper understanding of complex contexts.

Entity Linking Event Extraction

PaperRobot: Incremental Draft Generation of Scientific Ideas

2 code implementations ACL 2019 Qingyun Wang, Lifu Huang, Zhiying Jiang, Kevin Knight, Heng Ji, Mohit Bansal, Yi Luan

We present a PaperRobot who performs as an automatic research assistant by (1) conducting deep understanding of a large collection of human-written papers in a target domain and constructing comprehensive background knowledge graphs (KGs); (2) creating new ideas by predicting links from the background KGs, by combining graph attention and contextual text attention; (3) incrementally writing some key elements of a new paper based on memory-attention networks: from the input title along with predicted related entities to generate a paper abstract, from the abstract to generate conclusion and future work, and finally from future work to generate a title for a follow-on paper.

Graph Attention Knowledge Graphs +4

Genre Separation Network with Adversarial Training for Cross-genre Relation Extraction

no code implementations EMNLP 2018 Ge Shi, Chong Feng, Lifu Huang, Boliang Zhang, Heng Ji, Lejian Liao, He-Yan Huang

Relation Extraction suffers from dramatical performance decrease when training a model on one genre and directly applying it to a new genre, due to the distinct feature distributions.

Feature Engineering Relation +2

Chengyu Cloze Test

1 code implementation WS 2018 Zhiying Jiang, Boliang Zhang, Lifu Huang, Heng Ji

We present a neural recommendation model for Chengyu, which is a special type of Chinese idiom.

Cloze Test Sentence

Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension

no code implementations NAACL 2018 Bhavana Dalvi Mishra, Lifu Huang, Niket Tandon, Wen-tau Yih, Peter Clark

The new dataset, ProPara, is the first to contain natural (rather than machine-generated) text about a changing world along with a full annotation of entity states (location and existence) during those changes (81k datapoints).

Procedural Text Understanding

Paper Abstract Writing through Editing Mechanism

2 code implementations ACL 2018 Qingyun Wang, Zhi-Hao Zhou, Lifu Huang, Spencer Whitehead, Boliang Zhang, Heng Ji, Kevin Knight

We present a paper abstract writing system based on an attentive neural sequence-to-sequence model that can take a title as input and automatically generate an abstract.

Paper generation

Entity-aware Image Caption Generation

no code implementations EMNLP 2018 Di Lu, Spencer Whitehead, Lifu Huang, Heng Ji, Shih-Fu Chang

Current image captioning approaches generate descriptions which lack specific information, such as named entities that are involved in the images.

Caption Generation Image Captioning

Multi-lingual Common Semantic Space Construction via Cluster-consistent Word Embedding

no code implementations EMNLP 2018 Lifu Huang, Kyunghyun Cho, Boliang Zhang, Heng Ji, Kevin Knight

We construct a multilingual common semantic space based on distributional semantics, where words from multiple languages are projected into a shared space to enable knowledge and resource transfer across languages.

Clustering Word Alignment

Open Relation Extraction and Grounding

no code implementations IJCNLP 2017 Dian Yu, Lifu Huang, Heng Ji

Previous open Relation Extraction (open RE) approaches mainly rely on linguistic patterns and constraints to extract important relational triples from large-scale corpora.

Relation Relation Extraction +3

Learning Phrase Embeddings from Paraphrases with GRUs

no code implementations WS 2017 Zhihao Zhou, Lifu Huang, Heng Ji

Learning phrase representations has been widely explored in many Natural Language Processing (NLP) tasks (e. g., Sentiment Analysis, Machine Translation) and has shown promising improvements.

Machine Translation Sentiment Analysis +1

Improving Slot Filling Performance with Attentive Neural Networks on Dependency Structures

no code implementations EMNLP 2017 Lifu Huang, Avirup Sil, Heng Ji, Radu Florian

Slot Filling (SF) aims to extract the values of certain types of attributes (or slots, such as person:cities\_of\_residence) for a given entity from a large collection of source documents.

Relation Extraction Sentence +2

Zero-Shot Transfer Learning for Event Extraction

1 code implementation ACL 2018 Lifu Huang, Heng Ji, Kyunghyun Cho, Clare R. Voss

Most previous event extraction studies have relied heavily on features derived from annotated event mentions, thus cannot be applied to new event types without annotation effort.

Event Extraction Transfer Learning

Cannot find the paper you are looking for? You can Submit a new open access paper.