Search Results for author: Heng Ji

Found 241 papers, 99 papers with code

A Zero-Shot Claim Detection Framework Using Question Answering

no code implementations COLING 2022 Revanth Gangi Reddy, Sai Chetan Chinthakindi, Yi R. Fung, Kevin Small, Heng Ji

In recent years, there has been an increasing interest in claim detection as an important building block for misinformation detection.

Misinformation object-detection +2

Knowledge-Enriched Natural Language Generation

1 code implementation EMNLP (ACL) 2021 Wenhao Yu, Meng Jiang, Zhiting Hu, Qingyun Wang, Heng Ji, Nazneen Rajani

Knowledge-enriched text generation poses unique challenges in modeling and learning, driving active research in several core directions, ranging from integrated modeling of neural representations and symbolic information in the sequential/hierarchical/graphical structures, learning without direct supervisions due to the cost of structured annotation, efficient optimization and inference with massive and global constraints, to language grounding on multiple modalities, and generative reasoning with implicit commonsense knowledge and background knowledge.

Text Generation

Event Schema Induction with Double Graph Autoencoders

no code implementations NAACL 2022 Xiaomeng Jin, Manling Li, Heng Ji

To induce event schemas from historical events, previous work uses an event-by-event scheme, ignoring the global structure of the entire schema graph.

Enhancing Knowledge Selection for Grounded Dialogues via Document Semantic Graphs

no code implementations NAACL 2022 Sha Li, Mahdi Namazifar, Di Jin, Mohit Bansal, Heng Ji, Yang Liu, Dilek Hakkani-Tur

In this work, we propose to automatically convert the background knowledge documents into document semantic graphs and then perform knowledge selection over such graphs.

Multi-Task Learning Response Generation

New Frontiers of Information Extraction

no code implementations NAACL (ACL) 2022 Muhao Chen, Lifu Huang, Manling Li, Ben Zhou, Heng Ji, Dan Roth

This tutorial targets researchers and practitioners who are interested in AI and ML technologies for structural information extraction (IE) from unstructured textual sources.

Coreference by Appearance: Visually Grounded Event Coreference Resolution

no code implementations CRAC (ACL) 2021 Liming Wang, Shengyu Feng, Xudong Lin, Manling Li, Heng Ji, Shih-Fu Chang

Event coreference resolution is critical to understand events in the growing number of online news with multiple modalities including text, video, speech, etc.

coreference-resolution Coreference Resolution +3

Timeline Summarization based on Event Graph Compression via Time-Aware Optimal Transport

1 code implementation EMNLP 2021 Manling Li, Tengfei Ma, Mo Yu, Lingfei Wu, Tian Gao, Heng Ji, Kathleen McKeown

Timeline Summarization identifies major events from a news collection and describes them following temporal order, with key dates tagged.

Timeline Summarization

Lifelong Event Detection with Knowledge Transfer

1 code implementation EMNLP 2021 Pengfei Yu, Heng Ji, Prem Natarajan

We focus on lifelong event detection as an exemplar case and propose a new problem formulation that is also generalizable to other IE tasks.

Event Detection Transfer Learning

Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries

1 code implementation EMNLP 2021 Carl Edwards, ChengXiang Zhai, Heng Ji

Moreover, this can be viewed as an especially challenging cross-lingual retrieval problem by considering the molecules as a language with a very unique grammar.

Cross-Modal Retrieval Natural Language Queries +1

EventKE: Event-Enhanced Knowledge Graph Embedding

no code implementations Findings (EMNLP) 2021 Zixuan Zhang, Hongwei Wang, Han Zhao, Hanghang Tong, Heng Ji

Relations in most of the traditional knowledge graphs (KGs) only reflect static and factual connections, but fail to represent the dynamic activities and state changes about entities.

Knowledge Graph Embedding Knowledge Graphs

Personalized Entity Resolution with Dynamic Heterogeneous KnowledgeGraph Representations

no code implementations ACL (ECNLP) 2021 Ying Lin, Han Wang, Jiangning Chen, Tong Wang, Yue Liu, Heng Ji, Yang Liu, Premkumar Natarajan

We first build a cross-source heterogeneous knowledge graph from customer purchase history and product knowledge graph to jointly learn customer and product embeddings.

Entity Resolution

COVID-19 Claim Radar: A Structured Claim Extraction and Tracking System

1 code implementation ACL 2022 Manling Li, Revanth Gangi Reddy, Ziqi Wang, Yi-shyuan Chiang, Tuan Lai, Pengfei Yu, Zixuan Zhang, Heng Ji

To tackle the challenge of accurate and timely communication regarding the COVID-19 pandemic, we present a COVID-19 Claim Radar to automatically extract supporting and refuting claims on a daily basis.

Semi-supervised New Event Type Induction and Event Detection

no code implementations EMNLP 2020 Lifu Huang, Heng Ji

We design a Semi-Supervised Vector Quantized Variational Autoencoder framework to automatically learn a discrete latent type representation for each seen and unseen type and optimize them using seen type event annotations.

Event Detection Event Extraction +1

OpenPI-C: A Better Benchmark and Stronger Baseline for Open-Vocabulary State Tracking

1 code implementation1 Jun 2023 Xueqing Wu, Sha Li, Heng Ji

Open-vocabulary state tracking is a more practical version of state tracking that aims to track state changes of entities throughout a process without restricting the state space and entity space.

From Adversarial Arms Race to Model-centric Evaluation: Motivating a Unified Automatic Robustness Evaluation Framework

1 code implementation29 May 2023 Yangyi Chen, Hongcheng Gao, Ganqu Cui, Lifan Yuan, Dehan Kong, Hanlu Wu, Ning Shi, Bo Yuan, Longtao Huang, Hui Xue, Zhiyuan Liu, Maosong Sun, Heng Ji

In our experiments, we conduct a robustness evaluation of RoBERTa models to demonstrate the effectiveness of our evaluation framework, and further show the rationality of each component in the framework.

Adversarial Attack

Enhanced Chart Understanding in Vision and Language Task via Cross-modal Pre-training on Plot Table Pairs

no code implementations29 May 2023 Mingyang Zhou, Yi R. Fung, Long Chen, Christopher Thomas, Heng Ji, Shih-Fu Chang

Building cross-model intelligence that can understand charts and communicate the salient information hidden behind them is an appealing challenge in the vision and language(V+L) community.

Chart Question Answering Question Answering +1

Self Information Update for Large Language Models through Mitigating Exposure Bias

no code implementations29 May 2023 Pengfei Yu, Heng Ji

For instance, we can use the latest news articles to update the LLMs' existing knowledge.

Zero- and Few-Shot Event Detection via Prompt-Based Meta Learning

no code implementations27 May 2023 Zhenrui Yue, Huimin Zeng, Mengfei Lan, Heng Ji, Dong Wang

With emerging online topics as a source for numerous new events, detecting unseen / rare event types presents an elusive challenge for existing event detection methods, where only limited data access is provided for training.

Event Detection Meta-Learning

Non-Sequential Graph Script Induction via Multimedia Grounding

1 code implementation27 May 2023 Yu Zhou, Sha Li, Manling Li, Xudong Lin, Shih-Fu Chang, Mohit Bansal, Heng Ji

To automate the induction of such graph scripts for given tasks, we propose to take advantage of loosely aligned videos of people performing the tasks.

Measuring the Effect of Influential Messages on Varying Personas

no code implementations25 May 2023 Chenkai Sun, Jinning Li, Hou Pong Chan, ChengXiang Zhai, Heng Ji

Our analysis shows that the best-performing models are capable of predicting responses that are consistent with the personas, and as a byproduct, the task formulation also enables many interesting applications in the analysis of social network groups and their opinions, such as the discovery of extreme opinion groups.

Meta-review Generation with Checklist-guided Iterative Introspection

1 code implementation24 May 2023 Qi Zeng, Mankeerat Sidhu, Hou Pong Chan, Lu Wang, Heng Ji

Opinions in the scientific domain can be divergent, leading to controversy or consensus among reviewers.

Review Generation Text Generation

CREATOR: Disentangling Abstract and Concrete Reasonings of Large Language Models through Tool Creation

no code implementations23 May 2023 Cheng Qian, Chi Han, Yi R. Fung, Yujia Qin, Zhiyuan Liu, Heng Ji

Large Language Models (LLMs) have demonstrated significant progress in utilizing external APIs as tools for various tasks.

Transfer Learning

Interpretable Automatic Fine-grained Inconsistency Detection in Text Summarization

1 code implementation23 May 2023 Hou Pong Chan, Qi Zeng, Heng Ji

Motivated by how humans inspect factual inconsistency in summaries, we propose an interpretable fine-grained inconsistency detection model, FineGrainFact, which explicitly represents the facts in the documents and summaries with semantic frames extracted by semantic role labeling, and highlights the related semantic frames to predict inconsistency.

Semantic Role Labeling Text Summarization

ManiTweet: A New Benchmark for Identifying Manipulation of News on Social Media

no code implementations23 May 2023 Kung-Hsiang Huang, Hou Pong Chan, Kathleen McKeown, Heng Ji

We present a novel task, identifying manipulation of news on social media, which aims to detect manipulation in social media posts and identify manipulated or inserted information.

Fact Checking

LM-Switch: Lightweight Language Model Conditioning in Word Embedding Space

no code implementations22 May 2023 Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, Heng Ji

As pre-training and fine-tuning are costly and might negatively impact model performance, it is desired to efficiently adapt an existing model to different conditions such as styles, sentiments or narratives, when facing different audiences or scenarios.

Language Modelling Word Embeddings

In-Context Learning of Large Language Models Explained as Kernel Regression

no code implementations22 May 2023 Chi Han, Ziqi Wang, Han Zhao, Heng Ji

In this paper, we investigate the reason why a transformer-based language model can accomplish in-context learning after pre-training on a general language corpus by proposing one hypothesis that LLMs can simulate kernel regression algorithms when faced with in-context examples.

Bayesian Inference Language Modelling +2

Logical Entity Representation in Knowledge-Graphs for Differentiable Rule Learning

1 code implementation22 May 2023 Chi Han, Qizheng He, Charles Yu, Xinya Du, Hanghang Tong, Heng Ji

A LERP is designed as a vector of probabilistic logical functions on the entity's neighboring sub-graph.

Understanding the Effect of Data Augmentation on Knowledge Distillation

no code implementations21 May 2023 Ziqi Wang, Chi Han, Wenxuan Bao, Heng Ji

However, such data augmentation methods are sub-optimal for knowledge distillation since the teacher model could provide label distributions and is more tolerant to semantic shifts.

Data Augmentation Knowledge Distillation

Inference-time Re-ranker Relevance Feedback for Neural Information Retrieval

no code implementations19 May 2023 Revanth Gangi Reddy, Pradeep Dasigi, Md Arafat Sultan, Arman Cohan, Avirup Sil, Heng Ji, Hannaneh Hajishirzi

Neural information retrieval often adopts a retrieve-and-rerank framework: a bi-encoder network first retrieves K (e. g., 100) candidates that are then re-ranked using a more powerful cross-encoder model to rank the better candidates higher.

Information Retrieval Retrieval

Paxion: Patching Action Knowledge in Video-Language Foundation Models

1 code implementation18 May 2023 Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, Heng Ji

The Paxion framework utilizes a Knowledge Patcher network to encode new action knowledge and a Knowledge Fuser component to integrate the Patcher into frozen VidLMs without compromising their existing capabilities.

Action Understanding Object Recognition

LeTI: Learning to Generate from Textual Interactions

1 code implementation17 May 2023 Xingyao Wang, Hao Peng, Reyhaneh Jabbarvand, Heng Ji

LeTI iteratively fine-tunes the model, using the LM objective, on a concatenation of natural language instructions, LM-generated programs, and textual feedback, which is only provided when the generated program fails to solve the task.

Code Generation Event Argument Extraction

Zero-shot Faithful Factual Error Correction

1 code implementation13 May 2023 Kung-Hsiang Huang, Hou Pong Chan, Heng Ji

Faithfully correcting factual errors is critical for maintaining the integrity of textual knowledge bases and preventing hallucinations in sequence-to-sequence models.

SmartBook: AI-Assisted Situation Report Generation

1 code implementation25 Mar 2023 Revanth Gangi Reddy, Yi R. Fung, Qi Zeng, Manling Li, Ziqi Wang, Paul Sullivan, Heng Ji

Further, experiments show that expert analysts tend to add more information into the SmartBook reports, with only 2. 3% of the existing tokens being deleted, meaning SmartBook can serve as a useful foundation for analysts to build upon when creating intelligence reports.

Decision Making

GLEN: General-Purpose Event Detection for Thousands of Types

no code implementations16 Mar 2023 Qiusi Zhan, Sha Li, Kathryn Conger, Martha Palmer, Heng Ji, Jiawei Han

The development of event extraction systems has been hindered by the absence of wide-coverage, large-scale datasets.

Event Detection Event Extraction

Ensemble Transfer Learning for Multilingual Coreference Resolution

no code implementations22 Jan 2023 Tuan Manh Lai, Heng Ji

Leveraging the idea that the coreferential links naturally exist between anchor texts pointing to the same article, our method builds a sizeable distantly-supervised dataset for the target language that consists of tens of thousands of documents.

coreference-resolution Coreference Resolution +2

SumREN: Summarizing Reported Speech about Events in News

1 code implementation2 Dec 2022 Revanth Gangi Reddy, Heba Elfardy, Hou Pong Chan, Kevin Small, Heng Ji

A primary objective of news articles is to establish the factual record for an event, frequently achieved by conveying both the details of the specified event (i. e., the 5 Ws; Who, What, Where, When and Why regarding the event) and how people reacted to it (i. e., reported statements).

Document Summarization Multi-Document Summarization +1

Open Relation and Event Type Discovery with Type Abstraction

1 code implementation30 Nov 2022 Sha Li, Heng Ji, Jiawei Han

To tackle this problem, we introduce the idea of type abstraction, where the model is prompted to generalize and name the type.

Event Extraction Relation Extraction +1

ADEPT: A DEbiasing PrompT Framework

1 code implementation10 Nov 2022 Ke Yang, Charles Yu, Yi Fung, Manling Li, Heng Ji

Despite this, relatively few efforts have been made to debias PLMs by prompt tuning with continuous prompts compared to its discrete counterpart.

Language Modelling Word Embeddings

Zero-Shot Classification by Logical Reasoning on Natural Language Explanations

1 code implementation7 Nov 2022 Chi Han, Hengzhi Pei, Xinya Du, Heng Ji

To this end, we propose the framework CLORE (Classification by LOgical Reasoning on Explanations).

Classification Logical Reasoning +1

Video Event Extraction via Tracking Visual States of Arguments

no code implementations3 Nov 2022 Guang Yang, Manling Li, Jiajie Zhang, Xudong Lin, Shih-Fu Chang, Heng Ji

Video event extraction aims to detect salient events from a video and identify the arguments for each event as well as their semantic roles.

Event Extraction

Open-Vocabulary Argument Role Prediction for Event Extraction

1 code implementation3 Nov 2022 Yizhu Jiao, Sha Li, Yiqing Xie, Ming Zhong, Heng Ji, Jiawei Han

Specifically, we formulate the role prediction problem as an in-filling task and construct prompts for a pre-trained language model to generate candidate roles.

Event Extraction Language Modelling

A Close Look into the Calibration of Pre-trained Language Models

1 code implementation31 Oct 2022 Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, Heng Ji

We observe a consistent change in calibration performance across six factors.

Code4Struct: Code Generation for Few-Shot Event Structure Prediction

1 code implementation23 Oct 2022 Xingyao Wang, Sha Li, Heng Ji

As a case study, we formulate Event Argument Extraction (EAE) as converting text into event-argument structures that can be represented as a class object using code.

Code Generation Event Argument Extraction +2

Language Model Pre-Training with Sparse Latent Typing

1 code implementation23 Oct 2022 Liliang Ren, Zixuan Zhang, Han Wang, Clare R. Voss, ChengXiang Zhai, Heng Ji

Modern large-scale Pre-trained Language Models (PLMs) have achieved tremendous success on a wide range of downstream tasks.

Ranked #3 on Few-shot NER on Few-NERD (INTRA) (using extra training data)

Few-shot NER Language Modelling

Weakly-Supervised Temporal Article Grounding

1 code implementation22 Oct 2022 Long Chen, Yulei Niu, Brian Chen, Xudong Lin, Guangxing Han, Christopher Thomas, Hammad Ayyubi, Heng Ji, Shih-Fu Chang

Specifically, given an article and a relevant video, WSAG aims to localize all ``groundable'' sentences to the video, and these sentences are possibly at different semantic scales.

Natural Language Queries Video Grounding

Augmentation with Projection: Towards an Effective and Efficient Data Augmentation Paradigm for Distillation

1 code implementation21 Oct 2022 Ziqi Wang, Yuexin Wu, Frederick Liu, Daogao Liu, Le Hou, Hongkun Yu, Jing Li, Heng Ji

However, these data augmentation methods either potentially cause shifts in decision boundaries (representation interpolation), are not expressive enough (token replacement), or introduce too much computational overhead (augmentation with models).

Data Augmentation Knowledge Distillation

NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly

no code implementations16 Oct 2022 Yi R. Fung, Tuhin Chakraborty, Hao Guo, Owen Rambow, Smaranda Muresan, Heng Ji

Norm discovery is important for understanding and reasoning about the acceptable behaviors and potential violations in human communication and interactions.

Cultural Vocal Bursts Intensity Prediction Language Modelling

Towards a Unified Multi-Dimensional Evaluator for Text Generation

1 code implementation13 Oct 2022 Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, PengFei Liu, Chenguang Zhu, Heng Ji, Jiawei Han

We re-frame NLG evaluation as a Boolean Question Answering (QA) task, and by guiding the model with different questions, we can use one evaluator to evaluate from multiple dimensions.

Question Answering Response Generation +3

Learning to Decompose Visual Features with Latent Textual Prompts

no code implementations9 Oct 2022 Feng Wang, Manling Li, Xudong Lin, Hairong Lv, Alexander G. Schwing, Heng Ji

Recent advances in pre-training vision-language models like CLIP have shown great potential in learning transferable visual representations.

Retrieval

Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks

1 code implementation1 Oct 2022 Zhenhailong Wang, Xiaoman Pan, Dian Yu, Dong Yu, Jianshu Chen, Heng Ji

Notably, our proposed $\text{Zemi}_\text{LARGE}$ outperforms T0-3B by 16% on all seven evaluation tasks while being 3. 9x smaller in model size.

Language Modelling Retrieval +1

Dynamic Global Memory for Document-level Argument Extraction

1 code implementation ACL 2022 Xinya Du, Sha Li, Heng Ji

Extracting informative arguments of events from news articles is a challenging problem in information extraction, which requires a global contextual understanding of each document.

Event Argument Extraction

CONCRETE: Improving Cross-lingual Fact-checking with Cross-lingual Retrieval

1 code implementation COLING 2022 Kung-Hsiang Huang, ChengXiang Zhai, Heng Ji

Given the absence of cross-lingual information retrieval datasets with claim-like queries, we train the retriever with our proposed Cross-lingual Inverse Cloze Task (X-ICT), a self-supervised algorithm that creates training instances by translating the title of a passage.

Cross-lingual Fact-checking Cross-Lingual Information Retrieval +4

Incorporating Task-specific Concept Knowledge into Script Learning

1 code implementation31 Aug 2022 Chenkai Sun, Tie XU, ChengXiang Zhai, Heng Ji

In this paper, we present Tetris, a new task of Goal-Oriented Script Completion.

Contrastive Learning

Multimedia Generative Script Learning for Task Planning

2 code implementations25 Aug 2022 Qingyun Wang, Manling Li, Hou Pong Chan, Lifu Huang, Julia Hockenmaier, Girish Chowdhary, Heng Ji

Therefore, we propose a new task, Multimedia Generative Script Learning, to generate subsequent steps by tracking historical states in both text and vision modalities, as well as presenting the first benchmark containing 2, 338 tasks and 31, 496 steps with descriptive images.

Contrastive Learning Multimedia Generative Script Learning +2

Enhanced Knowledge Selection for Grounded Dialogues via Document Semantic Graphs

no code implementations15 Jun 2022 Sha Li, Mahdi Namazifar, Di Jin, Mohit Bansal, Heng Ji, Yang Liu, Dilek Hakkani-Tur

Providing conversation models with background knowledge has been shown to make open-domain dialogues more informative and engaging.

Multi-Task Learning Response Generation

Schema-Guided Event Graph Completion

no code implementations6 Jun 2022 Hongwei Wang, Zixuan Zhang, Sha Li, Jiawei Han, Yizhou Sun, Hanghang Tong, Joseph P. Olive, Heng Ji

Existing link prediction or graph completion methods have difficulty dealing with event graphs because they are usually designed for a single large graph such as a social network or a knowledge graph, rather than multiple small dynamic event graphs.

Link Prediction

Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval

no code implementations CVPR 2023 Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, Shih-Fu Chang

We surprisingly find that discrete text tokens coupled with a pretrained contrastive text model yields the best performance, which can even outperform state-of-the-art on the iVQA and How2QA datasets without additional training on millions of video-text data.

Retrieval Sentence Embeddings +1

Seeded Hierarchical Clustering for Expert-Crafted Taxonomies

no code implementations23 May 2022 Anish Saha, Amith Ananthram, Emily Allaway, Heng Ji, Kathleen McKeown

Practitioners from many disciplines (e. g., political science) use expert-crafted taxonomies to make sense of large, unlabeled corpora.

What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveys

no code implementations23 May 2022 Yubin Ge, Ziang Xiao, Jana Diesner, Heng Ji, Karrie Karahalios, Hari Sundaram

Conversational surveys, where an agent asks open-ended questions through natural language interfaces, offer a new way to collect information from people.

Question Generation Question-Generation

Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners

1 code implementation22 May 2022 Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, ZiYi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, Heng Ji

The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples, such as domain-specific captioning, question answering, and future event prediction.

Automatic Speech Recognition Automatic Speech Recognition (ASR) +5

Translation between Molecules and Natural Language

1 code implementation25 Apr 2022 Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, Heng Ji

We present $\textbf{MolT5}$ $-$ a self-supervised learning framework for pretraining models on a vast amount of unlabeled natural language text and molecule strings.

Molecule Captioning Self-Supervised Learning +2

Entity-Conditioned Question Generation for Robust Attention Distribution in Neural Information Retrieval

1 code implementation24 Apr 2022 Revanth Gangi Reddy, Md Arafat Sultan, Martin Franz, Avirup Sil, Heng Ji

On two public IR benchmarks, we empirically show that the proposed method helps improve both the model's attention patterns and retrieval performance, including in zero-shot settings.

Information Retrieval Question Generation +3

Faking Fake News for Real Fake News Detection: Propaganda-loaded Training Data Generation

1 code implementation10 Mar 2022 Kung-Hsiang Huang, Kathleen McKeown, Preslav Nakov, Yejin Choi, Heng Ji

Despite recent advances in detecting fake news generated by neural models, their results are not readily applicable to effective detection of human-written disinformation.

Fake News Detection Natural Language Inference +1

A Weibo Dataset for the 2022 Russo-Ukrainian Crisis

1 code implementation9 Mar 2022 Yi R. Fung, Heng Ji

Online social networks such as Twitter and Weibo play an important role in how people stay informed and exchange reactions.

Misinformation

Rethinking Task Sampling for Few-shot Vision-Language Transfer Learning

1 code implementation MMMPIE (COLING) 2022 Zhenhailong Wang, Hang Yu, Manling Li, Han Zhao, Heng Ji

While much literature has been devoted to exploring alternative optimization strategies, we identify another essential aspect towards effective few-shot transfer learning, task sampling, which is previously only be viewed as part of data pre-processing in MAML.

Few-Shot Learning Transfer Learning

Semi-supervised New Event Type Induction and Description via Contrastive Loss-Enforced Batch Attention

no code implementations12 Feb 2022 Carl Edwards, Heng Ji

In contrast, we present a novel approach to semi-supervised new event type induction using a masked contrastive loss, which learns similarities between event mentions by enforcing an attention mechanism over the data minibatch.

Event Extraction

CLIP-Event: Connecting Text and Images with Event Structures

1 code implementation CVPR 2022 Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, Shih-Fu Chang

Vision-language (V+L) pretraining models have achieved great success in supporting multimedia applications by understanding the alignments between images and text.

Contrastive Learning Event Extraction +2

MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding

2 code implementations20 Dec 2021 Revanth Gangi Reddy, Xilin Rui, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avirup Sil, Shih-Fu Chang, Alexander Schwing, Heng Ji

Specifically, the task involves multi-hop questions that require reasoning over image-caption pairs to identify the grounded visual object being referred to and then predicting a span from the news body text to answer the question.

Answer Generation Data Augmentation +2

Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences

1 code implementation NAACL 2022 Yifan Chen, Qi Zeng, Dilek Hakkani-Tur, Di Jin, Heng Ji, Yun Yang

Transformer-based models are not efficient in processing long sequences due to the quadratic space and time complexity of the self-attention modules.

Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification

1 code implementation5 Dec 2021 Zhenhailong Wang, Heng Ji

State-of-the-art brain-to-text systems have achieved great success in decoding language directly from brain signals using neural networks.

Electroencephalogram (EEG) Sentiment Analysis +1

Skyformer: Remodel Self-Attention with Gaussian Kernel and Nyström Method

1 code implementation NeurIPS 2021 Yifan Chen, Qi Zeng, Heng Ji, Yun Yang

Transformers are expensive to train due to the quadratic time and space complexity in the self-attention mechanism.

Learning Invariant Representations on Multilingual Language Models for Unsupervised Cross-Lingual Transfer

no code implementations ICLR 2022 Ruicheng Xian, Heng Ji, Han Zhao

Recent advances in neural modeling have produced deep multilingual language models capable of extracting cross-lingual knowledge from unparallel texts, as evidenced by their decent zero-shot transfer performance.

Cross-Lingual Transfer Domain Adaptation

Joint Multimedia Event Extraction from Video and Article

no code implementations Findings (EMNLP) 2021 Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji, Shih-Fu Chang

We introduce the new task of Video MultiMedia Event Extraction (Video M2E2) and propose two novel components to build the first system towards this task.

coreference-resolution Coreference Resolution +2

Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training

1 code implementation EMNLP 2021 Yu Meng, Yunyi Zhang, Jiaxin Huang, Xuan Wang, Yu Zhang, Heng Ji, Jiawei Han

We study the problem of training named entity recognition (NER) models using only distantly-labeled data, which can be automatically obtained by matching entity mentions in the raw text with entity types in a knowledge base.

Language Modelling named-entity-recognition +2

Corpus-based Open-Domain Event Type Induction

1 code implementation EMNLP 2021 Jiaming Shen, Yunyi Zhang, Heng Ji, Jiawei Han

As events of the same type could be expressed in multiple ways, we propose to represent each event type as a cluster of <predicate sense, object head> pairs.

Event Extraction Vocal Bursts Type Prediction

IMG2SMI: Translating Molecular Structure Images to Simplified Molecular-input Line-entry System

no code implementations3 Sep 2021 Daniel Campos, Heng Ji

A large portion of chemistry literature focuses on new molecules and reactions between molecules.

Image Captioning

A Unified Transformer-based Framework for Duplex Text Normalization

no code implementations23 Aug 2021 Tuan Manh Lai, Yang Zhang, Evelina Bakhturina, Boris Ginsburg, Heng Ji

In addition, we also create a cleaned dataset from the Spoken Wikipedia Corpora for German and report the performance of our systems on the dataset.

Automatic Speech Recognition Automatic Speech Recognition (ASR) +4

Fine-grained Information Extraction from Biomedical Literature based on Knowledge-enriched Abstract Meaning Representation

no code implementations ACL 2021 Zixuan Zhang, Nikolaus Parulian, Heng Ji, Ahmed Elsayed, Skatje Myers, Martha Palmer

In this paper, we propose a novel biomedical Information Extraction (IE) model to tackle these two challenges and extract scientific entities and events from English research papers.

Event Extraction Graph Attention

Event-Centric Natural Language Processing

no code implementations ACL 2021 Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji, Kathleen McKeown, Dan Roth

This tutorial targets researchers and practitioners who are interested in AI technologies that help machines understand natural language text, particularly real-world events described in the text.

HySPA: Hybrid Span Generation for Scalable Text-to-Graph Extraction

1 code implementation Findings (ACL) 2021 Liliang Ren, Chenkai Sun, Heng Ji, Julia Hockenmaier

Text-to-Graph extraction aims to automatically extract information graphs consisting of mentions and types from natural language texts.

 Ranked #1 on Relation Extraction on ACE 2005 (Sentence Encoder metric)

Joint Entity and Relation Extraction

Deep Learning on Graphs for Natural Language Processing

no code implementations NAACL 2021 Lingfei Wu, Yu Chen, Heng Ji, Yunyao Li

Due to its great power in modeling non-Euclidean data like graphs or manifolds, deep learning on graph techniques (i. e., Graph Neural Networks (GNNs)) have opened a new door to solving challenging graph-related NLP problems.

graph construction Graph Representation Learning +9

Event Time Extraction and Propagation via Graph Attention Networks

1 code implementation NAACL 2021 Haoyang Wen, Yanru Qu, Heng Ji, Qiang Ning, Jiawei Han, Avi Sil, Hanghang Tong, Dan Roth

Grounding events into a precise timeline is important for natural language understanding but has received limited attention in recent work.

Graph Attention Natural Language Understanding +2

Abstract Meaning Representation Guided Graph Encoding and Decoding for Joint Information Extraction

1 code implementation NAACL 2021 Zixuan Zhang, Heng Ji

The tasks of Rich Semantic Parsing, such as Abstract Meaning Representation (AMR), share similar goals with Information Extraction (IE) to convert natural language texts into structured semantic representations.

Semantic Parsing

Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr\"om Method

1 code implementation NeurIPS 2021 Yifan Chen, Qi Zeng, Heng Ji, Yun Yang

Transformers are expensive to train due to the quadratic time and space complexity in the self-attention mechanism.

Stage-wise Fine-tuning for Graph-to-Text Generation

1 code implementation ACL 2021 Qingyun Wang, Semih Yavuz, Victoria Lin, Heng Ji, Nazneen Rajani

Graph-to-text generation has benefited from pre-trained language models (PLMs) in achieving better performance than structured graph encoders.

Ranked #3 on Data-to-Text Generation on WebNLG (using extra training data)

Data-to-Text Generation KB-to-Language Generation +1

VAULT: VAriable Unified Long Text Representation for Machine Reading Comprehension

no code implementations ACL 2021 Haoyang Wen, Anthony Ferritto, Heng Ji, Radu Florian, Avirup Sil

Existing models on Machine Reading Comprehension (MRC) require complex model architecture for effectively modeling long texts with paragraph representation and classification, thereby making inference computationally inefficient for production use.

Machine Reading Comprehension Natural Questions

Learning Shared Semantic Space for Speech-to-Text Translation

2 code implementations Findings (ACL) 2021 Chi Han, Mingxuan Wang, Heng Ji, Lei LI

By projecting audio and text features to a common semantic representation, Chimera unifies MT and ST tasks and boosts the performance on ST benchmarks, MuST-C and Augmented Librispeech, to a new state-of-the-art.

Machine Translation Speech-to-Text Translation +1

Towards Robust Neural Retrieval Models with Synthetic Pre-Training

no code implementations15 Apr 2021 Revanth Gangi Reddy, Vikas Yadav, Md Arafat Sultan, Martin Franz, Vittorio Castelli, Heng Ji, Avirup Sil

Recent work has shown that commonly available machine reading comprehension (MRC) datasets can be used to train high-performance neural information retrieval (IR) systems.

Information Retrieval Machine Reading Comprehension +1

The Future is not One-dimensional: Complex Event Schema Induction by Graph Modeling for Event Prediction

1 code implementation EMNLP 2021 Manling Li, Sha Li, Zhenhailong Wang, Lifu Huang, Kyunghyun Cho, Heng Ji, Jiawei Han, Clare Voss

We introduce a new concept of Temporal Complex Event Schema: a graph-based schema representation that encompasses events, arguments, temporal connections and argument relations.

Document-Level Event Argument Extraction by Conditional Generation

1 code implementation NAACL 2021 Sha Li, Heng Ji, Jiawei Han

On the task of argument extraction, we achieve an absolute gain of 7. 6% F1 and 5. 7% F1 over the next best model on the RAMS and WikiEvents datasets respectively.

Document-level Event Extraction Event Argument Extraction +1

Personalized Entity Resolution with Dynamic Heterogeneous Knowledge Graph Representations

no code implementations6 Apr 2021 Ying Lin, Han Wang, Jiangning Chen, Tong Wang, Yue Liu, Heng Ji, Yang Liu, Premkumar Natarajan

For example, with "add milk to my cart", a customer may refer to a certain organic product, while some customers may want to re-order products they regularly purchase.

Entity Resolution

Efficient Attentions for Long Document Summarization

1 code implementation NAACL 2021 Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, Lu Wang

The quadratic computational and memory complexities of large Transformers have limited their scalability for long document summarization.

Document Summarization

Controllable and Diverse Text Generation in E-commerce

no code implementations23 Feb 2021 Huajie Shao, Jun Wang, Haohong Lin, Xuezhou Zhang, Aston Zhang, Heng Ji, Tarek Abdelzaher

The algorithm is injected into a Conditional Variational Autoencoder (CVAE), allowing \textit{Apex} to control both (i) the order of keywords in the generated sentences (conditioned on the input keywords and their order), and (ii) the trade-off between diversity and accuracy.

Text Generation

White Paper: Challenges and Considerations for the Creation of a Large Labelled Repository of Online Videos with Questionable Content

no code implementations25 Jan 2021 Thamar Solorio, Mahsa Shafaei, Christos Smailis, Mona Diab, Theodore Giannakopoulos, Heng Ji, Yang Liu, Rada Mihalcea, Smaranda Muresan, Ioannis Kakadiaris

This white paper presents a summary of the discussions regarding critical considerations to develop an extensive repository of online videos annotated with labels indicating questionable content.

MUSE: Textual Attributes Guided Portrait Painting Generation

1 code implementation9 Nov 2020 Xiaodan Hu, Pengfei Yu, Kevin Knight, Heng Ji, Bo Li, Honghui Shi

Experiments show that our approach can accurately illustrate 78% textual attributes, which also help MUSE capture the subject in a more creative and expressive way.

KompaRe: A Knowledge Graph Comparative Reasoning System

no code implementations6 Nov 2020 Lihui Liu, Boxin Du, Heng Ji, Hanghang Tong

In detail, we develop KompaRe, the first of its kind prototype system that provides comparative reasoning capability over large knowledge graphs.

Knowledge Graphs

Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation

2 code implementations24 Oct 2020 Yuning Mao, Xiang Ren, Heng Ji, Jiawei Han

Despite significant progress, state-of-the-art abstractive summarization methods are still prone to hallucinate content inconsistent with the source document.

Abstractive Text Summarization Keyphrase Extraction

Global Attention for Name Tagging

no code implementations CONLL 2018 Boliang Zhang, Spencer Whitehead, Lifu Huang, Heng Ji

Many name tagging approaches use local contextual information with much success, but fail when the local context is ambiguous or limited.

Text Classification Using Label Names Only: A Language Model Self-Training Approach

2 code implementations EMNLP 2020 Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, Jiawei Han

In this paper, we explore the potential of only using the label name of each class to train classification models on unlabeled data, without using any labeled documents.

Document Classification General Classification +6

ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis

1 code implementation INLG (ACL) 2020 Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, Nazneen Fatema Rajani

To assist human review process, we build a novel ReviewRobot to automatically assign a review score and write comments for multiple categories such as novelty and meaningful comparison.

Review Generation

A Survey of Knowledge-Enhanced Text Generation

3 code implementations9 Oct 2020 Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, Meng Jiang

To address this issue, researchers have considered incorporating various forms of knowledge beyond the input text into the generation models.

Text Generation

Minimize Exposure Bias of Seq2Seq Models in Joint Entity and Relation Extraction

1 code implementation Findings of the Association for Computational Linguistics 2020 Ranran Haoran Zhang, Qianying Liu, Aysa Xuemo Fan, Heng Ji, Daojian Zeng, Fei Cheng, Daisuke Kawahara, Sadao Kurohashi

We propose a novel Sequence-to-Unordered-Multi-Tree (Seq2UMTree) model to minimize the effects of exposure bias by limiting the decoding length to three within a triplet and removing the order among triplets.

Joint Entity and Relation Extraction

GAIA: A Fine-grained Multimedia Knowledge Extraction System

no code implementations ACL 2020 Manling Li, Alireza Zareian, Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng Ji, Shih-Fu Chang, Clare Voss, Daniel Napierski, Marjorie Freedman

We present the first comprehensive, open source multimedia knowledge extraction system that takes a massive stream of unstructured, heterogeneous multimedia data from various sources and languages as input, and creates a coherent, structured knowledge base, indexing entities, relations, and events, following a rich, fine-grained ontology.

A Joint Neural Model for Information Extraction with Global Features

no code implementations ACL 2020 Ying Lin, Heng Ji, Fei Huang, Lingfei Wu

OneIE performs end-to-end IE in four stages: (1) Encoding a given sentence as contextualized word representations; (2) Identifying entity mentions and event triggers as nodes; (3) Computing label scores for all nodes and their pairwise links using local classifiers; (4) Searching for the globally optimal graph with a beam decoder.

Cross-media Structured Common Space for Multimedia Event Extraction

no code implementations ACL 2020 Manling Li, Alireza Zareian, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, Shih-Fu Chang

We introduce a new task, MultiMedia Event Extraction (M2E2), which aims to extract events and their arguments from multimedia documents.

Event Extraction

Training with Streaming Annotation

no code implementations11 Feb 2020 Tongtao Zhang, Heng Ji, Shih-Fu Chang, Marjorie Freedman

In this paper, we address a practical scenario where training data is released in a sequence of small-scale batches and annotation in earlier phases has lower quality than the later counterparts.

Event Extraction

Cross-lingual Joint Entity and Word Embedding to Improve Entity Linking and Parallel Sentence Mining

no code implementations WS 2019 Xiaoman Pan, Thamme Gowda, Heng Ji, Jonathan May, Scott Miller

Because this multilingual common space directly relates the semantics of contextual words in the source language to that of entities in the target language, we leverage it for unsupervised cross-lingual entity linking.

Cross-Lingual Entity Linking Entity Linking

An Attentive Fine-Grained Entity Typing Model with Latent Type Representation

no code implementations IJCNLP 2019 Ying Lin, Heng Ji

In addition, we propose a two-step mention-aware attention mechanism to enable the model to focus on important words in mentions and contexts.

Entity Typing Type prediction +2

Cross-lingual Structure Transfer for Relation and Event Extraction

no code implementations IJCNLP 2019 Ananya Subburathinam, Di Lu, Heng Ji, Jonathan May, Shih-Fu Chang, Avirup Sil, Clare Voss

The identification of complex semantic structures such as events and entity relations, already a challenging Information Extraction task, is doubly difficult from sources written in under-resourced and under-annotated languages.

Event Extraction Relation Extraction

Low-Resource Name Tagging Learned with Weakly Labeled Data

1 code implementation IJCNLP 2019 Yixin Cao, Zikun Hu, Tat-Seng Chua, Zhiyuan Liu, Heng Ji

Name tagging in low-resource languages or domains suffers from inadequate training data.

TAG

Raw-to-End Name Entity Recognition in Social Media

1 code implementation14 Aug 2019 Liyuan Liu, Zihan Wang, Jingbo Shang, Dandong Yin, Heng Ji, Xiang Ren, Shaowen Wang, Jiawei Han

Our model neither requires the conversion from character sequences to word sequences, nor assumes tokenizer can correctly detect all word boundaries.

named-entity-recognition Named Entity Recognition +1

Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization

no code implementations ACL 2019 Manling Li, Lingyu Zhang, Heng Ji, Richard J. Radke

Transcripts of natural, multi-person meetings differ significantly from documents like news articles, which can make Natural Language Generation models for generating summaries unfocused.

Meeting Summarization Text Generation

Multilingual Entity, Relation, Event and Human Value Extraction

no code implementations NAACL 2019 Manling Li, Ying Lin, Joseph Hoover, Spencer Whitehead, Clare Voss, Morteza Dehghani, Heng Ji

This paper demonstrates a state-of-the-art end-to-end multilingual (English, Russian, and Ukrainian) knowledge extraction system that can perform entity discovery and linking, relation extraction, event extraction, and coreference.

Event Extraction Relation Extraction