Search Results for author: Heng Ji

Found 298 papers, 141 papers with code

Personalized Entity Resolution with Dynamic Heterogeneous KnowledgeGraph Representations

no code implementations ACL (ECNLP) 2021 Ying Lin, Han Wang, Jiangning Chen, Tong Wang, Yue Liu, Heng Ji, Yang Liu, Premkumar Natarajan

We first build a cross-source heterogeneous knowledge graph from customer purchase history and product knowledge graph to jointly learn customer and product embeddings.

Entity Resolution

Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries

1 code implementation EMNLP 2021 Carl Edwards, ChengXiang Zhai, Heng Ji

Moreover, this can be viewed as an especially challenging cross-lingual retrieval problem by considering the molecules as a language with a very unique grammar.

Cross-Modal Retrieval Natural Language Queries +1

Coreference by Appearance: Visually Grounded Event Coreference Resolution

no code implementations CRAC (ACL) 2021 Liming Wang, Shengyu Feng, Xudong Lin, Manling Li, Heng Ji, Shih-Fu Chang

Event coreference resolution is critical to understand events in the growing number of online news with multiple modalities including text, video, speech, etc.

coreference-resolution Event Coreference Resolution +2

Knowledge-Enriched Natural Language Generation

1 code implementation EMNLP (ACL) 2021 Wenhao Yu, Meng Jiang, Zhiting Hu, Qingyun Wang, Heng Ji, Nazneen Rajani

Knowledge-enriched text generation poses unique challenges in modeling and learning, driving active research in several core directions, ranging from integrated modeling of neural representations and symbolic information in the sequential/hierarchical/graphical structures, learning without direct supervisions due to the cost of structured annotation, efficient optimization and inference with massive and global constraints, to language grounding on multiple modalities, and generative reasoning with implicit commonsense knowledge and background knowledge.

Text Generation

Semi-supervised New Event Type Induction and Event Detection

no code implementations EMNLP 2020 Lifu Huang, Heng Ji

We design a Semi-Supervised Vector Quantized Variational Autoencoder framework to automatically learn a discrete latent type representation for each seen and unseen type and optimize them using seen type event annotations.

Event Detection Event Extraction +1

COVID-19 Claim Radar: A Structured Claim Extraction and Tracking System

1 code implementation ACL 2022 Manling Li, Revanth Gangi Reddy, Ziqi Wang, Yi-shyuan Chiang, Tuan Lai, Pengfei Yu, Zixuan Zhang, Heng Ji

To tackle the challenge of accurate and timely communication regarding the COVID-19 pandemic, we present a COVID-19 Claim Radar to automatically extract supporting and refuting claims on a daily basis.

New Frontiers of Information Extraction

no code implementations NAACL (ACL) 2022 Muhao Chen, Lifu Huang, Manling Li, Ben Zhou, Heng Ji, Dan Roth

This tutorial targets researchers and practitioners who are interested in AI and ML technologies for structural information extraction (IE) from unstructured textual sources.

Enhancing Knowledge Selection for Grounded Dialogues via Document Semantic Graphs

no code implementations NAACL 2022 Sha Li, Mahdi Namazifar, Di Jin, Mohit Bansal, Heng Ji, Yang Liu, Dilek Hakkani-Tur

In this work, we propose to automatically convert the background knowledge documents into document semantic graphs and then perform knowledge selection over such graphs.

Multi-Task Learning Response Generation +1

EventKE: Event-Enhanced Knowledge Graph Embedding

no code implementations Findings (EMNLP) 2021 Zixuan Zhang, Hongwei Wang, Han Zhao, Hanghang Tong, Heng Ji

Relations in most of the traditional knowledge graphs (KGs) only reflect static and factual connections, but fail to represent the dynamic activities and state changes about entities.

Knowledge Graph Embedding Knowledge Graphs +1

Lifelong Event Detection with Knowledge Transfer

1 code implementation EMNLP 2021 Pengfei Yu, Heng Ji, Prem Natarajan

We focus on lifelong event detection as an exemplar case and propose a new problem formulation that is also generalizable to other IE tasks.

Event Detection Transfer Learning

Timeline Summarization based on Event Graph Compression via Time-Aware Optimal Transport

1 code implementation EMNLP 2021 Manling Li, Tengfei Ma, Mo Yu, Lingfei Wu, Tian Gao, Heng Ji, Kathleen McKeown

Timeline Summarization identifies major events from a news collection and describes them following temporal order, with key dates tagged.

Timeline Summarization

Event Schema Induction with Double Graph Autoencoders

no code implementations NAACL 2022 Xiaomeng Jin, Manling Li, Heng Ji

To induce event schemas from historical events, previous work uses an event-by-event scheme, ignoring the global structure of the entire schema graph.

A Zero-Shot Claim Detection Framework Using Question Answering

no code implementations COLING 2022 Revanth Gangi Reddy, Sai Chetan Chinthakindi, Yi R. Fung, Kevin Small, Heng Ji

In recent years, there has been an increasing interest in claim detection as an important building block for misinformation detection.

Misinformation Object +3

Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models

no code implementations10 Jul 2024 Yuji Zhang, Sha Li, Jiateng Liu, Pengfei Yu, Yi R. Fung, Jing Li, Manling Li, Heng Ji

This phenomenon partially stems from training data imbalance, which we verify on both pretrained models and fine-tuned models, over a wide range of LM model families and sizes. From a theoretical point of view, knowledge overshadowing can be interpreted as over-generalization of the dominant conditions (patterns).

Hallucination Language Modelling

Eliminating Position Bias of Language Models: A Mechanistic Approach

no code implementations1 Jul 2024 Ziqi Wang, HANLIN ZHANG, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham M. Kakade, Hao Peng, Heng Ji

Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings.

object-detection Object Detection +3

FIRST: Faster Improved Listwise Reranking with Single Token Decoding

1 code implementation21 Jun 2024 Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, Heng Ji

Further, we incorporate a learning-to-rank loss during training, prioritizing ranking accuracy for the more relevant passages.

Information Retrieval Language Modelling +1

MACAROON: Training Vision-Language Models To Be Your Engaged Partners

1 code implementation20 Jun 2024 Shujin Wu, Yi R. Fung, Sha Li, Yixin Wan, Kai-Wei Chang, Heng Ji

Large vision-language models (LVLMs), while proficient in following instructions and responding to diverse questions, invariably generate detailed responses even when questions are ambiguous or unanswerable, leading to hallucinations and bias issues.

GLaD: Synergizing Molecular Graphs and Language Descriptors for Enhanced Power Conversion Efficiency Prediction in Organic Photovoltaic Devices

no code implementations23 May 2024 Thao Nguyen, Tiara Torres-Flores, Changhyun Hwang, Carl Edwards, Ying Diao, Heng Ji

Especially, GLaD proves valuable for tasks in low-data regimes within the chemical space, as it enriches molecular representations by incorporating molecular property descriptions learned from large-scale pretraining.

Decision Making Efficient Exploration +2

Weak-to-Strong Extrapolation Expedites Alignment

1 code implementation25 Apr 2024 Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, Nanyun Peng

Through experiments with twelve open-source LLMs on HuggingFace, we demonstrate that ExPO consistently improves off-the-shelf DPO/RLHF models, as evaluated on the mainstream LLM benchmarks AlpacaEval 2. 0 and MT-Bench.

Text-Based Reasoning About Vector Graphics

no code implementations9 Apr 2024 Zhenhailong Wang, Joy Hsu, Xingyao Wang, Kuan-Hao Huang, Manling Li, Jiajun Wu, Heng Ji

By casting an image to a text-based representation, we can leverage the power of language models to learn alignment from SVG to visual primitives and generalize to unseen question-answering tasks.

Descriptive Language Modelling +2

Fact Checking Beyond Training Set

1 code implementation27 Mar 2024 Payam Karisani, Heng Ji

We then focus on the reader component and propose to train it such that it is insensitive towards the order of claims and evidence documents.

Domain Adaptation Fact Checking

From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models

1 code implementation18 Mar 2024 Kung-Hsiang Huang, Hou Pong Chan, Yi R. Fung, Haoyi Qiu, Mingyang Zhou, Shafiq Joty, Shih-Fu Chang, Heng Ji

This survey paper serves as a comprehensive resource for researchers and practitioners in the fields of natural language processing, computer vision, and data analysis, providing valuable insights and directions for future research in chart understanding leveraging large foundation models.

Chart Understanding Data Visualization

Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models

no code implementations26 Feb 2024 Jeonghwan Kim, Heng Ji

Recent advances in instruction-tuned Large Vision-Language Models (LVLMs) have imbued the models with the ability to generate high-level, image-grounded explanations with ease.

Attribute Fine-Grained Visual Categorization +1

L+M-24: Building a Dataset for Language + Molecules @ ACL 2024

1 code implementation22 Feb 2024 Carl Edwards, Qingyun Wang, Lawrence Zhao, Heng Ji

Language-molecule models have emerged as an exciting direction for molecular discovery and understanding.

Entity Linking Property Prediction

EVEDIT: Event-based Knowledge Editing with Deductive Editing Boundaries

no code implementations17 Feb 2024 Jiateng Liu, Pengfei Yu, Yuji Zhang, Sha Li, Zixuan Zhang, Heng Ji

The dynamic nature of real-world information necessitates efficient knowledge editing (KE) in large language models (LLMs) for knowledge updating.

knowledge editing

Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinement

no code implementations16 Feb 2024 Chenkai Sun, Ke Yang, Revanth Gangi Reddy, Yi R. Fung, Hou Pong Chan, ChengXiang Zhai, Heng Ji

The increasing demand for personalized interactions with large language models (LLMs) calls for the development of methodologies capable of accurately and efficiently identifying user opinions and preferences.

Language Modelling Large Language Model +1

Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking

1 code implementation14 Feb 2024 Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, Heng Ji

Pretrained large language models have revolutionized many applications but still face challenges related to cultural bias and a lack of cultural commonsense knowledge crucial for guiding cross-culture communication and interactions.

Benchmarking Language Modelling +1

Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debate

no code implementations12 Feb 2024 Kyungha Kim, Sangyun Lee, Kung-Hsiang Huang, Hou Pong Chan, Manling Li, Heng Ji

Fact-checking research has extensively explored verification but less so the generation of natural-language explanations, crucial for user trust.

Fact Checking Text Generation

Masked LoGoNet: Fast and Accurate 3D Image Analysis for Medical Domain

no code implementations9 Feb 2024 Amin Karimi Monsefi, Payam Karisani, Mengxi Zhou, Stacey Choi, Nathan Doble, Heng Ji, Srinivasan Parthasarathy, Rajiv Ramnath

In this paper, we introduce a new neural network architecture, termed LoGoNet, with a tailored self-supervised learning (SSL) method to mitigate such challenges.

Contrastive Learning Image Segmentation +4

Executable Code Actions Elicit Better LLM Agents

2 code implementations1 Feb 2024 Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, Heng Ji

LLM agents are typically prompted to produce actions by generating JSON or text in a pre-defined format, which is usually limited by constrained action space (e. g., the scope of pre-defined tools) and restricted flexibility (e. g., inability to compose multiple tools).

Language Modelling Large Language Model

Named Entity Recognition Under Domain Shift via Metric Learning for Life Sciences

1 code implementation19 Jan 2024 Hongyi Liu, Qingyun Wang, Payam Karisani, Heng Ji

In our experiments, we observed that such a model is prone to mislabeling the source entities, which can often appear in the text, as the target entities.

Contrastive Learning Few-Shot Learning +4

If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents

no code implementations1 Jan 2024 Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang, Yi R. Fung, Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang, Yiquan Wang, Heng Ji, ChengXiang Zhai

The prominent large language models (LLMs) of today differ from past language models not only in size, but also in the fact that they are trained on a combination of natural language and formal language (code).

Code Generation

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

3 code implementations18 Dec 2023 Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, Tong Zhang

We investigate its behavior in three distinct settings -- offline, online, and hybrid -- and propose efficient algorithms with finite-sample theoretical guarantees.

Language Modelling Large Language Model

Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning

2 code implementations15 Dec 2023 Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R. Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, Heng Ji

This work inaugurates a new domain in factual error correction for chart captions, presenting a novel evaluation mechanism, and demonstrating an effective approach to ensuring the factuality of generated chart captions.

Factual Inconsistency Detection in Chart Captioning Image Captioning +1

RESIN-EDITOR: A Schema-guided Hierarchical Event Graph Visualizer and Editor

1 code implementation5 Dec 2023 Khanh Duy Nguyen, Zixuan Zhang, Reece Suchocki, Sha Li, Martha Palmer, Susan Brown, Jiawei Han, Heng Ji

In this paper, we present RESIN-EDITOR, an interactive event graph visualizer and editor designed for analyzing complex events.

Large Language Models on Graphs: A Comprehensive Survey

1 code implementation5 Dec 2023 Bowen Jin, Gang Liu, Chi Han, Meng Jiang, Heng Ji, Jiawei Han

Besides, although LLMs have shown their pure text-based reasoning ability, it is underexplored whether such ability can be generalized to graphs (i. e., graph-based reasoning).

Language Modelling

InfoPattern: Unveiling Information Propagation Patterns in Social Media

no code implementations27 Nov 2023 Chi Han, Jialiang Xu, Manling Li, Hanning Zhang, Tarek Abdelzaher, Heng Ji

Social media play a significant role in shaping public opinion and influencing ideological communities through information propagation.

Stance Detection

ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation

1 code implementation22 Nov 2023 Yangyi Chen, Xingyao Wang, Manling Li, Derek Hoiem, Heng Ji

We adopt a weakly-supervised approach to directly generate visual event structures from captions for ViStruct training, capitalizing on abundant image-caption pairs from the web.

R-Tuning: Instructing Large Language Models to Say `I Don't Know'

1 code implementation16 Nov 2023 Hanning Zhang, Shizhe Diao, Yong Lin, Yi R. Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, Tong Zhang

This approach is formalized by first identifying the disparity in knowledge encompassed by pre-trained parameters compared to that of instruction tuning data.

Hallucination Sentence

TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction

1 code implementation16 Nov 2023 Kuan-Hao Huang, I-Hung Hsu, Tanmay Parekh, Zhiyu Xie, Zixuan Zhang, Premkumar Natarajan, Kai-Wei Chang, Nanyun Peng, Heng Ji

In this work, we identify and address evaluation challenges, including inconsistency due to varying data assumptions or preprocessing steps, the insufficiency of current evaluation frameworks that may introduce dataset or data split bias, and the low reproducibility of some previous approaches.

Benchmarking Event Extraction

Defining a New NLP Playground

no code implementations31 Oct 2023 Sha Li, Chi Han, Pengfei Yu, Carl Edwards, Manling Li, Xingyao Wang, Yi R. Fung, Charles Yu, Joel R. Tetreault, Eduard H. Hovy, Heng Ji

The recent explosion of performance of large language models (LLMs) has changed the field of Natural Language Processing (NLP) more abruptly and seismically than any other shift in the field's 80-year history.

Instruct and Extract: Instruction Tuning for On-Demand Information Extraction

1 code implementation24 Oct 2023 Yizhu Jiao, Ming Zhong, Sha Li, Ruining Zhao, Siru Ouyang, Heng Ji, Jiawei Han

However, when it comes to information extraction - a classic task in natural language processing - most task-specific systems cannot align well with long-tail ad hoc extraction use cases for non-expert users.

Instruction Following

Monte Carlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Design

1 code implementation22 Oct 2023 Henry W. Sprueill, Carl Edwards, Mariefel V. Olarte, Udishnu Sanyal, Heng Ji, Sutanay Choudhury

Discovering novel catalysts requires complex reasoning involving multiple chemical properties and resultant trade-offs, leading to a combinatorial growth in the search space.

Instruction Following Language Modelling +1

Decoding the Silent Majority: Inducing Belief Augmented Social Graph with Large Language Model for Response Forecasting

1 code implementation20 Oct 2023 Chenkai Sun, Jinning Li, Yi R. Fung, Hou Pong Chan, Tarek Abdelzaher, ChengXiang Zhai, Heng Ji

Automatic response forecasting for news media plays a crucial role in enabling content producers to efficiently predict the impact of news releases and prevent unexpected negative outcomes such as social conflict and moral injury.

Language Modelling Large Language Model

The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions

1 code implementation19 Oct 2023 Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, Jiawei Han

Recent progress in Large Language Models (LLMs) has produced models that exhibit remarkable performance across a variety of NLP tasks.

Enabling Language Models to Implicitly Learn Self-Improvement

no code implementations2 Oct 2023 Ziqi Wang, Le Hou, Tianjian Lu, Yuexin Wu, Yunxuan Li, Hongkun Yu, Heng Ji

Specifically, we reformulate the training objective of reinforcement learning from human feedback (RLHF) -- instead of maximizing response quality for a given input, we maximize the quality gap of the response conditioned on a reference response.

Text Generation

Parameter-Efficient Tuning Helps Language Model Alignment

no code implementations1 Oct 2023 Tianci Xue, Ziqi Wang, Heng Ji

To this end, prior works incorporate controllable generations for alignment to make language models learn multiple preferences and provide outputs with different preferences during inference if asked.

Language Modelling

CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets

1 code implementation29 Sep 2023 Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R. Fung, Hao Peng, Heng Ji

It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets to enhance their capability to solve complex tasks.

Language Modelling Mathematical Reasoning

MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback

1 code implementation19 Sep 2023 Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, Heng Ji

However, current evaluation protocols often emphasize benchmark performance with single-turn exchanges, neglecting the nuanced interactions among the user, LLMs, and external tools, while also underestimating the importance of natural language feedback from users.

Decision Making

Mitigating the Alignment Tax of RLHF

no code implementations12 Sep 2023 Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan YAO, Tong Zhang

Building on the analysis and the observation that averaging different layers of the transformer leads to significantly different reward-tax trade-offs, we propose Adaptive Model Averaging (AMA) to adaptively find various combination ratios of model layers.

Common Sense Reasoning Continual Learning

Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models

1 code implementation8 Sep 2023 Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, Ajay Divakaran

Based on this pipeline and the existing coarse-grained annotated dataset, we build the CURE benchmark to measure both the zero-shot reasoning performance and consistency of VLMs.

Visual Reasoning

LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models

1 code implementation30 Aug 2023 Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, Sinong Wang

As a result, their performance suffers drastically on inputs longer than those encountered during training, substantially limiting their applications in real-world tasks involving long contexts such as encoding scientific articles, code repositories, or long dialogues.

2k 4k +1

Making Pre-trained Language Models both Task-solvers and Self-calibrators

1 code implementation21 Jul 2023 Yangyi Chen, Xingyao Wang, Heng Ji

In this work, we consider the practical scenario that we need to effectively utilize training samples to make PLMs both task-solvers and self-calibrators.

Adversarial Defense

Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration

2 code implementations11 Jul 2023 Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, Heng Ji

In this work, we propose Solo Performance Prompting (SPP), which transforms a single LLM into a cognitive synergist by engaging in multi-turn self-collaboration with multiple personas.

Hallucination Logic Grid Puzzle

C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation

1 code implementation27 Jun 2023 Liliang Ren, Mankeerat Sidhu, Qi Zeng, Revanth Gangi Reddy, Heng Ji, ChengXiang Zhai

Existing reference-free turn-level evaluation metrics for chatbots inadequately capture the interaction between the user and the system.

Dialogue Evaluation

SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design

no code implementations19 Jun 2023 Carl Edwards, Aakanksha Naik, Tushar Khot, Martin Burke, Heng Ji, Tom Hope

We are given a small "personalized dataset" of 10-20 drug synergy relationships in the context of specific cancer cell targets.

In-Context Learning Language Modelling

OpenPI-C: A Better Benchmark and Stronger Baseline for Open-Vocabulary State Tracking

1 code implementation1 Jun 2023 Xueqing Wu, Sha Li, Heng Ji

Open-vocabulary state tracking is a more practical version of state tracking that aims to track state changes of entities throughout a process without restricting the state space and entity space.

From Adversarial Arms Race to Model-centric Evaluation: Motivating a Unified Automatic Robustness Evaluation Framework

1 code implementation29 May 2023 Yangyi Chen, Hongcheng Gao, Ganqu Cui, Lifan Yuan, Dehan Kong, Hanlu Wu, Ning Shi, Bo Yuan, Longtao Huang, Hui Xue, Zhiyuan Liu, Maosong Sun, Heng Ji

In our experiments, we conduct a robustness evaluation of RoBERTa models to demonstrate the effectiveness of our evaluation framework, and further show the rationality of each component in the framework.

Adversarial Attack

Information Association for Language Model Updating by Mitigating LM-Logical Discrepancy

no code implementations29 May 2023 Pengfei Yu, Heng Ji

To evaluate and address the core challenge, we propose a new task formulation of the information updating task that only requires the provision of an unstructured updating corpus and evaluates the performance of information updating on the generalizability to question-answer pairs pertaining to the updating information.

Answer Generation knowledge editing +3

Enhanced Chart Understanding in Vision and Language Task via Cross-modal Pre-training on Plot Table Pairs

no code implementations29 May 2023 Mingyang Zhou, Yi R. Fung, Long Chen, Christopher Thomas, Heng Ji, Shih-Fu Chang

Building cross-model intelligence that can understand charts and communicate the salient information hidden behind them is an appealing challenge in the vision and language(V+L) community.

Chart Question Answering Chart Understanding +2

Zero- and Few-Shot Event Detection via Prompt-Based Meta Learning

1 code implementation27 May 2023 Zhenrui Yue, Huimin Zeng, Mengfei Lan, Heng Ji, Dong Wang

With emerging online topics as a source for numerous new events, detecting unseen / rare event types presents an elusive challenge for existing event detection methods, where only limited data access is provided for training.

Event Detection Meta-Learning

Non-Sequential Graph Script Induction via Multimedia Grounding

1 code implementation27 May 2023 Yu Zhou, Sha Li, Manling Li, Xudong Lin, Shih-Fu Chang, Mohit Bansal, Heng Ji

To automate the induction of such graph scripts for given tasks, we propose to take advantage of loosely aligned videos of people performing the tasks.

Measuring the Effect of Influential Messages on Varying Personas

1 code implementation25 May 2023 Chenkai Sun, Jinning Li, Hou Pong Chan, ChengXiang Zhai, Heng Ji

Our analysis shows that the best-performing models are capable of predicting responses that are consistent with the personas, and as a byproduct, the task formulation also enables many interesting applications in the analysis of social network groups and their opinions, such as the discovery of extreme opinion groups.

Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluation

1 code implementation24 May 2023 Qi Zeng, Mankeerat Sidhu, Ansel Blume, Hou Pong Chan, Lu Wang, Heng Ji

To address this gap, we propose the task of scientific opinion summarization, where research paper reviews are synthesized into meta-reviews.

Opinion Summarization Review Generation +1

ManiTweet: A New Benchmark for Identifying Manipulation of News on Social Media

no code implementations23 May 2023 Kung-Hsiang Huang, Hou Pong Chan, Kathleen McKeown, Heng Ji

We present a novel task, identifying manipulation of news on social media, which aims to detect manipulation in social media posts and identify manipulated or inserted information.

Fact Checking

CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models

2 code implementations23 May 2023 Cheng Qian, Chi Han, Yi R. Fung, Yujia Qin, Zhiyuan Liu, Heng Ji

Additionally, we introduce the Creation Challenge dataset, featuring 2K diverse questions, to emphasize the necessity and benefits of LLMs' tool creation ability.

2k Math +1

Interpretable Automatic Fine-grained Inconsistency Detection in Text Summarization

1 code implementation23 May 2023 Hou Pong Chan, Qi Zeng, Heng Ji

Motivated by how humans inspect factual inconsistency in summaries, we propose an interpretable fine-grained inconsistency detection model, FineGrainFact, which explicitly represents the facts in the documents and summaries with semantic frames extracted by semantic role labeling, and highlights the related semantic frames to predict inconsistency.

Semantic Role Labeling Text Summarization

SciMON: Scientific Inspiration Machines Optimized for Novelty

1 code implementation23 May 2023 Qingyun Wang, Doug Downey, Heng Ji, Tom Hope

We explore and enhance the ability of neural language models to generate novel scientific directions grounded in literature.

Contextualized Literature-based Discovery Link Prediction +1

Logical Entity Representation in Knowledge-Graphs for Differentiable Rule Learning

1 code implementation22 May 2023 Chi Han, Qizheng He, Charles Yu, Xinya Du, Hanghang Tong, Heng Ji

A LERP is designed as a vector of probabilistic logical functions on the entity's neighboring sub-graph.

Link Prediction

Word Embeddings Are Steers for Language Models

1 code implementation22 May 2023 Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, Heng Ji

In this work, we theoretically and empirically revisit output word embeddings and find that their linear transformations are equivalent to steering language model generation styles.

Language Modelling Word Embeddings

Understanding the Effect of Data Augmentation on Knowledge Distillation

no code implementations21 May 2023 Ziqi Wang, Chi Han, Wenxuan Bao, Heng Ji

However, such data augmentation methods are sub-optimal for knowledge distillation since the teacher model could provide label distributions and is more tolerant to semantic shifts.

Data Augmentation Knowledge Distillation

ReFIT: Relevance Feedback from a Reranker during Inference

no code implementations19 May 2023 Revanth Gangi Reddy, Pradeep Dasigi, Md Arafat Sultan, Arman Cohan, Avirup Sil, Heng Ji, Hannaneh Hajishirzi

Retrieve-and-rerank is a prevalent framework in neural information retrieval, wherein a bi-encoder network initially retrieves a pre-defined number of candidates (e. g., K=100), which are then reranked by a more powerful cross-encoder model.

Information Retrieval Retrieval

LeTI: Learning to Generate from Textual Interactions

1 code implementation17 May 2023 Xingyao Wang, Hao Peng, Reyhaneh Jabbarvand, Heng Ji

We explore LMs' potential to learn from textual interactions (LETI) that not only check their correctness with binary labels but also pinpoint and explain errors in their outputs through textual feedback.

Code Generation Event Argument Extraction

Zero-shot Faithful Factual Error Correction

1 code implementation13 May 2023 Kung-Hsiang Huang, Hou Pong Chan, Heng Ji

Faithfully correcting factual errors is critical for maintaining the integrity of textual knowledge bases and preventing hallucinations in sequence-to-sequence models.

SmartBook: AI-Assisted Situation Report Generation for Intelligence Analysts

1 code implementation25 Mar 2023 Revanth Gangi Reddy, Daniel Lee, Yi R. Fung, Khanh Duy Nguyen, Qi Zeng, Manling Li, Ziqi Wang, Clare Voss, Heng Ji

Timely and comprehensive understanding of emerging events is crucial for effective decision-making; automating situation report generation can significantly reduce the time, effort, and cost for intelligence analysts.

Decision Making Language Modelling +1

GLEN: General-Purpose Event Detection for Thousands of Types

1 code implementation16 Mar 2023 Qiusi Zhan, Sha Li, Kathryn Conger, Martha Palmer, Heng Ji, Jiawei Han

Finally, we perform error analysis and show that label noise is still the largest challenge for improving performance for this new dataset.

Event Detection Event Extraction

Ensemble Transfer Learning for Multilingual Coreference Resolution

no code implementations22 Jan 2023 Tuan Manh Lai, Heng Ji

Leveraging the idea that the coreferential links naturally exist between anchor texts pointing to the same article, our method builds a sizeable distantly-supervised dataset for the target language that consists of tens of thousands of documents.

coreference-resolution Question Answering +1

SumREN: Summarizing Reported Speech about Events in News

1 code implementation2 Dec 2022 Revanth Gangi Reddy, Heba Elfardy, Hou Pong Chan, Kevin Small, Heng Ji

A primary objective of news articles is to establish the factual record for an event, frequently achieved by conveying both the details of the specified event (i. e., the 5 Ws; Who, What, Where, When and Why regarding the event) and how people reacted to it (i. e., reported statements).

Document Summarization Multi-Document Summarization +2

Open Relation and Event Type Discovery with Type Abstraction

1 code implementation30 Nov 2022 Sha Li, Heng Ji, Jiawei Han

To tackle this problem, we introduce the idea of type abstraction, where the model is prompted to generalize and name the type.

Event Extraction Relation +2

ADEPT: A DEbiasing PrompT Framework

1 code implementation10 Nov 2022 Ke Yang, Charles Yu, Yi Fung, Manling Li, Heng Ji

Despite this, relatively few efforts have been made to debias PLMs by prompt tuning with continuous prompts compared to its discrete counterpart.

Attribute Language Modelling +1

Zero-Shot Classification by Logical Reasoning on Natural Language Explanations

1 code implementation7 Nov 2022 Chi Han, Hengzhi Pei, Xinya Du, Heng Ji

To this end, we propose the framework CLORE (Classification by LOgical Reasoning on Explanations).

Classification Logical Reasoning +1

Video Event Extraction via Tracking Visual States of Arguments

no code implementations3 Nov 2022 Guang Yang, Manling Li, Jiajie Zhang, Xudong Lin, Shih-Fu Chang, Heng Ji

Video event extraction aims to detect salient events from a video and identify the arguments for each event as well as their semantic roles.

Event Extraction

Open-Vocabulary Argument Role Prediction for Event Extraction

1 code implementation3 Nov 2022 Yizhu Jiao, Sha Li, Yiqing Xie, Ming Zhong, Heng Ji, Jiawei Han

Specifically, we formulate the role prediction problem as an in-filling task and construct prompts for a pre-trained language model to generate candidate roles.

Event Extraction Language Modelling

A Close Look into the Calibration of Pre-trained Language Models

2 code implementations31 Oct 2022 Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, Heng Ji

We observe a consistent change in calibration performance across six factors.

Code4Struct: Code Generation for Few-Shot Event Structure Prediction

1 code implementation23 Oct 2022 Xingyao Wang, Sha Li, Heng Ji

As a case study, we formulate Event Argument Extraction (EAE) as converting text into event-argument structures that can be represented as a class object using code.

Code Generation Event Argument Extraction +3

Language Model Pre-Training with Sparse Latent Typing

1 code implementation23 Oct 2022 Liliang Ren, Zixuan Zhang, Han Wang, Clare R. Voss, ChengXiang Zhai, Heng Ji

Modern large-scale Pre-trained Language Models (PLMs) have achieved tremendous success on a wide range of downstream tasks.

Ranked #6 on Few-shot NER on Few-NERD (INTRA) (using extra training data)

Few-shot NER Language Modelling +1

Weakly-Supervised Temporal Article Grounding

1 code implementation22 Oct 2022 Long Chen, Yulei Niu, Brian Chen, Xudong Lin, Guangxing Han, Christopher Thomas, Hammad Ayyubi, Heng Ji, Shih-Fu Chang

Specifically, given an article and a relevant video, WSAG aims to localize all ``groundable'' sentences to the video, and these sentences are possibly at different semantic scales.

Natural Language Queries Sentence +1

Augmentation with Projection: Towards an Effective and Efficient Data Augmentation Paradigm for Distillation

1 code implementation21 Oct 2022 Ziqi Wang, Yuexin Wu, Frederick Liu, Daogao Liu, Le Hou, Hongkun Yu, Jing Li, Heng Ji

However, these data augmentation methods either potentially cause shifts in decision boundaries (representation interpolation), are not expressive enough (token replacement), or introduce too much computational overhead (augmentation with models).

Data Augmentation Diversity +1

NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly

1 code implementation16 Oct 2022 Yi R. Fung, Tuhin Chakraborty, Hao Guo, Owen Rambow, Smaranda Muresan, Heng Ji

Norm discovery is important for understanding and reasoning about the acceptable behaviors and potential violations in human communication and interactions.

Cultural Vocal Bursts Intensity Prediction Hallucination +1

Towards a Unified Multi-Dimensional Evaluator for Text Generation

2 code implementations13 Oct 2022 Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, PengFei Liu, Chenguang Zhu, Heng Ji, Jiawei Han

We re-frame NLG evaluation as a Boolean Question Answering (QA) task, and by guiding the model with different questions, we can use one evaluator to evaluate from multiple dimensions.

nlg evaluation Question Answering +4

Learning to Decompose Visual Features with Latent Textual Prompts

no code implementations9 Oct 2022 Feng Wang, Manling Li, Xudong Lin, Hairong Lv, Alexander G. Schwing, Heng Ji

Recent advances in pre-training vision-language models like CLIP have shown great potential in learning transferable visual representations.


Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks

1 code implementation1 Oct 2022 Zhenhailong Wang, Xiaoman Pan, Dian Yu, Dong Yu, Jianshu Chen, Heng Ji

Notably, our proposed $\text{Zemi}_\text{LARGE}$ outperforms T0-3B by 16% on all seven evaluation tasks while being 3. 9x smaller in model size.

Language Modelling Retrieval +2

Dynamic Global Memory for Document-level Argument Extraction

1 code implementation ACL 2022 Xinya Du, Sha Li, Heng Ji

Extracting informative arguments of events from news articles is a challenging problem in information extraction, which requires a global contextual understanding of each document.

Event Argument Extraction Sentence

CONCRETE: Improving Cross-lingual Fact-checking with Cross-lingual Retrieval

1 code implementation COLING 2022 Kung-Hsiang Huang, ChengXiang Zhai, Heng Ji

Given the absence of cross-lingual information retrieval datasets with claim-like queries, we train the retriever with our proposed Cross-lingual Inverse Cloze Task (X-ICT), a self-supervised algorithm that creates training instances by translating the title of a passage.

Cross-lingual Fact-checking Cross-Lingual Information Retrieval +4

Multimedia Generative Script Learning for Task Planning

1 code implementation25 Aug 2022 Qingyun Wang, Manling Li, Hou Pong Chan, Lifu Huang, Julia Hockenmaier, Girish Chowdhary, Heng Ji

Goal-oriented generative script learning aims to generate subsequent steps to reach a particular goal, which is an essential task to assist robots or humans in performing stereotypical activities.

Contrastive Learning Decoder +5

Enhanced Knowledge Selection for Grounded Dialogues via Document Semantic Graphs

no code implementations15 Jun 2022 Sha Li, Mahdi Namazifar, Di Jin, Mohit Bansal, Heng Ji, Yang Liu, Dilek Hakkani-Tur

Providing conversation models with background knowledge has been shown to make open-domain dialogues more informative and engaging.

Multi-Task Learning Response Generation +1

Schema-Guided Event Graph Completion

no code implementations6 Jun 2022 Hongwei Wang, Zixuan Zhang, Sha Li, Jiawei Han, Yizhou Sun, Hanghang Tong, Joseph P. Olive, Heng Ji

Existing link prediction or graph completion methods have difficulty dealing with event graphs because they are usually designed for a single large graph such as a social network or a knowledge graph, rather than multiple small dynamic event graphs.

Link Prediction

Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval

1 code implementation CVPR 2023 Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, Shih-Fu Chang

We surprisingly find that discrete text tokens coupled with a pretrained contrastive text model yields the best performance, which can even outperform state-of-the-art on the iVQA and How2QA datasets without additional training on millions of video-text data.

Retrieval Sentence +2

What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveys

no code implementations23 May 2022 Yubin Ge, Ziang Xiao, Jana Diesner, Heng Ji, Karrie Karahalios, Hari Sundaram

We constructed a new human-annotated dataset of human-written follow-up questions with dialogue history and labeled knowledge in the context of conversational surveys.

Question Generation Question-Generation

Seeded Hierarchical Clustering for Expert-Crafted Taxonomies

no code implementations23 May 2022 Anish Saha, Amith Ananthram, Emily Allaway, Heng Ji, Kathleen McKeown

Practitioners from many disciplines (e. g., political science) use expert-crafted taxonomies to make sense of large, unlabeled corpora.


Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners

1 code implementation22 May 2022 Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, ZiYi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, Heng Ji

The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples, such as domain-specific captioning, question answering, and future event prediction.

Attribute Automatic Speech Recognition +6

Translation between Molecules and Natural Language

1 code implementation25 Apr 2022 Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, Heng Ji

We present $\textbf{MolT5}$ $-$ a self-supervised learning framework for pretraining models on a vast amount of unlabeled natural language text and molecule strings.

Molecule Captioning Self-Supervised Learning +2

Entity-Conditioned Question Generation for Robust Attention Distribution in Neural Information Retrieval

1 code implementation24 Apr 2022 Revanth Gangi Reddy, Md Arafat Sultan, Martin Franz, Avirup Sil, Heng Ji

On two public IR benchmarks, we empirically show that the proposed method helps improve both the model's attention patterns and retrieval performance, including in zero-shot settings.

Information Retrieval Question Generation +3

Faking Fake News for Real Fake News Detection: Propaganda-loaded Training Data Generation

1 code implementation10 Mar 2022 Kung-Hsiang Huang, Kathleen McKeown, Preslav Nakov, Yejin Choi, Heng Ji

Despite recent advances in detecting fake news generated by neural models, their results are not readily applicable to effective detection of human-written disinformation.

Fake News Detection Natural Language Inference +1

Rethinking Task Sampling for Few-shot Vision-Language Transfer Learning

1 code implementation MMMPIE (COLING) 2022 Zhenhailong Wang, Hang Yu, Manling Li, Han Zhao, Heng Ji

While much literature has been devoted to exploring alternative optimization strategies, we identify another essential aspect towards effective few-shot transfer learning, task sampling, which is previously only be viewed as part of data pre-processing in MAML.

Few-Shot Learning Transfer Learning

A Weibo Dataset for the 2022 Russo-Ukrainian Crisis

1 code implementation9 Mar 2022 Yi R. Fung, Heng Ji

Online social networks such as Twitter and Weibo play an important role in how people stay informed and exchange reactions.


Semi-supervised New Event Type Induction and Description via Contrastive Loss-Enforced Batch Attention

no code implementations12 Feb 2022 Carl Edwards, Heng Ji

In contrast, we present a novel approach to semi-supervised new event type induction using a masked contrastive loss, which learns similarities between event mentions by enforcing an attention mechanism over the data minibatch.

Event Extraction

CLIP-Event: Connecting Text and Images with Event Structures

1 code implementation CVPR 2022 Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, Shih-Fu Chang

Vision-language (V+L) pretraining models have achieved great success in supporting multimedia applications by understanding the alignments between images and text.

Contrastive Learning Event Extraction +2

MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding

2 code implementations20 Dec 2021 Revanth Gangi Reddy, Xilin Rui, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avirup Sil, Shih-Fu Chang, Alexander Schwing, Heng Ji

Specifically, the task involves multi-hop questions that require reasoning over image-caption pairs to identify the grounded visual object being referred to and then predicting a span from the news body text to answer the question.

Answer Generation Data Augmentation +2

Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences

1 code implementation NAACL 2022 Yifan Chen, Qi Zeng, Dilek Hakkani-Tur, Di Jin, Heng Ji, Yun Yang

Transformer-based models are not efficient in processing long sequences due to the quadratic space and time complexity of the self-attention modules.

Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification

1 code implementation5 Dec 2021 Zhenhailong Wang, Heng Ji

State-of-the-art brain-to-text systems have achieved great success in decoding language directly from brain signals using neural networks.

EEG Sentence +3

Skyformer: Remodel Self-Attention with Gaussian Kernel and Nyström Method

1 code implementation NeurIPS 2021 Yifan Chen, Qi Zeng, Heng Ji, Yun Yang

Transformers are expensive to train due to the quadratic time and space complexity in the self-attention mechanism.