Search Results for author: Tat-Seng Chua

Found 233 papers, 149 papers with code

Re-examining the Role of Schema Linking in Text-to-SQL

no code implementations EMNLP 2020 Wenqiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, Tat-Seng Chua

By providing a schema linking corpus based on the Spider text-to-SQL dataset, we systematically study the role of schema linking.

Text-To-SQL

A Picture Is Worth a Graph: Blueprint Debate on Graph for Multimodal Reasoning

no code implementations22 Mar 2024 Changmeng Zheng, Dayong Liang, WengYu Zhang, Xiao-Yong Wei, Tat-Seng Chua, Qing Li

The study addresses two key challenges: the trivialization of opinions resulting from excessive summarization and the diversion of focus caused by distractor concepts introduced from images.

Multimodal Reasoning

LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images

1 code implementation18 Mar 2024 Ruyi Xu, Yuan YAO, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, Gao Huang

To address the challenges, we present LLaVA-UHD, a large multimodal model that can efficiently perceive images in any aspect ratio and high resolution.

Think Twice Before Assure: Confidence Estimation for Large Language Models through Reflection on Multiple Answers

no code implementations15 Mar 2024 Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, Tat-Seng Chua

Confidence estimation aiming to evaluate output trustability is crucial for the application of large language models (LLM), especially the black-box ones.

FashionReGen: LLM-Empowered Fashion Report Generation

no code implementations11 Mar 2024 Yujuan Ding, Yunshan Ma, Wenqi Fan, Yige Yao, Tat-Seng Chua, Qing Li

Fashion analysis refers to the process of examining and evaluating trends, styles, and elements within the fashion industry to understand and interpret its current state, generating fashion reports.

Strength Lies in Differences! Towards Effective Non-collaborative Dialogues via Tailored Strategy Planning

no code implementations11 Mar 2024 Tong Zhang, Chen Huang, Yang Deng, Hongru Liang, Jia Liu, Zujie Wen, Wenqiang Lei, Tat-Seng Chua

We investigate non-collaborative dialogue agents that must engage in tailored strategic planning for diverse users to secure a favorable agreement.

Discriminative Probing and Tuning for Text-to-Image Generation

no code implementations7 Mar 2024 Leigang Qu, Wenjie Wang, Yongqi Li, Hanwang Zhang, Liqiang Nie, Tat-Seng Chua

We present a discriminative adapter built on T2I models to probe their discriminative abilities on two representative tasks and leverage discriminative fine-tuning to improve their text-image alignment.

Text-to-Image Generation

Uplift Modeling for Target User Attacks on Recommender Systems

1 code implementation5 Mar 2024 Wenjie Wang, Changsheng Wang, Fuli Feng, Wentao Shi, Daizong Ding, Tat-Seng Chua

UBA estimates the treatment effect on each target user and optimizes the allocation of fake user budgets to maximize the attack performance.

Recommendation Systems

Learning to Ask Critical Questions for Assisting Product Search

no code implementations5 Mar 2024 Zixuan Li, Lizi Liao, Tat-Seng Chua

In this paper, we propose a dual-learning model that hybrids the best from both implicit session feedback and proactively clarifying with users on the most critical questions.

Information Retrieval

Contrastive Pre-training for Deep Session Data Understanding

no code implementations5 Mar 2024 Zixuan Li, Lizi Liao, Yunshan Ma, Tat-Seng Chua

In this work, we delve into deep session data understanding via scrutinizing the various clues inside the rich information in user sessions.

Contrastive Learning

Abductive Ego-View Accident Video Understanding for Safe Driving Perception

no code implementations1 Mar 2024 Jianwu Fang, Lei-Lei Li, Junfei Zhou, Junbin Xiao, Hongkai Yu, Chen Lv, Jianru Xue, Tat-Seng Chua

This model involves a contrastive interaction loss to learn the pair co-occurrence of normal, near-accident, accident frames with the corresponding text descriptions, such as accident reasons, prevention advice, and accident categories.

Object object-detection +3

Prospect Personalized Recommendation on Large Language Model-based Agent Platform

1 code implementation28 Feb 2024 Jizhi Zhang, Keqin Bao, Wenjie Wang, Yang Zhang, Wentao Shi, Wanhong Xu, Fuli Feng, Tat-Seng Chua

Additionally, we prospect the evolution of Rec4Agentverse and conceptualize it into three stages based on the enhancement of the interaction and information exchange among Agent Items, Agent Recommender, and the user.

Language Modelling Large Language Model +1

A Survey on Neural Question Generation: Methods, Applications, and Prospects

no code implementations28 Feb 2024 Shasha Guo, Lizi Liao, Cuiping Li, Tat-Seng Chua

In this survey, we present a detailed examination of the advancements in Neural Question Generation (NQG), a field leveraging neural network techniques to generate relevant questions from diverse inputs like knowledge bases, texts, and images.

Question Generation Question-Generation

Gotcha! Don't trick me with unanswerable questions! Self-aligning Large Language Models for Responding to Unknown Questions

no code implementations23 Feb 2024 Yang Deng, Yong Zhao, Moxin Li, See-Kiong Ng, Tat-Seng Chua

Despite the remarkable abilities of Large Language Models (LLMs) to answer questions, they often display a considerable level of overconfidence even when the question does not have a definitive answer.

On the Multi-turn Instruction Following for Conversational Web Agents

1 code implementation23 Feb 2024 Yang Deng, Xuan Zhang, Wenxuan Zhang, Yifei Yuan, See-Kiong Ng, Tat-Seng Chua

Web agents powered by Large Language Models (LLMs) have demonstrated remarkable abilities in planning and executing multi-step interactions within complex web-based environments, fulfilling a wide range of web navigation tasks.

Conversational Web Navigation Instruction Following

GraphEdit: Large Language Models for Graph Structure Learning

1 code implementation23 Feb 2024 Zirui Guo, Lianghao Xia, Yanhua Yu, Yuling Wang, Zixuan Yang, Wei Wei, Liang Pang, Tat-Seng Chua, Chao Huang

Graph Structure Learning (GSL) focuses on capturing intrinsic dependencies and interactions among nodes in graph-structured data by generating novel graph structures.

Graph structure learning

Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond

no code implementations16 Feb 2024 Yongqi Li, Wenjie Wang, Leigang Qu, Liqiang Nie, Wenjie Li, Tat-Seng Chua

Building upon this capability, we propose to enable multimodal large language models (MLLMs) to memorize and recall images within their parameters.

Cross-Modal Retrieval Retrieval

Distillation Enhanced Generative Retrieval

no code implementations16 Feb 2024 Yongqi Li, Zhen Zhang, Wenjie Wang, Liqiang Nie, Wenjie Li, Tat-Seng Chua

Generative retrieval is a promising new paradigm in text retrieval that generates identifier strings of relevant passages as the retrieval target.

Retrieval Text Retrieval

LLM-based Federated Recommendation

no code implementations15 Feb 2024 Jujia Zhao, Wenjie Wang, Chen Xu, Zhaochun Ren, See-Kiong Ng, Tat-Seng Chua

Nevertheless, applying Fed4Rec to LLM-based recommendation presents two main challenges: first, an increase in the imbalance of performance across clients, affecting the system's efficiency over time, and second, a high demand on clients' computational and storage resources for local training and inference of LLMs.

Federated Learning Language Modelling +2

Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models

1 code implementation6 Feb 2024 Kelvin J. L. Koa, Yunshan Ma, Ritchie Ng, Tat-Seng Chua

The training samples for the PPO trainer are also the responses generated during the reflective process, which eliminates the need for human annotators.

Stock Prediction

Data-efficient Fine-tuning for LLM-based Recommendation

no code implementations30 Jan 2024 Xinyu Lin, Wenjie Wang, Yongqi Li, Shuo Yang, Fuli Feng, Yinwei Wei, Tat-Seng Chua

To pursue the two objectives, we propose a novel data pruning method based on two scores, i. e., influence score and effort score, to efficiently identify the influential samples.

Improving Expressive Power of Spectral Graph Neural Networks with Eigenvalue Correction

no code implementations28 Jan 2024 Kangkang Lu, Yanhua Yu, Hao Fei, Xuan Li, Zixuan Yang, Zirui Guo, Meiyu Liang, Mengran Yin, Tat-Seng Chua

Moreover, we theoretically establish that the number of distinguishable eigenvalues plays a pivotal role in determining the expressive power of spectral graph neural networks.

Node Classification

Towards 3D Molecule-Text Interpretation in Language Models

1 code implementation25 Jan 2024 Sihang Li, Zhiyuan Liu, Yanchen Luo, Xiang Wang, Xiangnan He, Kenji Kawaguchi, Tat-Seng Chua, Qi Tian

Through 3D molecule-text alignment and 3D molecule-centric instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder and LM.

Instruction Following Language Modelling +3

TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data

no code implementations24 Jan 2024 Fengbin Zhu, Ziyang Liu, Fuli Feng, Chao Wang, Moxin Li, Tat-Seng Chua

In this work, we address question answering (QA) over a hybrid of tabular and textual data that are very common content on the Web (e. g. SEC filings), where discrete reasoning capabilities are often required.

Language Modelling Question Answering

Instilling Multi-round Thinking to Text-guided Image Generation

no code implementations16 Jan 2024 Lidong Zeng, Zhedong Zheng, Yinwei Wei, Tat-Seng Chua

This paper delves into the text-guided image editing task, focusing on modifying a reference image according to user-specified textual feedback to embody specific attributes.

Image Generation text-guided-generation +1

GOODAT: Towards Test-time Graph Out-of-Distribution Detection

1 code implementation10 Jan 2024 Luzhi Wang, Dongxiao He, He Zhang, Yixin Liu, Wenjie Wang, Shirui Pan, Di Jin, Tat-Seng Chua

To identify and reject OOD samples with GNNs, recent studies have explored graph OOD detection, often focusing on training a specific model or modifying the data on top of a well-trained GNN.

Out-of-Distribution Detection

Temporally and Distributionally Robust Optimization for Cold-Start Recommendation

1 code implementation15 Dec 2023 Xinyu Lin, Wenjie Wang, Jujia Zhao, Yongqi Li, Fuli Feng, Tat-Seng Chua

They learn a feature extractor on warm-start items to align feature representations with interactions, and then leverage the feature extractor to extract the feature representations of cold-start items for interaction prediction.

Collaborative Filtering

Towards Goal-oriented Intelligent Tutoring Systems in Online Education

no code implementations3 Dec 2023 Yang Deng, Zifeng Ren, An Zhang, Wenqiang Lei, Tat-Seng Chua

In this work, we investigate a new task, named Goal-oriented Intelligent Tutoring Systems (GITS), which aims to enable the student's mastery of a designated concept by strategically planning a customized sequence of exercises and assessment.

cognitive diagnosis Representation Learning

Structured, Complex and Time-complete Temporal Event Forecasting

1 code implementation2 Dec 2023 Yunshan Ma, Chenchen Ye, Zijian Wu, Xiang Wang, Yixin Cao, Liang Pang, Tat-Seng Chua

To address these limitations, we introduce a novel formulation for Structured, Complex, and Time-complete Temporal Event (SCTc-TE).

RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback

2 code implementations1 Dec 2023 Tianyu Yu, Yuan YAO, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, Tat-Seng Chua

Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in multimodal understanding, reasoning, and interaction.

Hallucination

MultiCBR: Multi-view Contrastive Learning for Bundle Recommendation

1 code implementation28 Nov 2023 Yunshan Ma, Yingzhi He, Xiang Wang, Yinwei Wei, Xiaoyu Du, Yuyangzi Fu, Tat-Seng Chua

It does, however, have two limitations: 1) the two-view formulation does not fully exploit all the heterogeneous relations among users, bundles and items; and 2) the "early contrast and late fusion" framework is less effective in capturing user preference and difficult to generalize to multiple views.

Contrastive Learning Representation Learning

Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching

no code implementations21 Nov 2023 Meng Chu, Zhedong Zheng, Wei Ji, Tingyu Wang, Tat-Seng Chua

Navigating drones through natural language commands remains challenging due to the dearth of accessible multi-modal datasets and the stringent precision requirements for aligning visual and textual data.

Drone navigation Language Modelling +2

Do LLMs Implicitly Exhibit User Discrimination in Recommendation? An Empirical Study

no code implementations13 Nov 2023 Chen Xu, Wenjie Wang, Yuxin Li, Liang Pang, Jun Xu, Tat-Seng Chua

Recently, Large Language Models (LLMs) have enhanced user interaction, enabling seamless information retrieval and recommendations.

Information Retrieval Recommendation Systems +1

NExT-Chat: An LMM for Chat, Detection and Segmentation

1 code implementation8 Nov 2023 Ao Zhang, Yuan YAO, Wei Ji, Zhiyuan Liu, Tat-Seng Chua

The development of large language models (LLMs) has greatly advanced the field of multimodal understanding, leading to the emergence of large multimodal models (LMMs).

Referring Expression Referring Expression Segmentation +1

Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents

1 code implementation1 Nov 2023 Yang Deng, Wenxuan Zhang, Wai Lam, See-Kiong Ng, Tat-Seng Chua

Proactive dialogues serve as a practical yet challenging dialogue problem in the era of large language models (LLMs), where the dialogue policy planning is the key to improving the proactivity of LLMs.

Language Modelling Large Language Model

Empowering Collaborative Filtering with Principled Adversarial Contrastive Loss

1 code implementation NeurIPS 2023 An Zhang, Leheng Sheng, Zhibo Cai, Xiang Wang, Tat-Seng Chua

To bridge the gap, we delve into the reasons underpinning the success of contrastive loss in CF, and propose a principled Adversarial InfoNCE loss (AdvInfoNCE), which is a variant of InfoNCE, specially tailored for CF methods.

Collaborative Filtering Contrastive Learning +3

Leveraging Multimodal Features and Item-level User Feedback for Bundle Construction

1 code implementation28 Oct 2023 Yunshan Ma, Xiaohao Liu, Yinwei Wei, Zhulin Tao, Xiang Wang, Tat-Seng Chua

Specifically, we use self-attention modules to combine the multimodal and multi-item features, and then leverage both item- and bundle-level contrastive learning to enhance the representation learning, thus to counter the modality missing, noise, and sparsity problems.

Contrastive Learning Representation Learning

Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules

1 code implementation NeurIPS 2023 Zhiyuan Liu, Yaorui Shi, An Zhang, Enzhi Zhang, Kenji Kawaguchi, Xiang Wang, Tat-Seng Chua

Our results show that a subgraph-level tokenizer and a sufficiently expressive decoder with remask decoding have a large impact on the encoder's representation learning.

Representation Learning Self-Supervised Learning

A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction

1 code implementation18 Oct 2023 Ruihao Shui, Yixin Cao, Xiang Wang, Tat-Seng Chua

Large language models (LLMs) have demonstrated great potential for domain-specific applications, such as the law domain.

Information Retrieval Legal Reasoning +1

Robust Collaborative Filtering to Popularity Distribution Shift

1 code implementation16 Oct 2023 An Zhang, Wenchang Ma, Jingnan Zheng, Xiang Wang, Tat-Seng Chua

The popularity shortcut tricks are good for in-distribution (ID) performance but poorly generalized to out-of-distribution (OOD) data, i. e., when popularity distribution of test data shifts w. r. t.

Collaborative Filtering

On Generative Agents in Recommendation

1 code implementation16 Oct 2023 An Zhang, Leheng Sheng, Yuxin Chen, Hao Li, Yang Deng, Xiang Wang, Tat-Seng Chua

Recommender systems are the cornerstone of today's information dissemination, yet a disconnect between offline metrics and online performance greatly hinders their development.

Collaborative Filtering Movie Recommendation +1

Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators

1 code implementation11 Oct 2023 Liang Chen, Yang Deng, Yatao Bian, Zeyu Qin, Bingzhe Wu, Tat-Seng Chua, Kam-Fai Wong

Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks when being prompted to generate world knowledge.

Information Retrieval Informativeness +4

A Multi-facet Paradigm to Bridge Large Language Model and Recommendation

no code implementations10 Oct 2023 Xinyu Lin, Wenjie Wang, Yongqi Li, Fuli Feng, See-Kiong Ng, Tat-Seng Chua

To combat these issues, we propose a novel multi-facet paradigm, namely TransRec, to bridge the LLMs to recommendation.

Attribute Language Modelling +2

Progressive Text-to-3D Generation for Automatic 3D Prototyping

1 code implementation26 Sep 2023 Han Yi, Zhedong Zheng, Xiangyu Xu, Tat-Seng Chua

We aspire for our work to pave the way for automatic 3D prototyping via natural language descriptions.

Text to 3D

NExT-GPT: Any-to-Any Multimodal LLM

1 code implementation11 Sep 2023 Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, Tat-Seng Chua

While recently Multimodal Large Language Models (MM-LLMs) have made exciting strides, they mostly fall prey to the limitation of only input-side multimodal understanding, without the ability to produce content in multiple modalities.

Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs

no code implementations26 Aug 2023 Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Tat-Seng Chua

In this work, we investigate strengthening the awareness of video dynamics for DMs, for high-quality T2V generation.

In-Context Learning Video Generation

Diffusion Variational Autoencoder for Tackling Stochasticity in Multi-Step Regression Stock Price Prediction

1 code implementation18 Aug 2023 Kelvin J. L. Koa, Yunshan Ma, Ritchie Ng, Tat-Seng Chua

The hierarchical VAE allows us to learn the complex and low-level latent variables for stock prediction, while the diffusion probabilistic model trains the predictor to handle stock price stochasticity by progressively adding random noise to the stock data.

regression Stock Prediction +1

Context-aware Event Forecasting via Graph Disentanglement

1 code implementation12 Aug 2023 Yunshan Ma, Chenchen Ye, Zijian Wu, Xiang Wang, Yixin Cao, Tat-Seng Chua

The task of event forecasting aims to model the relational and temporal patterns based on historical events and makes forecasting to what will happen in the future.

Disentanglement Link Prediction

Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labeling

no code implementations9 Aug 2023 Yu Zhao, Hao Fei, Yixin Cao, Bobo Li, Meishan Zhang, Jianguo Wei, Min Zhang, Tat-Seng Chua

A scene-event mapping mechanism is first designed to bridge the gap between the underlying scene structure and the high-level event semantic structure, resulting in an overall hierarchical scene-event (termed ICE) graph structure.

Semantic Role Labeling

LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation

no code implementations9 Aug 2023 Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nie, Tat-Seng Chua

Afterward, we propose a fine-grained object-interaction diffusion method to synthesize high-faithfulness images conditioned on the prompt and the automatically generated layout.

In-Context Learning Text-to-Image Generation

Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognition

no code implementations8 Aug 2023 Bobo Li, Hao Fei, Lizi Liao, Yu Zhao, Chong Teng, Tat-Seng Chua, Donghong Ji, Fei Li

On the other hand, during the feature fusion stage, we propose a Contribution-aware Fusion Mechanism (CFM) and a Context Refusion Mechanism (CRM) for multimodal and context integration, respectively.

Contrastive Learning Disentanglement +1

Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions

1 code implementation8 Aug 2023 Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Yueting Zhuang

This shortcoming results in MLLMs' underperformance in comprehending demonstrative instructions consisting of multiple, interleaved, and multimodal instructions that demonstrate the required context to complete a task.

Image Captioning Instruction Following

SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning

2 code implementations3 Aug 2023 Keyu Duan, Qian Liu, Tat-Seng Chua, Shuicheng Yan, Wei Tsang Ooi, Qizhe Xie, Junxian He

More recently, with the rapid development of language models (LMs), researchers have focused on leveraging LMs to facilitate the learning of TGs, either by jointly training them in a computationally intensive framework (merging the two stages), or designing complex self-supervised training tasks for feature extraction (enhancing the first stage).

Feature Engineering Graph Learning +3

XNLP: An Interactive Demonstration System for Universal Structured NLP

no code implementations3 Aug 2023 Hao Fei, Meishan Zhang, Min Zhang, Tat-Seng Chua

Structured Natural Language Processing (XNLP) is an important subset of NLP that entails understanding the underlying semantic or syntactic structure of texts, which serves as a foundational component for many downstream applications.

Revisiting Conversation Discourse for Dialogue Disentanglement

no code implementations6 Jun 2023 Bobo Li, Hao Fei, Fei Li, Shengqiong Wu, Lizi Liao, Yinwei Wei, Tat-Seng Chua, Donghong Ji

Conversation utterances are essentially organized and described by the underlying discourse, and thus dialogue disentanglement requires the full understanding and harnessing of the intrinsic discourse attribute.

Attribute Disentanglement

Computational Technologies for Fashion Recommendation: A Survey

no code implementations6 Jun 2023 Yujuan Ding, Zhihui Lai, P. Y. Mok, Tat-Seng Chua

Fashion recommendation is a key research field in computational fashion research and has attracted considerable interest in the computer vision, multimedia, and information retrieval communities in recent years.

Information Retrieval Product Recommendation

Discovering Dynamic Causal Space for DAG Structure Learning

1 code implementation5 Jun 2023 Fangfu Liu, Wenchang Ma, An Zhang, Xiang Wang, Yueqi Duan, Tat-Seng Chua

Discovering causal structure from purely observational data (i. e., causal discovery), aiming to identify causal relationships among variables, is a fundamental task in machine learning.

Causal Discovery Combinatorial Optimization

Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration

1 code implementation23 May 2023 Yang Deng, Lizi Liao, Liang Chen, Hongru Wang, Wenqiang Lei, Tat-Seng Chua

Conversational systems based on Large Language Models (LLMs), such as ChatGPT, show exceptional proficiency in context understanding and response generation.

Descriptive Response Generation

Robust Prompt Optimization for Large Language Models Against Distribution Shifts

no code implementations23 May 2023 Moxin Li, Wenjie Wang, Fuli Feng, Yixin Cao, Jizhi Zhang, Tat-Seng Chua

In this light, we propose a new problem of robust prompt optimization for LLMs against distribution shifts, which requires the prompt optimized over the labeled source group can simultaneously generalize to an unlabeled target group.

Language Modelling Large Language Model

MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Space

1 code implementation22 May 2023 Hanxing Ding, Liang Pang, Zihao Wei, HuaWei Shen, Xueqi Cheng, Tat-Seng Chua

Multi-aspect controllable text generation aims to generate fluent sentences that possess multiple desired attributes simultaneously.

Attribute Text Generation

InstructVid2Vid: Controllable Video Editing with Natural Language Instructions

no code implementations21 May 2023 Bosheng Qin, Juncheng Li, Siliang Tang, Tat-Seng Chua, Yueting Zhuang

To improve the consistency between adjacent frames of generated videos, we propose the Frame Difference Loss, which is incorporated during the training process.

Attribute Image Generation +2

Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination

1 code implementation20 May 2023 Hao Fei, Qian Liu, Meishan Zhang, Min Zhang, Tat-Seng Chua

In this work, we investigate a more realistic unsupervised multimodal machine translation (UMMT) setup, inference-time image-free UMMT, where the model is trained with source-text image pairs, and tested with only source-text inputs.

Hallucination Multimodal Machine Translation +1

Constructing Code-mixed Universal Dependency Forest for Unbiased Cross-lingual Relation Extraction

no code implementations20 May 2023 Hao Fei, Meishan Zhang, Min Zhang, Tat-Seng Chua

Latest efforts on cross-lingual relation extraction (XRE) aggressively leverage the language-consistent structural features from the universal dependency (UD) resource, while they may largely suffer from biased transfer (e. g., either target-biased or source-biased) due to the inevitable linguistic disparity between languages.

Relation Relation Extraction +1

Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment

no code implementations20 May 2023 Shengqiong Wu, Hao Fei, Wei Ji, Tat-Seng Chua

Unpaired cross-lingual image captioning has long suffered from irrelevancy and disfluency issues, due to the inconsistencies of the semantic scene and syntax attributes during transfer.

Image Captioning Translation

Generating Visual Spatial Description via Holistic 3D Scene Understanding

1 code implementation19 May 2023 Yu Zhao, Hao Fei, Wei Ji, Jianguo Wei, Meishan Zhang, Min Zhang, Tat-Seng Chua

With an external 3D scene extractor, we obtain the 3D objects and scene features for input images, based on which we construct a target object-centered 3D spatial scene graph (Go3D-S2G), such that we model the spatial semantics of target objects within the holistic 3D scenes.

Scene Understanding Text Generation

Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling

1 code implementation19 May 2023 Shengqiong Wu, Hao Fei, Yixin Cao, Lidong Bing, Tat-Seng Chua

First, we represent the fine-grained semantic structures of the input image and text with the visual and textual scene graphs, which are further fused into a unified cross-modal graph (CMG).

Denoising Relation +1

Reasoning Implicit Sentiment with Chain-of-Thought Prompting

1 code implementation18 May 2023 Hao Fei, Bobo Li, Qian Liu, Lidong Bing, Fei Li, Tat-Seng Chua

While sentiment analysis systems try to determine the sentiment polarities of given targets based on the key opinion expressions in input texts, in implicit sentiment analysis (ISA) the opinion cues come in an implicit and obscure manner.

Common Sense Reasoning Sentiment Analysis

Dual Semantic Knowledge Composed Multimodal Dialog Systems

no code implementations17 May 2023 Xiaolin Chen, Xuemeng Song, Yinwei Wei, Liqiang Nie, Tat-Seng Chua

Thereafter, considering that the attribute knowledge and relation knowledge can benefit the responding to different levels of questions, we design a multi-level knowledge composition module in MDS-S2 to obtain the latent composed response representation.

Attribute Relation +1

A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospects

no code implementations4 May 2023 Yang Deng, Wenqiang Lei, Wai Lam, Tat-Seng Chua

Proactive dialogue systems, related to a wide range of real-world conversational applications, equip the conversational agent with the capability of leading the conversation direction towards achieving pre-defined targets or fulfilling certain goals from the system side.

Doc2SoarGraph: Discrete Reasoning over Visually-Rich Table-Text Documents via Semantic-Oriented Hierarchical Graphs

1 code implementation3 May 2023 Fengbin Zhu, Chao Wang, Fuli Feng, Zifeng Ren, Moxin Li, Tat-Seng Chua

Discrete reasoning over table-text documents (e. g., financial reports) gains increasing attention in recent two years.

VPGTrans: Transfer Visual Prompt Generator across LLMs

1 code implementation NeurIPS 2023 Ao Zhang, Hao Fei, Yuan YAO, Wei Ji, Li Li, Zhiyuan Liu, Tat-Seng Chua

While developing a new multimodal LLM (MLLM) by pre-training on tremendous image-text pairs from scratch can be exceedingly resource-consuming, connecting an existing LLM with a comparatively lightweight visual prompt generator (VPG) becomes a feasible paradigm.

Transfer Learning

Search-in-the-Chain: Interactively Enhancing Large Language Models with Search for Knowledge-intensive Tasks

1 code implementation28 Apr 2023 Shicheng Xu, Liang Pang, HuaWei Shen, Xueqi Cheng, Tat-Seng Chua

This paper proposes a novel framework named \textbf{Search-in-the-Chain} (SearChain) for the interaction between LLM and IR to solve the challenges.

Fact Checking Information Retrieval +7

Learnable Pillar-based Re-ranking for Image-Text Retrieval

1 code implementation25 Apr 2023 Leigang Qu, Meng Liu, Wenjie Wang, Zhedong Zheng, Liqiang Nie, Tat-Seng Chua

Image-text retrieval aims to bridge the modality gap and retrieve cross-modal content based on semantic similarities.

Re-Ranking Retrieval +1

On the Robustness of Aspect-based Sentiment Analysis: Rethinking Model, Data, and Training

no code implementations19 Apr 2023 Hao Fei, Tat-Seng Chua, Chenliang Li, Donghong Ji, Meishan Zhang, Yafeng Ren

In this study, we propose to enhance the ABSA robustness by systematically rethinking the bottlenecks from all possible angles, including model, data, and training.

Aspect-Based Sentiment Analysis Aspect-Based Sentiment Analysis (ABSA) +2

LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Model

1 code implementation13 Apr 2023 Hao Fei, Shengqiong Wu, Jingye Li, Bobo Li, Fei Li, Libo Qin, Meishan Zhang, Min Zhang, Tat-Seng Chua

Universally modeling all typical information extraction tasks (UIE) with one generative language model (GLM) has revealed great potential by the latest study, where various IE predictions are unified into a linearized hierarchical expression under a GLM.

Language Modelling UIE

Diffusion Recommender Model

1 code implementation11 Apr 2023 Wenjie Wang, Yiyan Xu, Fuli Feng, Xinyu Lin, Xiangnan He, Tat-Seng Chua

In light of the impressive advantages of Diffusion Models (DMs) over traditional generative models in image synthesis, we propose a novel Diffusion Recommender Model (named DiffRec) to learn the generative process in a denoising manner.

Denoising Image Generation +1

Generative Recommendation: Towards Next-generation Recommender Paradigm

1 code implementation7 Apr 2023 Wenjie Wang, Xinyu Lin, Fuli Feng, Xiangnan He, Tat-Seng Chua

However, such a retrieval-based recommender paradigm faces two limitations: 1) the human-generated items in the corpus might fail to satisfy the users' diverse information needs, and 2) users usually adjust the recommendations via inefficient passive feedback, e. g., clicks.

Recommendation Systems Retrieval +1

Causal Disentangled Recommendation Against User Preference Shifts

1 code implementation28 Mar 2023 Wenjie Wang, Xinyu Lin, Liuhui Wang, Fuli Feng, Yunshan Ma, Tat-Seng Chua

Inspired by the causal graph, our key considerations to handle preference shifts lie in modeling the interaction generation procedure by: 1) capturing the preference shifts across environments for accurate preference prediction, and 2) disentangling the sparse influence from user preference to interactions for accurate effect estimation of preference.

Recommendation Systems

Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models

no code implementations ICCV 2023 Juncheng Li, Minghe Gao, Longhui Wei, Siliang Tang, Wenqiao Zhang, Mengze Li, Wei Ji, Qi Tian, Tat-Seng Chua, Yueting Zhuang

Prompt tuning, a recently emerging paradigm, enables the powerful vision-language pre-training models to adapt to downstream tasks in a parameter -- and data -- efficient way, by learning the ``soft prompts'' to condition frozen pre-training models.

Domain Generalization Few-Shot Learning +1

Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting

1 code implementation6 Mar 2023 An Zhang, Fangfu Liu, Wenchang Ma, Zhibo Cai, Xiang Wang, Tat-Seng Chua

Despite great success in low-dimensional linear systems, it has been observed that these approaches overly exploit easier-to-fit samples, thus inevitably learning spurious edges.

Bilevel Optimization Causal Discovery

Contrastive Video Question Answering via Video Graph Transformer

1 code implementation27 Feb 2023 Junbin Xiao, Pan Zhou, Angela Yao, Yicong Li, Richang Hong, Shuicheng Yan, Tat-Seng Chua

CoVGT's uniqueness and superiority are three-fold: 1) It proposes a dynamic graph transformer module which encodes video by explicitly capturing the visual objects, their relations and dynamics, for complex spatio-temporal reasoning.

Ranked #11 on Video Question Answering on NExT-QA (using extra training data)

Contrastive Learning Question Answering +1

Invariant Collaborative Filtering to Popularity Distribution Shift

1 code implementation10 Feb 2023 An Zhang, Jingnan Zheng, Xiang Wang, Yancheng Yuan, Tat-Seng Chua

Collaborative Filtering (CF) models, despite their great success, suffer from severe performance drops due to popularity distribution shifts, where these changes are ubiquitous and inevitable in real-world scenarios.

Collaborative Filtering Representation Learning

Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding

no code implementations22 Jan 2023 Juncheng Li, Siliang Tang, Linchao Zhu, Wenqiao Zhang, Yi Yang, Tat-Seng Chua, Fei Wu, Yueting Zhuang

To systematically benchmark the compositional generalizability of temporal grounding models, we introduce a new Compositional Temporal Grounding task and construct two new dataset splits, i. e., Charades-CG and ActivityNet-CG.

Semantic correspondence Sentence

Are Binary Annotations Sufficient? Video Moment Retrieval via Hierarchical Uncertainty-Based Active Learning

1 code implementation CVPR 2023 Wei Ji, Renjie Liang, Zhedong Zheng, Wenqiao Zhang, Shengyu Zhang, Juncheng Li, Mengze Li, Tat-Seng Chua

Moreover, we treat the uncertainty score of frames in a video as a whole, and estimate the difficulty of each video, which can further relieve the burden of video selection.

Active Learning Moment Retrieval +1

MRTNet: Multi-Resolution Temporal Network for Video Sentence Grounding

no code implementations26 Dec 2022 Wei Ji, Long Chen, Yinwei Wei, Yiming Wu, Tat-Seng Chua

In this work, we propose a novel multi-resolution temporal video sentence grounding network: MRTNet, which consists of a multi-modal feature encoder, a Multi-Resolution Temporal (MRT) module, and a predictor module.

Descriptive Sentence

Causal Inference for Knowledge Graph based Recommendation

1 code implementation20 Dec 2022 Yinwei Wei, Xiang Wang, Liqiang Nie, Shaoyu Li, Dingxian Wang, Tat-Seng Chua

Knowledge Graph (KG), as a side-information, tends to be utilized to supplement the collaborative filtering (CF) based recommendation model.

Collaborative Filtering counterfactual +1

Cognitive Accident Prediction in Driving Scenes: A Multimodality Benchmark

1 code implementation19 Dec 2022 Jianwu Fang, Lei-Lei Li, Kuan Yang, Zhedong Zheng, Jianru Xue, Tat-Seng Chua

In particular, the text description provides a dense semantic description guidance for the primary context of the traffic scene, while the driver attention provides a traction to focus on the critical region closely correlating with safe driving.

Decision Making

Visually Grounded Commonsense Knowledge Acquisition

1 code implementation22 Nov 2022 Yuan YAO, Tianyu Yu, Ao Zhang, Mengdi Li, Ruobing Xie, Cornelius Weber, Zhiyuan Liu, Hai-Tao Zheng, Stefan Wermter, Tat-Seng Chua, Maosong Sun

In this work, we present CLEVER, which formulates CKE as a distantly supervised multi-instance learning problem, where models learn to summarize commonsense relations from a bag of images about an entity pair without any human annotation on image instances.

Language Modelling

Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization

1 code implementation14 Nov 2022 Yiyang Chen, Zhedong Zheng, Wei Ji, Leigang Qu, Tat-Seng Chua

The key idea underpinning the proposed method is to integrate fine- and coarse-grained retrieval as matching data points with small and large fluctuations, respectively.

Composed Image Retrieval (CoIR) Image Retrieval with Multi-Modal Query +1

PiPa: Pixel- and Patch-wise Self-supervised Learning for Domain Adaptative Semantic Segmentation

1 code implementation14 Nov 2022 Mu Chen, Zhedong Zheng, Yi Yang, Tat-Seng Chua

In an attempt to fill this gap, we propose a unified pixel- and patch-wise self-supervised learning framework, called PiPa, for domain adaptive semantic segmentation that facilitates intra-image pixel-wise correlations and patch-wise semantic consistency against different contexts.

Self-Supervised Learning Semantic Segmentation +2

Behavioral Intention Prediction in Driving Scenes: A Survey

no code implementations1 Nov 2022 Jianwu Fang, Fan Wang, Jianru Xue, Tat-Seng Chua

Behavioral Intention Prediction (BIP) simulates such a human consideration process and fulfills the early prediction of specific behaviors.

Trajectory Prediction

Incorporating Bias-aware Margins into Contrastive Loss for Collaborative Filtering

1 code implementation20 Oct 2022 An Zhang, Wenchang Ma, Xiang Wang, Tat-Seng Chua

Collaborative filtering (CF) models easily suffer from popularity bias, which makes recommendation deviate from users' actual preferences.

Collaborative Filtering

Re4: Learning to Re-contrast, Re-attend, Re-construct for Multi-interest Recommendation

1 code implementation17 Aug 2022 Shengyu Zhang, Lingxiao Yang, Dong Yao, Yujie Lu, Fuli Feng, Zhou Zhao, Tat-Seng Chua, Fei Wu

Specifically, Re4 encapsulates three backward flows, i. e., 1) Re-contrast, which drives each interest embedding to be distinct from other interests using contrastive learning; 2) Re-attend, which ensures the interest-item correlation estimation in the forward flow to be consistent with the criterion used in final recommendation; and 3) Re-construct, which ensures that each interest embedding can semantically reflect the information of representative items that relate to the corresponding interest.

Contrastive Learning Recommendation Systems

CCL4Rec: Contrast over Contrastive Learning for Micro-video Recommendation

no code implementations17 Aug 2022 Shengyu Zhang, Bofang Li, Dong Yao, Fuli Feng, Jieming Zhu, Wenyan Fan, Zhou Zhao, Xiaofei He, Tat-Seng Chua, Fei Wu

Micro-video recommender systems suffer from the ubiquitous noises in users' behaviors, which might render the learned user representation indiscriminating, and lead to trivial recommendations (e. g., popular items) or even weird ones that are far beyond users' interests.

Contrastive Learning Recommendation Systems

Equivariant and Invariant Grounding for Video Question Answering

1 code implementation26 Jul 2022 Yicong Li, Xiang Wang, Junbin Xiao, Tat-Seng Chua

Specifically, the equivariant grounding encourages the answering to be sensitive to the semantic changes in the causal scene and question; in contrast, the invariant grounding enforces the answering to be insensitive to the changes in the environment scene.

Question Answering Video Question Answering

Towards Complex Document Understanding By Discrete Reasoning

no code implementations25 Jul 2022 Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, Tat-Seng Chua

Document Visual Question Answering (VQA) aims to understand visually-rich documents to answer questions in natural language, which is an emerging research topic for both Natural Language Processing and Computer Vision.

document understanding Question Answering +1

Video Graph Transformer for Video Question Answering

1 code implementation12 Jul 2022 Junbin Xiao, Pan Zhou, Tat-Seng Chua, Shuicheng Yan

VGT's uniqueness are two-fold: 1) it designs a dynamic graph transformer module which encodes video by explicitly capturing the visual objects, their relations, and dynamics for complex spatio-temporal reasoning; and 2) it exploits disentangled video and text Transformers for relevance comparison between the video and text to perform QA, instead of entangled cross-modal Transformer for answer classification.

Ranked #18 on Video Question Answering on NExT-QA (using extra training data)

Question Answering Relation +2

Structured and Natural Responses Co-generation for Conversational Search

1 code implementation ACM SIGIR Conference on Research and Development in Information Retrieval 2022 Chenchen Ye, Lizi Liao, Fuli Feng, Wei Ji, Tat-Seng Chua

Existing approaches either 1) predict structured dialog acts first and then generate natural response; or 2) map conversation context to natural responses directly in an end-to-end manner.

Conversational Search

Intelligent Request Strategy Design in Recommender System

no code implementations23 Jun 2022 Xufeng Qian, Yue Xu, Fuyu Lv, Shengyu Zhang, Ziwen Jiang, Qingwen Liu, Xiaoyi Zeng, Tat-Seng Chua, Fei Wu

RSs typically put a large number of items into one page to reduce excessive resource consumption from numerous paging requests, which, however, would diminish the RSs' ability to timely renew the recommendations according to users' real-time interest and lead to a poor user experience.

Causal Inference counterfactual +1

Let Invariant Rationale Discovery Inspire Graph Contrastive Learning

1 code implementation16 Jun 2022 Sihang Li, Xiang Wang, An Zhang, Yingxin Wu, Xiangnan He, Tat-Seng Chua

Specifically, without supervision signals, RGCL uses a rationale generator to reveal salient features about graph instance-discrimination as the rationale, and then creates rationale-aware views for contrastive learning.

Contrastive Learning

Invariant Grounding for Video Question Answering

1 code implementation CVPR 2022 Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, Tat-Seng Chua

At its core is understanding the alignments between visual scenes in video and linguistic semantics in question to yield the answer.

Question Answering Video Question Answering

CrossCBR: Cross-view Contrastive Learning for Bundle Recommendation

1 code implementation1 Jun 2022 Yunshan Ma, Yingzhi He, An Zhang, Xiang Wang, Tat-Seng Chua

Recent methods usually take advantage of both user-bundle and user-item interactions information to obtain informative representations for users and bundles, corresponding to bundle view and item view, respectively.

Contrastive Learning Graph Learning

Differentiable Invariant Causal Discovery

no code implementations31 May 2022 Yu Wang, An Zhang, Xiang Wang, Yancheng Yuan, Xiangnan He, Tat-Seng Chua

This paper proposes Differentiable Invariant Causal Discovery (DICD), utilizing the multi-environment information based on a differentiable framework to avoid learning spurious edges and wrong causal directions.

Causal Discovery

PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models

1 code implementation23 May 2022 Yuan YAO, Qianyu Chen, Ao Zhang, Wei Ji, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun

We show that PEVL enables state-of-the-art performance of detector-free VLP models on position-sensitive tasks such as referring expression comprehension and phrase grounding, and also improves the performance on position-insensitive tasks with grounded inputs.

Language Modelling Object +7

3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective

1 code implementation27 Apr 2022 Zhedong Zheng, Jiayin Zhu, Wei Ji, Yi Yang, Tat-Seng Chua

This research aims to study a self-supervised 3D clothing reconstruction method, which recovers the geometry shape and texture of human clothing from a single image.

3D Reconstruction Person Re-Identification +2

Reinforced Causal Explainer for Graph Neural Networks

1 code implementation23 Apr 2022 Xiang Wang, Yingxin Wu, An Zhang, Fuli Feng, Xiangnan He, Tat-Seng Chua

Such reward accounts for the dependency of the newly-added edge and the previously-added edges, thus reflecting whether they collaborate together and form a coalition to pursue better explanations.

Graph Classification

Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue Policy

no code implementations7 Apr 2022 Wenqiang Lei, Yao Zhang, Feifan Song, Hongru Liang, Jiaxin Mao, Jiancheng Lv, Zhenglu Yang, Tat-Seng Chua

To this end, we contribute to advance the study of the proactive dialogue policy to a more natural and challenging setting, i. e., interacting dynamically with users.

Video Question Answering: Datasets, Algorithms and Challenges

1 code implementation2 Mar 2022 Yaoyao Zhong, Junbin Xiao, Wei Ji, Yicong Li, Weihong Deng, Tat-Seng Chua

Video Question Answering (VideoQA) aims to answer natural language questions according to the given videos.

Question Answering Video Question Answering

KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems

3 code implementations22 Feb 2022 Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, Tat-Seng Chua

The progress of recommender systems is hampered mainly by evaluation as it requires real-time interactions between humans and systems, which is too laborious and expensive.

Recommendation Systems User Simulation

Discovering Invariant Rationales for Graph Neural Networks

1 code implementation ICLR 2022 Ying-Xin Wu, Xiang Wang, An Zhang, Xiangnan He, Tat-Seng Chua

Intrinsic interpretability of graph neural networks (GNNs) is to find a small subset of the input graph's features -- rationale -- which guides the model prediction.

Graph Classification

Deconfounding to Explanation Evaluation in Graph Neural Networks

no code implementations21 Jan 2022 Ying-Xin Wu, Xiang Wang, An Zhang, Xia Hu, Fuli Feng, Xiangnan He, Tat-Seng Chua

In this work, we propose Deconfounded Subgraph Evaluation (DSE) which assesses the causal effect of an explanatory subgraph on the model prediction.

A Dual Prompt Learning Framework for Few-Shot Dialogue State Tracking

no code implementations15 Jan 2022 Yuting Yang, Wenqiang Lei, Pei Huang, Juan Cao, Jintao Li, Tat-Seng Chua

In this paper, we focus on how to utilize the language understanding and generation ability of pre-trained language models for DST.

Dialogue State Tracking Language Modelling

Training Free Graph Neural Networks for Graph Matching

1 code implementation14 Jan 2022 Zhiyuan Liu, Yixin Cao, Fuli Feng, Xiang Wang, Jie Tang, Kenji Kawaguchi, Tat-Seng Chua

We present a framework of Training Free Graph Matching (TFGM) to boost the performance of Graph Neural Networks (GNNs) based graph matching, providing a fast promising solution without training (training-free).

Entity Alignment Graph Matching +1

Causal Attention for Interpretable and Generalizable Graph Classification

1 code implementation30 Dec 2021 Yongduo Sui, Xiang Wang, Jiancan Wu, Min Lin, Xiangnan He, Tat-Seng Chua

To endow the classifier with better interpretation and generalization, we propose the Causal Attention Learning (CAL) strategy, which discovers the causal patterns and mitigates the confounding effect of shortcuts.

Graph Attention Graph Classification

Video as Conditional Graph Hierarchy for Multi-Granular Question Answering

1 code implementation12 Dec 2021 Junbin Xiao, Angela Yao, Zhiyuan Liu, Yicong Li, Wei Ji, Tat-Seng Chua

To align with the multi-granular essence of linguistic concepts in language queries, we propose to model video as a conditional graph hierarchy which weaves together visual facts of different granularity in a level-wise manner, with the guidance of corresponding textual cues.

Question Answering Video Question Answering +1

Rethinking the Two-Stage Framework for Grounded Situation Recognition

1 code implementation10 Dec 2021 Meng Wei, Long Chen, Wei Ji, Xiaoyu Yue, Tat-Seng Chua

Since each verb is associated with a specific set of semantic roles, all existing GSR methods resort to a two-stage framework: predicting the verb in the first stage and detecting the semantic roles in the second stage.

Grounded Situation Recognition Object Recognition +1

Learning Robust Recommender from Noisy Implicit Feedback

1 code implementation2 Dec 2021 Wenjie Wang, Fuli Feng, Xiangnan He, Liqiang Nie, Tat-Seng Chua

Inspired by this observation, we propose a new training strategy named Adaptive Denoising Training (ADT), which adaptively prunes the noisy interactions by two paradigms (i. e., Truncated Loss and Reweighted Loss).

Denoising Recommendation Systems

Towards Multi-Grained Explainability for Graph Neural Networks

1 code implementation NeurIPS 2021 Xiang Wang, Yingxin Wu, An Zhang, Xiangnan He, Tat-Seng Chua

A performant paradigm towards multi-grained explainability is until-now lacking and thus a focus of our work.

Decoupling Strategy and Surface Realization for Task-oriented Dialogues

no code implementations29 Sep 2021 Chenchen Ye, Lizi Liao, Fuli Feng, Wei Ji, Tat-Seng Chua

The core is to construct a latent content space for strategy optimization and disentangle the surface style from it.

Reinforcement Learning (RL) Style Transfer +1

Inductive Lottery Ticket Learning for Graph Neural Networks

no code implementations29 Sep 2021 Yongduo Sui, Xiang Wang, Tianlong Chen, Xiangnan He, Tat-Seng Chua

In this work, we propose a simple and effective learning paradigm, Inductive Co-Pruning of GNNs (ICPG), to endow graph lottery tickets with inductive pruning capacity.

Graph Classification Node Classification +1

Why Do We Click: Visual Impression-aware News Recommendation

1 code implementation26 Sep 2021 Jiahao Xun, Shengyu Zhang, Zhou Zhao, Jieming Zhu, Qi Zhang, Jingjie Li, Xiuqiang He, Xiaofei He, Tat-Seng Chua, Fei Wu

In this work, inspired by the fact that users make their click decisions mostly based on the visual impression they perceive when browsing news, we propose to capture such visual impression information with visual-semantic modeling for news recommendation.

Decision Making News Recommendation

CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models

1 code implementation24 Sep 2021 Yuan YAO, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun

Pre-Trained Vision-Language Models (VL-PTMs) have shown promising capabilities in grounding natural language in image data, facilitating a broad variety of cross-modal tasks.

Visual Grounding

CauseRec: Counterfactual User Sequence Synthesis for Sequential Recommendation

no code implementations11 Sep 2021 Shengyu Zhang, Dong Yao, Zhou Zhao, Tat-Seng Chua, Fei Wu

In this paper, we propose to learn accurate and robust user representations, which are required to be less sensitive to (attack on) noisy behaviors and trust more on the indispensable ones, by modeling counterfactual data distribution.

counterfactual Representation Learning +1

How Knowledge Graph and Attention Help? A Qualitative Analysis into Bag-level Relation Extraction

1 code implementation ACL 2021 Zikun Hu, Yixin Cao, Lifu Huang, Tat-Seng Chua

In this paper, we contribute a dataset and propose a paradigm to quantitatively evaluate the effect of attention and KG on bag-level relation extraction (RE).

Relation Relation Extraction

How Knowledge Graph and Attention Help? A Quantitative Analysis into Bag-level Relation Extraction

1 code implementation26 Jul 2021 Zikun Hu, Yixin Cao, Lifu Huang, Tat-Seng Chua

In this paper, we contribute a dataset and propose a paradigm to quantitatively evaluate the effect of attention and KG on bag-level relation extraction (RE).

Relation Relation Extraction

MConv: An Environment for Multimodal Conversational Search across Multiple Domains

1 code implementation SIGIR 2021 Lizi Liao, Le Hong Long, Zheng Zhang, Minlie Huang, Tat-Seng Chua

Second, a set of benchmark results for dialogue state tracking, conversational recommendation, response generation as well as a unified model for multiple tasks are reported.

Conversational Search Dialogue State Tracking +1

NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions

1 code implementation CVPR 2021 Junbin Xiao, Xindi Shang, Angela Yao, Tat-Seng Chua

We introduce NExT-QA, a rigorously designed video question answering (VideoQA) benchmark to advance video understanding from describing to explaining the temporal actions.

Question Answering Video Question Answering +2

Empowering Language Understanding with Counterfactual Reasoning

1 code implementation Findings (ACL) 2021 Fuli Feng, Jizhi Zhang, Xiangnan He, Hanwang Zhang, Tat-Seng Chua

Present language understanding methods have demonstrated extraordinary ability of recognizing patterns in texts via machine learning.

counterfactual Counterfactual Reasoning +2

Deconfounded Video Moment Retrieval with Causal Intervention

1 code implementation3 Jun 2021 Xun Yang, Fuli Feng, Wei Ji, Meng Wang, Tat-Seng Chua

To fill the research gap, we propose a causality-inspired VMR framework that builds structural causal model to capture the true effect of query and video content on the prediction.

Moment Retrieval Retrieval

Reproducibility Companion Paper: Knowledge Enhanced Neural Fashion Trend Forecasting

1 code implementation25 May 2021 Yunshan Ma, Yujuan Ding, Xun Yang, Lizi Liao, Wai Keung Wong, Tat-Seng Chua, Jinyoung Moon, Hong-Han Shuai

This companion paper supports the replication of the fashion trend forecasting experiments with the KERN (Knowledge Enhanced Recurrent Network) method that we presented in the ICMR 2020.

Deconfounded Recommendation for Alleviating Bias Amplification

1 code implementation22 May 2021 Wenjie Wang, Fuli Feng, Xiangnan He, Xiang Wang, Tat-Seng Chua

In this work, we scrutinize the cause-effect factors for bias amplification, identifying the main reason lies in the confounder effect of imbalanced item distribution on user representation and prediction score.

Fairness Recommendation Systems

NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions

2 code implementations18 May 2021 Junbin Xiao, Xindi Shang, Angela Yao, Tat-Seng Chua

We introduce NExT-QA, a rigorously designed video question answering (VideoQA) benchmark to advance video understanding from describing to explaining the temporal actions.

Question Answering Video Question Answering +2

Leveraging Two Types of Global Graph for Sequential Fashion Recommendation

no code implementations17 May 2021 Yujuan Ding, Yunshan Ma, Wai Keung Wong, Tat-Seng Chua

Sequential fashion recommendation is of great significance in online fashion shopping, which accounts for an increasing portion of either fashion retailing or online e-commerce.

Vocal Bursts Valence Prediction

TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance

1 code implementation ACL 2021 Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, Tat-Seng Chua

In this work, we extract samples from real financial reports to build a new large-scale QA dataset containing both Tabular And Textual data, named TAT-QA, where numerical reasoning is usually required to infer the answer, such as addition, subtraction, multiplication, division, counting, comparison/sorting, and the compositions.

Question Answering

Leveraging Multiple Relations for Fashion Trend Forecasting Based on Social Media

no code implementations7 May 2021 Yujuan Ding, Yunshan Ma, Lizi Liao, Wai Keung Wong, Tat-Seng Chua

Towards insightful fashion trend forecasting, previous work [1] proposed to analyze more fine-grained fashion elements which can informatively reveal fashion trends.

Time Series Analysis

A-FMI: Learning Attributions from Deep Networks via Feature Map Importance

no code implementations12 Apr 2021 An Zhang, Xiang Wang, Chengfang Fang, Jie Shi, Tat-Seng Chua, Zehua Chen

Gradient-based attribution methods can aid in the understanding of convolutional neural networks (CNNs).

Conditional Hyper-Network for Blind Super-Resolution with Multiple Degradations

1 code implementation8 Apr 2021 Guanghao Yin, Wei Wang, Zehuan Yuan, Wei Ji, Dongdong Yu, Shouqian Sun, Tat-Seng Chua, Changhu Wang

We extract degradation prior at task-level with the proposed ConditionNet, which will be used to adapt the parameters of the basic SR network (BaseNet).

Blind Super-Resolution Image Super-Resolution

Learning Intents behind Interactions with Knowledge Graph for Recommendation

2 code implementations14 Feb 2021 Xiang Wang, Tinglin Huang, Dingxian Wang, Yancheng Yuan, Zhenguang Liu, Xiangnan He, Tat-Seng Chua

In this study, we explore intents behind a user-item interaction by using auxiliary item knowledge, and propose a new model, Knowledge Graph-based Intent Network (KGIN).

Recommendation Systems Relation

Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering

no code implementations4 Jan 2021 Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, Tat-Seng Chua

Open-domain Question Answering (OpenQA) is an important task in Natural Language Processing (NLP), which aims to answer a question in the form of natural language based on large-scale unstructured documents.

Machine Reading Comprehension Open-Domain Question Answering

Causal Screening to Interpret Graph Neural Networks

no code implementations1 Jan 2021 Xiang Wang, Yingxin Wu, An Zhang, Xiangnan He, Tat-Seng Chua

In this work, we focus on the causal interpretability in GNNs and propose a method, Causal Screening, from the perspective of cause-effect.

Explanation Generation

Learning Relation Prototype from Unlabeled Texts for Long-tail Relation Extraction

1 code implementation27 Nov 2020 Yixin Cao, Jun Kuang, Ming Gao, Aoying Zhou, Yonggang Wen, Tat-Seng Chua

In this paper, we propose a general approach to learn relation prototypesfrom unlabeled texts, to facilitate the long-tail relation extraction by transferring knowledge from the relation types with sufficient trainingdata.

Relation Relation Extraction +1

Should Graph Convolution Trust Neighbors? A Simple Causal Inference Method

1 code implementation22 Oct 2020 Fuli Feng, Weiran Huang, Xiangnan He, Xin Xin, Qifan Wang, Tat-Seng Chua

To this end, we analyze the working mechanism of GCN with causal graph, estimating the causal effect of a node's local structure for the prediction.

Blocking Causal Inference +4

Clicks can be Cheating: Counterfactual Recommendation for Mitigating Clickbait Issue

1 code implementation21 Sep 2020 Wenjie Wang, Fuli Feng, Xiangnan He, Hanwang Zhang, Tat-Seng Chua

However, we argue that there is a significant gap between clicks and user satisfaction -- it is common that a user is "cheated" to click an item by the attractive title/cover of the item.

Click-Through Rate Prediction counterfactual +1

Multi-modal Cooking Workflow Construction for Food Recipes

no code implementations20 Aug 2020 Liangming Pan, Jingjing Chen, Jianlong Wu, Shaoteng Liu, Chong-Wah Ngo, Min-Yen Kan, Yu-Gang Jiang, Tat-Seng Chua

Understanding food recipe requires anticipating the implicit causal effects of cooking actions, such that the recipe can be converted into a graph describing the temporal workflow of the recipe.

Common Sense Reasoning

Neural Sparse Voxel Fields

1 code implementation NeurIPS 2020 Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, Christian Theobalt

We also demonstrate several challenging tasks, including multi-scene learning, free-viewpoint rendering of a moving human, and large-scale scene rendering.

Tree-Augmented Cross-Modal Encoding for Complex-Query Video Retrieval

no code implementations6 Jul 2020 Xun Yang, Jianfeng Dong, Yixin Cao, Xun Wang, Meng Wang, Tat-Seng Chua

To facilitate video retrieval with complex queries, we propose a Tree-augmented Cross-modal Encoding method by jointly learning the linguistic structure of queries and the temporal representation of videos.

Retrieval Video Retrieval

Disentangled Graph Collaborative Filtering

2 code implementations3 Jul 2020 Xiang Wang, Hongye Jin, An Zhang, Xiangnan He, Tong Xu, Tat-Seng Chua

Such uniform approach to model user interests easily results in suboptimal representations, failing to model diverse relationships and disentangle user intents in representations.

Collaborative Filtering Disentanglement

Interactive Path Reasoning on Graph for Conversational Recommendation

no code implementations1 Jul 2020 Wenqiang Lei, Gangyi Zhang, Xiangnan He, Yisong Miao, Xiang Wang, Liang Chen, Tat-Seng Chua

Traditional recommendation systems estimate user preference on items from past interaction history, thus suffering from the limitations of obtaining fine-grained and dynamic user preference.

Attribute Recommendation Systems

Few-shot 3D Point Cloud Semantic Segmentation

1 code implementation CVPR 2021 Na Zhao, Tat-Seng Chua, Gim Hee Lee

These fully supervised approaches heavily rely on large amounts of labeled training data that are difficult to obtain and cannot segment new classes after training.

Few-shot 3D Point Cloud Semantic Segmentation Segmentation +1

Denoising Implicit Feedback for Recommendation

1 code implementation7 Jun 2020 Wenjie Wang, Fuli Feng, Xiangnan He, Liqiang Nie, Tat-Seng Chua

In this work, we explore the central theme of denoising implicit feedback for recommender training.

Denoising Recommendation Systems

Rethinking Dialogue State Tracking with Reasoning

no code implementations27 May 2020 Lizi Liao, Yunshan Ma, Wenqiang Lei, Tat-Seng Chua

Tracking dialogue states to better interpret user goals and feed downstream policy learning is a bottleneck in dialogue management.

Dialogue Management Dialogue State Tracking +1

Hierarchical Fashion Graph Network for Personalized Outfit Recommendation

1 code implementation26 May 2020 Xingchen Li, Xiang Wang, Xiangnan He, Long Chen, Jun Xiao, Tat-Seng Chua

Fashion outfit recommendation has attracted increasing attentions from online shopping services and fashion communities. Distinct from other scenarios (e. g., social networking or content sharing) which recommend a single item (e. g., a friend or picture) to a user, outfit recommendation predicts user preference on a set of well-matched fashion items. Hence, performing high-quality personalized outfit recommendation should satisfy two requirements -- 1) the nice compatibility of fashion items and 2) the consistence with user preference.

Seamlessly Unifying Attributes and Items: Conversational Recommendation for Cold-Start Users

1 code implementation23 May 2020 Shijun Li, Wenqiang Lei, Qingyun Wu, Xiangnan He, Peng Jiang, Tat-Seng Chua

In this work, we consider the conversational recommendation for cold-start users, where a system can both ask the attributes from and recommend items to a user interactively.

Collaborative Filtering Thompson Sampling

Knowledge Enhanced Neural Fashion Trend Forecasting

1 code implementation7 May 2020 Yunshan Ma, Yujuan Ding, Xun Yang, Lizi Liao, Wai Keung Wong, Tat-Seng Chua

Further-more, to effectively model the time series data of fashion elements with rather complex patterns, we propose a Knowledge EnhancedRecurrent Network model (KERN) which takes advantage of the capability of deep recurrent neural networks in modeling time-series data.

Time Series Time Series Analysis

Semantic Graphs for Generating Deep Questions

1 code implementation ACL 2020 Liangming Pan, Yuxi Xie, Yansong Feng, Tat-Seng Chua, Min-Yen Kan

This paper proposes the problem of Deep Question Generation (DQG), which aims to generate complex questions that require reasoning over multiple pieces of information of the input passage.

Question Generation Question-Generation

Learning Goal-oriented Dialogue Policy with Opposite Agent Awareness

no code implementations Asian Chapter of the Association for Computational Linguistics 2020 Zheng Zhang, Lizi Liao, Xiaoyan Zhu, Tat-Seng Chua, Zitao Liu, Yan Huang, Minlie Huang

Most existing approaches for goal-oriented dialogue policy learning used reinforcement learning, which focuses on the target agent policy and simply treat the opposite agent policy as part of the environment.

Decision Making

Reinforced Negative Sampling over Knowledge Graph for Recommendation

1 code implementation12 Mar 2020 Xiang Wang, Yaokun Xu, Xiangnan He, Yixin Cao, Meng Wang, Tat-Seng Chua

Properly handling missing data is a fundamental challenge in recommendation.

Cross-GCN: Enhancing Graph Convolutional Network with $k$-Order Feature Interactions

no code implementations5 Mar 2020 Fuli Feng, Xiangnan He, Hanwang Zhang, Tat-Seng Chua

Graph Convolutional Network (GCN) is an emerging technique that performs learning and reasoning on graph data.

Document Classification

Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems

no code implementations21 Feb 2020 Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, Tat-Seng Chua

Recommender systems are embracing conversational technologies to obtain user preferences dynamically, and to overcome inherent limitations of their static models.

Recommendation Systems

Multi-source Domain Adaptation for Visual Sentiment Classification

no code implementations12 Jan 2020 Chuang Lin, Sicheng Zhao, Lei Meng, Tat-Seng Chua

Existing domain adaptation methods on visual sentiment classification typically are investigated under the single-source scenario, where the knowledge learned from a source domain of sufficient labeled data is transferred to the target domain of loosely labeled or unlabeled data.

Classification Domain Adaptation +4

SESS: Self-Ensembling Semi-Supervised 3D Object Detection

1 code implementation CVPR 2020 Na Zhao, Tat-Seng Chua, Gim Hee Lee

The performance of existing point cloud-based 3D object detection methods heavily relies on large-scale high-quality 3D annotations.

3D Object Detection Object +2

Heuristic Black-box Adversarial Attacks on Video Recognition Models

1 code implementation21 Nov 2019 Zhipeng Wei, Jingjing Chen, Xingxing Wei, Linxi Jiang, Tat-Seng Chua, Fengfeng Zhou, Yu-Gang Jiang

To overcome this challenge, we propose a heuristic black-box attack model that generates adversarial perturbations only on the selected frames and regions.

Adversarial Attack Video Recognition

Mining Unfollow Behavior in Large-Scale Online Social Networks via Spatial-Temporal Interaction

1 code implementation17 Nov 2019 Haozhe Wu, Zhiyuan Hu, Jia Jia, Yaohua Bu, Xiangnan He, Tat-Seng Chua

Next, we define user's attributes as two categories: spatial attributes (e. g., social role of user) and temporal attributes (e. g., post content of user).

Informativeness

Semi-supervised Entity Alignment via Joint Knowledge Embedding Model and Cross-graph Model

1 code implementation IJCNLP 2019 Chengjiang Li, Yixin Cao, Lei Hou, Jiaxin Shi, Juanzi Li, Tat-Seng Chua

Specifically, as for the knowledge embedding model, we utilize TransE to implicitly complete two KGs towards consistency and learn relational constraints between entities.

Entity Alignment Graph Attention +1

MMGCN: Multi-modal Graph Convolution Network for Personalized Recommendation of Micro-video

1 code implementation ACM International Conference on Multimedia 2019 Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, Tat-Seng Chua

Existing works on multimedia recommendation largely exploit multi-modal contents to enrich item representations, while less effort is made to leverage information interchange between users and items to enhance user representations and further capture user's fine-grained preferences on different modalities.

Microvideo Recommendation Micro-video recommendations +4

Meta-Transfer Learning through Hard Tasks

1 code implementation7 Oct 2019 Qianru Sun, Yaoyao Liu, Zhaozheng Chen, Tat-Seng Chua, Bernt Schiele

In this paper, we propose a novel approach called meta-transfer learning (MTL) which learns to transfer the weights of a deep NN for few-shot learning tasks.

Few-Shot Learning Transfer Learning

Low-Resource Name Tagging Learned with Weakly Labeled Data

1 code implementation IJCNLP 2019 Yixin Cao, Zikun Hu, Tat-Seng Chua, Zhiyuan Liu, Heng Ji

Name tagging in low-resource languages or domains suffers from inadequate training data.

TAG

PS^2-Net: A Locally and Globally Aware Network for Point-Based Semantic Segmentation

1 code implementation15 Aug 2019 Na Zhao, Tat-Seng Chua, Gim Hee Lee

In this paper, we present the PS^2-Net -- a locally and globally aware deep learning framework for semantic segmentation on 3D scene-level point clouds.

Scene Segmentation Segmentation

Who, Where, and What to Wear? Extracting Fashion Knowledge from Social Media

no code implementations12 Aug 2019 Yunshan Ma, Xun Yang, Lizi Liao, Yixin Cao, Tat-Seng Chua

We unify three tasks of occasion, person and clothing discovery from multiple modalities of images, texts and metadata.

Human Detection

Automatic Fashion Knowledge Extraction from Social Media

no code implementations12 Aug 2019 Yunshan Ma, Lizi Liao, Tat-Seng Chua

Fashion knowledge plays a pivotal role in helping people in their dressing.

Modeling Embedding Dimension Correlations via Convolutional Neural Collaborative Filtering

1 code implementation26 Jun 2019 Xiaoyu Du, Xiangnan He, Fajie Yuan, Jinhui Tang, Zhiguang Qin, Tat-Seng Chua

In this work, we emphasize on modeling the correlations among embedding dimensions in neural networks to pursue higher effectiveness for CF.

Collaborative Filtering Recommendation Systems

Deep Conversational Recommender in Travel

no code implementations25 Jun 2019 Lizi Liao, Ryuichi Takanobu, Yunshan Ma, Xun Yang, Minlie Huang, Tat-Seng Chua

When traveling to a foreign country, we are often in dire need of an intelligent conversational agent to provide instant and informative responses to our various queries.

Response Generation

Learning to Self-Train for Semi-Supervised Few-Shot Classification

1 code implementation NeurIPS 2019 Xinzhe Li, Qianru Sun, Yaoyao Liu, Shibao Zheng, Qin Zhou, Tat-Seng Chua, Bernt Schiele

On each task, we train a few-shot model to predict pseudo labels for unlabeled data, and then iterate the self-training steps on labeled and pseudo-labeled data with each step followed by fine-tuning.

Classification General Classification +1

Recent Advances in Neural Question Generation

no code implementations22 May 2019 Liangming Pan, Wenqiang Lei, Tat-Seng Chua, Min-Yen Kan

Emerging research in Neural Question Generation (NQG) has started to integrate a larger variety of inputs, and generating questions requiring higher levels of cognition.

Question Generation Question-Generation

KGAT: Knowledge Graph Attention Network for Recommendation

7 code implementations20 May 2019 Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, Tat-Seng Chua

To provide more accurate, diverse, and explainable recommendation, it is compulsory to go beyond modeling user-item interactions and take side information into account.

Explainable Recommendation Knowledge Graphs +1

Neural Graph Collaborative Filtering

19 code implementations20 May 2019 Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, Tat-Seng Chua

Further analysis verifies the importance of embedding propagation for learning better user and item representations, justifying the rationality and effectiveness of NGCF.

Collaborative Filtering Link Prediction +1

Graph Adversarial Training: Dynamically Regularizing Based on Graph Structure

1 code implementation20 Feb 2019 Fuli Feng, Xiangnan He, Jie Tang, Tat-Seng Chua

Adversarial Training (AT), a dynamic regularization technique, can resist the worst-case perturbations on input features and is a promising choice to improve model robustness and generalization.

General Classification Node Classification

Cannot find the paper you are looking for? You can Submit a new open access paper.