Search Results for author: Maosong Sun

Found 210 papers, 126 papers with code

BMInf: An Efficient Toolkit for Big Model Inference and Tuning

1 code implementation ACL 2022 Xu Han, Guoyang Zeng, Weilin Zhao, Zhiyuan Liu, Zhengyan Zhang, Jie zhou, Jun Zhang, Jia Chao, Maosong Sun

In recent years, large-scale pre-trained language models (PLMs) containing billions of parameters have achieved promising results on various NLP tasks.


Pass off Fish Eyes for Pearls: Attacking Model Selection of Pre-trained Models

1 code implementation ACL 2022 Biru Zhu, Yujia Qin, Fanchao Qi, Yangdong Deng, Zhiyuan Liu, Maosong Sun, Ming Gu

To validate our viewpoints, we design two methods to evaluate the robustness of FMS: (1) model disguise attack, which post-trains an inferior PTM with a contrastive objective, and (2) evaluation data selection, which selects a subset of the data points for FMS evaluation based on K-means clustering.

Backdoor Attack Model Selection

TopWORDS-Seg: Simultaneous Text Segmentation and Word Discovery for Open-Domain Chinese Texts via Bayesian Inference

no code implementations ACL 2022 Changzai Pan, Maosong Sun, Ke Deng

Processing open-domain Chinese texts has been a critical bottleneck in computational linguistics for decades, partially because text segmentation and word discovery often entangle with each other in this challenging scenario.

Bayesian Inference Text Segmentation

Going “Deeper”: Structured Sememe Prediction via Transformer with Tree Attention

1 code implementation Findings (ACL) 2022 Yining Ye, Fanchao Qi, Zhiyuan Liu, Maosong Sun

However, all existing sememe prediction studies ignore the hierarchical structures of sememes, which are important in the sememe-based semantic description system.

CodRED: A Cross-Document Relation Extraction Dataset for Acquiring Knowledge in the Wild

1 code implementation EMNLP 2021 Yuan YAO, Jiaju Du, Yankai Lin, Peng Li, Zhiyuan Liu, Jie zhou, Maosong Sun

Existing relation extraction (RE) methods typically focus on extracting relational facts between entity pairs within single sentences or documents.

Relation Extraction

Self-Supervised Quality Estimation for Machine Translation

no code implementations EMNLP 2021 Yuanhang Zheng, Zhixing Tan, Meng Zhang, Mieradilijiang Maimaiti, Huanbo Luan, Maosong Sun, Qun Liu, Yang Liu

Quality estimation (QE) of machine translation (MT) aims to evaluate the quality of machine-translated sentences without references and is important in practical applications of MT.

Machine Translation Translation

Prompt Tuning for Discriminative Pre-trained Language Models

1 code implementation Findings (ACL) 2022 Yuan YAO, Bowen Dong, Ao Zhang, Zhengyan Zhang, Ruobing Xie, Zhiyuan Liu, Leyu Lin, Maosong Sun, Jianyong Wang

Recent works have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing (NLP) tasks.

Language Modelling Question Answering +1

A Template-based Method for Constrained Neural Machine Translation

no code implementations23 May 2022 Shuo Wang, Peng Li, Zhixing Tan, Zhaopeng Tu, Maosong Sun, Yang Liu

In this work, we propose a template-based method that can yield results with high translation quality and match accuracy while keeping the decoding speed.

Machine Translation Translation

PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models

1 code implementation23 May 2022 Yuan YAO, Qianyu Chen, Ao Zhang, Wei Ji, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun

We show that PEVL enables state-of-the-art performance of detector-free VLP models on position-sensitive tasks such as referring expression comprehension and phrase grounding, and also improves the performance on position-insensitive tasks with grounded inputs.

Language Modelling Phrase Grounding +3

Sampling with Attribute-Related Information for Controlling Language Models

1 code implementation12 May 2022 Shangda Wu, Maosong Sun

We propose a new simple guided decoding method, Gamma Sampling, which does not require complex engineering and any extra data.

Symphony Generation with Permutation Invariant Language Model

1 code implementation10 May 2022 Jiafeng Liu, Yuanliang Dong, Zehua Cheng, Xinran Zhang, Xiaobing Li, Feng Yu, Maosong Sun

In this work, we present a symbolic symphony music generation solution, SymphonyNet, based on a permutation invariant language model.

Audio Generation Language Modelling +2

LEVEN: A Large-Scale Chinese Legal Event Detection Dataset

1 code implementation Findings (ACL) 2022 Feng Yao, Chaojun Xiao, Xiaozhi Wang, Zhiyuan Liu, Lei Hou, Cunchao Tu, Juanzi Li, Yun Liu, Weixing Shen, Maosong Sun

However, existing Legal Event Detection (LED) datasets only concern incomprehensive event types and have limited annotated data, which restricts the development of LED methods and their downstream applications.

Event Detection

A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models

1 code implementation ACL 2022 Deming Ye, Yankai Lin, Peng Li, Maosong Sun, Zhiyuan Liu

Pre-trained language models (PLMs) cannot well recall rich factual knowledge of entities exhibited in large-scale corpora, especially those rare entities.

Domain Adaptation

QuoteR: A Benchmark of Quote Recommendation for Writing

1 code implementation ACL 2022 Fanchao Qi, Yanhui Yang, Jing Yi, Zhili Cheng, Zhiyuan Liu, Maosong Sun

To facilitate the research on this task, we build a large and fully open quote recommendation dataset called QuoteR, which comprises three parts including English, standard Chinese and classical Chinese.

Chord-Conditioned Melody Choralization with Controllable Harmonicity and Polyphonicity

1 code implementation17 Feb 2022 Shangda Wu, Xiaobing Li, Maosong Sun

Melody choralization, i. e. generating a four-part chorale based on a user-given melody, has long been closely associated with J. S.

YACLC: A Chinese Learner Corpus with Multidimensional Annotation

no code implementations30 Dec 2021 Yingying Wang, Cunliang Kong, Liner Yang, Yijun Wang, Xiaorong Lu, Renfen Hu, Shan He, Zhenghao Liu, Yun Chen, Erhong Yang, Maosong Sun

This resource is of great relevance for second language acquisition research, foreign-language teaching, and automatic grammatical error correction.

Grammatical Error Correction Language Acquisition

On Transferability of Prompt Tuning for Natural Language Processing

1 code implementation12 Nov 2021 Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, Jie zhou

To explore whether we can improve PT via prompt transfer, we empirically investigate the transferability of soft prompts across different downstream tasks and PLMs in this work.

Natural Language Understanding Pretrained Language Models +1

OpenPrompt: An Open-source Framework for Prompt-learning

1 code implementation ACL 2022 Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun

Prompt-learning has become a new paradigm in modern natural language processing, which directly adapts pre-trained language models (PLMs) to $cloze$-style prediction, autoregressive modeling, or sequence to sequence generation, resulting in promising performances on various tasks.

Exploring Universal Intrinsic Task Subspace via Prompt Tuning

1 code implementation15 Oct 2021 Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, Jie zhou

In the experiments, we study diverse few-shot NLP tasks and surprisingly find that in a 250-dimensional subspace found with 100 tasks, by only tuning 250 free parameters, we can recover 97% and 83% of the full prompt tuning performance for 100 seen tasks (using different training data) and 20 unseen tasks, respectively, showing great generalization ability of the found intrinsic task subspace.

Textual Backdoor Attacks Can Be More Harmful via Two Simple Tricks

no code implementations15 Oct 2021 Yangyi Chen, Fanchao Qi, Zhiyuan Liu, Maosong Sun

In this paper, we find two simple tricks that can make existing textual backdoor attacks much more harmful.

Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer

1 code implementation EMNLP 2021 Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, Maosong Sun

In this paper, we make the first attempt to conduct adversarial and backdoor attacks based on text style transfer, which is aimed at altering the style of a sentence while preserving its meaning.

Backdoor Attack Style Transfer +1

CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models

1 code implementation24 Sep 2021 Yuan YAO, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun

Pre-Trained Vision-Language Models (VL-PTMs) have shown promising capabilities in grounding natural language in image data, facilitating a broad variety of cross-modal tasks.

Visual Grounding

Packed Levitated Marker for Entity and Relation Extraction

1 code implementation ACL 2022 Deming Ye, Yankai Lin, Peng Li, Maosong Sun

In particular, we propose a neighborhood-oriented packing strategy, which considers the neighbor spans integrally to better model the entity boundary information.

Joint Entity and Relation Extraction Named Entity Recognition

Lingxi: A Diversity-aware Chinese Modern Poetry Generation System

no code implementations27 Aug 2021 Xinran Zhang, Maosong Sun, Jiafeng Liu, Xiaobing Li

We propose nucleus sampling with randomized head (NS-RH) algorithm, which randomizes the high frequency part ("head") of the predicted distribution, in order to emphasize on the "comparatively low frequency" words.

Semantic Similarity Semantic Textual Similarity +1

Language Models are Good Translators

no code implementations25 Jun 2021 Shuo Wang, Zhaopeng Tu, Zhixing Tan, Wenxuan Wang, Maosong Sun, Yang Liu

Inspired by the recent progress of large-scale pre-trained language models on machine translation in a limited scenario, we firstly demonstrate that a single language model (LM4MT) can achieve comparable performance with strong encoder-decoder NMT models on standard machine translation benchmarks, using the same training data and similar amount of model parameters.

Language Modelling Machine Translation +1

CPM-2: Large-scale Cost-effective Pre-trained Language Models

2 code implementations20 Jun 2021 Zhengyan Zhang, Yuxian Gu, Xu Han, Shengqi Chen, Chaojun Xiao, Zhenbo Sun, Yuan YAO, Fanchao Qi, Jian Guan, Pei Ke, Yanzheng Cai, Guoyang Zeng, Zhixing Tan, Zhiyuan Liu, Minlie Huang, Wentao Han, Yang Liu, Xiaoyan Zhu, Maosong Sun

We present a suite of cost-effective techniques for the use of PLMs to deal with the efficiency issues of pre-training, fine-tuning, and inference.

Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution

1 code implementation ACL 2021 Fanchao Qi, Yuan YAO, Sophia Xu, Zhiyuan Liu, Maosong Sun

Recent studies show that neural natural language processing (NLP) models are vulnerable to backdoor attacks.

On the Language Coverage Bias for Neural Machine Translation

no code implementations Findings (ACL) 2021 Shuo Wang, Zhaopeng Tu, Zhixing Tan, Shuming Shi, Maosong Sun, Yang Liu

Language coverage bias, which indicates the content-dependent differences between sentence pairs originating from the source and target languages, is important for neural machine translation (NMT) because the target-original training data is not well exploited in current practice.

Data Augmentation Machine Translation +1

CCPM: A Chinese Classical Poetry Matching Dataset

1 code implementation3 Jun 2021 Wenhao Li, Fanchao Qi, Maosong Sun, Xiaoyuan Yi, Jiarui Zhang

We hope this dataset can further enhance the study on incorporating deep semantics into the understanding and generation system of Chinese classical poetry.


Open Hierarchical Relation Extraction

1 code implementation NAACL 2021 Kai Zhang, Yuan YAO, Ruobing Xie, Xu Han, Zhiyuan Liu, Fen Lin, Leyu Lin, Maosong Sun

To establish the bidirectional connections between OpenRE and relation hierarchy, we propose the task of open hierarchical relation extraction and present a novel OHRE framework for the task.

Relation Extraction

Sub-Character Tokenization for Chinese Pretrained Language Models

no code implementations1 Jun 2021 Chenglei Si, Zhengyan Zhang, Yingfa Chen, Fanchao Qi, Xiaozhi Wang, Zhiyuan Liu, Yasheng Wang, Qun Liu, Maosong Sun

2) Pronunciation-based SubChar tokenizers can encode Chinese homophones into the same transliteration sequences and produce the same tokenization output, hence being robust to all homophone typos.

Chinese Word Segmentation Language Modelling +2

Fully Hyperbolic Neural Networks

1 code implementation ACL 2022 Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, Jie zhou

Hyperbolic neural networks have shown great potential for modeling complex data.

Transfer Learning for Sequence Generation: from Single-source to Multi-source

1 code implementation ACL 2021 Xuancheng Huang, Jingfang Xu, Maosong Sun, Yang Liu

Although directly finetuning pretrained models on MSG tasks and concatenating multiple sources into a single long sequence is regarded as a simple method to transfer pretrained models to MSG tasks, we conjecture that the direct finetuning method leads to catastrophic forgetting and solely relying on pretrained self-attention layers to capture cross-source information is not sufficient.

Automatic Post-Editing Document Summarization +3

Knowledge Inheritance for Pre-trained Language Models

2 code implementations28 May 2021 Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, Jie zhou

Specifically, we introduce a pre-training framework named "knowledge inheritance" (KI) and explore how could knowledge distillation serve as auxiliary supervision during pre-training to efficiently learn larger PLMs.

Domain Adaptation Knowledge Distillation +2

Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger

2 code implementations ACL 2021 Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, Maosong Sun

As far as we know, almost all existing textual backdoor attack methods insert additional contents into normal samples as triggers, which causes the trigger-embedded samples to be detected and the backdoor attacks to be blocked without much effort.

Backdoor Attack

Automatic Construction of Sememe Knowledge Bases via Dictionaries

1 code implementation Findings (ACL) 2021 Fanchao Qi, Yangyi Chen, Fengyu Wang, Zhiyuan Liu, Xiao Chen, Maosong Sun

We use this method to build an English SKB and a French SKB, and conduct comprehensive evaluations from both intrinsic and extrinsic perspectives.

TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference

1 code implementation NAACL 2021 Deming Ye, Yankai Lin, Yufei Huang, Maosong Sun

To address this issue, we propose a dynamic token reduction approach to accelerate PLMs' inference, named TR-BERT, which could flexibly adapt the layer number of each token in inference to avoid redundant calculation.

PTR: Prompt Tuning with Rules for Text Classification

1 code implementation24 May 2021 Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, Maosong Sun

This indicates that PTR is a promising approach to take advantage of both human prior knowledge and PLMs for those complicated classification tasks.

Classification Natural Language Inference +3

Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction

1 code implementation Findings (ACL) 2021 Tianyu Gao, Xu Han, Keyue Qiu, Yuzhuo Bai, Zhiyu Xie, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, Jie zhou

Distantly supervised (DS) relation extraction (RE) has attracted much attention in the past few years as it can utilize large-scale auto-labeled data.

Relation Extraction

Dynamic Multi-Branch Layers for On-Device Neural Machine Translation

1 code implementation14 May 2021 Zhixing Tan, Zeyuan Yang, Meng Zhang, Qun Liu, Maosong Sun, Yang Liu

With the rapid development of artificial intelligence (AI), there is a trend in moving AI applications, such as neural machine translation (NMT), from cloud to mobile devices.

Machine Translation Translation

Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents

1 code implementation9 May 2021 Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, Maosong Sun

Legal artificial intelligence (LegalAI) aims to benefit legal systems with the technology of artificial intelligence, especially natural language processing (NLP).

Language Modelling Question Answering +1

Visual Distant Supervision for Scene Graph Generation

1 code implementation ICCV 2021 Yuan YAO, Ao Zhang, Xu Han, Mengdi Li, Cornelius Weber, Zhiyuan Liu, Stefan Wermter, Maosong Sun

In this work, we propose visual distant supervision, a novel paradigm of visual relation learning, which can train scene graph models without any human-labeled data.

Graph Generation Predicate Classification +1

Equality before the Law: Legal Judgment Consistency Analysis for Fairness

no code implementations25 Mar 2021 Yuzhong Wang, Chaojun Xiao, Shirong Ma, Haoxi Zhong, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, Maosong Sun

We propose to simulate judges from different groups with legal judgment prediction (LJP) models and measure the judicial inconsistency with the disagreement of the judgment results given by LJP models trained on different groups.


Optimal Embedding Calibration for Symbolic Music Similarity

no code implementations13 Mar 2021 Xinran Zhang, Maosong Sun, Jiafeng Liu, Xiaobing Li

In natural language processing (NLP), the semantic similarity task requires large-scale, high-quality human-annotated labels for fine-tuning or evaluation.

Language Modelling Representation Learning +2

Improving Diversity of Neural Text Generation via Inverse Probability Weighting

no code implementations13 Mar 2021 Xinran Zhang, Maosong Sun, Jiafeng Liu, Xiaobing Li

Traditional stochastic sampling methods only focus on truncating the unreliable "tail" of the distribution, and do not address the "head" part, which we show might contain tedious or even repetitive candidates with high probability that lead to repetition loops.

Language Modelling Text Generation

UPRec: User-Aware Pre-training for Recommender Systems

no code implementations22 Feb 2021 Chaojun Xiao, Ruobing Xie, Yuan YAO, Zhiyuan Liu, Maosong Sun, Xu Zhang, Leyu Lin

Existing sequential recommendation methods rely on large amounts of training data and usually suffer from the data sparsity problem.

Self-Supervised Learning Sequential Recommendation

Representation Learning for Natural Language Processing

no code implementations7 Feb 2021 Zhiyuan Liu, Yankai Lin, Maosong Sun

This book aims to review and present the recent advances of distributed representation learning for NLP, including why representation learning can improve NLP, how representation learning takes part in various important topics of NLP, and what challenges are still not well addressed by distributed representation.

Representation Learning

CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models

1 code implementation7 Feb 2021 Yusheng Su, Xu Han, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Peng Li, Jie zhou, Maosong Sun

We then perform contrastive semi-supervised learning on both the retrieved unlabeled and original labeled instances to help PLMs capture crucial task-related semantic features.

OpenMatch: An Open Source Library for Neu-IR Research

1 code implementation30 Jan 2021 Zhenghao Liu, Kaitao Zhang, Chenyan Xiong, Zhiyuan Liu, Maosong Sun

OpenMatch is a Python-based library that serves for Neural Information Retrieval (Neu-IR) research.

Document Ranking Information Retrieval

Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks

1 code implementation ICML Workshop AML 2021 Zhengyan Zhang, Guangxuan Xiao, Yongwei Li, Tian Lv, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Xin Jiang, Maosong Sun

In this work, we demonstrate the universal vulnerability of PTMs, where fine-tuned PTMs can be easily controlled by backdoor attacks in arbitrary downstream tasks.

Backdoor Attack

Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning

1 code implementation31 Dec 2020 Chenglei Si, Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Qun Liu, Maosong Sun

In this work, we propose a simple and effective method to cover a much larger proportion of the attack search space, called Adversarial and Mixup Data Augmentation (AMDA).

Adversarial Robustness Pretrained Language Models +2

Neural Machine Translation: A Review of Methods, Resources, and Tools

no code implementations31 Dec 2020 Zhixing Tan, Shuo Wang, Zonghan Yang, Gang Chen, Xuancheng Huang, Maosong Sun, Yang Liu

Machine translation (MT) is an important sub-field of natural language processing that aims to translate natural languages using computers.

Data Augmentation Machine Translation +1

Towards a Universal Continuous Knowledge Base

no code implementations25 Dec 2020 Gang Chen, Maosong Sun, Yang Liu

In this work, we propose a method for building a continuous knowledge base (CKB) that can store knowledge imported from multiple, diverse neural networks.

Knowledge Distillation Text Classification +1

Mask-Align: Self-Supervised Neural Word Alignment

1 code implementation ACL 2021 Chi Chen, Maosong Sun, Yang Liu

Word alignment, which aims to align translationally equivalent words between source and target sentences, plays an important role in many natural language processing tasks.

Machine Translation Translation +1

Try to Substitute: An Unsupervised Chinese Word Sense Disambiguation Method Based on HowNet

1 code implementation COLING 2020 Bairu Hou, Fanchao Qi, Yuan Zang, Xurui Zhang, Zhiyuan Liu, Maosong Sun

In this paper, we propose a new unsupervised method for HowNet-based Chinese WSD, which exploits the masked language model task of pre-trained language models.

Language Modelling Word Sense Disambiguation

Denoising Relation Extraction from Document-level Distant Supervision

1 code implementation EMNLP 2020 Chaojun Xiao, Yuan YAO, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin, Leyu Lin

Distant supervision (DS) has been widely used to generate auto-labeled data for sentence-level relation extraction (RE), which improves RE performance.

Denoising Relation Extraction

Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads

no code implementations7 Nov 2020 Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Qun Liu, Maosong Sun

To measure the informativeness of attention heads, we train our Single-Shot Meta-Pruner (SMP) with a meta-learning paradigm aiming to maintain the distribution of text representations after pruning.

Informativeness Meta-Learning +1

Towards Interpretable Natural Language Understanding with Explanations as Latent Variables

1 code implementation NeurIPS 2020 Wangchunshu Zhou, Jinyi Hu, HANLIN ZHANG, Xiaodan Liang, Maosong Sun, Chenyan Xiong, Jian Tang

In this paper, we develop a general framework for interpretable natural language understanding that requires only a small set of human annotated explanations for training.

Explanation Generation Natural Language Understanding

Learning from Context or Names? An Empirical Study on Neural Relation Extraction

1 code implementation EMNLP 2020 Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, Jie zhou

We find that (i) while context is the main source to support the predictions, RE models also heavily rely on the information from entity mentions, most of which is type information, and (ii) existing datasets may leak shallow heuristics via entity mentions and thus contribute to the high performance on RE benchmarks.

Relation Extraction

IsOBS: An Information System for Oracle Bone Script

no code implementations EMNLP 2020 Xu Han, Yuzhuo Bai, Keyue Qiu, Zhiyuan Liu, Maosong Sun

Oracle bone script (OBS) is the earliest known ancient Chinese writing system and the ancestor of modern Chinese.

Few-Shot Learning

WantWords: An Open-source Online Reverse Dictionary System

1 code implementation EMNLP 2020 Fanchao Qi, Lei Zhang, Yanhui Yang, Zhiyuan Liu, Maosong Sun

A reverse dictionary takes descriptions of words as input and outputs words semantically matching the input descriptions.

CokeBERT: Contextual Knowledge Selection and Embedding towards Enhanced Pre-Trained Language Models

1 code implementation29 Sep 2020 Yusheng Su, Xu Han, Zhengyan Zhang, Peng Li, Zhiyuan Liu, Yankai Lin, Jie zhou, Maosong Sun

In this paper, we propose a novel framework named Coke to dynamically select contextual knowledge and embed knowledge context according to textual context for PLMs, which can avoid the effect of redundant and ambiguous knowledge in KGs that cannot match the input text.

Knowledge Graphs

Knowledge Transfer via Pre-training for Recommendation: A Review and Prospect

no code implementations19 Sep 2020 Zheni Zeng, Chaojun Xiao, Yuan YAO, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, Maosong Sun

Recommender systems aim to provide item recommendations for users, and are usually faced with data sparsity problem (e. g., cold start) in real-world scenarios.

Recommendation Systems Transfer Learning

Country Image in COVID-19 Pandemic: A Case Study of China

1 code implementation12 Sep 2020 Huimin Chen, Zeyu Zhu, Fanchao Qi, Yining Ye, Zhiyuan Liu, Maosong Sun, Jianbin Jin

Therefore, in this study, we take China as a specific and typical case and investigate its image with aspect-based sentiment analysis on a large-scale Twitter dataset.

Aspect-Based Sentiment Analysis

Modeling Voting for System Combination in Machine Translation

1 code implementation14 Jul 2020 Xuancheng Huang, Jiacheng Zhang, Zhixing Tan, Derek F. Wong, Huanbo Luan, Jingfang Xu, Maosong Sun, Yang Liu

System combination is an important technique for combining the hypotheses of different machine translation systems to improve translation performance.

Machine Translation Translation

Continual Relation Learning via Episodic Memory Activation and Reconsolidation

no code implementations ACL 2020 Xu Han, Yi Dai, Tianyu Gao, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, Jie zhou

Continual relation learning aims to continually train a model on new data to learn incessantly emerging novel relations while avoiding catastrophically forgetting old relations.

Continual Learning

Fast Network Embedding Enhancement via High Order Proximity Approximation

2 code implementations ‏‏‎ ‎ 2020 Cheng Yang, Maosong Sun, Zhiyuan Liu, Cunchao Tu

Many Network Representation Learning (NRL) methods have been proposed to learn vector representations for vertices in a network recently.

Dimensionality Reduction Link Prediction +2

KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion

1 code implementation Findings (ACL) 2021 Jie Zhou, Shengding Hu, Xin Lv, Cheng Yang, Zhiyuan Liu, Wei Xu, Jie Jiang, Juanzi Li, Maosong Sun

Based on the datasets, we propose novel tasks such as multi-hop knowledge abstraction (MKA), multi-hop knowledge concretization (MKC) and then design a comprehensive benchmark.

Knowledge Graphs Transfer Learning

How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence

2 code implementations ACL 2020 Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, Maosong Sun

Legal Artificial Intelligence (LegalAI) focuses on applying the technology of artificial intelligence, especially natural language processing, to benefit tasks in the legal domain.

Train No Evil: Selective Masking for Task-Guided Pre-Training

1 code implementation EMNLP 2020 Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, Maosong Sun

In this stage, the model is trained by masked language modeling on in-domain unsupervised data to learn domain-specific patterns and we propose a novel selective masking strategy to learn task-specific patterns.

Language Modelling Masked Language Modeling +1

Coreferential Reasoning Learning for Language Representation

2 code implementations EMNLP 2020 Deming Ye, Yankai Lin, Jiaju Du, Zheng-Hao Liu, Peng Li, Maosong Sun, Zhiyuan Liu

Language representation models such as BERT could effectively capture contextual semantic information from plain text, and have been proved to achieve promising results in lots of downstream NLP tasks with appropriate fine-tuning.

Relation Extraction

MixPoet: Diverse Poetry Generation via Learning Controllable Mixed Latent Space

no code implementations13 Mar 2020 Xiaoyuan Yi, Ruoyu Li, Cheng Yang, Wenhao Li, Maosong Sun

Though recent neural models make prominent progress in some criteria of poetry quality, generated poems still suffer from the problem of poor diversity.

Generating Major Types of Chinese Classical Poetry in a Uniformed Framework

no code implementations LREC 2020 Jinyi Hu, Maosong Sun

In this paper, we propose a GPT-2 based uniformed framework for generating major types of Chinese classical poems.

Text Generation

Lexical Sememe Prediction using Dictionary Definitions by Capturing Local Semantic Correspondence

1 code implementation16 Jan 2020 Jiaju Du, Fanchao Qi, Maosong Sun, Zhiyuan Liu

We find that sememes of each word are usually semantically matched to different words in its dictionary definition, and we name this matching relationship local semantic correspondence.

Semantic correspondence

Multi-channel Reverse Dictionary Model

1 code implementation18 Dec 2019 Lei Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Qun Liu, Maosong Sun

A reverse dictionary takes the description of a target word as input and outputs the target word together with other words that match the description.

Learning to Predict Explainable Plots for Neural Story Generation

no code implementations5 Dec 2019 Gang Chen, Yang Liu, Huanbo Luan, Meng Zhang, Qun Liu, Maosong Sun

While the use of neural networks has proven effective in improving story generation, how to learn to generate an explainable high-level plot still remains a major challenge.

Story Generation

JEC-QA: A Legal-Domain Question Answering Dataset

no code implementations27 Nov 2019 Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, Maosong Sun

We present JEC-QA, the largest question answering dataset in the legal domain, collected from the National Judicial Examination of China.

Question Answering Reading Comprehension

Neural Machine Translation with Explicit Phrase Alignment

no code implementations26 Nov 2019 Jiacheng Zhang, Huanbo Luan, Maosong Sun, FeiFei Zhai, Jingfang Xu, Yang Liu

The lack of alignment in NMT models leads to three problems: it is hard to (1) interpret the translation process, (2) impose lexical constraints, and (3) impose structural constraints.

Machine Translation Translation

Learning to Copy for Automatic Post-Editing

2 code implementations IJCNLP 2019 Xuancheng Huang, Yang Liu, Huanbo Luan, Jingfang Xu, Maosong Sun

To better identify translation errors, our method learns the representations of source sentences and system outputs in an interactive way.

Automatic Post-Editing Translation

Multi-Paragraph Reasoning with Knowledge-enhanced Graph Neural Network

no code implementations6 Nov 2019 Deming Ye, Yankai Lin, Zheng-Hao Liu, Zhiyuan Liu, Maosong Sun

Multi-paragraph reasoning is indispensable for open-domain question answering (OpenQA), which receives less attention in the current OpenQA systems.

Open-Domain Question Answering

Adversarial Language Games for Advanced Natural Language Intelligence

no code implementations5 Nov 2019 Yuan Yao, Haoxi Zhong, Zhengyan Zhang, Xu Han, Xiaozhi Wang, Chaojun Xiao, Guoyang Zeng, Zhiyuan Liu, Maosong Sun

In this work, we propose a challenging adversarial language game called Adversarial Taboo as an example, in which an attacker and a defender compete around a target word.

Board Games

HMEAE: Hierarchical Modular Event Argument Extraction

1 code implementation IJCNLP 2019 Xiaozhi Wang, Ziqi Wang, Xu Han, Zhiyuan Liu, Juanzi Li, Peng Li, Maosong Sun, Jie zhou, Xiang Ren

Existing event extraction methods classify each argument role independently, ignoring the conceptual correlations between different argument roles.

Event Extraction

Word-level Textual Adversarial Attacking as Combinatorial Optimization

1 code implementation ACL 2020 Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, Maosong Sun

Also, further experiments show our model has higher transferability and can bring more robustness enhancement to victim models by adversarial training.

Adversarial Attack Combinatorial Optimization +3

Fine-grained Fact Verification with Kernel Graph Attention Network

1 code implementation ACL 2020 Zhenghao Liu, Chenyan Xiong, Maosong Sun, Zhiyuan Liu

Fact Verification requires fine-grained natural language inference capability that finds subtle clues to identify the syntactical and semantically correct but not well-supported claims.

Fact Verification Graph Attention +1

Improving Sequence Modeling Ability of Recurrent Neural Networks via Sememes

1 code implementation20 Oct 2019 Yujia Qin, Fanchao Qi, Sicong Ouyang, Zhiyuan Liu, Cheng Yang, Yasheng Wang, Qun Liu, Maosong Sun

Sememes, the minimum semantic units of human languages, have been successfully utilized in various natural language processing applications.

Adversarial Attack Language Modelling +2

FewRel 2.0: Towards More Challenging Few-Shot Relation Classification

1 code implementation IJCNLP 2019 Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, Jie zhou

We present FewRel 2. 0, a more challenging task to investigate two aspects of few-shot relation classification models: (1) Can they adapt to a new domain with only a handful of instances?

Classification Domain Adaptation +2

OpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction

1 code implementation IJCNLP 2019 Xu Han, Tianyu Gao, Yuan YAO, Demin Ye, Zhiyuan Liu, Maosong Sun

OpenNRE is an open-source and extensible toolkit that provides a unified framework to implement neural models for relation extraction (RE).

Information Retrieval Question Answering +1

Using BERT for Word Sense Disambiguation

no code implementations18 Sep 2019 Jiaju Du, Fanchao Qi, Maosong Sun

Word Sense Disambiguation (WSD), which aims to identify the correct sense of a given polyseme, is a long-standing problem in NLP.

Word Sense Disambiguation

Improving Back-Translation with Uncertainty-based Confidence Estimation

1 code implementation IJCNLP 2019 Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, Maosong Sun

While back-translation is simple and effective in exploiting abundant monolingual corpora to improve low-resource neural machine translation (NMT), the synthetic bilingual corpora generated by NMT models trained on limited authentic bilingual data are inevitably noisy.

Low-Resource Neural Machine Translation Translation

Neural Snowball for Few-Shot Relation Learning

1 code implementation29 Aug 2019 Tianyu Gao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, Maosong Sun

To address new relations with few-shot instances, we propose a novel bootstrapping approach, Neural Snowball, to learn new relations by transferring semantic knowledge about existing relations.

Knowledge Graphs Relation Extraction

Explore Entity Embedding Effectiveness in Entity Retrieval

no code implementations28 Aug 2019 Zhenghao Liu, Chenyan Xiong, Maosong Sun, Zhiyuan Liu

Entity embedding learns lots of semantic information from the knowledge graph and represents entities with a low-dimensional representation, which provides an opportunity to establish interactions between query related entities and candidate entities for entity retrieval.

Entity Retrieval Learning-To-Rank

GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification

2 code implementations ACL 2019 Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, LiFeng Wang, Changcheng Li, Maosong Sun

Fact verification (FV) is a challenging task which requires to retrieve relevant evidence from plain text and use the evidence to verify given claims.

Fact Verification

Quantifying Similarity between Relations with Fact Distribution

1 code implementation ACL 2019 Weize Chen, Hao Zhu, Xu Han, Zhiyuan Liu, Maosong Sun

We introduce a conceptually simple and effective method to quantify the similarity between relations in knowledge bases.

General Classification Open Information Extraction

Modeling Semantic Compositionality with Sememe Knowledge

1 code implementation ACL 2019 Fanchao Qi, Jun-Jie Huang, Chenghao Yang, Zhiyuan Liu, Xiao Chen, Qun Liu, Maosong Sun

In this paper, we verify the effectiveness of sememes, the minimum semantic units of human languages, in modeling SC by a confirmatory experiment.

multi-word expression embedding multi-word expression sememe prediction

Jiuge: A Human-Machine Collaborative Chinese Classical Poetry Generation System

no code implementations ACL 2019 Guo Zhipeng, Xiaoyuan Yi, Maosong Sun, Wenhao Li, Cheng Yang, Jiannan Liang, Huimin Chen, Yuhui Zhang, Ruoyu Li

By exposing the options of poetry genres, styles and revision modes, Jiuge, acting as a professional assistant, allows constant and active participation of users in poetic creation.

XQA: A Cross-lingual Open-domain Question Answering Dataset

1 code implementation ACL 2019 Jiahua Liu, Yankai Lin, Zhiyuan Liu, Maosong Sun

Experimental results show that the multilingual BERT model achieves the best results in almost all target languages, while the performance of cross-lingual OpenQA is still much lower than that of English.

Machine Translation Open-Domain Question Answering +1

DocRED: A Large-Scale Document-Level Relation Extraction Dataset

4 code implementations ACL 2019 Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zheng-Hao Liu, Zhiyuan Liu, Lixin Huang, Jie zhou, Maosong Sun

Multiple entities in a document generally exhibit complex inter-sentence relations, and cannot be well handled by existing relation extraction (RE) methods that typically focus on extracting intra-sentence relations for single entity pairs.

Relation Extraction

COS960: A Chinese Word Similarity Dataset of 960 Word Pairs

1 code implementation1 Jun 2019 Junjie Huang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Maosong Sun

Word similarity computation is a widely recognized task in the field of lexical semantics.

POS Word Similarity

Adversarial Training for Weakly Supervised Event Detection

1 code implementation NAACL 2019 Xiaozhi Wang, Xu Han, Zhiyuan Liu, Maosong Sun, Peng Li

Modern weakly supervised methods for event detection (ED) avoid time-consuming human annotation and achieve promising results by learning from auto-labeled data.

Event Detection

ERNIE: Enhanced Language Representation with Informative Entities

1 code implementation ACL 2019 Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, Qun Liu

Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks.

Entity Linking Entity Typing +5

OpenHowNet: An Open Sememe-based Lexical Knowledge Base

1 code implementation28 Jan 2019 Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Qiang Dong, Maosong Sun, Zhendong Dong

In this paper, we present an open sememe-based lexical knowledge base OpenHowNet.

Knowledge Representation Learning: A Quantitative Review

2 code implementations28 Dec 2018 Yankai Lin, Xu Han, Ruobing Xie, Zhiyuan Liu, Maosong Sun

Knowledge representation learning (KRL) aims to represent entities and relations in knowledge graph in low-dimensional semantic space, which have been widely used in massive knowledge-driven tasks.

General Classification Information Retrieval +7

Neural Diffusion Model for Microscopic Cascade Prediction

1 code implementation21 Dec 2018 Cheng Yang, Maosong Sun, Haoran Liu, Shiyi Han, Zhiyuan Liu, Huanbo Luan

The strong assumptions oversimplify the complex diffusion mechanism and prevent these models from better fitting real-world cascade data.

Social and Information Networks Physics and Society

Graph Neural Networks: A Review of Methods and Applications

5 code implementations20 Dec 2018 Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, LiFeng Wang, Changcheng Li, Maosong Sun

Lots of learning tasks require dealing with graph data which contains rich relation information among elements.

Graph Attention

Bandit Learning with Implicit Feedback

1 code implementation NeurIPS 2018 Yi Qi, Qingyun Wu, Hongning Wang, Jie Tang, Maosong Sun

Implicit feedback, such as user clicks, although abundant in online information service systems, does not provide substantial evidence on users' evaluation of system's output.

Bayesian Inference

CED: Credible Early Detection of Social Media Rumors

no code implementations10 Nov 2018 Changhe Song, Cunchao Tu, Cheng Yang, Zhiyuan Liu, Maosong Sun

By regarding all reposts to a rumor candidate as a sequence, the proposed model will seek an early point-in-time for making a credible prediction.

Social and Information Networks

Prior Knowledge Integration for Neural Machine Translation using Posterior Regularization

1 code implementation ACL 2017 Jiacheng Zhang, Yang Liu, Huanbo Luan, Jingfang Xu, Maosong Sun

Although neural machine translation has made significant progress recently, how to integrate multiple overlapping, arbitrary prior knowledge sources remains a challenge.

Machine Translation Translation

OpenKE: An Open Toolkit for Knowledge Embedding

1 code implementation EMNLP 2018 Xu Han, Shulin Cao, Xin Lv, Yankai Lin, Zhiyuan Liu, Maosong Sun, Juanzi Li

We release an open toolkit for knowledge embedding (OpenKE), which provides a unified framework and various fundamental models to embed knowledge graphs into a continuous low-dimensional space.

Information Retrieval Knowledge Graphs +2

Language Modeling with Sparse Product of Sememe Experts

1 code implementation EMNLP 2018 Yihong Gu, Jun Yan, Hao Zhu, Zhiyuan Liu, Ruobing Xie, Maosong Sun, Fen Lin, Leyu Lin

Most language modeling methods rely on large-scale data to statistically learn the sequential patterns of words.

Language Modelling

Enhancing Stock Movement Prediction with Adversarial Training

1 code implementation13 Oct 2018 Fuli Feng, Huimin Chen, Xiangnan He, Ji Ding, Maosong Sun, Tat-Seng Chua

The key novelty is that we propose to employ adversarial training to improve the generalization of a neural network prediction model.

Stock Prediction

Overview of CAIL2018: Legal Judgment Prediction Competition

2 code implementations13 Oct 2018 Haoxi Zhong, Chaojun Xiao, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, Jianfeng Xu

In this paper, we give an overview of the Legal Judgment Prediction (LJP) competition at Chinese AI and Law challenge (CAIL2018).

Improving the Transformer Translation Model with Document-Level Context

3 code implementations EMNLP 2018 Jiacheng Zhang, Huanbo Luan, Maosong Sun, FeiFei Zhai, Jingfang Xu, Min Zhang, Yang Liu

Although the Transformer translation model (Vaswani et al., 2017) has achieved state-of-the-art performance in a variety of translation tasks, how to use document-level context to deal with discourse phenomena problematic for Transformer still remains a challenge.


Cross-lingual Lexical Sememe Prediction

1 code implementation EMNLP 2018 Fanchao Qi, Yankai Lin, Maosong Sun, Hao Zhu, Ruobing Xie, Zhiyuan Liu

We propose a novel framework to model correlations between sememes and multi-lingual words in low-dimensional semantic space for sememe prediction.

cross-lingual sememe prediction Learning Word Embeddings +1

A Multi-answer Multi-task Framework for Real-world Machine Reading Comprehension

no code implementations EMNLP 2018 Jiahua Liu, Wan Wei, Maosong Sun, Hao Chen, Yantao Du, Dekang Lin

The task of machine reading comprehension (MRC) has evolved from answering simple questions from well-edited text to answering real questions from users out of web data.

Information Retrieval Machine Reading Comprehension +1

Hierarchical Relation Extraction with Coarse-to-Fine Grained Attention

1 code implementation EMNLP 2018 Xu Han, Pengfei Yu, Zhiyuan Liu, Maosong Sun, Peng Li

In this paper, we aim to incorporate the hierarchical information of relations for distantly supervised relation extraction and propose a novel hierarchical attention scheme.

Knowledge Graphs Relation Extraction

Automatic Poetry Generation with Mutual Reinforcement Learning

no code implementations EMNLP 2018 Xiaoyuan Yi, Maosong Sun, Ruoyu Li, Wenhao Li

Human experts evaluate poetry in terms of some specific criteria, instead of word-level likelihood.