Search Results for author:
Found 29 papers, 13 papers with code
KFCNet: Knowledge Filtering and Contrastive Learning for Generative Commonsense Reasoning
Pre-trained language models have led to substantial gains over a broad range of natural language processing (NLP) tasks, but have been shown to have limitations for natural language generation tasks with high-quality requirements on the output, such as commonsense generation and ad keyword generation.
CULG: Commercial Universal Language Generation
Pre-trained language models (PLMs) have dramatically improved performance for many natural language processing (NLP) tasks in domains such as finance and healthcare.
Enhancing Self-Attention with Knowledge-Assisted Attention Maps
Large-scale pre-trained language models have attracted extensive attentions in the research community and shown promising results on various tasks of natural language processing.
XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation
In this paper, we introduce XGLUE, a new benchmark dataset to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora, and evaluate their performance across a diverse set of cross-lingual tasks.
Healing Unsafe Dialogue Responses with Weak Supervision Signals
Recent years have seen increasing concerns about the unsafe response generation of large-scale dialogue systems, where agents will learn offensive or biased behaviors from the real-world corpus.
MERGE: Fast Private Text Generation
Recent years have seen increasing concerns about the private inference of NLP services and Transformer models.
SwiftPruner: Reinforced Evolutionary Pruning for Efficient Ad Relevance
Remarkably, under our latency requirement of 1900us on CPU, SwiftPruner achieves a 0. 86% higher AUC than the state-of-the-art uniform sparse baseline for BERT-Mini on a large scale real-world dataset.
A Self-Paced Mixed Distillation Method for Non-Autoregressive Generation
To further illustrate the commercial value of our approach, we conduct experiments on three generation tasks in real-world advertisements applications.
Taming Sparsely Activated Transformer with Stochastic Experts
While most on-going research focuses on improving SAMs models by exploring methods of routing inputs to experts, our analysis reveals that such research might not lead to the solution we expect, i. e., the commonly-used routing methods based on gating mechanisms do not work better than randomly routing inputs to experts.
KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning
Pre-trained language models have led to substantial gains over a broad range of natural language processing (NLP) tasks, but have been shown to have limitations for natural language generation tasks with high-quality requirements on the output, such as commonsense generation and ad keyword generation.
FastSeq: Make Sequence Generation Faster
Transformer-based models have made tremendous impacts in natural language generation.
EL-Attention: Memory Efficient Lossless Attention for Generation
Transformer model with multi-head attention requires caching intermediate results for efficient inference in generation tasks.
ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation
ProphetNet is a pre-training based natural language generation method which shows powerful performance on English text summarization and question generation tasks.
Mask Attention Networks: Rethinking and Strengthen Transformer
We therefore introduce a new layer named dynamic mask attention network (DMAN) with a learnable mask matrix which is able to model localness adaptively.
Ranked #11 on
Machine Translation
on WMT2014 English-German
TextGNN: Improving Text Encoder via Graph Neural Network in Sponsored Search
Text encoders based on C-DSSM or transformers have demonstrated strong performance in many Natural Language Processing (NLP) tasks.
BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
In this paper, we propose BANG, a new pretraining model to Bridge the gap between Autoregressive (AR) and Non-autoregressive (NAR) Generation.
An Enhanced Knowledge Injection Model for Commonsense Generation
Commonsense generation aims at generating plausible everyday scenario description based on a set of provided concepts.
GLGE: A New General Language Generation Evaluation Benchmark
Multi-task benchmarks such as GLUE and SuperGLUE have driven great progress of pretraining and transfer learning in Natural Language Processing (NLP).
ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training
This paper presents a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism.
ProphetNet-Ads: A Looking Ahead Strategy for Generative Retrieval Models in Sponsored Search Engine
We build a dataset from a real-word sponsored search engine and carry out experiments to analyze different generative retrieval models.
AutoADR: Automatic Model Design for Ad Relevance
We add the model designed by AutoADR as a sub-model into the production Ad Relevance model.
HittER: Hierarchical Transformers for Knowledge Graph Embeddings
Our proposed model consists of two different Transformer blocks: the bottom block extracts features of each entity-relation pair in the local neighborhood of the source entity and the top block aggregates the relational information from outputs of the bottom block.
Ranked #1 on
Link Prediction
on FB15k-237
(Hit@10 metric)
XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
In this paper, we introduce XGLUE, a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora and evaluate their performance across a diverse set of cross-lingual tasks.
TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
Experimental results showed that the inference time was significantly reduced and was firstly controlled around 20ms on CPUs while at the same time the performance gain from fine-tuned BERT-Base model was mostly retained.
ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
This paper presents a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism.
Ranked #6 on
Question Generation
on SQuAD1.1
(using extra training data)
DeepProbe: Information Directed Sequence Understanding and Chatbot Design via Recurrent Neural Networks
Three applications, namely a rewritter, a relevance scorer and a chatbot for ad recommendation, were built around DeepProbe, with the first two serving as precursory building blocks for the third.
Large-Scale Multi-Label Learning with Incomplete Label Assignments
Unlike prior works, our method can effectively and efficiently consider missing labels and label correlations simultaneously, and is very scalable, that has linear time complexities over the size of the data.
A Lipschitz Exploration-Exploitation Scheme for Bayesian Optimization
The exploration phase aims to select samples that shrink the search space as much as possible.
Learning to Rank by Optimizing NDCG Measure
Learning to rank is a relatively new field of study, aiming to learn a ranking function from a set of training data with relevancy labels.
