Search Results for author:
Found 53 papers, 23 papers with code
MINER: Multi-Interest Matching Network for News Recommendation
Accurately matching user’s interests and candidate news is the key to news recommendation.
Controlled Text Generation Using Dictionary Prior in Variational Autoencoders
While variational autoencoders (VAEs) have been widely applied in text generation tasks, they are troubled by two challenges: insufficient representation capacity and poor controllability.
MTRec: Multi-Task Learning over BERT for News Recommendation
With the adoption of large pre-trained models like BERT in news recommendation, the above way to incorporate multi-field information may encounter challenges: the shallow feature encoding to compress the category and entity information is not compatible with the deep BERT encoding.
Prompt-Based Length Controlled Generation with Reinforcement Learning
Recently, large language models (LLMs) like ChatGPT and GPT-4 have attracted great attention given their surprising improvement and performance.
NewsDialogues: Towards Proactive News Grounded Conversation
In this paper, we propose a novel task, Proactive News Grounded Conversation, in which a dialogue system can proactively lead the conversation based on some key topics of the news.
AutoConv: Automatically Generating Information-seeking Conversations with Large Language Models
Information-seeking conversation, which aims to help users gather information through conversation, has achieved great progress in recent years.
Aligning Large Language Models with Human: A Survey
(2) Training methodologies: a detailed review of the prevailing training methods employed for LLM alignment.
Enhancing Coherence of Extractive Summarization with Multitask Learning
This study proposes a multitask learning architecture for extractive summarization with coherence boosting.
Retrieval-based Disentanglement with Distant Supervision
Disentangled representation learning remains challenging as ground truth factors of variation do not naturally exist.
G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks
Recently, domain-specific PLMs have been proposed to boost the task performance of specific domains (e. g., biomedical and computer science) by continuing to pre-train general PLMs with domain-specific corpora.
Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation
In order to detect such toxic generations, existing methods rely on templates, real-world data extraction, crowdsourcing workers, or automatic generation to construct adversarial contexts that are likely to induce toxic generations.
LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling
Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks.
Pre-training Language Models with Deterministic Factual Knowledge
Further experiments on question-answering datasets show that trying to learn a deterministic relationship with the proposed methods can also help other knowledge-intensive tasks.
Exploring Extreme Parameter Compression for Pre-trained Language Models
A tiny version achieves $96. 7\%$ performance of BERT-base with $ {1}/{48} $ encoder parameters (i. e., less than 2M parameters excluding the embedding layer) and $2. 7 \times$ faster on inference.
How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis
We check the words that have three typical associations with the missing words: knowledge-dependent, positionally close, and highly co-occurred.
PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model
We investigate different aspects of responses generated by PanGu-Bot, including response quality, knowledge, and safety.
Compression of Generative Pre-trained Language Models via Quantization
We find that previous quantization methods fail on generative tasks due to the \textit{homogeneous word embeddings} caused by reduced capacity, and \textit{varied distribution of weights}.
Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answering
To alleviate the data scarcity problem in training question answering systems, recent works propose additional intermediate pre-training for dense passage retrieval (DPR).
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.
Read before Generate! Faithful Long Form Question Answering with Machine Reading
Long-form question answering (LFQA) aims to generate a paragraph-length answer for a given question.
Ranked #1 on
Question Answering
on KILT: ELI5
bert2BERT: Towards Reusable Pretrained Language Models
However, large language model pre-training costs intensive computational resources and most of the models are trained from scratch without reusing the existing pre-trained models, which is wasteful.
Towards Efficient Post-training Quantization of Pre-trained Language Models
Experiments on GLUE and SQuAD benchmarks show that our proposed PTQ solution not only performs close to QAT, but also enjoys significant reductions in training time, memory overhead, and data consumption.
DyLex: Incorporating Dynamic Lexicons into BERT for Sequence Labeling
Incorporating lexical knowledge into deep learning models has been proved to be very effective for sequence labeling tasks.
Improving Unsupervised Question Answering via Summarization-Informed Question Generation
Template-based QG uses linguistically-informed heuristics to transform declarative sentences into interrogatives, whereas supervised QG uses existing Question Answering (QA) datasets to train a system to generate a question given a passage and an answer.
Integrating Regular Expressions with Neural Networks via DFA
Human-designed rules are widely used to build industry applications.
Generate & Rank: A Multi-task Framework for Math Word Problems
Math word problem (MWP) is a challenging and critical task in natural language processing.
Ranked #1 on
Math Word Problem Solving
on Math23K
GhostBERT: Generate More Features with Cheap Operations for BERT
Transformer-based pre-trained language models like BERT, though powerful in many tasks, are expensive in both memory and computation, due to their large number of parameters.
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
Specifically, we carefully design the techniques of one-shot learning and the search space to provide an adaptive and efficient development way of tiny PLMs for various latency constraints.
A Mutual Information Maximization Approach for the Spurious Solution Problem in Weakly Supervised Question Answering
This setting gives rise to the spurious solution problem: there may exist many spurious solutions that coincidentally derive the correct answer, but training on such solutions can hurt model performance (e. g., producing wrong solutions or answers).
Improved OOD Generalization via Adversarial Training and Pre-training
In this paper, after defining OOD generalization via Wasserstein distance, we theoretically show that a model robust to input perturbation generalizes well on OOD data.
Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation
Task-agnostic knowledge distillation, a teacher-student framework, has been proved effective for BERT compression.
Reweighting Augmented Samples by Minimizing the Maximal Expected Loss
Inspired by adversarial training, we minimize this maximal expected loss (MMEL) and obtain a simple and interpretable closed-form solution: more attention should be paid to augmented samples with large loss values (i. e., harder examples).
LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
The multilingual pre-trained language models (e. g, mBERT, XLM and XLM-R) have shown impressive performance on cross-lingual natural language understanding tasks.
Non-invasive Self-attention for Side Information Fusion in Sequential Recommendation
It is still an open question to leverage various types of information under the BERT framework.
On Position Embeddings in BERT
Various Position Embeddings (PEs) have been proposed in Transformer based architectures~(e. g. BERT) to model word order.
HopRetriever: Retrieve Hops over Wikipedia to Answer Complex Questions
In this paper, we propose a new retrieval target, hop, to collect the hidden reasoning evidence from Wikipedia for complex question answering.
Ranked #8 on
Question Answering
on HotpotQA
BinaryBERT: Pushing the Limit of BERT Quantization
In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization.
Improving Task-Agnostic BERT Distillation with Layer Mapping Search
Comprehensive experiments on the evaluation benchmarks demonstrate that 1) layer mapping strategy has a significant effect on task-agnostic BERT distillation and different layer mappings can result in quite different performances; 2) the optimal layer mapping strategy from the proposed search process consistently outperforms the other heuristic ones; 3) with the optimal layer mapping, our student model achieves state-of-the-art performance on the GLUE tasks.
SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval
Term-based sparse representations dominate the first-stage text retrieval in industrial applications, due to its advantage in efficiency, interpretability, and exact term matching.
TernaryBERT: Distillation-aware Ultra-low Bit BERT
Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks. However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices.
Enriching Large-Scale Eventuality Knowledge Graph with Entailment Relations
Computational and cognitive studies suggest that the abstraction of eventualities (activities, states, and events) is crucial for humans to understand daily eventualities.
DynaBERT: Dynamic BERT with Adaptive Width and Depth
The pre-trained language models like BERT, though powerful in many natural language processing tasks, are both computation and memory expensive.
Neural Subgraph Isomorphism Counting
In this paper, we study a new graph learning problem: learning to count subgraph isomorphisms.
TinyBERT: Distilling BERT for Natural Language Understanding
To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models.
Ranked #1 on
Natural Language Inference
on MultiNLI Dev
Dialog State Tracking with Reinforced Data Augmentation
Neural dialog state trackers are generally limited due to the lack of quantity and diversity of annotated training data.
Decomposable Neural Paraphrase Generation
Paraphrasing exists at different granularity levels, such as lexical level, phrasal level and sentential level.
An Investigation of Few-Shot Learning in Spoken Term Classification
In this paper, we investigate the feasibility of applying few-shot learning algorithms to a speech task.
Paraphrase Generation with Deep Reinforcement Learning
The generator, built as a sequence-to-sequence learning model, can produce paraphrases given a sentence.
Neural Machine Translation with Reconstruction
Although end-to-end Neural Machine Translation (NMT) has achieved remarkable progress in the past two years, it suffers from a major drawback: translations generated by NMT systems often lack of adequacy.
Neural Generative Question Answering
Empirical study shows the proposed model can effectively deal with the variations of questions and answers, and generate right and natural answers by referring to the facts in the knowledge-base.
Multimodal Convolutional Neural Networks for Matching Image and Sentence
In this paper, we propose multimodal convolutional neural networks (m-CNNs) for matching image and sentence.
Ranked #15 on
Image Retrieval
on Flickr30K 1K test
Neural Responding Machine for Short-Text Conversation
We propose Neural Responding Machine (NRM), a neural network-based response generator for Short-Text Conversation.
On Approximate Inference for Generalized Gaussian Process Models
In this paper, we consider efficient algorithms for approximate inference on GGPMs using the general form of the EFD.
