Search Results for author:
Found 68 papers, 34 papers with code
TinyBERT: Distilling BERT for Natural Language Understanding
To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models.
Ranked #1 on
Natural Language Inference
on MultiNLI Dev
DynaBERT: Dynamic BERT with Adaptive Width and Depth
The pre-trained language models like BERT, though powerful in many natural language processing tasks, are both computation and memory expensive.
TernaryBERT: Distillation-aware Ultra-low Bit BERT
Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks. However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices.
BinaryBERT: Pushing the Limit of BERT Quantization
In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization.
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
Specifically, we carefully design the techniques of one-shot learning and the search space to provide an adaptive and efficient development way of tiny PLMs for various latency constraints.
PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model
We investigate different aspects of responses generated by PanGu-Bot, including response quality, knowledge, and safety.
Neural Responding Machine for Short-Text Conversation
We propose Neural Responding Machine (NRM), a neural network-based response generator for Short-Text Conversation.
DyLex: Incorporating Dynamic Lexicons into BERT for Sequence Labeling
Incorporating lexical knowledge into deep learning models has been proved to be very effective for sequence labeling tasks.
Aligning Large Language Models with Human: A Survey
(2) Training methodologies: a detailed review of the prevailing training methods employed for LLM alignment.
Multimodal Convolutional Neural Networks for Matching Image and Sentence
In this paper, we propose multimodal convolutional neural networks (m-CNNs) for matching image and sentence.
Ranked #16 on
Image Retrieval
on Flickr30K 1K test
Data Management For Large Language Models: A Survey
Data plays a fundamental role in the training of Large Language Models (LLMs).
Neural Machine Translation with Reconstruction
Although end-to-end Neural Machine Translation (NMT) has achieved remarkable progress in the past two years, it suffers from a major drawback: translations generated by NMT systems often lack of adequacy.
Neural Generative Question Answering
Empirical study shows the proposed model can effectively deal with the variations of questions and answers, and generate right and natural answers by referring to the facts in the knowledge-base.
Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answering
To alleviate the data scarcity problem in training question answering systems, recent works propose additional intermediate pre-training for dense passage retrieval (DPR).
Neural Subgraph Isomorphism Counting
In this paper, we study a new graph learning problem: learning to count subgraph isomorphisms.
FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models
To fill this research gap, in this paper, we propose FollowBench, a Multi-level Fine-grained Constraints Following Benchmark for LLMs.
MINER: Multi-Interest Matching Network for News Recommendation
Accurately matching user’s interests and candidate news is the key to news recommendation.
Exploring Extreme Parameter Compression for Pre-trained Language Models
A tiny version achieves $96. 7\%$ performance of BERT-base with $ {1}/{48} $ encoder parameters (i. e., less than 2M parameters excluding the embedding layer) and $2. 7 \times$ faster on inference.
M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models
In this paper, we propose M4LE, a Multi-ability, Multi-range, Multi-task, Multi-domain benchmark for Long-context Evaluation.
Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios
The recent trend of using Large Language Models (LLMs) as tool agents in real-world applications underscores the necessity for comprehensive evaluations of their capabilities, particularly in complex scenarios involving planning, creating, and using tools.
Learning to Edit: Aligning LLMs with Knowledge Editing
Knowledge editing techniques, aiming to efficiently modify a minor proportion of knowledge in large language models (LLMs) without negatively impacting performance across other inputs, have garnered widespread attention.
G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks
Recently, domain-specific PLMs have been proposed to boost the task performance of specific domains (e. g., biomedical and computer science) by continuing to pre-train general PLMs with domain-specific corpora.
An Investigation of Few-Shot Learning in Spoken Term Classification
In this paper, we investigate the feasibility of applying few-shot learning algorithms to a speech task.
Enriching Large-Scale Eventuality Knowledge Graph with Entailment Relations
Computational and cognitive studies suggest that the abstraction of eventualities (activities, states, and events) is crucial for humans to understand daily eventualities.
MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models
Large language models (LLMs) are increasingly relied upon for complex multi-turn conversations across diverse real-world applications.
Preparing Lessons for Progressive Training on Language Models
The rapid progress of Transformers in artificial intelligence has come at the cost of increased resource consumption and greenhouse gas emissions due to growing model sizes.
Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation
In order to detect such toxic generations, existing methods rely on templates, real-world data extraction, crowdsourcing workers, or automatic generation to construct adversarial contexts that are likely to induce toxic generations.
PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models
LLMs are prompted to generate extensive content in response to these meta-questions.
mCLIP: Multilingual CLIP via Cross-lingual Transfer
Furthermore, to enhance the token- and sentence-level multilingual representation of the MTE, we propose to train it with machine translation and contrastive learning jointly before the TriKD to provide a better initialization.
NewsDialogues: Towards Proactive News Grounded Conversation
In this paper, we propose a novel task, Proactive News Grounded Conversation, in which a dialogue system can proactively lead the conversation based on some key topics of the news.
A Mutual Information Maximization Approach for the Spurious Solution Problem in Weakly Supervised Question Answering
This setting gives rise to the spurious solution problem: there may exist many spurious solutions that coincidentally derive the correct answer, but training on such solutions can hurt model performance (e. g., producing wrong solutions or answers).
Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignment
Inspired by previous work which identified that feed-forward networks (FFNs) within Transformers are responsible for factual knowledge expressions, we investigate two methods to efficiently improve the factual expression capability {of FFNs} by knowledge enhancement and alignment respectively.
Reweighting Augmented Samples by Minimizing the Maximal Expected Loss
Inspired by adversarial training, we minimize this maximal expected loss (MMEL) and obtain a simple and interpretable closed-form solution: more attention should be paid to augmented samples with large loss values (i. e., harder examples).
MTRec: Multi-Task Learning over BERT for News Recommendation
With the adoption of large pre-trained models like BERT in news recommendation, the above way to incorporate multi-field information may encounter challenges: the shallow feature encoding to compress the category and entity information is not compatible with the deep BERT encoding.
Paraphrase Generation with Deep Reinforcement Learning
The generator, built as a sequence-to-sequence learning model, can produce paraphrases given a sentence.
On Approximate Inference for Generalized Gaussian Process Models
In this paper, we consider efficient algorithms for approximate inference on GGPMs using the general form of the EFD.
Decomposable Neural Paraphrase Generation
Paraphrasing exists at different granularity levels, such as lexical level, phrasal level and sentential level.
Dialog State Tracking with Reinforced Data Augmentation
Neural dialog state trackers are generally limited due to the lack of quantity and diversity of annotated training data.
SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval
Term-based sparse representations dominate the first-stage text retrieval in industrial applications, due to its advantage in efficiency, interpretability, and exact term matching.
On Position Embeddings in BERT
Various Position Embeddings (PEs) have been proposed in Transformer based architectures~(e. g. BERT) to model word order.
Improving Task-Agnostic BERT Distillation with Layer Mapping Search
Comprehensive experiments on the evaluation benchmarks demonstrate that 1) layer mapping strategy has a significant effect on task-agnostic BERT distillation and different layer mappings can result in quite different performances; 2) the optimal layer mapping strategy from the proposed search process consistently outperforms the other heuristic ones; 3) with the optimal layer mapping, our student model achieves state-of-the-art performance on the GLUE tasks.
HopRetriever: Retrieve Hops over Wikipedia to Answer Complex Questions
In this paper, we propose a new retrieval target, hop, to collect the hidden reasoning evidence from Wikipedia for complex question answering.
Ranked #6 on
Question Answering
on HotpotQA
Non-invasive Self-attention for Side Information Fusion in Sequential Recommendation
It is still an open question to leverage various types of information under the BERT framework.
LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
The multilingual pre-trained language models (e. g, mBERT, XLM and XLM-R) have shown impressive performance on cross-lingual natural language understanding tasks.
Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation
Task-agnostic knowledge distillation, a teacher-student framework, has been proved effective for BERT compression.
Improved OOD Generalization via Adversarial Training and Pre-training
In this paper, after defining OOD generalization via Wasserstein distance, we theoretically show that a model robust to input perturbation generalizes well on OOD data.
Generate & Rank: A Multi-task Framework for Math Word Problems
Math word problem (MWP) is a challenging and critical task in natural language processing.
Ranked #2 on
Math Word Problem Solving
on Math23K
Integrating Regular Expressions with Neural Networks via DFA
Human-designed rules are widely used to build industry applications.
GhostBERT: Generate More Features with Cheap Operations for BERT
Transformer-based pre-trained language models like BERT, though powerful in many tasks, are expensive in both memory and computation, due to their large number of parameters.
Improving Unsupervised Question Answering via Summarization-Informed Question Generation
Template-based QG uses linguistically-informed heuristics to transform declarative sentences into interrogatives, whereas supervised QG uses existing Question Answering (QA) datasets to train a system to generate a question given a passage and an answer.
Towards Efficient Post-training Quantization of Pre-trained Language Models
Experiments on GLUE and SQuAD benchmarks show that our proposed PTQ solution not only performs close to QAT, but also enjoys significant reductions in training time, memory overhead, and data consumption.
bert2BERT: Towards Reusable Pretrained Language Models
However, large language model pre-training costs intensive computational resources and most of the models are trained from scratch without reusing the existing pre-trained models, which is wasteful.
Read before Generate! Faithful Long Form Question Answering with Machine Reading
Long-form question answering (LFQA) aims to generate a paragraph-length answer for a given question.
Ranked #1 on
Question Answering
on KILT: ELI5
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.
Compression of Generative Pre-trained Language Models via Quantization
We find that previous quantization methods fail on generative tasks due to the \textit{homogeneous word embeddings} caused by reduced capacity, and \textit{varied distribution of weights}.
How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis
We check the words that have three typical associations with the missing words: knowledge-dependent, positionally close, and highly co-occurred.
Controlled Text Generation Using Dictionary Prior in Variational Autoencoders
While variational autoencoders (VAEs) have been widely applied in text generation tasks, they are troubled by two challenges: insufficient representation capacity and poor controllability.
Pre-training Language Models with Deterministic Factual Knowledge
Further experiments on question-answering datasets show that trying to learn a deterministic relationship with the proposed methods can also help other knowledge-intensive tasks.
LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling
Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks.
Retrieval-based Disentangled Representation Learning with Natural Language Supervision
In light of this, we present Vocabulary Disentangled Retrieval (VDR), a retrieval-based framework that harnesses natural language as proxies of the underlying data variation to drive disentangled representation learning.
Enhancing Coherence of Extractive Summarization with Multitask Learning
This study proposes a multitask learning architecture for extractive summarization with coherence boosting.
AutoConv: Automatically Generating Information-seeking Conversations with Large Language Models
Information-seeking conversation, which aims to help users gather information through conversation, has achieved great progress in recent years.
Prompt-Based Length Controlled Generation with Reinforcement Learning
Large language models (LLMs) like ChatGPT and GPT-4 have attracted great attention given their surprising performance on a wide range of NLP tasks.
SELF: Self-Evolution with Language Feedback
SELF initiates with a meta-skill learning process that equips the LLMs with capabilities for self-feedback and self-refinement.
Exploring the Usage of Chinese Pinyin in Pretraining
Through delicate pretraining tasks, the characters and pinyin representation are fused, which can enhance the error tolerance for SSP errors.
Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis
The rapid development of large language models (LLMs) has not only provided numerous opportunities but also presented significant challenges.
YODA: Teacher-Student Progressive Learning for Language Models
With the teacher's guidance, the student learns to iteratively refine its answer with feedback, and forms a robust and comprehensive understanding of the posed questions.
Does the Generator Mind its Contexts? An Analysis of Generative Model Faithfulness under Context Transfer
The present study introduces the knowledge-augmented generator, which is specifically designed to produce information that remains grounded in contextual knowledge, regardless of alterations in the context.
