Search Results for author:
Found 41 papers, 29 papers with code
BMInf: An Efficient Toolkit for Big Model Inference and Tuning
In recent years, large-scale pre-trained language models (PLMs) containing billions of parameters have achieved promising results on various NLP tasks.
Robust and Scalable Model Editing for Large Language Models
Large language models (LLMs) can make predictions using parametric knowledge--knowledge encoded in the model weights--or contextual knowledge--knowledge presented in the context.
ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models
Some recent efforts have explored introducing ReLU or its variants as the substitutive activation function to help LLMs achieve activation sparsity and inference acceleration, but few can simultaneously obtain high sparsity and comparable model performance.
InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory
To alleviate these issues, existing efforts employ sliding attention windows and discard distant tokens to achieve the processing of extremely long sequences.
ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs
To find the most efficient activation function for sparse computation, we propose a systematic framework to examine the sparsity of LLMs from three aspects: the trade-off between sparsity and performance, the predictivity of sparsity, and the hardware affinity.
Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules
Pre-trained language models (PLMs) have achieved remarkable results on NLP tasks but at the expense of huge parameter sizes and the consequent computational costs.
CPET: Effective Parameter-Efficient Tuning for Compressed Large Language Models
Parameter-efficient tuning (PET) has been widely explored in recent years because it tunes much fewer parameters (PET modules) than full-parameter fine-tuning (FT) while still stimulating sufficient knowledge from large language models (LLMs) for downstream tasks.
Plug-and-Play Document Modules for Pre-trained Models
By inserting document plugins into the backbone PTM for downstream tasks, we can encode a document one time to handle multiple tasks, which is more efficient than conventional encoding-task coupling methods that simultaneously encode documents and input queries using task-specific encoders.
Plug-and-Play Knowledge Injection for Pre-trained Language Models
Experimental results on three knowledge-driven NLP tasks show that existing injection methods are not suitable for the new paradigm, while map-tuning effectively improves the performance of downstream models.
Emergent Modularity in Pre-trained Transformers
In analogy to human brains, we consider two main characteristics of modularity: (1) functional specialization of neurons: we evaluate whether each neuron is mainly specialized in a certain function, and find that the answer is yes.
UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models
In this paper, we propose a UNified knowledge inTERface, UNTER, to provide a unified perspective to exploit both structured knowledge and unstructured knowledge.
Bridging the Language Gap: Knowledge Injected Multilingual Question Answering
As a popular topic in natural language processing tasks, extractive question answering task (extractive QA) has gained extensive attention in the past few years.
READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises
In order to fill this important gap, we construct READIN: a Chinese multi-task benchmark with REalistic And Diverse Input Noises.
Finding Skill Neurons in Pre-trained Transformer-based Language Models
Furthermore, we demonstrate the skill neurons are most likely generated in pre-training rather than fine-tuning by showing that the skill neurons found with prompt tuning are also crucial for other fine-tuning methods freezing neuron weights, such as the adapter-based tuning and BitFit.
Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models
Prompting, which casts downstream applications as language modeling tasks, has shown to be sample efficient compared to standard fine-tuning with pre-trained models.
Effective Few-Shot Named Entity Linking by Meta-Learning
In this paper, we endeavor to solve the problem of few-shot entity linking, which only requires a minimal amount of in-domain labeled data and is more practical in real situations.
A Unified Understanding of Deep NLP Models for Text Classification
The rapid development of deep natural language processing (NLP) models for text classification has led to an urgent need for a unified understanding of these models proposed individually.
Prompt Tuning for Discriminative Pre-trained Language Models
Recent works have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing (NLP) tasks.
A Roadmap for Big Model
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm.
CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark
We argue that for general-purpose language intelligence evaluation, the benchmark itself needs to be comprehensive and systematic.
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
In this work, we study the computational patterns of FFNs and observe that most inputs only activate a tiny ratio of neurons of FFNs.
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
Pre-Trained Vision-Language Models (VL-PTMs) have shown promising capabilities in grounding natural language in image data, facilitating a broad variety of cross-modal tasks.
CPM-2: Large-scale Cost-effective Pre-trained Language Models
We present a suite of cost-effective techniques for the use of PLMs to deal with the efficiency issues of pre-training, fine-tuning, and inference.
Pre-Trained Models: Past, Present and Future
Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI).
Sub-Character Tokenization for Chinese Pretrained Language Models
2) Pronunciation-based SubChar tokenizers can encode Chinese homophones into the same transliteration sequences and produce the same tokenization output, hence being robust to homophone typos.
Knowledge Inheritance for Pre-trained Language Models
Specifically, we introduce a pre-training framework named "knowledge inheritance" (KI) and explore how could knowledge distillation serve as auxiliary supervision during pre-training to efficiently learn larger PLMs.
Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger
As far as we know, almost all existing textual backdoor attack methods insert additional contents into normal samples as triggers, which causes the trigger-embedded samples to be detected and the backdoor attacks to be blocked without much effort.
CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
We then perform contrastive semi-supervised learning on both the retrieved unlabeled and original labeled instances to help PLMs capture crucial task-related semantic features.
Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks
In this work, we demonstrate the universal vulnerability of PTMs, where fine-tuned PTMs can be easily controlled by backdoor attacks in arbitrary downstream tasks.
Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning
In this work, we propose a simple and effective method to cover a much larger proportion of the attack search space, called Adversarial and Mixup Data Augmentation (AMDA).
CPM: A Large-scale Generative Chinese Pre-trained Language Model
However, applying GPT-3 to address Chinese NLP tasks is still challenging, as the training corpus of GPT-3 is primarily English, and the parameters are not publicly available.
Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads
To measure the informativeness of attention heads, we train our Single-Shot Meta-Pruner (SMP) with a meta-learning paradigm aiming to maintain the distribution of text representations after pruning.
CokeBERT: Contextual Knowledge Selection and Embedding towards Enhanced Pre-Trained Language Models
In this paper, we propose a novel framework named Coke to dynamically select contextual knowledge and embed knowledge context according to textual context for PLMs, which can avoid the effect of redundant and ambiguous knowledge in KGs that cannot match the input text.
Train No Evil: Selective Masking for Task-Guided Pre-Training
In this stage, the model is trained by masked language modeling on in-domain unsupervised data to learn domain-specific patterns and we propose a novel selective masking strategy to learn task-specific patterns.
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text.
Adversarial Language Games for Advanced Natural Language Intelligence
In this work, we propose a challenging adversarial language game called Adversarial Taboo as an example, in which an attacker and a defender compete around a target word.
ERNIE: Enhanced Language Representation with Informative Entities
Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks.
Ranked #1 on
Entity Linking
on FIGER
COSINE: Compressive Network Embedding on Large-scale Information Networks
There is recently a surge in approaches that learn low-dimensional embeddings of nodes in networks.
Graph Neural Networks: A Review of Methods and Applications
Lots of learning tasks require dealing with graph data which contains rich relation information among elements.
A Unified Framework for Community Detection and Network Representation Learning
Network representation learning (NRL) aims to learn low-dimensional vectors for vertices in a network.
Social and Information Networks Physics and Society
