Search Results for author:
Found 60 papers, 32 papers with code
Rethinking Denoised Auto-Encoding in Language Pre-Training
Pre-trained self-supervised models such as BERT have achieved striking success in learning sequence representations, especially for natural language processing.
Qwen2 Technical Report
This report introduces the Qwen2 series, the latest addition to our large language models and large multimodal models.
Qwen Technical Report
Large language models (LLMs) have revolutionized the field of artificial intelligence, enabling natural language processing tasks that were previously thought to be exclusive to humans.
Ranked #3 on
Multi-Label Text Classification
on CC3M-TagMask
Transferring General Multimodal Pretrained Models to Text Recognition
This paper proposes a new method, OFA-OCR, to transfer multimodal pretrained models to text recognition.
OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models
As a starting point, we provide presets of 7 different modalities and 23 highly-diverse example tasks in OFASys, with which we also develop a first-in-kind, single model, OFA+, that can handle text, image, speech, video, and motion data.
DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention
To enhance the correlation between vision and language in disentangled spaces, we introduce the visual concepts to DiMBERT which represent visual information in textual format.
Prophet Attention: Predicting Attention with Future Attention for Image Captioning
Especially for image captioning, the attention based models are expected to ground correct image regions with proper generated words.
From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models
We then design a Model Uncertainty--aware Knowledge Integration (MUKI) framework to recover the golden supervision for the student.
Delving into the Openness of CLIP
However, evaluating the openness of CLIP-like models is challenging, as the models are open to arbitrary vocabulary in theory, but their accuracy varies in practice.
Hierarchical Inductive Transfer for Continual Dialogue Learning
However, for the continual increase of online chit-chat scenarios, directly fine-tuning these models for each of the new tasks not only explodes the capacity of the dialogue system on the embedded devices but also causes knowledge forgetting on pre-trained models and knowledge interference among diverse dialogue tasks.
Model Uncertainty-Aware Knowledge Amalgamation for Pre-Trained Language Models
As many fine-tuned pre-trained language models~(PLMs) with promising performance are generously released, investigating better ways to reuse these models is vital as it can greatly reduce the retraining computational cost and the potential environmental side-effects.
Well-classified Examples are Underestimated in Classification with Deep Neural Networks
The conventional wisdom behind learning deep classification models is to focus on bad-classified examples and ignore well-classified examples that are far from the decision boundary.
Topology-Imbalance Learning for Semi-Supervised Node Classification
The class imbalance problem, as an important issue in learning node representations, has drawn increasing attention from the community.
Adversarial Parameter Defense by Multi-Step Risk Minimization
To enhance neural networks, we propose the adversarial parameter defense algorithm that minimizes the average risk of multiple adversarial parameter corruptions.
O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning
Video captioning combines video understanding and language generation.
Neural Network Surgery: Injecting Data Patterns into Pre-trained Models with Minimal Instance-wise Side Effects
Motivated by neuroscientific evidence and theoretical results, we demonstrate that side effects can be controlled by the number of changed parameters and thus, we propose to conduct \textit{neural network surgery} by only modifying a limited number of parameters.
A Global Past-Future Early Exit Method for Accelerating Inference of Pre-trained Language Models
We first take into consideration all the linguistic information embedded in the past layers and then take a further step to engage the future information which is originally inaccessible for predictions.
Rethinking Skip Connection with Layer Normalization in Transformers and ResNets
In this work, we investigate how the scale factors in the effectiveness of the skip connection and reveal that a trivial adjustment of the scale will lead to spurious gradient exploding or vanishing in line with the deepness of the models, which could be addressed by normalization, in particular, layer normalization, which induces consistent improvements over the plain skip connection.
Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models
However, in this paper, we find that it is possible to hack the model in a data-free way by modifying one single word embedding vector, with almost no accuracy sacrificed on clean samples.
Multi-View Feature Representation for Dialogue Generation with Bidirectional Distillation
The finding of general knowledge is further hindered by the unidirectional distillation, as the student should obey the teacher and may discard some knowledge that is truly general but refuted by the teacher.
High-Likelihood Area Matters --- Rewarding Near-Correct Predictions Under Imbalanced Distributions
We find in practice that the high-likelihood area contains correct predictions for tail classes and it plays a vital role in learning imbalanced class distributions.
Rethinking the Promotion Brought by Contrastive Learning to Semi-Supervised Node Classification
Graph Contrastive Learning (GCL) has proven highly effective in promoting the performance of Semi-Supervised Node Classification (SSNC).
Rethinking Skip Connection with Layer Normalization
In this work, we investigate how the scale factors in the effectiveness of the skip connection and reveal that a trivial adjustment of the scale will lead to spurious gradient exploding or vanishing in line with the deepness of the models, which could by addressed by normalization, in particular, layer normalization, which induces consistent improvements over the plain skip connection.
Prophet Attention: Predicting Attention with Future Attention
Especially for image captioning, the attention based models are expected to ground correct image regions with proper generated words.
CAPT: Contrastive Pre-Training for Learning Denoised Sequence Representations
Pre-trained self-supervised models such as BERT have achieved striking success in learning sequence representations, especially for natural language processing.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Human dialogues are scenario-based and appropriate responses generally relate to the latent context knowledge entailed by the specific scenario.
Collaborative Group Learning
Collaborative learning has successfully applied knowledge transfer to guide a pool of small student networks towards robust local minima.
Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption
We argue that the vulnerability of model parameters is of crucial value to the study of model robustness and generalization but little research has been devoted to understanding this matter.
Rethinking and Improving Natural Language Generation with Layer-Wise Multi-View Decoding
While it is common practice to draw information from only the last encoder layer, recent work has proposed to use representations from different encoder layers for diversified levels of information.
Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection
Self-attention based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks.
An Adaptive and Momental Bound Method for Stochastic Learning
The dynamic learning rate bounds are based on the exponential moving averages of the adaptive learning rates themselves, which smooth out unexpected large learning rates and stabilize the training of deep neural networks.
Sparse Transformer: Concentrated Attention Through Explicit Selection
Extensive experimental results on a series of natural language processing tasks, including neural machine translation, image captioning, and language modeling, all demonstrate the advantages of Sparse Transformer in model performance.
PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation
Through this method, we generate synthetic data using a large amount of unlabeled data in the target domain and then obtain a word segmentation model for the target domain.
A Hierarchical Reinforced Sequence Operation Method for Unsupervised Text Style Transfer
Unsupervised text style transfer aims to alter text styles while preserving the content, without aligned data for supervision.
Memorized Sparse Backpropagation
Neural network learning is usually time-consuming since backpropagation needs to compute full gradients and backpropagate them across multiple layers.
Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations
In vision-and-language grounding problems, fine-grained representations of the image are considered to be of paramount importance.
Diversity-Promoting GAN: A Cross-Entropy Based Generative Adversarial Network for Diversified Text Generation
Existing text generation methods tend to produce repeated and {''}boring{''} expressions.
Evaluating Semantic Rationality of a Sentence: A Sememe-Word-Matching Neural Network based on HowNet
To evaluate the effectiveness of the proposed model, we build a large-scale rationality evaluation dataset.
simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions
The encode-decoder framework has shown recent success in image captioning.
Review-Driven Multi-Label Music Style Classification by Exploiting Style Correlations
This task requires the system to identify multiple styles of music based on its reviews on websites.
Learning When to Concentrate or Divert Attention: Self-Adaptive Attention Temperature for Neural Machine Translation
Most of the Neural Machine Translation (NMT) models are based on the sequence-to-sequence (Seq2Seq) model with an encoder-decoder framework equipped with the attention mechanism.
Ranked #7 on
Machine Translation
on IWSLT2015 English-Vietnamese
A Skeleton-Based Model for Promoting Coherence Among Sentences in Narrative Story Generation
Compared to the state-of-the-art models, our skeleton-based model can generate significantly more coherent text according to human evaluation and automatic evaluation.
Sememe Prediction: Learning Semantic Knowledge from Unstructured Textual Wiki Descriptions
In the experiments, we take a real-world sememe knowledge base HowNet and the corresponding descriptions of the words in Baidu Wiki for training and evaluation.
Deconvolution-Based Global Decoding for Neural Machine Translation
A great proportion of sequence-to-sequence (Seq2Seq) models for Neural Machine Translation (NMT) adopt Recurrent Neural Network (RNN) to generate translation word by word following a sequential order.
Ranked #9 on
Machine Translation
on IWSLT2015 English-Vietnamese
Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach
We evaluate our approach on two review datasets, Yelp and Amazon.
Ranked #6 on
Unsupervised Text Style Transfer
on Yelp
Regularizing Output Distribution of Abstractive Chinese Social Media Text Summarization for Improved Semantic Consistency
Especially, the proposed approach improves the semantic consistency by 4\% in terms of human evaluation.
A Hierarchical End-to-End Model for Jointly Improving Text Summarization and Sentiment Classification
Text summarization and sentiment classification both aim to capture the main ideas of the text but at different levels.
Structure Regularized Neural Network for Entity Relation Classification for Chinese Literature Text
In this paper, we propose the task of relation classification for Chinese literature text.
Query and Output: Generating Words by Querying Distributed Word Representations for Paraphrase Generation
The existing sequence-to-sequence model tends to memorize the words and the patterns in the training dataset instead of learning the meaning of the words.
DP-GAN: Diversity-Promoting Generative Adversarial Network for Generating Informative and Diversified Text
Existing text generation methods tend to produce repeated and "boring" expressions.
Building an Ellipsis-aware Chinese Dependency Treebank for Web Text
Web 2. 0 has brought with it numerous user-produced data revealing one's thoughts, experiences, and knowledge, which are a great source for many tasks, such as information extraction, and knowledge base construction.
Hybrid Oracle: Making Use of Ambiguity in Transition-based Chinese Dependency Parsing
In the training of transition-based dependency parsers, an oracle is used to predict a transition sequence for a sentence and its gold tree.
Complex Structure Leads to Overfitting: A Structure Regularization Decoding Method for Natural Language Processing
The decoding of the complex structure model is regularized by the additionally trained simple structure model.
Does Higher Order LSTM Have Better Accuracy for Segmenting and Labeling Sequence Data?
In our work, we first design a new model called "high order LSTM" to predict multiple tags for the current token which contains not only the current tag but also the previous several tags.
Training Simplification and Model Simplification for Deep Learning: A Minimal Effort Back Propagation Method
Based on the sparsified gradients, we further simplify the model by eliminating the rows or columns that are seldom updated, which will reduce the computational cost both in the training and decoding, and potentially accelerate decoding in real-world applications.
Label Embedding Network: Learning Label Representation for Soft Training of Deep Networks
We propose a method, called Label Embedding Network, which can learn label representation (label embedding) during the training process of deep networks.
Minimal Effort Back Propagation for Convolutional Neural Networks
As traditional neural network consumes a significant amount of computing resources during back propagation, \citet{Sun2017mePropSB} propose a simple yet effective technique to alleviate this problem.
meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
In back propagation, only a small subset of the full gradient is computed to update the model parameters.
Towards Easier and Faster Sequence Labeling for Natural Language Processing: A Search-based Probabilistic Online Learning Framework (SAPO)
We show that this method with fast training and theoretical guarantee of convergence, which is easy to implement, can support search-based optimization and obtain top accuracy.
