Search Results for author:
Found 20 papers, 7 papers with code
CTRL: Connect Tabular and Language Model for CTR Prediction
Both tabular data and converted textual data are regarded as two different modalities and are separately fed into the collaborative CTR model and pre-trained language model.
Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding
While various vision-language pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far.
Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
To this end, we formulate a novel quantization training paradigm to compress the embeddings from the training stage, termed low-precision training (LPT).
LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling
Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks.
Compression of Generative Pre-trained Language Models via Quantization
We find that previous quantization methods fail on generative tasks due to the \textit{homogeneous word embeddings} caused by reduced capacity, and \textit{varied distribution of weights}.
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.
Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark
Experiments show that Wukong can serve as a promising Chinese pre-training dataset and benchmark for different cross-modal learning methods.
Ranked #6 on
Zero-shot Image Retrieval
on COCO-CN
FILIP: Fine-grained Interactive Language-Image Pre-Training
In this paper, we introduce a large-scale Fine-grained Interactive Language-Image Pre-training (FILIP) to achieve finer-level alignment through a cross-modal late interaction mechanism, which uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective.
Towards Efficient Post-training Quantization of Pre-trained Language Models
Experiments on GLUE and SQuAD benchmarks show that our proposed PTQ solution not only performs close to QAT, but also enjoys significant reductions in training time, memory overhead, and data consumption.
GhostBERT: Generate More Features with Cheap Operations for BERT
Transformer-based pre-trained language models like BERT, though powerful in many tasks, are expensive in both memory and computation, due to their large number of parameters.
Improved OOD Generalization via Adversarial Training and Pre-training
In this paper, after defining OOD generalization via Wasserstein distance, we theoretically show that a model robust to input perturbation generalizes well on OOD data.
Reweighting Augmented Samples by Minimizing the Maximal Expected Loss
Inspired by adversarial training, we minimize this maximal expected loss (MMEL) and obtain a simple and interpretable closed-form solution: more attention should be paid to augmented samples with large loss values (i. e., harder examples).
BinaryBERT: Pushing the Limit of BERT Quantization
In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization.
TernaryBERT: Distillation-aware Ultra-low Bit BERT
Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks. However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices.
DynaBERT: Dynamic BERT with Adaptive Width and Depth
The pre-trained language models like BERT, though powerful in many natural language processing tasks, are both computation and memory expensive.
Normalization Helps Training of Quantized LSTM
The long-short-term memory (LSTM), though powerful, is memory and computa\x02tion expensive.
Analysis of Quantized Models
We show that (i) weight-quantized networks converge to an error related to the weight quantization resolution and weight dimension; (ii) quantizing gradients slows convergence by a factor related to the gradient quantization resolution and dimension; and (iii) clipping the gradient before quantization renders this factor dimension-free, thus allowing the use of fewer bits for gradient quantization.
Power Law in Sparsified Deep Neural Networks
The power law has been observed in the degree distributions of many biological neural networks.
Loss-aware Weight Quantization of Deep Networks
The huge size of deep networks hinders their use in small computing devices.
Loss-aware Binarization of Deep Networks
Deep neural network models, though very powerful and highly successful, are computationally expensive in terms of space and time.
