Search Results for author:
Found 36 papers, 20 papers with code
UNIT: Unifying Image and Text Recognition in One Vision Encoder
Starting with a vision encoder pre-trained with image recognition tasks, UNIT introduces a lightweight language decoder for predicting text outputs and a lightweight vision decoder to prevent catastrophic forgetting of the original image encoding capabilities.
Embedding Compression in Recommender Systems: A Survey
In this survey, we provide a comprehensive review of embedding compression approaches in recommender systems.
HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models
High-resolution inputs enable Large Vision-Language Models (LVLMs) to discern finer visual details, enhancing their comprehension capabilities.
DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models
Specifically, we trace back the semantic relevance flow from generated language tokens to raw visual encoder patches and the intermediate outputs produced by projectors.
OAC: Output-adaptive Calibration for Accurate Post-training Quantization
The Hessian is also used for detecting the most salient weights to quantization.
Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modality
Video paragraph captioning (VPC) involves generating detailed narratives for long videos, utilizing supportive modalities such as speech and event boundaries.
Visually Guided Generative Text-Layout Pre-training for Document Intelligence
Prior study shows that pre-training techniques can boost the performance of visual document understanding (VDU), which typically requires models to gain abilities to perceive and reason both document texts and layouts (e. g., locations of texts and table-cells).
MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric
We find that directly using smaller pre-trained models and applying magnitude-based pruning on CLIP models leads to inflexibility and inferior performance.
IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact
Such outliers are found to allocate most of the attention scores on initial tokens of input, termed as pivot tokens, which are crucial to the performance of quantized LLMs.
TempCompass: Do Video LLMs Really Understand Videos?
Motivated by these two problems, we propose the \textbf{TempCompass} benchmark, which introduces a diversity of temporal aspects and task formats.
Extending Context Window of Large Language Models via Semantic Compression
Transformer-based Large Language Models (LLMs) often impose limitations on the length of the text input to ensure the generation of fluent and relevant responses.
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
This work proposes TimeChat, a time-sensitive multimodal large language model specifically designed for long video understanding.
Ranked #2 on
Video-Text Retrieval
on Test-of-Time
(using extra training data)
VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Models
The ability to perceive how objects change over time is a crucial ingredient in human intelligence.
FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation
The multi-aspect categorization of FETV enables fine-grained analysis of the metrics' reliability in different scenarios.
TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding
TESTA can reduce the number of visual tokens by 75% and thus accelerate video encoding.
Ranked #1 on
Video Retrieval
on Condensed Movies
(using extra training data)
mCLIP: Multilingual CLIP via Cross-lingual Transfer
Furthermore, to enhance the token- and sentence-level multilingual representation of the MTE, we propose to train it with machine translation and contrastive learning jointly before the TriKD to provide a better initialization.
CTRL: Connect Collaborative and Language Model for CTR Prediction
Both tabular data and converted textual data are regarded as two different modalities and are separately fed into the collaborative CTR model and pre-trained language model.
Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding
While various vision-language pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far.
Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
To this end, we formulate a novel quantization training paradigm to compress the embeddings from the training stage, termed low-precision training (LPT).
LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling
Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks.
Compression of Generative Pre-trained Language Models via Quantization
We find that previous quantization methods fail on generative tasks due to the \textit{homogeneous word embeddings} caused by reduced capacity, and \textit{varied distribution of weights}.
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.
Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark
Experiments show that Wukong can serve as a promising Chinese pre-training dataset and benchmark for different cross-modal learning methods.
Ranked #6 on
Image Retrieval
on MUGE Retrieval
FILIP: Fine-grained Interactive Language-Image Pre-Training
In this paper, we introduce a large-scale Fine-grained Interactive Language-Image Pre-training (FILIP) to achieve finer-level alignment through a cross-modal late interaction mechanism, which uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective.
Towards Efficient Post-training Quantization of Pre-trained Language Models
Experiments on GLUE and SQuAD benchmarks show that our proposed PTQ solution not only performs close to QAT, but also enjoys significant reductions in training time, memory overhead, and data consumption.
GhostBERT: Generate More Features with Cheap Operations for BERT
Transformer-based pre-trained language models like BERT, though powerful in many tasks, are expensive in both memory and computation, due to their large number of parameters.
Improved OOD Generalization via Adversarial Training and Pre-training
In this paper, after defining OOD generalization via Wasserstein distance, we theoretically show that a model robust to input perturbation generalizes well on OOD data.
Reweighting Augmented Samples by Minimizing the Maximal Expected Loss
Inspired by adversarial training, we minimize this maximal expected loss (MMEL) and obtain a simple and interpretable closed-form solution: more attention should be paid to augmented samples with large loss values (i. e., harder examples).
BinaryBERT: Pushing the Limit of BERT Quantization
In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization.
TernaryBERT: Distillation-aware Ultra-low Bit BERT
Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks. However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices.
DynaBERT: Dynamic BERT with Adaptive Width and Depth
The pre-trained language models like BERT, though powerful in many natural language processing tasks, are both computation and memory expensive.
Normalization Helps Training of Quantized LSTM
The long-short-term memory (LSTM), though powerful, is memory and computa\x02tion expensive.
Analysis of Quantized Models
We show that (i) weight-quantized networks converge to an error related to the weight quantization resolution and weight dimension; (ii) quantizing gradients slows convergence by a factor related to the gradient quantization resolution and dimension; and (iii) clipping the gradient before quantization renders this factor dimension-free, thus allowing the use of fewer bits for gradient quantization.
Power Law in Sparsified Deep Neural Networks
The power law has been observed in the degree distributions of many biological neural networks.
Loss-aware Weight Quantization of Deep Networks
The huge size of deep networks hinders their use in small computing devices.
Loss-aware Binarization of Deep Networks
Deep neural network models, though very powerful and highly successful, are computationally expensive in terms of space and time.
