Search Results for author:
Found 34 papers, 20 papers with code
MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
The resultant models, denoted as MobileLLM-LS, demonstrate a further accuracy enhancement of 0. 7%/0. 8% than MobileLLM 125M/350M.
On The Open Prompt Challenge In Conditional Audio Generation
Text-to-audio generation (TTA) produces audio from a text description, learning from pairs of audio samples and hand-annotated text.
LLM-FP4: 4-Bit Floating-Point Quantized Transformers
Our method, for the first time, can quantize both weights and activations in the LLaMA-13B to only 4-bit and achieves an average score of 63. 1 on the common sense zero-shot reasoning tasks, which is only 5. 8 lower than the full-precision model, significantly outperforming the previous state-of-the-art by 12. 7 points.
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning
Motivated by this, we target to build a unified interface for completing many vision-language tasks including image description, visual question answering, and visual grounding, among others.
Ranked #10 on
Visual Question Answering
on BenchLMM
Efficient Quantization-aware Training with Adaptive Coreset Selection
Compared with previous coreset selection methods, our method significantly improves QAT performance with different dataset fractions.
Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts
In addition, the proposed method achieves the SOTA performance in NAS for building fast machine translation models, yielding better latency-BLEU tradeoff compared to HAT, state-of-the-art NAS for MT.
Binary and Ternary Natural Language Generation
For machine translation, we achieved BLEU scores of 21. 7 and 17. 6 on the WMT16 En-Ro benchmark, compared with a full precision mBART model score of 26. 8.
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
Several post-training quantization methods have been applied to large language models (LLMs), and have been shown to perform well down to 8-bits.
EBSR: Enhanced Binary Neural Network for Image Super-Resolution
While the performance of deep convolutional neural networks for image super-resolution (SR) has improved significantly, the rapid increase of memory and computation requirements hinders their deployment on resource-constrained devices.
Oscillation-free Quantization for Low-bit Vision Transformers
In addition, we also found that the interdependence between quantized weights in $\textit{query}$ and $\textit{key}$ of a self-attention layer makes ViT vulnerable to oscillation.
SDQ: Stochastic Differentiable Quantization with Mixed Precision
In order to deploy deep models in a computationally efficient manner, model quantization approaches have been frequently used.
BiT: Robustly Binarized Multi-distilled Transformer
Modern pre-trained transformers have rapidly advanced the state-of-the-art in machine learning, but have also grown in parameters and computational complexity, making them increasingly difficult to deploy in resource-constrained environments.
Stereo Neural Vernier Caliper
We propose a new object-centric framework for learning-based stereo 3D object detection.
Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space
This paper explores the feasibility of finding an optimal sub-model from a vision transformer and introduces a pure vision transformer slimming (ViT-Slim) framework.
Data-Free Neural Architecture Search via Recursive Label Calibration
We identify that the NAS task requires the synthesized data (we target at image domain here) with enough semantics, diversity, and a minimal domain gap from the natural images.
Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
The nonuniform quantization strategy for compressing neural networks usually achieves better performance than its counterpart, i. e., uniform strategy, due to its superior representational capacity.
Sliced Recursive Transformer
The proposed weight sharing mechanism by sliced recursion structure allows us to build a transformer with more than 100 or even 1000 shared layers with ease while keeping a compact size (13~15M), to avoid optimization difficulties when the model is too large.
Ranked #270 on
Image Classification
on ImageNet
How Do Adam and Training Strategies Help BNNs Optimization?
We show the regularization effect of second-order momentum in Adam is crucial to revitalize the weights that are dead due to the activation saturation in BNNs.
"BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
However, the BN layer is costly to calculate and is typically implemented with non-binary parameters, leaving a hurdle for the efficient implementation of BNN training.
Ranked #167 on
Image Classification
on CIFAR-10
Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study
This work aims to empirically clarify a recently discovered perspective that label smoothing is incompatible with knowledge distillation.
S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration
In this paper, we focus on this more difficult scenario: learning networks where both weights and activations are binary, meanwhile, without any human annotated labels.
Partial Is Better Than All: Revisiting Fine-tuning Strategy for Few-shot Learning
A common practice for this task is to train a model on the base set first and then transfer to novel classes through fine-tuning (Here fine-tuning procedure is defined as transferring knowledge from base to novel data, i. e. learning to transfer in few-shot scenario.)
Conditional Link Prediction of Category-Implicit Keypoint Detection
Keypoints of objects reflect their concise abstractions, while the corresponding connection links (CL) build the skeleton by detecting the intrinsic relations between keypoints.
Weight-dependent Gates for Network Pruning
In this paper, a simple yet effective network pruning framework is proposed to simultaneously address the problems of pruning indicator, pruning ratio, and efficiency constraint.
Joint Multi-Dimension Pruning via Numerical Gradient Update
To tackle these three naturally different dimensions, we proposed a general framework by defining pruning as seeking the best pruning vector (i. e., the numerical value of layer-wise channel number, spacial size, depth) and construct a unique mapping from the pruning vector to the pruned network structures.
Binarizing MobileNet via Evolution-based Searching
Inspired by one-shot architecture search frameworks, we manipulate the idea of group convolution to design efficient 1-Bit Convolutional Neural Networks (CNNs), assuming an approximately optimal trade-off between computational cost and model accuracy.
Attentive CutMix: An Enhanced Data Augmentation Approach for Deep Learning Based Image Classification
In this paper, we propose Attentive CutMix, a naturally enhanced augmentation strategy based on CutMix.
Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning
This drawback hinders the model from learning subtle variance and fine-grained information.
ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
In this paper, we propose several ideas for enhancing a binary network to close its accuracy gap from real-valued networks without incurring any additional computational cost.
Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
Together, the redefinition of latent weights as inertia and the introduction of Bop enable a better understanding of BNN optimization and open up the way for further improvements in training methodologies for BNNs.
Single Path One-Shot Neural Architecture Search with Uniform Sampling
It is easy to train and fast to search.
Ranked #88 on
Neural Architecture Search
on ImageNet
(Accuracy metric)
MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
In this paper, we propose a novel meta learning approach for automatic channel pruning of very deep neural networks.
Bi-Real Net: Binarizing Deep Network Towards Real-Network Performance
To address the training difficulty, we propose a training algorithm using a tighter approximation to the derivative of the sign function, a magnitude-aware gradient for weight updating, a better initialization method, and a two-step scheme for training a deep network.
Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm
In this work, we study the 1-bit convolutional neural networks (CNNs), of which both the weights and activations are binary.
