Search Results for author:
Found 51 papers, 40 papers with code
Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning
Our empirical studies, including instruction tuning experiments with models such as Pythia and LLaMA, demonstrate the effectiveness of the proposed framework.
MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers
This is made possible by utilizing an alternative method for feature transformation to replace the linear projection of fully-connected layers.
Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMs
Conversely, training-free approaches offer a more efficient alternative by adapting pre-trained image-LLMs models for video tasks without additional training, but they face inference efficiency bottlenecks due to the large number of visual tokens generated from video frames.
Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuning
Token compression expedites the training and inference of Vision Transformers (ViTs) by reducing the number of the redundant tokens, e. g., pruning inattentive tokens or merging similar tokens.
Multi-Granularity Semantic Revision for Large Language Model Distillation
At the sequence level, we propose a sequence correction and re-generation (SCRG) strategy.
ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking
For instance, we achieve approximately $70\times$ compression for the Pythia-410M model, with the final performance being as accurate as the original model on various downstream tasks.
SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization
However, replacing LayerNorm with more efficient BatchNorm in transformer often leads to inferior performance and collapse in training.
No Time to Waste: Squeeze Time into Channel for Mobile Video Understanding
In this paper, we propose to squeeze the time axis of a video sequence into the channel dimension and present a lightweight video recognition network, term as \textit{SqueezeTime}, for mobile video understanding.
EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models
Speculative decoding emerges as a pivotal technique for enhancing the inference speed of Large Language Models (LLMs).
Context-Guided Spatial Feature Reconstruction for Efficient Semantic Segmentation
A shape self-calibration function is designed to make the key areas closer to foreground objects.
Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning
Current solutions for efficiently constructing large vision-language (VL) models follow a two-step paradigm: projecting the output of pre-trained vision encoders to the input space of pre-trained language models as visual prompts; and then transferring the models to downstream VL tasks via end-to-end parameter-efficient fine-tuning (PEFT).
Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting
It is noteworthy that the inference latency of the self-draft model may no longer be negligible compared to the large model, necessitating strategies to increase the token acceptance rate while minimizing the drafting steps of the small model.
GhostNetV3: Exploring the Training Strategies for Compact Models
In this paper, by systematically investigating the impact of different training ingredients, we introduce a strong training strategy for compact models.
SAM-DiffSR: Structure-Modulated Diffusion Model for Image Super-Resolution
In this paper, we propose the SAM-DiffSR model, which can utilize the fine-grained structure information from SAM in the process of sampling noise to improve the image quality without additional computational cost during inference.
DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models
Large language models (LLMs) face a daunting challenge due to the excessive computational and memory requirements of the commonly used Transformer architecture.
Data-efficient Large Vision Models through Sequential Autoregression
Training general-purpose vision models on purely sequential visual data, eschewing linguistic inputs, has heralded a new frontier in visual understanding.
Rethinking Optimization and Architecture for Tiny Language Models
Several design formulas are empirically proved especially effective for tiny language models, including tokenizer compression, architecture tweaking, parameter inheritance and multiple-round training.
A Survey on Transformer Compression
Model compression methods reduce the memory and computational cost of Transformer, which is a necessary step to implement large language/vision models on practical devices.
Circuit Design and Efficient Simulation of Quantum Inner Product and Empirical Studies of Its Effect on Near-Term Hybrid Quantum-Classic Machine Learning
However it remains unclear for the embodiment of the quantum circuits (QC) for QIP let alone a (thorough) evaluation of the QIP circuits especially in a practical context in the NISQ era by applying QIP to ML via hybrid quantum-classic pipelines.
PanGu-$π$: Enhancing Language Model Architectures via Nonlinearity Compensation
We then demonstrate that the proposed approach is significantly effective for enhancing the model nonlinearity through carefully designed ablations; thus, we present a new efficient model architecture for establishing modern, namely, PanGu-$\pi$.
TinySAM: Pushing the Envelope for Efficient Segment Anything Model
Extensive experiments on various zero-shot transfer tasks demonstrate the significantly advantageous performance of our TinySAM against counterpart methods.
CBQ: Cross-Block Quantization for Large Language Models
Post-training quantization (PTQ) has played a key role in compressing large language models (LLMs) with ultra-low costs.
LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Models
Moreover, based on the observation that the accuracy of CLIP model does not increase correspondingly as the parameters of text encoder increase, an extra objective of masked language modeling (MLM) is leveraged for maximizing the potential of the shortened text encoder.
One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation
To tackle the challenge in distilling heterogeneous models, we propose a simple yet effective one-for-all KD framework called OFA-KD, which significantly improves the distillation performance between heterogeneous architectures.
GPT4Image: Can Large Pre-trained Models Help Vision Models on Perception Tasks?
We present a new learning paradigm in which the knowledge extracted from large pre-trained models are utilized to help models like CNN and ViT learn enhanced representations and achieve better performance.
Masked Image Modeling with Local Multi-Scale Reconstruction
The lower layers are not explicitly guided and the interaction among their patches is only used for calculating new activations.
Network Expansion for Practical Training Acceleration
Recently, the sizes of deep neural networks and training datasets both increase drastically to pursue better performance in a practical sense.
FastMIM: Expediting Masked Image Modeling Pre-training for Vision
This paper presents FastMIM, a simple and generic framework for expediting masked image modeling with the following two steps: (i) pre-training vision backbones with low-resolution input images; and (ii) reconstructing Histograms of Oriented Gradients (HOG) feature instead of original RGB values of the input images.
GhostNetV2: Enhance Cheap Operation with Long-Range Attention
The convolutional operation can only capture local information in a window region, which prevents performance from being further improved.
Spatial-Channel Token Distillation for Vision MLPs
Recently, neural architectures with all Multi-layer Perceptrons (MLPs) have attracted great research interest from the computer vision community.
Ranked #532 on
Image Classification
on ImageNet
Vision GNN: An Image is Worth Graph of Nodes
In this paper, we propose to represent the image as a graph structure and introduce a new Vision GNN (ViG) architecture to extract graph-level feature for visual tasks.
Ranked #395 on
Image Classification
on ImageNet
Source-Free Domain Adaptation via Distribution Estimation
Domain Adaptation aims to transfer the knowledge learned from a labeled source domain to an unlabeled target domain whose data distributions are different.
From Quantum Graph Computing to Quantum Graph Learning: A Survey
Quantum computing (QC) is a new computational paradigm whose foundations relate to quantum physics.
PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture
We hope this new baseline will be helpful to the further research and application of vision transformer.
An Image Patch is a Wave: Phase-Aware Vision MLP
To dynamically aggregate tokens, we propose to represent each token as a wave function with two parts, amplitude and phase.
Hire-MLP: Vision MLP via Hierarchical Rearrangement
Previous vision MLPs such as MLP-Mixer and ResMLP accept linearly flattened image patches as input, making them inflexible for different input sizes and hard to capture spatial information.
Homogeneous Architecture Augmentation for Neural Predictor
Specifically, a homogeneous architecture augmentation algorithm is proposed in HAAP to generate sufficient training data taking the use of homogeneous representation.
CMT: Convolutional Neural Networks Meet Vision Transformers
Vision transformers have been successfully applied to image recognition tasks due to their ability to capture long-range dependencies within an image.
Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation
Specifically, we train a tiny student model to match a pre-trained teacher model in the patch-level manifold space.
Augmented Shortcuts for Vision Transformers
Transformer models have achieved great progress on computer vision tasks recently.
ReNAS: Relativistic Evaluation of Neural Architecture Search
An effective and efficient architecture performance evaluation scheme is essential for the success of Neural Architecture Search (NAS).
Patch Slimming for Efficient Vision Transformers
We first identify the effective patches in the last layer and then use them to guide the patch selection process of previous layers.
Ranked #8 on
Efficient ViTs
on ImageNet-1K (with DeiT-T)
Vision Transformer Pruning
Vision transformer has achieved competitive performance on a variety of computer vision applications.
Manifold Regularized Dynamic Network Pruning
Then, the manifold relationship between instances and the pruned sub-networks will be aligned in the training procedure.
Learning Frequency Domain Approximation for Binary Neural Networks
Binary neural networks (BNNs) represent original full-precision weights and activations into 1-bit with sign function.
A Survey on Visual Transformer
Transformer, first applied to the field of natural language processing, is a type of deep neural network mainly based on the self-attention mechanism.
SCOP: Scientific Control for Reliable Neural Network Pruning
To increase the reliability of the results, we prefer to have a more rigorous research design by including a scientific control group as an essential part to minimize the effect of all factors except the association between the filter and expected network output.
A Semi-Supervised Assessor of Neural Architectures
A graph convolutional neural network is introduced to predict the performance of architectures based on the learned representations and their relation modeled by the graph.
Beyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks
On one hand, massive trainable parameters significantly enhance the performance of these deep networks.
ReNAS:Relativistic Evaluation of Neural Architecture Search
An effective and efficient architecture performance evaluation scheme is essential for the success of Neural Architecture Search (NAS).
Bringing Giant Neural Networks Down to Earth with Unlabeled Data
Specifically, we exploit the unlabeled data to mimic the classification characteristics of giant networks, so that the original capacity can be preserved nicely.
