Search Results for author:
Found 14 papers, 6 papers with code
HETHUB: A Distributed Training System with Heterogeneous Cluster for Large-Scale Models
We train the Llama-140B model on a heterogeneous cluster with 768 GPU-accelerators(128 AMD and 640 GPU-accelerator A).
You Only Cache Once: Decoder-Decoder Architectures for Language Models
We introduce a decoder-decoder architecture, YOCO, for large language models, which only caches key-value pairs once.
Efficient Large Language Models: A Survey
We hope our survey can serve as a valuable resource to help researchers and practitioners gain a systematic understanding of efficient LLMs research and inspire them to contribute to this important and exciting field.
AutoTaskFormer: Searching Vision Transformers for Multi-task Learning
Vision Transformers have shown great performance in single tasks such as classification and segmentation.
ElasticViT: Conflict-aware Supernet Training for Deploying Fast Vision Transformer on Diverse Mobile Devices
However, prior supernet training methods that rely on uniform sampling suffer from the gradient conflict issue: the sampled subnets can have vastly different model sizes (e. g., 50M vs. 2G FLOPs), leading to different optimization directions and inferior performance.
SpaceEvo: Hardware-Friendly Search Space Design for Efficient INT8 Inference
The combination of Neural Architecture Search (NAS) and quantization has proven successful in automatically designing low-FLOPs INT8 quantized neural networks (QNN).
PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformation
Dynamic sparsity, where the sparsity patterns are unknown until runtime, poses a significant challenge to deep learning.
SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstraction
SuperScaler is a system that facilitates the design and generation of highly flexible parallelization plans.
Nesting Forward Automatic Differentiation for Memory-Efficient Deep Neural Network Training
An activation function is an element-wise mathematical function and plays a crucial role in deep neural networks (DNN).
Privacy-preserving Online AutoML for Domain-Specific Face Detection
Thanks to HyperFD, each local task (client) is able to effectively leverage the learning "experience" of previous tasks without uploading raw images to the platform; meanwhile, the meta-feature extractor is continuously learned to better trade off the bias and variance.
Learning to Rank Ace Neural Architectures via Normalized Discounted Cumulative Gain
One of the key challenges in Neural Architecture Search (NAS) is to efficiently rank the performances of architectures.
How Does Supernet Help in Neural Architecture Search?
Weight sharing, as an approach to speed up architecture performance estimation has received wide attention.
LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression
BERT is a cutting-edge language representation model pre-trained by a large corpus, which achieves superior performances on various natural language understanding tasks.
Deeper Insights into Weight Sharing in Neural Architecture Search
With the success of deep neural networks, Neural Architecture Search (NAS) as a way of automatic model design has attracted wide attention.
