Search Results for author:
Found 13 papers, 5 papers with code
A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules
Training large models is both resource-intensive and time-consuming, making it crucial to understand the quantitative relationship between model performance and hyperparameters.
Decoupling Knowledge and Reasoning in Transformers: A Modular Architecture with Generalized Cross-Attention
Transformers have achieved remarkable success across diverse domains, but their monolithic architecture presents challenges in interpretability, adaptability, and scalability.
Compression for Better: A General and Stable Lossless Compression Framework
A key challenge is effectively leveraging compression errors and defining the boundaries for lossless compression to minimize model loss.
KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation
The recently developed retrieval-augmented generation (RAG) technology has enabled the efficient construction of domain-specific applications.
A Comprehensive Survey on Distributed Training of Graph Neural Networks
This situation poses a considerable challenge for newcomers, hindering their ability to grasp a comprehensive understanding of the workflows, computational patterns, communication strategies, and optimization techniques employed in distributed GNN training.
GLM-130B: An Open Bilingual Pre-trained Model
We introduce GLM-130B, a bilingual (English and Chinese) pre-trained language model with 130 billion parameters.
Ranked #1 on
Language Modelling
on CLUE (OCNLI_50K)
Mixed-Precision Inference Quantization: Radically Towards Faster inference speed, Lower Storage requirement, and Lower Loss
Based on the model's resilience to computational noise, model quantization is important for compressing models and improving computing speed.
Quantization in Layer's Input is Matter
In this paper, we will show that the quantization in layer's input is more important than parameters' quantization for loss function.
GraphTheta: A Distributed Graph Neural Network Learning System With Flexible Training Strategy
Herein, we present GraphTheta, the first distributed and scalable graph learning system built upon vertex-centric distributed graph processing with neural network operators implemented as user-defined functions.
AIPerf: Automated machine learning as an AI-HPC benchmark
The de facto HPC benchmark LINPACK can not reflect AI computing power and I/O performance without representative workload.
Bridging the Gap Between Neural Networks and Neuromorphic Hardware with A Neural Network Compiler
Different from developing neural networks (NNs) for general-purpose processors, the development for NN chips usually faces with some hardware-specific restrictions, such as limited precision of network signals and parameters, constrained computation scale, and limited types of non-linear functions.
SaberLDA: Sparsity-Aware Learning of Topic Models on GPUs
Latent Dirichlet Allocation (LDA) is a popular tool for analyzing discrete count data such as text and images.
WarpLDA: a Cache Efficient O(1) Algorithm for Latent Dirichlet Allocation
We then develop WarpLDA, an LDA sampler which achieves both the best O(1) time complexity per token and the best O(K) scope of random access.
