Search Results for author:
Found 53 papers, 30 papers with code
ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing
Sparsely activated Mixture-of-Experts (MoE) models are widely adopted to scale up model capacity without increasing the computation budget.
SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization
Second, we propose a method to smooth $Q$, enhancing the accuracy of INT4 $QK$.
Consistency Diffusion Bridge Models
Recently, diffusion denoising bridge models (DDBMs), a new formulation of generative modeling that builds stochastic processes between fixed data endpoints based on a reference diffusion process, have achieved empirical success across tasks with coupled data distribution, such as image-to-image translation.
COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training
FP8 training has emerged as a promising method for improving training efficiency.
Beyond 2:4: exploring V:N:M sparsity for efficient transformer inference on GPUs
In this study, we thoroughly investigate the application of V:N:M sparsity in vision models and LLMs across multiple tasks, from pertaining to downstream tasks.
FrameBridge: Improving Image-to-Video Generation with Bridge Models
By formulating I2V synthesis as a frames-to-frames generation task and modelling it with a data-to-data process, we fully exploit the information in input image and facilitate the generative model to learn the image animation process.
On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent
We also show that Adam behaves similarly to SignGD in terms of both optimization and generalization in this setting.
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration
Although quantization has proven to be an effective method for accelerating model inference, existing quantization methods primarily focus on optimizing the linear layer.
S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-training
Training deep neural networks (DNNs) is costly.
1-Bit FQT: Pushing the Limit of Fully Quantized Training to 1-bit
The strategy leverages the heterogeneity of gradients by pruning less informative gradients and enhancing the numerical precision of remaining gradients to mitigate gradient variance.
Pruning Large Language Models with Semi-Structural Adaptive Sparse Training
However, these methods often suffer from considerable performance degradation on complex language understanding tasks, raising concerns about the feasibility of pruning in LLMs.
SparseDM: Toward Sparse Efficient Diffusion Models
Diffusion models have been extensively used in data generation tasks and are recognized as one of the best generative models.
Accelerating Transformer Pre-training with 2:4 Sparsity
Utilizing this metric, we propose three techniques to preserve accuracy: to modify the sparse-refined straight-through estimator by applying the masked decay term on gradients, to determine a feasible decay factor in warm-up stage, and to enhance the model's quality by a dense fine-tuning procedure near the end of pre-training.
Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization
Pretraining transformers are generally time-consuming.
Efficient Backpropagation with Variance-Controlled Adaptive Sampling
On all the tasks, VCAS can preserve the original training loss trajectory and validation accuracy with an up to 73. 87% FLOPs reduction of BP and 49. 58% FLOPs reduction of the whole training process.
C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory
Generative Adversarial Imitation Learning (GAIL) trains a generative policy to mimic a demonstrator.
DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics
In this work, we propose a novel formulation towards the optimal parameterization during sampling that minimizes the first-order discretization error of the ODE solution.
Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting
Despite the significant progress made in practical applications of aligned language models (LMs), they tend to be overconfident in output answers compared to the corresponding pre-trained LMs.
Training Transformers with 4-bit Integers
To achieve this, we carefully analyze the specific structures of activation and gradients in transformers to propose dedicated quantizers for them.
Stabilizing GANs' Training with Brownian Motion Controller
We theoretically prove that the training process of DiracGANs-BMC is globally exponential stable and derive bounds on the rate of convergence.
Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Models
In light of these observations, we evaluate the calibration of several methods that preserve pre-trained features and show that preserving pre-trained features can improve the calibration of fine-tuned language models.
Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs
The probability flow ordinary differential equation (ODE) of diffusion models (i. e., diffusion ODEs) is a particular case of continuous normalizing flows (CNFs), which enables deterministic inference and exact likelihood evaluation.
Ranked #2 on
Density Estimation
on ImageNet 32x32
Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning
The main challenge for this setting is that the intermediate guidance during the diffusion sampling procedure, which is jointly defined by the sampling distribution and the energy function, is unknown and is hard to estimate.
DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
The commonly-used fast sampler for guided sampling is DDIM, a first-order diffusion ODE solver that generally needs 100 to 250 steps for high-quality samples.
GACT: Activation Compressed Training for Generic Network Architectures
Training large neural network (NN) models requires extensive memory resources, and Activation Compressed Training (ACT) is a promising approach to reduce training memory footprint.
Fast Lossless Neural Compression with Integer-Only Discrete Flows
In this work, we propose Integer-only Discrete Flows (IODF), an efficient neural compressor with integer-only arithmetic.
Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matching
To fill up this gap, we show that the negative likelihood of the ODE can be bounded by controlling the first, second, and third-order score matching errors; and we further present a novel high-order denoising score matching method to enable maximum likelihood training of score-based diffusion ODEs.
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
In this work, we propose an exact formulation of the solution of diffusion ODEs.
Deep Ensemble as a Gaussian Process Approximate Posterior
In this way, we relate DE to Bayesian inference to enjoy reliable Bayesian uncertainty.
Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models
This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs, dubbed as delta tuning in this paper.
Deep Ensemble as a Gaussian Process Posterior
Deep Ensemble (DE) is a flexible, feasible, and effective alternative to Bayesian neural networks (BNNs) for uncertainty estimation in deep learning.
ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training
On all these tasks, ActNN compresses the activation to 2 bits on average, with negligible accuracy loss.
Implicit Normalizing Flows
Through theoretical analysis, we show that the function space of ImpFlow is strictly richer than that of ResFlows.
BARS: Joint Search of Cell Topology and Layout for Accurate and Efficient Binary ARchitectures
We also design the micro-level search space to strengthen the information flow for BNN.
A Statistical Framework for Low-bitwidth Training of Deep Neural Networks
We show that the FQT gradient is an unbiased estimator of the QAT gradient, and we discuss the impact of gradient quantization on its variance.
Ranked #9 on
Semantic Textual Similarity
on STS Benchmark
BEV-Seg: Bird's Eye View Semantic Segmentation Using Geometry and Semantic Point Cloud
Bird's-eye-view (BEV) is a powerful and widely adopted representation for road scenes that captures surrounding objects and their spatial locations, along with overall context in the scene.
VFlow: More Expressive Generative Flows with Variational Data Augmentation
Generative flows are promising tractable models for density modeling that define probabilistic distributions with invertible transformations.
Stochastic Expectation Maximization with Variance Reduction
However, sEM has a slower asymptotic convergence rate than batch EM, and requires a decreasing sequence of step sizes, which is difficult to tune.
Towards Training Probabilistic Topic Models on Neuromorphic Multi-chip Systems
We also propose an extension to train pLSI and a method to prune the network to obey the limited fan-in of some NMSs.
Stochastic Training of Graph Convolutional Networks
Previous attempts on reducing the receptive field size by subsampling neighbors do not have any convergence guarantee, and their receptive field size per node is still in the order of hundreds.
Population Matching Discrepancy and Applications in Deep Learning
In this paper, we propose population matching discrepancy (PMD) for estimating the distribution distance based on samples, as well as an algorithm to learn the parameters of the distributions using PMD as an objective.
Stochastic Training of Graph Convolutional Networks with Variance Reduction
Previous attempts on reducing the receptive field size by subsampling neighbors do not have a convergence guarantee, and their receptive field size per node is still in the order of hundreds.
ZhuSuan: A Library for Bayesian Deep Learning
In this paper we introduce ZhuSuan, a python probabilistic programming library for Bayesian deep learning, which conjoins the complimentary advantages of Bayesian methods and deep learning.
Scalable Inference for Nested Chinese Restaurant Process Topic Models
Finally, we propose an efficient distributed implementation of PCGS through vectorization, pre-processing, and a careful design of the concurrent data structures and communication strategy.
SaberLDA: Sparsity-Aware Learning of Topic Models on GPUs
Latent Dirichlet Allocation (LDA) is a popular tool for analyzing discrete count data such as text and images.
Scaling up Dynamic Topic Models
Dynamic topic models (DTMs) are very effective in discovering topics and capturing their evolution trends in time series data.
Streaming Gibbs Sampling for LDA Model
Streaming variational Bayes (SVB) is successful in learning LDA models in an online manner.
WarpLDA: a Cache Efficient O(1) Algorithm for Latent Dirichlet Allocation
We then develop WarpLDA, an LDA sampler which achieves both the best O(1) time complexity per token and the best O(K) scope of random access.
Dropout Training for SVMs with Data Augmentation
Empirical results on several real datasets demonstrate the effectiveness of dropout training on significantly boosting the classification accuracy of both linear and nonlinear SVMs.
Big Learning with Bayesian Methods
Explosive growth in data and availability of cheap computing resources have sparked increasing interest in Big learning, an emerging subfield that studies scalable machine learning algorithms, systems, and applications with Big Data.
Dropout Training for Support Vector Machines
To deal with the intractable expectation of the non-smooth hinge loss under corrupting distributions, we develop an iteratively re-weighted least square (IRLS) algorithm by exploring data augmentation techniques.
Scalable Inference for Logistic-Normal Topic Models
Logistic-normal topic models can effectively discover correlation structures among latent topics.
