Search Results for author:
Found 8 papers, 0 papers with code
Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
Large Language Models (LLMs) have shown immense potential in enhancing various aspects of our daily lives, from conversational AI to search and AI assistants.
Fine-grained Attention I/O Complexity: Comprehensive Analysis for Backward Passes
For small cache sizes, we provide an algorithm that improves over existing methods and achieves the tight bounds.
Looped ReLU MLPs May Be All You Need as Practical Programmable Computers
In contrast, the multi-layer perceptrons with $\mathsf{ReLU}$ activation ($\mathsf{ReLU}$-$\mathsf{MLP}$), one of the most fundamental components of neural networks, is known to be expressive; specifically, a two-layer neural network is a universal approximator given an exponentially large number of hidden neurons.
Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time
The computational complexity of the self-attention mechanism in popular transformer architectures poses significant challenges for training and inference, and becomes the bottleneck for long inputs.
Differential Privacy of Cross-Attention with Provable Guarantee
In addition, our data structure can guarantee that the process of answering user query satisfies $(\epsilon, \delta)$-DP with $\widetilde{O}(n^{-1} \epsilon^{-1} \alpha^{-1/2} R^{2s} R_w r^2)$ additive error and $n^{-1} (\alpha + \epsilon_s)$ relative error between our output and the true answer.
Unraveling the Smoothness Properties of Diffusion Models: A Gaussian Mixture Perspective
We prove that if the target distribution is a $k$-mixture of Gaussians, the density of the entire diffusion process will also be a $k$-mixture of Gaussians.
Tensor Attention Training: Provably Efficient Learning of Higher-order Transformers
Tensor Attention, a multi-view attention that is able to capture high-order correlations among multiple modalities, can overcome the representational limitations of classical matrix attention.
Differentially Private Attention Computation
Large language models (LLMs), especially those based on the Transformer architecture, have had a profound impact on various aspects of daily life, such as natural language processing, content generation, research methodologies, and more.
