Search Results for author:
Found 35 papers, 11 papers with code
The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexity
In this paper, we analyze the computational limitations of Mamba and State-space Models (SSMs) by using the circuit complexity framework.
Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond
The application of transformer-based models on time series forecasting (TSF) tasks has long been popular to study.
On the Expressive Power of Modern Hopfield Networks
Modern Hopfield networks (MHNs) have emerged as powerful tools in deep learning, capable of replacing components such as pooling layers, LSTMs, and attention mechanisms.
Circuit Complexity Bounds for RoPE-based Transformer Architecture
In this work, we establish a circuit complexity bound for Transformers with $\mathsf{RoPE}$ attention.
Advancing the Understanding of Fixed Point Iterations in Deep Neural Networks: A Detailed Analytical Study
Recent empirical studies have identified fixed point iteration phenomena in deep neural networks, where the hidden state tends to stabilize after several layers, showing minimal change in subsequent layers.
Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
Large Language Models (LLMs) have shown immense potential in enhancing various aspects of our daily lives, from conversational AI to search and AI assistants.
Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent
Our results demonstrate that as long as the input data has a constant condition number, e. g., $n = O(d)$, the linear looped Transformers can achieve a small error by multi-step gradient descent during in-context learning.
HSR-Enhanced Sparse Attention Acceleration
Our approach achieves a running time of $O(mn^{4/5})$ significantly faster than the naive approach $O(mn)$ for attention generation, where $n$ is the context length, $m$ is the query length, and $d$ is the hidden dimension.
Looped ReLU MLPs May Be All You Need as Practical Programmable Computers
In contrast, the multi-layer perceptrons with $\mathsf{ReLU}$ activation ($\mathsf{ReLU}$-$\mathsf{MLP}$), one of the most fundamental components of neural networks, is known to be expressive; specifically, a two-layer neural network is a universal approximator given an exponentially large number of hidden neurons.
Fine-grained Attention I/O Complexity: Comprehensive Analysis for Backward Passes
For small cache sizes, we provide an algorithm that improves over existing methods and achieves the tight bounds.
Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
Our research introduces a novel approach for the long context bottleneck to accelerate LLM inference and reduce GPU memory consumption.
Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time
The computational complexity of the self-attention mechanism in popular transformer architectures poses significant challenges for training and inference, and becomes the bottleneck for long inputs.
A Tighter Complexity Analysis of SparseGPT
In this work, we improved the analysis of the running time of SparseGPT [Frantar, Alistarh ICML 2023] from $O(d^{3})$ to $O(d^{\omega} + d^{2+a+o(1)} + d^{1+\omega(1, 1, a)-a})$ for any $a \in [0, 1]$, where $\omega$ is the exponent of matrix multiplication.
Fast John Ellipsoid Computation with Differential Privacy Optimization
Determining the John ellipsoid - the largest volume ellipsoid contained within a convex polytope - is a fundamental problem with applications in machine learning, optimization, and data analytics.
Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability
In this study, we delve into the ICL capabilities of LLMs on composite tasks, with only simple tasks as in-context examples.
Differential Privacy of Cross-Attention with Provable Guarantee
In addition, our data structure can guarantee that the process of answering user query satisfies $(\epsilon, \delta)$-DP with $\widetilde{O}(n^{-1} \epsilon^{-1} \alpha^{-1/2} R^{2s} R_w r^2)$ additive error and $n^{-1} (\alpha + \epsilon_s)$ relative error between our output and the true answer.
Differential Privacy Mechanisms in Neural Tangent Kernel Regression
Training data privacy is a fundamental problem in modern Artificial Intelligence (AI) applications, such as face recognition, recommendation systems, language generation, and many others, as it may contain sensitive user information related to legal issues.
Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
Large language models (LLMs) and vision-language models (VLMs) have demonstrated remarkable performance across a wide range of tasks and domains.
Towards Infinite-Long Prefix in Transformer
Prompting and context-based fine-tuning methods, which we call Prefix Learning, have been proposed to enhance the performance of language models on various downstream tasks.
Why Larger Language Models Do In-context Learning Differently?
This sheds light on where transformers pay attention to and how that affects ICL.
Unraveling the Smoothness Properties of Diffusion Models: A Gaussian Mixture Perspective
We prove that if the target distribution is a $k$-mixture of Gaussians, the density of the entire diffusion process will also be a $k$-mixture of Gaussians.
Tensor Attention Training: Provably Efficient Learning of Higher-order Transformers
Tensor Attention, a multi-view attention that is able to capture high-order correlations among multiple modalities, can overcome the representational limitations of classical matrix attention.
Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers
We then design a fast algorithm to approximate the attention matrix via a sum of such $k$ convolution matrices.
Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyond
The softmax activation function plays a crucial role in the success of large language models (LLMs), particularly in the self-attention mechanism of the widely adopted Transformer architecture.
Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning
An emerging solution with recent success in vision and NLP involves finetuning a foundation model on a selection of relevant tasks, before its adaptation to a target task with limited labeled samples.
Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs
We direct our focus to the complex algebraic learning task of modular addition involving $k$ inputs.
A Graph-Theoretic Framework for Understanding Open-World Semi-Supervised Learning
Open-world semi-supervised learning aims at inferring both known and novel classes in unlabeled data, by harnessing prior knowledge from a labeled set with known classes.
When and How Does Known Class Help Discover Unknown Ones? Provable Understanding Through Spectral Analysis
This paper bridges the gap by providing an analytical framework to formalize and investigate when and how known classes can help discover novel classes.
Domain Generalization via Nuclear Norm Regularization
In this paper, we propose a simple and effective regularization method based on the nuclear norm of the learned features for domain generalization.
The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning
foundation models) has recently become a prevalent learning paradigm, where one first pre-trains a representation using large-scale unlabeled data, and then learns simple predictors on top of the representation using small labeled data from the downstream tasks.
A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Features
These results provide theoretical evidence showing that feature learning in neural networks depends strongly on the input structure and leads to the superior performance.
Attentive Walk-Aggregating Graph Neural Networks
Our experiments demonstrate the strong performance of AWARE in graph-level prediction tasks in the standard setting in the domains of molecular property prediction and social networks.
Deep Online Fused Video Stabilization
We present a deep neural network (DNN) that uses both sensor data (gyroscope) and image content (optical flow) to stabilize videos through unsupervised learning.
SF-Net: Structured Feature Network for Continuous Sign Language Recognition
The proposed SF-Net extracts features in a structured manner and gradually encodes information at the frame level, the gloss level and the sentence level into the feature representation.
DAWN: Dual Augmented Memory Network for Unsupervised Video Object Tracking
Our Dual Augmented Memory Network (DAWN) is unique in remembering both target and background, and using an improved attention LSTM memory to guide the focus on memorized features.
