Search Results for author:
Found 23 papers, 9 papers with code
Attention Sinks and Outlier Features: A 'Catch, Tag, and Release' Mechanism for Embeddings
Two prominent features of large language models (LLMs) is the presence of large-norm (outlier) features and the tendency for tokens to attend very strongly to a select few tokens.
OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition
The recent paradigm shift to large-scale foundation models has brought about a new era for deep learning that, while has found great success in practice, has also been plagued by prohibitively expensive costs in terms of high memory consumption and compute.
Transformer Block Coupling and its Correlation with Generalization in LLMs
Large Language Models (LLMs) have made significant strides in natural language processing, and a precise understanding of the internal mechanisms driving their success is essential.
Sparsest Models Elude Pruning: An Exposé of Pruning's Current Capabilities
Pruning has emerged as a promising approach for compressing large-scale models, yet its effectiveness in recovering the sparsest of models has not yet been explored.
Breach By A Thousand Leaks: Unsafe Information Leakage in `Safe' AI Responses
To quantify these risks, we introduce a new safety evaluation framework based on impermissible information leakage of model outputs and demonstrate how our proposed question-decomposition attack can extract dangerous knowledge from a censored LLM more effectively than traditional jailbreaking.
Linguistic Collapse: Neural Collapse in (Large) Language Models
Neural collapse ($\mathcal{NC}$) is a phenomenon observed in classification tasks where top-layer representations collapse into their class means, which become equinorm, equiangular and aligned with the classifiers.
Pushing Boundaries: Mixup's Influence on Neural Collapse
These findings are unexpected, as mixed-up features are not simple convex combinations of feature class means (as one might get, for example, by training mixup with the mean squared error loss).
Residual Alignment: Uncovering the Mechanisms of Residual Networks
Our measurements reveal a process called Residual Alignment (RA) characterized by four properties: (RA1) intermediate representations of a given input are equispaced on a line, embedded in high dimensional space, as observed by Gai and Zhang [2021]; (RA2) top left and right singular vectors of Residual Jacobians align with each other and across different depths; (RA3) Residual Jacobians are at most rank C for fully-connected ResNets, where C is the number of classes; and (RA4) top singular values of Residual Jacobians scale inversely with depth.
Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift
Designing deep neural network classifiers that perform robustly on distributions differing from the available training data is an active area of machine learning research.
LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?
Specifically, we demonstrate that semantic censorship can be perceived as an undecidable problem, highlighting the inherent challenges in censorship that arise due to LLMs' programmatic and instruction-following capabilities.
Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path
The analytically-tractable MSE loss offers more mathematical opportunities than the hard-to-analyze CE loss, inspiring us to leverage MSE loss towards the theoretical investigation of NC.
Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
Numerous researchers recently applied empirical spectral analysis to the study of modern deep learning classifiers.
Prevalence of Neural Collapse during the terminal phase of deep learning training
Modern practice for training classification deepnets involves a Terminal Phase of Training (TPT), which begins at the epoch where training error first vanishes; During TPT, the training error stays effectively zero while training loss is pushed towards zero.
Degrees of Freedom Analysis of Unrolled Neural Networks
Leveraging the Stein's Unbiased Risk Estimator (SURE), this paper analyzes the generalization risk with its bias and variance components for recurrent unrolled networks.
Measuring the Spectrum of Deepnet Hessians
We apply state-of-the-art tools in modern high-dimensional numerical linear algebra to approximate efficiently the spectrum of the Hessian of modern deepnets, with tens of millions of parameters, trained on real data.
Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
We expose a structure in the derivative of the logits with respect to the parameters of the model, which is used to explain the existence of outliers in the spectrum of the Hessian.
The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
We decompose the Hessian into different components and study the dynamics with training and sample size of each term individually.
Neural Proximal Gradient Descent for Compressive Imaging
Recovering high-resolution images from limited sensory data typically leads to a serious ill-posed inverse problem, demanding inversion algorithms that effectively capture the prior information.
Recurrent Generative Adversarial Networks for Proximal Learning and Automated Compressive Image Recovery
Building effective priors is however challenged by the low train and test overhead dictated by real-time tasks; and the need for retrieving visually "plausible" and physically "feasible" images with minimal hallucination.
Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning
We show that the training of the filters is essential to allow for non-trivial signals in the model, and we derive an online algorithm to learn the dictionaries from real data, effectively resulting in cascaded sparse convolutional layers.
Convolutional Dictionary Learning via Local Processing
Convolutional Sparse Coding (CSC) is an increasingly popular model in the signal and image processing communities, tackling some of the limitations of traditional patch-based sparse representations.
Multimodal Latent Variable Analysis
The first step in our analysis is to find the common source of variability present in all sensor measurements.
Convolutional Neural Networks Analyzed via Convolutional Sparse Coding
This is shown to be tightly connected to CNN, so much so that the forward pass of the CNN is in fact the thresholding pursuit serving the ML-CSC model.
