Search Results for author:
Found 17 papers, 6 papers with code
Pushing Boundaries: Mixup's Influence on Neural Collapse
These findings are unexpected, as mixed-up features are not simple convex combinations of feature class means (as one might get, for example, by training mixup with the mean squared error loss).
Residual Alignment: Uncovering the Mechanisms of Residual Networks
Our measurements reveal a process called Residual Alignment (RA) characterized by four properties: (RA1) intermediate representations of a given input are equispaced on a line, embedded in high dimensional space, as observed by Gai and Zhang [2021]; (RA2) top left and right singular vectors of Residual Jacobians align with each other and across different depths; (RA3) Residual Jacobians are at most rank C for fully-connected ResNets, where C is the number of classes; and (RA4) top singular values of Residual Jacobians scale inversely with depth.
Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift
Designing deep neural network classifiers that perform robustly on distributions differing from the available training data is an active area of machine learning research.
LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?
Specifically, we demonstrate that semantic censorship can be perceived as an undecidable problem, highlighting the inherent challenges in censorship that arise due to LLMs' programmatic and instruction-following capabilities.
Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path
The analytically-tractable MSE loss offers more mathematical opportunities than the hard-to-analyze CE loss, inspiring us to leverage MSE loss towards the theoretical investigation of NC.
Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
Numerous researchers recently applied empirical spectral analysis to the study of modern deep learning classifiers.
Prevalence of Neural Collapse during the terminal phase of deep learning training
Modern practice for training classification deepnets involves a Terminal Phase of Training (TPT), which begins at the epoch where training error first vanishes; During TPT, the training error stays effectively zero while training loss is pushed towards zero.
Degrees of Freedom Analysis of Unrolled Neural Networks
Leveraging the Stein's Unbiased Risk Estimator (SURE), this paper analyzes the generalization risk with its bias and variance components for recurrent unrolled networks.
Measuring the Spectrum of Deepnet Hessians
We apply state-of-the-art tools in modern high-dimensional numerical linear algebra to approximate efficiently the spectrum of the Hessian of modern deepnets, with tens of millions of parameters, trained on real data.
Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
We expose a structure in the derivative of the logits with respect to the parameters of the model, which is used to explain the existence of outliers in the spectrum of the Hessian.
The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
We decompose the Hessian into different components and study the dynamics with training and sample size of each term individually.
Neural Proximal Gradient Descent for Compressive Imaging
Recovering high-resolution images from limited sensory data typically leads to a serious ill-posed inverse problem, demanding inversion algorithms that effectively capture the prior information.
Recurrent Generative Adversarial Networks for Proximal Learning and Automated Compressive Image Recovery
Building effective priors is however challenged by the low train and test overhead dictated by real-time tasks; and the need for retrieving visually "plausible" and physically "feasible" images with minimal hallucination.
Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning
We show that the training of the filters is essential to allow for non-trivial signals in the model, and we derive an online algorithm to learn the dictionaries from real data, effectively resulting in cascaded sparse convolutional layers.
Convolutional Dictionary Learning via Local Processing
Convolutional Sparse Coding (CSC) is an increasingly popular model in the signal and image processing communities, tackling some of the limitations of traditional patch-based sparse representations.
Multimodal Latent Variable Analysis
The first step in our analysis is to find the common source of variability present in all sensor measurements.
Convolutional Neural Networks Analyzed via Convolutional Sparse Coding
This is shown to be tightly connected to CNN, so much so that the forward pass of the CNN is in fact the thresholding pursuit serving the ML-CSC model.
