Search Results for author:
Found 22 papers, 10 papers with code
Deep Neural Network Initialization with Sparsity Inducing Activations
Inducing and leveraging sparse activations during training and inference is a promising avenue for improving the computational efficiency of deep networks, which is increasingly important as network sizes continue to grow and their application becomes more widespread.
Beyond IID weights: sparse and low-rank deep Neural Networks are also Gaussian Processes
The infinitely wide neural network has been proven a useful and manageable mathematical model that enables the understanding of many phenomena appearing in deep learning.
Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs
Inspired by the Dynamic Sparse Training, DSnoT minimizes the reconstruction error between the dense and sparse LLMs, in the fashion of performing iterative weight pruning-and-growing on top of sparse LLMs.
On the Initialisation of Wide Low-Rank Feedforward Neural Networks
The edge-of-chaos dynamics of wide randomly initialized low-rank feedforward networks are analyzed.
Optimal Approximation Complexity of High-Dimensional Functions with Neural Networks
We investigate properties of neural networks that use both ReLU and $x^2$ as activation functions and build upon previous results to show that both analytic functions and functions in Sobolev spaces can be approximated by such networks of constant depth to arbitrary accuracy, demonstrating optimal order approximation rates across all nonlinear approximators, including standard ReLU networks.
Improved Projection Learning for Lower Dimensional Feature Maps
The requirement to repeatedly move large feature maps off- and on-chip during inference with convolutional neural networks (CNNs) imposes high costs in terms of both energy and time.
Tuning-free multi-coil compressed sensing MRI with Parallel Variable Density Approximate Message Passing (P-VDAMP)
To our knowledge, P-VDAMP is the first algorithm for multi-coil MRI data that obeys a state evolution with accurately tracked parameters.
Activation function design for deep networks: linearity and effective initialisation
The activation function deployed in a deep neural network has great influence on the performance of the network at initialisation, which in turn has implications for training.
Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset
We show that standard training of networks built with these layers, and pruned at initialization, achieves state-of-the-art accuracy for extreme sparsities on a variety of benchmark network architectures and datasets.
Mutual Information of Neural Network Initialisations: Mean Field Approximations
The ability to train randomly initialised deep neural networks is known to depend strongly on the variance of the weight matrices and biases as well as the choice of nonlinear activation.
Information Theory Information Theory
An Empirical Study of Derivative-Free-Optimization Algorithms for Targeted Black-Box Attacks in Deep Neural Networks
We perform a comprehensive study on the performance of derivative free optimization (DFO) algorithms for the generation of targeted black-box adversarial attacks on Deep Neural Network (DNN) classifiers assuming the perturbation energy is bounded by an $\ell_\infty$ constraint and the number of queries to the network is limited.
Compressed sensing of low-rank plus sparse matrices
This manuscript develops similar guarantees showing that $m\times n$ matrices that can be expressed as the sum of a rank-$r$ matrix and a $s$-sparse matrix can be recovered by computationally tractable methods from $\mathcal{O}(r(m+n-r)+s)\log(mn/s)$ linear measurements.
Encoder blind combinatorial compressed sensing
In this paper we consider the problem of designing a decoder to recover a set of sparse codes from their linear measurements alone, that is without access to encoder matrix.
Approximate Message Passing with a Colored Aliasing Model for Variable Density Fourier Sampled Images
Central to AMP is its "state evolution", which guarantees that the difference between the current estimate and ground truth (the "aliasing") at every iteration obeys a Gaussian distribution that can be fully characterized by a scalar.
A Model-Based Derivative-Free Approach to Black-Box Adversarial Examples: BOBYQA
We demonstrate that model-based derivative free optimisation algorithms can generate adversarial targeted misclassification of deep networks using fewer network queries than non-model-based methods.
Trajectory growth lower bounds for random sparse deep ReLU networks
This paper considers the growth in the length of one-dimensional trajectories as they are passed through deep ReLU neural networks, which, among other things, is one measure of the expressivity of deep networks.
An Approximate Message Passing Algorithm for Rapid Parameter-Free Compressed Sensing MRI
In response we present an algorithm based on Orthogonal AMP constructed specifically for variable density partial Fourier sensing matrices.
Trajectory growth through random deep ReLU networks
This paper considers the growth in the length of one-dimensional trajectories as they are passed through deep ReLU neural networks, which, among other things, is one measure of the expressivity of deep networks.
Geometric anomaly detection in data
This paper describes the systematic application of local topological methods for detecting interfaces and related anomalies in complicated high-dimensional data.
Algebraic Topology Algebraic Geometry 57N80
Multispectral snapshot demosaicing via non-convex matrix completion
Snapshot mosaic multispectral imagery acquires an undersampled data cube by acquiring a single spectral measurement per spatial pixel.
Towards an understanding of CNNs: analysing the recovery of activation pathways via Deep Convolutional Sparse Coding
Deep Convolutional Sparse Coding (D-CSC) is a framework reminiscent of deep convolutional neural networks (DCNNs), but by omitting the learning of the dictionaries one can more transparently analyse the role of the activation function and its ability to recover activation paths through the layers.
Parallel multi-scale reduction of persistent homology filtrations
The persistent homology pipeline includes the reduction of a, so-called, boundary matrix.
Algebraic Topology
