Search Results for author: Jared Tanner

Found 22 papers, 10 papers with code

Deep Neural Network Initialization with Sparsity Inducing Activations

no code implementations25 Feb 2024 Ilan Price, Nicholas Daultry Ball, Samuel C. H. Lam, Adam C. Jones, Jared Tanner

Inducing and leveraging sparse activations during training and inference is a promising avenue for improving the computational efficiency of deep networks, which is increasingly important as network sizes continue to grow and their application becomes more widespread.

Computational Efficiency

Beyond IID weights: sparse and low-rank deep Neural Networks are also Gaussian Processes

no code implementations25 Oct 2023 Thiziri Nait-Saada, Alireza Naderi, Jared Tanner

The infinitely wide neural network has been proven a useful and manageable mathematical model that enables the understanding of many phenomena appearing in deep learning.

Gaussian Processes

Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs

1 code implementation13 Oct 2023 Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, Rongrong Ji

Inspired by the Dynamic Sparse Training, DSnoT minimizes the reconstruction error between the dense and sparse LLMs, in the fashion of performing iterative weight pruning-and-growing on top of sparse LLMs.

Network Pruning

On the Initialisation of Wide Low-Rank Feedforward Neural Networks

no code implementations31 Jan 2023 Thiziri Nait Saada, Jared Tanner

The edge-of-chaos dynamics of wide randomly initialized low-rank feedforward networks are analyzed.

Optimal Approximation Complexity of High-Dimensional Functions with Neural Networks

no code implementations30 Jan 2023 Vincent P. H. Goverse, Jad Hamdan, Jared Tanner

We investigate properties of neural networks that use both ReLU and $x^2$ as activation functions and build upon previous results to show that both analytic functions and functions in Sobolev spaces can be approximated by such networks of constant depth to arbitrary accuracy, demonstrating optimal order approximation rates across all nonlinear approximators, including standard ReLU networks.

Vocal Bursts Intensity Prediction

Improved Projection Learning for Lower Dimensional Feature Maps

no code implementations27 Oct 2022 Ilan Price, Jared Tanner

The requirement to repeatedly move large feature maps off- and on-chip during inference with convolutional neural networks (CNNs) imposes high costs in terms of both energy and time.

Tuning-free multi-coil compressed sensing MRI with Parallel Variable Density Approximate Message Passing (P-VDAMP)

1 code implementation8 Mar 2022 Charles Millard, Mark Chiew, Jared Tanner, Aaron T. Hess, Boris Mailhe

To our knowledge, P-VDAMP is the first algorithm for multi-coil MRI data that obeys a state evolution with accurately tracked parameters.

Activation function design for deep networks: linearity and effective initialisation

1 code implementation17 May 2021 Michael Murray, Vinayak Abrol, Jared Tanner

The activation function deployed in a deep neural network has great influence on the performance of the network at initialisation, which in turn has implications for training.

Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset

2 code implementations12 Feb 2021 Ilan Price, Jared Tanner

We show that standard training of networks built with these layers, and pruned at initialization, achieves state-of-the-art accuracy for extreme sparsities on a variety of benchmark network architectures and datasets.

Mutual Information of Neural Network Initialisations: Mean Field Approximations

no code implementations8 Feb 2021 Jared Tanner, Giuseppe Ughi

The ability to train randomly initialised deep neural networks is known to depend strongly on the variance of the weight matrices and biases as well as the choice of nonlinear activation.

Information Theory Information Theory

An Empirical Study of Derivative-Free-Optimization Algorithms for Targeted Black-Box Attacks in Deep Neural Networks

no code implementations3 Dec 2020 Giuseppe Ughi, Vinayak Abrol, Jared Tanner

We perform a comprehensive study on the performance of derivative free optimization (DFO) algorithms for the generation of targeted black-box adversarial attacks on Deep Neural Network (DNN) classifiers assuming the perturbation energy is bounded by an $\ell_\infty$ constraint and the number of queries to the network is limited.

Compressed sensing of low-rank plus sparse matrices

2 code implementations18 Jul 2020 Jared Tanner, Simon Vary

This manuscript develops similar guarantees showing that $m\times n$ matrices that can be expressed as the sum of a rank-$r$ matrix and a $s$-sparse matrix can be recovered by computationally tractable methods from $\mathcal{O}(r(m+n-r)+s)\log(mn/s)$ linear measurements.

Matrix Completion

Encoder blind combinatorial compressed sensing

no code implementations10 Apr 2020 Michael Murray, Jared Tanner

In this paper we consider the problem of designing a decoder to recover a set of sparse codes from their linear measurements alone, that is without access to encoder matrix.

Community Detection Computational Efficiency +2

Approximate Message Passing with a Colored Aliasing Model for Variable Density Fourier Sampled Images

1 code implementation3 Mar 2020 Charles Millard, Aaron T Hess, Boris Mailhé, Jared Tanner

Central to AMP is its "state evolution", which guarantees that the difference between the current estimate and ground truth (the "aliasing") at every iteration obeys a Gaussian distribution that can be fully characterized by a scalar.

A Model-Based Derivative-Free Approach to Black-Box Adversarial Examples: BOBYQA

1 code implementation24 Feb 2020 Giuseppe Ughi, Vinayak Abrol, Jared Tanner

We demonstrate that model-based derivative free optimisation algorithms can generate adversarial targeted misclassification of deep networks using fewer network queries than non-model-based methods.

Trajectory growth lower bounds for random sparse deep ReLU networks

no code implementations25 Nov 2019 Ilan Price, Jared Tanner

This paper considers the growth in the length of one-dimensional trajectories as they are passed through deep ReLU neural networks, which, among other things, is one measure of the expressivity of deep networks.

An Approximate Message Passing Algorithm for Rapid Parameter-Free Compressed Sensing MRI

1 code implementation4 Nov 2019 Charles Millard, Aaron T Hess, Boris Mailhé, Jared Tanner

In response we present an algorithm based on Orthogonal AMP constructed specifically for variable density partial Fourier sensing matrices.

Trajectory growth through random deep ReLU networks

no code implementations25 Sep 2019 Ilan Price, Jared Tanner

This paper considers the growth in the length of one-dimensional trajectories as they are passed through deep ReLU neural networks, which, among other things, is one measure of the expressivity of deep networks.

Geometric anomaly detection in data

1 code implementation25 Aug 2019 Bernadette J. Stolz, Jared Tanner, Heather A. Harrington, Vidit Nanda

This paper describes the systematic application of local topological methods for detecting interfaces and related anomalies in complicated high-dimensional data.

Algebraic Topology Algebraic Geometry 57N80

Multispectral snapshot demosaicing via non-convex matrix completion

no code implementations28 Feb 2019 Giancarlo A. Antonucci, Simon Vary, David Humphreys, Robert A. Lamb, Jonathan Piper, Jared Tanner

Snapshot mosaic multispectral imagery acquires an undersampled data cube by acquiring a single spectral measurement per spatial pixel.

Demosaicking Matrix Completion

Towards an understanding of CNNs: analysing the recovery of activation pathways via Deep Convolutional Sparse Coding

no code implementations26 Jun 2018 Michael Murray, Jared Tanner

Deep Convolutional Sparse Coding (D-CSC) is a framework reminiscent of deep convolutional neural networks (DCNNs), but by omitting the learning of the dictionaries one can more transparently analyse the role of the activation function and its ability to recover activation paths through the layers.

Parallel multi-scale reduction of persistent homology filtrations

1 code implementation15 Aug 2017 Rodrigo Mendoza-Smith, Jared Tanner

The persistent homology pipeline includes the reduction of a, so-called, boundary matrix.

Algebraic Topology

Cannot find the paper you are looking for? You can Submit a new open access paper.