Search Results for author:
Found 17 papers, 3 papers with code
A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Features
We prove that training neural networks on 1-D data is equivalent to solving a convex Lasso problem with a fixed, explicitly defined dictionary matrix of features.
The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Models
We also show that all the stationary of the nonconvex training objective can be characterized as the global optimum of a subsampled convex program.
Globally Optimal Training of Neural Networks with Threshold Activation Functions
We first show that regularized deep threshold network training problems can be equivalently formulated as a standard convex optimization problem, which parallels the LASSO method, provided that the last hidden layer width exceeds a certain threshold.
Convexifying Transformers: Improving optimization and understanding of transformer networks
To this end, we study the training problem of attention/transformer networks and introduce a novel convex analytic approach to improve the understanding and optimization of these networks.
GLEAM: Greedy Learning for Large-Scale Accelerated MRI Reconstruction
However, they require several iterations of a large neural network to handle high-dimensional imaging tasks such as 3D MRI.
Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers
Vision transformers using self-attention or its proposed alternatives have demonstrated promising results in many image related tasks.
Greedy Learning for Large-Scale Neural MRI Reconstruction
Model-based deep learning approaches have recently shown state-of-the-art performance for accelerated MRI reconstruction.
Parallel Deep Neural Networks Have Zero Duality Gap
Recent work has proven that the strong duality holds (which means zero duality gap) for regularized finite-width two-layer ReLU networks and consequently provided an equivalent convex training problem.
Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programs
We first show that the training of multiple three-layer ReLU sub-networks with weight decay regularization can be equivalently cast as a convex optimization problem in a higher dimensional space, where sparsity is enforced via a group $\ell_1$-norm regularization.
Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions
In this work, we analyze the training of Wasserstein GANs with two-layer neural network discriminators through the lens of convex duality, and for a variety of generators expose the conditions under which Wasserstein GANs can be solved exactly with convex optimization approaches, or can be represented as convex-concave games.
Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization
Batch Normalization (BN) is a commonly used technique to accelerate and stabilize training of deep neural networks.
Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms
We describe the convex semi-infinite dual of the two-layer vector-output ReLU neural network training problem.
Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Time
We study training of Convolutional Neural Networks (CNNs) with ReLU activations and introduce exact convex optimization formulations with a polynomial complexity with respect to the number of data samples, the number of neurons, and data dimension.
Convex Geometry and Duality of Over-parameterized Neural Networks
Our analysis also shows that optimal network parameters can be also characterized as interpretable closed-form formulas in some practically relevant special cases.
Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networks
We develop exact representations of training two-layer neural networks with rectified linear units (ReLUs) in terms of a single convex program with number of variables polynomial in the number of training samples and the number of hidden neurons.
Revealing the Structure of Deep Neural Networks via Convex Duality
We show that a set of optimal hidden layer weights for a norm regularized DNN training problem can be explicitly found as the extreme points of a convex set.
Unsupervised and Semi-supervised Anomaly Detection with LSTM Neural Networks
We investigate anomaly detection in an unsupervised framework and introduce Long Short Term Memory (LSTM) neural network based algorithms.
Semi-supervised Anomaly Detection
Supervised Anomaly Detection
+1
