Search Results for author:
Found 39 papers, 12 papers with code
A Survey of Optimization Methods for Training DL Models: Theoretical Perspective on Convergence and Generalization
In this paper, we provide an extensive summary of the theoretical foundations of optimization methods in DL, including presenting various methodologies, their convergence analyses, and generalization abilities.
AD-L-JEPA: Self-Supervised Spatial World Models with Joint Embedding Predictive Architecture for Autonomous Driving with LiDAR Data
In this paper, we present AD-L-JEPA (aka Autonomous Driving with LiDAR data via a Joint Embedding Predictive Architecture), a novel self-supervised pre-training framework for autonomous driving with LiDAR data that, as opposed to existing methods, is neither generative nor contrastive.
Adjacent Leader Decentralized Stochastic Gradient Descent
This work focuses on the decentralized deep learning optimization framework.
GRAWA: Gradient-based Weighted Averaging for Distributed Training of Deep Learning Models
We study distributed training of deep learning models in time-constrained environments.
TAME: Task Agnostic Continual Learning using Multiple Experts
Our algorithm, which we call TAME (Task-Agnostic continual learning using Multiple Experts), automatically detects the shift in data distributions and switches between task expert networks in an online manner.
ERASE-Net: Efficient Segmentation Networks for Automotive Radar Signals
The core of our approach is the novel detect-then-segment method for raw radar signals.
Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape
In this paper, we study the sharpness of a deep learning (DL) loss landscape around local minima in order to reveal systematic mechanisms underlying the generalization abilities of DL models.
AutoDrop: Training Deep Learning Models with Automatic Learning Rate Drop
Modern deep learning (DL) architectures are trained using variants of the SGD algorithm that is run with a $\textit{manually}$ defined learning rate schedule, i. e., the learning rate is dropped at the pre-defined epochs, typically when the training loss is expected to saturate.
A Theoretical-Empirical Approach to Estimating Sample Complexity of DNNs
It is however unclear how to extend these measures to DNNs and therefore the existing analyses are applicable to simple neural networks, which are not used in practice, e. g., linear or shallow ones or otherwise multi-layer perceptrons.
Overcoming Catastrophic Forgetting via Direction-Constrained Optimization
We study the principal directions of the trajectory of the optimizer after convergence and show that traveling along a few top principal directions can quickly bring the parameters outside the cone but this is not the case for the remaining directions.
Backdoor Attacks on the DNN Interpretation System
In this paper we design a backdoor attack that alters the saliency map produced by the network for an input image only with injected trigger that is invisible to the naked eye while maintaining the prediction accuracy.
SGB: Stochastic Gradient Bound Method for Optimizing Partition Functions
The update of the proposed method, that we refer to as Stochastic Partition Function Bound (SPFB), resembles scaled stochastic gradient descent where the scaling factor relies on a second order term that is however different from the Hessian.
Multi-modal Experts Network for Autonomous Driving
The other is the phenomena of network overfitting to the simplest and most informative input.
Behavior-Guided Reinforcement Learning
We introduce a new approach for comparing reinforcement learning policies, using Wasserstein distances (WDs) in a newly defined latent behavioral space.
Learning to Score Behaviors for Guided Policy Optimization
We introduce a new approach for comparing reinforcement learning policies, using Wasserstein distances (WDs) in a newly defined latent behavioral space.
Leader Stochastic Gradient Descent for Distributed Training of Deep Learning Models: Extension
Finally, we implement an asynchronous version of our algorithm and extend it to the multi-leader setting, where we form groups of workers, each represented by its own local leader (the best performer in a group), and update each worker with a corrective direction comprised of two attractive forces: one to the local, and one to the global leader (the best performer among all workers).
LdSM: Logarithm-depth Streaming Multi-label Decision Trees
In this paper we develop the LdSM algorithm for the construction and training of multi-label decision trees, where in every node of the tree we optimize a novel objective function that favors balanced splits, maintains high class purity of children nodes, and allows sending examples to multiple directions but with a penalty that prevents tree over-growth.
Towards Automated Melanoma Detection with Deep Learning: Data Purification and Augmentation
Melanoma is one of the ten most common cancers in the US.
Adversarial Learning-Based On-Line Anomaly Monitoring for Assured Autonomy
The paper proposes an on-line monitoring framework for continuous real-time safety/security in learning-based control systems (specifically application to a unmanned ground vehicle).
Beyond Backprop: Online Alternating Minimization with Auxiliary Variables
Despite significant recent advances in deep neural networks, training them remains a challenge due to the highly non-convex nature of the objective function.
VisualBackProp for learning using privileged information with CNNs
In many machine learning applications, from medical diagnostics to autonomous driving, the availability of prior knowledge can be used to improve the predictive performance of learning algorithms and incorporate `physical,' `domain knowledge,' or `common sense' concepts into training of machine learning systems as well as verify constraints/properties of the systems.
LSALSA: Accelerated Source Separation via Learned Sparse Coding
We propose an efficient algorithm for the generalized sparse coding (SC) inference problem.
Invertible Autoencoder for domain adaptation
However, it does not explicitly enforce $F_{BA}$ to be an inverse operation to $F_{AB}$.
A Deep Unsupervised Learning Approach Toward MTBI Identification Using Diffusion MRI
Mild traumatic brain injury is a growing public health problem with an estimated incidence of over 1. 7 million people annually in US.
Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car
This eliminates the need for human engineers to anticipate what is important in an image and foresee all the necessary rules for safe driving.
VisualBackProp: efficient visualization of CNNs
We furthermore justify our approach with theoretical arguments and theoretically confirm that the proposed method identifies sets of input pixels, rather than individual pixels, that collaboratively contribute to the prediction.
Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape.
Structured adaptive and random spinners for fast machine learning computations
We consider an efficient computational framework for speeding up several machine learning algorithms with almost no loss of accuracy.
Simultaneous Learning of Trees and Representations for Extreme Classification and Density Estimation
We consider multi-class classification where the predictor has a hierarchical structure that allows for a very large number of labels both at train and test time.
On the boosting ability of top-down decision tree learning algorithm for multiclass classification
We analyze the performance of the top-down multiclass classification algorithm for decision tree learning called LOMtree, recently proposed in the literature Choromanska and Langford (2014) for solving efficiently classification problems with very large number of classes.
Binary embeddings with structured hashed projections
We prove several theoretical results showing that projections via various structured matrices followed by nonlinear mappings accurately preserve the angular distance between input high-dimensional vectors.
Deep learning with Elastic Averaging SGD
We empirically demonstrate that in the deep learning setting, due to the existence of many local optima, allowing more exploration can lead to the improved performance.
The Loss Surfaces of Multilayer Networks
We show that for large-size decoupled networks the lowest critical values of the random loss function form a layered structure and they are located in a well-defined band lower-bounded by the global minimum.
Notes on using Determinantal Point Processes for Clustering with Applications to Text Clustering
In this paper, we compare three initialization schemes for the KMEANS clustering algorithm: 1) random initialization (KMEANSRAND), 2) KMEANS++, and 3) KMEANSD++.
Differentially- and non-differentially-private random decision trees
We consider supervised learning with random decision trees, where the tree construction is completely random.
Logarithmic Time Online Multiclass prediction
We develop top-down tree construction approaches for constructing logarithmic depth trees.
Stochastic Bound Majorization
Recently a majorization method for optimizing partition functions of log-linear models was proposed alongside a novel quadratic variational upper-bound.
Semistochastic Quadratic Bound Methods
Batch methods based on the quadratic bound were recently proposed for this class of problems, and performed favorably in comparison to state-of-the-art techniques.
Majorization for CRFs and Latent Likelihoods
The partition function plays a key role in probabilistic modeling including conditional random fields, graphical models, and maximum likelihood estimation.
