Search Results for author:
Found 81 papers, 46 papers with code
DiscLDA: Discriminative Learning for Dimensionality Reduction and Classification
By using the transformed topic mixture proportions as a new representation of documents, we obtain a supervised dimensionality reduction algorithm that uncovers the latent structure in a document collection while preserving predictive power for the task of classification.
Gaussian Probabilities and Expectation Propagation
We consider these unexpected results empirically and theoretically, both for the problem of Gaussian probabilities and for EP more generally.
SiGMa: Simple Greedy Matching for Aligning Large Knowledge Bases
The Internet has enabled the creation of a growing number of large-scale knowledge bases in a variety of domains containing complementary information.
SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
In this work we introduce a new optimisation method called SAGA in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates.
On Pairwise Costs for Network Flow Multi-Object Tracking
Multi-object tracking has been recently approached with the min-cost network flow optimization techniques.
Sequential Kernel Herding: Frank-Wolfe Optimization for Particle Filtering
Recently, the Frank-Wolfe optimization algorithm was suggested as a procedure to obtain adaptive quadrature rules for integrals of functions in a reproducing kernel Hilbert space (RKHS) with a potentially faster rate of convergence than Monte Carlo integration (and "kernel herding" was shown to be a special case of this procedure).
Variance Reduced Stochastic Gradient Descent with Neighbors
As a side-product we provide a unified convergence analysis for a family of variance reduction algorithms, which we call memorization algorithms.
Unsupervised Learning from Narrated Instruction Videos
Third, we experimentally demonstrate that the proposed method can automatically discover, in an unsupervised manner, the main steps to achieve the task and locate the steps in the input videos.
Ranked #7 on
Temporal Action Localization
on CrossTask
Rethinking LDA: moment matching for discrete ICA
We consider moment matching techniques for estimation in Latent Dirichlet Allocation (LDA).
Barrier Frank-Wolfe for Marginal Inference
We introduce a globally-convergent algorithm for optimizing the tree-reweighted (TRW) variational objective over the marginal polytope.
On the Global Linear Convergence of Frank-Wolfe Optimization Variants
In this paper, we highlight and clarify several variants of the Frank-Wolfe optimization algorithm that have been successfully applied in practice: away-steps FW, pairwise FW, fully-corrective FW and Wolfe's minimum norm point algorithm, and prove for the first time that they all enjoy global linear convergence, under a weaker condition than strong convexity of the objective.
Beyond CCA: Moment Matching for Multi-View Models
We introduce three novel semi-parametric extensions of probabilistic canonical correlation analysis with identifiability guarantees.
PAC-Bayesian Theory Meets Bayesian Inference
That is, for the negative log-likelihood loss function, we show that the minimization of PAC-Bayesian generalization risk bounds maximizes the Bayesian marginal likelihood.
Minding the Gaps for Block Frank-Wolfe Optimization of Structured SVMs
In this paper, we propose several improvements on the block-coordinate Frank-Wolfe (BCFW) algorithm from Lacoste-Julien et al. (2013) recently used to optimize the structured support vector machine (SSVM) objective in the context of structured prediction, though it has wider applications.
ASAGA: Asynchronous Parallel SAGA
We describe ASAGA, an asynchronous parallel version of the incremental gradient algorithm SAGA that enjoys fast linear convergence rates.
Convergence Rate of Frank-Wolfe for Non-Convex Objectives
We give a simple proof that the Frank-Wolfe algorithm obtains a stationary point at a rate of $O(1/\sqrt{t})$ on non-convex objectives with a Lipschitz continuous gradient.
Frank-Wolfe Algorithms for Saddle Point Problems
We extend the Frank-Wolfe (FW) optimization algorithm to solve constrained smooth convex-concave saddle point (SP) problems.
Joint Discovery of Object States and Manipulation Actions
We assume a consistent temporal order for the changes in object states and manipulation actions, and introduce new optimization techniques to learn model parameters without additional supervision.
On Structured Prediction Theory with Calibrated Convex Surrogate Losses
We provide novel theoretical insights on structured prediction in the context of efficient convex surrogate loss minimization with consistency guarantees.
SEARNN: Training RNNs with Global-Local Losses
We propose SEARNN, a novel training algorithm for recurrent neural networks (RNNs) inspired by the "learning to search" (L2S) approach to structured prediction.
A Closer Look at Memorization in Deep Networks
We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness.
Breaking the Nonsmooth Barrier: A Scalable Parallel Method for Composite Optimization
Due to their simplicity and excellent performance, parallel asynchronous variants of stochastic gradient descent have become popular methods to solve a wide range of large-scale optimization problems on multi-core architectures.
Parametric Adversarial Divergences are Good Losses for Generative Modeling
Parametric adversarial divergences, which are a generalization of the losses used to train generative adversarial networks (GANs), have often been described as being approximations of their nonparametric counterparts, such as the Jensen-Shannon divergence, which can be derived under the so-called optimal discriminator assumption.
Adaptive Stochastic Dual Coordinate Ascent for Conditional Random Fields
In this paper, we adapt SDCA to train CRFs, and we enhance it with an adaptive non-uniform sampling strategy based on block duality gaps.
Improved asynchronous parallel optimization analysis for stochastic incremental methods
Notably, we prove that ASAGA and KROMAGNON can obtain a theoretical linear speedup on multi-core systems even without sparsity assumptions.
A3T: Adversarially Augmented Adversarial Training
Recent research showed that deep neural networks are highly sensitive to so-called adversarial perturbations, which are tiny perturbations of the input data purposely designed to fool a machine learning classifier.
A Variational Inequality Perspective on Generative Adversarial Networks
Generative adversarial networks (GANs) form a generative modeling approach known for producing appealing samples, but they are notably difficult to train.
Frank-Wolfe Splitting via Augmented Lagrangian Method
In this work, we develop and analyze the Frank-Wolfe Augmented Lagrangian (FW-AL) algorithm, a method for minimizing a smooth function over convex compact sets related by a "linear consistency" constraint that only requires access to a linear minimization oracle over the individual constraints.
Negative Momentum for Improved Game Dynamics
Games generalize the single-objective optimization paradigm by introducing different objective functions for different players.
Predicting Tactical Solutions to Operational Planning Problems under Imperfect Information
We aim to predict at a high speed the expected TDOS associated with the second stage problem, conditionally on the first stage variables.
Scattering Networks for Hybrid Representation Learning
In particular, by working in scattering space, we achieve competitive results both for supervised and unsupervised learning tasks, while making progress towards constructing more interpretable CNNs.
A Modern Take on the Bias-Variance Tradeoff in Neural Networks
The bias-variance tradeoff tells us that as model complexity increases, bias falls and variances increases, leading to a U-shaped test error curve.
Quantifying Learning Guarantees for Convex but Inconsistent Surrogates
We study consistency properties of machine learning methods based on minimizing convex surrogates.
Predicting Tactical Solutions to Operational Planning Problems under Imperfect Information
We formulate the problem as a two-stage optimal prediction stochastic program whose solution we predict with a supervised machine learning algorithm.
Are Few-Shot Learning Benchmarks too Simple ? Solving them without Task Supervision at Test-Time
We show that several popular few-shot learning benchmarks can be solved with varying degrees of success without using support set Labels at Test-time (LT).
Reducing Noise in GAN Training with Variance Reduced Extragradient
We study the effect of the stochastic gradient noise on the training of generative adversarial networks (GANs) and show that it can prevent the convergence of standard game optimization methods, while the batch version converges.
Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks
When optimizing over-parameterized models, such as deep neural networks, a large set of parameters can achieve zero training error.
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates
To improve the proposed methods' practical performance, we give heuristics to use larger step-sizes and acceleration.
Gradient-Based Neural DAG Learning
We propose a novel score-based approach to learning a directed acyclic graph (DAG) from observational data.
A Closer Look at the Optimization Landscapes of Generative Adversarial Networks
Generative adversarial networks have been very successful in generative modeling, however they remain relatively challenging to train compared to standard deep neural networks.
A Tight and Unified Analysis of Gradient-Based Methods for a Whole Spectrum of Games
We provide new analyses of the EG's local and global convergence properties and use is to get a tighter global convergence rate for OG and CO. Our analysis covers the whole range of settings between bilinear and strongly monotone games.
GAIT: A Geometric Approach to Information Theory
We advocate the use of a notion of entropy that reflects the relative abundances of the symbols in an alphabet, as well as the similarities between them.
Are Few-shot Learning Benchmarks Too Simple ?
We argue that the widely used Omniglot and miniImageNet benchmarks are too simple because their class semantics do not vary across episodes, which defeats their intended purpose of evaluating few-shot classification methods.
Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation
Under this condition, we show that the regularized subsampled Newton method (R-SSN) achieves global linear convergence with an adaptive step-size and a constant batch-size.
Accelerating Smooth Games by Manipulating Spectral Shapes
Using this perspective, we propose an optimal algorithm for bilinear games.
Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence
Consequently, the proposed stochastic Polyak step-size (SPS) is an attractive choice for setting the learning rate for stochastic gradient descent (SGD).
An Analysis of the Adaptation Speed of Causal Models
When the intervention is on the effect variable, we characterize the relative adaptation speed.
Adaptive Gradient Methods Converge Faster with Over-Parameterization (but you should do a line-search)
In this setting, we prove that AMSGrad with constant step-size and momentum converges to the minimizer at a faster $O(1/T)$ rate.
To Each Optimizer a Norm, To Each Norm its Generalization
For under-parameterized linear classification, we prove that for any linear classifier separating the data, there exists a family of quadratic norms ||.||_P such that the classifier's direction is the same as that of the maximum P-margin solution.
Adversarial Example Games
We introduce Adversarial Example Games (AEG), a framework that models the crafting of adversarial examples as a min-max game between a generator of attacks and a classifier.
Differentiable Causal Discovery from Interventional Data
This work constitutes a new step in this direction by proposing a theoretically-grounded method based on neural networks that can leverage interventional data.
Stochastic Hamiltonian Gradient Methods for Smooth Games
The success of adversarial formulations in machine learning has brought renewed motivation for smooth games.
Implicit Regularization via Neural Feature Alignment
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint.
Flight-connection Prediction for Airline Crew Scheduling to Construct Initial Clusters for OR Optimizer
We present a case study of using machine learning classification algorithms to initialize a large-scale commercial solver (GENCOL) based on column generation in the context of the airline crew pairing problem, where small savings of as little as 1% translate to increasing annual revenue by dozens of millions of dollars in a large airline.
Adaptive Gradient Methods Converge Faster with Over-Parameterization (and you can do a line-search)
Under an interpolation assumption, we prove that AMSGrad with a constant step-size and momentum can converge to the minimizer at the faster $O(1/T)$ rate for smooth, convex functions.
Machine Learning in Airline Crew Pairing to Construct Initial Clusters for Dynamic Constraint Aggregation
The crew pairing problem (CPP) is generally modelled as a set partitioning problem where the flights have to be partitioned in pairings.
Geometry-Aware Universal Mirror-Prox
Mirror-prox (MP) is a well-known algorithm to solve variational inequality (VI) problems.
On the Convergence of Continuous Constrained Optimization for Structure Learning
Recently, structure learning of directed acyclic graphs (DAGs) has been formulated as a continuous optimization problem by leveraging an algebraic characterization of acyclicity.
SVRG Meets AdaGrad: Painless Variance Reduction
Variance reduction (VR) methods for finite-sum minimization typically require the knowledge of problem-dependent constants that are often unknown and difficult to estimate.
Online Adversarial Attacks
Adversarial attacks expose important vulnerabilities of deep learning models, yet little attention has been paid to settings where data arrives as a stream.
Repurposing Pretrained Models for Robust Out-of-domain Few-Shot Learning
Model-agnostic meta-learning (MAML) is a popular method for few-shot learning but assumes that we have access to the meta-training set.
Structured Convolutional Kernel Networks for Airline Crew Scheduling
Motivated by the needs from an airline crew scheduling application, we introduce structured convolutional kernel networks (Struct-CKN), which combine CKNs from Mairal et al. (2014) in a structured prediction framework that supports constraints on the outputs.
Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity
Two of the most prominent algorithms for solving unconstrained smooth games are the classical stochastic gradient descent-ascent (SGDA) and the recently introduced stochastic consensus optimization (SCO) [Mescheder et al., 2017].
Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA
This work introduces a novel principle we call disentanglement via mechanism sparsity regularization, which can be applied when the latent factors of interest depend sparsely on past latent factors and/or observed auxiliary variables.
A Survey of Self-Supervised and Few-Shot Object Detection
Labeling data is often expensive and time-consuming, especially for tasks such as object detection and instance segmentation, which require dense labeling of the image.
Convergence Rates for the MAP of an Exponential Family and Stochastic Mirror Descent -- an Open Problem
We consider the problem of upper bounding the expected log-likelihood sub-optimality of the maximum likelihood estimate (MLE), or a conjugate maximum a posteriori (MAP) for an exponential family, in a non-asymptotic way.
Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation
Most set prediction models in deep learning use set-equivariant operations, but they actually operate on multisets.
Bayesian Structure Learning with Generative Flow Networks
In Bayesian structure learning, we are interested in inferring a distribution over the directed acyclic graph (DAG) structure of Bayesian networks, from data.
Data-Efficient Structured Pruning via Submodular Optimization
Structured pruning is an effective approach for compressing large pre-trained neural networks without significantly affecting their performance.
Partial Disentanglement via Mechanism Sparsity
In this work, we introduce a generalization of this theory which applies to any ground-truth graph and specifies qualitatively how disentangled the learned representation is expected to be, via a new equivalence relation over models we call consistency.
Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints
The performance of trained neural networks is robust to harsh levels of pruning.
Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning
Although disentangled representations are often said to be beneficial for downstream tasks, current empirical and theoretical understanding is limited.
CrossSplit: Mitigating Label Noise Memorization through Data Splitting
We approach the problem of improving robustness of deep learning algorithms in the presence of label noise.
Unlocking Slot Attention by Changing Optimal Transport Costs
Slot attention is a powerful method for object-centric modeling in images and videos.
Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?
Pretraining a neural network on a large dataset is becoming a cornerstone in machine learning that is within the reach of only a few communities with large-resources.
PopulAtion Parameter Averaging (PAPA)
Based on this idea, we propose PopulAtion Parameter Averaging (PAPA): a method that combines the generality of ensembling with the efficiency of weight averaging.
On the Identifiability of Quantized Factors
We introduce this novel form of identifiability, termed quantized factor identifiability, and provide a comprehensive proof of the recovery of the quantized factors.
Promoting Exploration in Memory-Augmented Adam using Critical Momenta
Adaptive gradient-based optimizers, particularly Adam, have left their mark in training large-scale deep learning models.
Balancing Act: Constraining Disparate Impact in Sparse Models
Model pruning is a popular approach to enable the deployment of large deep learning models on edge devices with restricted computational or storage capacities.
Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependencies
We develop a nonparametric identifiability theory that formalizes this principle and shows that the latent factors can be recovered by regularizing the learned causal graph to be sparse.
