Search Results for author:
Found 34 papers, 7 papers with code
Learning disconnected manifolds: a no GAN's land
Typical architectures of Generative Adversarial Networks make use of a unimodal latent/input distribution transformed by a continuous generator.
The Pitfalls of Memorization: When Memorization Hurts Generalization
Neural networks often learn simple explanations that fit the majority of the data while memorizing exceptions that deviate from these explanations. This behavior leads to poor generalization when the learned explanations rely on spurious correlations.
Strong Model Collapse
Within the scaling laws paradigm, which underpins the training of large neural networks like ChatGPT and Llama, we consider a supervised regression setting and establish the existance of a strong form of the model collapse phenomenon, a critical performance degradation due to synthetic data in the training corpus.
An Effective Theory of Bias Amplification
Machine learning models may capture and amplify biases present in data, leading to disparate test performance across social groups.
Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification
Large Language Models (LLM) are increasingly trained on data generated by other LLM, either because generated text and images become part of the pre-training corpus, or because synthetized data is used as a replacement for expensive human-annotation.
Model Collapse Demystified: The Case of Regression
In the era of proliferation of large language and image generation models, the phenomenon of "model collapse" refers to the situation whereby as a model is trained recursively on data generated from previous generations of itself over time, its performance degrades until the model eventually becomes completely useless, i. e the model collapses.
A Tale of Tails: Model Collapse as a Change of Scaling Laws
We discover a wide range of decay phenomena, analyzing loss of scaling, shifted scaling with number of generations, the ''un-learning" of skills, and grokking when mixing human and synthesized data.
Scaling Laws for Associative Memories
Learning arguably involves the discovery and memorization of abstract rules.
Robust Linear Regression: Phase-Transitions and Precise Tradeoffs for General Norms
In this paper, we investigate the impact of test-time adversarial attacks on linear regression models and determine the optimal level of robustness that any model can reach while maintaining a given level of standard predictive performance (accuracy).
Robust Linear Regression: Gradient-descent, Early-stopping, and Beyond
However, we show that this strategy can be arbitrarily sub-optimal in the case of general Mahalanobis attacks.
An Adversarial Robustness Perspective on the Topology of Neural Networks
In this paper, we investigate the impact of neural networks (NNs) topology on adversarial robustness.
Contextual bandits with concave rewards, and an application to fair ranking
We consider Contextual Bandits with Concave Rewards (CBCR), a multi-objective bandit problem where the desired trade-off between the rewards is defined by a known concave objective function, and the reward vector depends on an observed stochastic context.
Fast online ranking with fairness of exposure
In this paper, we propose the first efficient online algorithm to optimize concave objective functions in the space of rankings which applies to every concave and smooth objective function, such as the ones found for fairness of exposure.
Scalable MCMC Sampling for Nonsymmetric Determinantal Point Processes
In this work, we develop a scalable MCMC sampling algorithm for $k$-NDPPs with low-rank kernels, thus enabling runtime that is sublinear in $n$.
Origins of Low-dimensional Adversarial Perturbations
Finally, we show that if a decision-region is compact, then it admits a universal adversarial perturbation with $L_2$ norm which is $\sqrt{d}$ times smaller than the typical $L_2$ norm of a data point.
On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimes
To better understand these factors, we provide a precise study of the adversarial robustness in different scenarios, from initialization to the end of training in different regimes, as well as intermediate scenarios, where initialization still plays a role due to "lazy" training.
Scalable Sampling for Nonsymmetric Determinantal Point Processes
However, existing work leaves open the question of scalable NDPP sampling.
Fundamental tradeoffs between memorization and robustness in random features and neural tangent regimes
More precisely, if $n$ is the number of training examples, $d$ is the input dimension, and $k$ is the number of hidden neurons in a two-layer neural network, we prove for a large class of activation functions that, if the model memorizes even a fraction of the training, then its Sobolev-seminorm is lower-bounded by (i) $\sqrt{n}$ in case of infinite-width random features (RF) or neural tangent kernel (NTK) with $d \gtrsim n$; (ii) $\sqrt{n}$ in case of finite-width RF with proportionate scaling of $d$ and $k$; and (iii) $\sqrt{n/k}$ in case of finite-width NTK with proportionate scaling of $d$ and $k$.
On the Convergence of Smooth Regularized Approximate Value Iteration Schemes
Entropy regularization, smoothing of Q-values and neural network function approximator are key components of the state-of-the-art reinforcement learning (RL) algorithms, such as Soft Actor-Critic~\cite{haarnoja2018soft}.
Implicit bias of any algorithm: bounding bias via margin
We measure the quality of such a hyperplane by its margin $\gamma(w)$, defined as minimum distance between any of the points $x_i$ and the hyperplane.
Scalable Learning and MAP Inference for Nonsymmetric Determinantal Point Processes
Determinantal point processes (DPPs) have attracted significant attention in machine learning for their ability to model subsets drawn from a large item collection.
Classifier-independent Lower-Bounds for Adversarial Robustness
(1) We use optimal transport theory to derive variational formulae for the Bayes-optimal error a classifier can make on a given classification problem, subject to adversarial attacks.
Learning disconnected manifolds: a no GANs land
Typical architectures of Generative AdversarialNetworks make use of a unimodal latent distribution transformed by a continuous generator.
On the Convergence of Approximate and Regularized Policy Iteration Schemes
Entropy regularized algorithms such as Soft Q-learning and Soft Actor-Critic, recently showed state-of-the-art performance on a number of challenging reinforcement learning (RL) tasks.
Adversarial Robustness via Label-Smoothing
We study Label-Smoothing as a means for improving adversarial robustness of supervised deep-learning models.
Distributionally Robust Counterfactual Risk Minimization
This manuscript introduces the idea of using Distributionally Robust Optimization (DRO) for the Counterfactual Risk Minimization (CRM) problem.
Learning Nonsymmetric Determinantal Point Processes
Our method imposes a particular decomposition of the nonsymmetric kernel that enables such tractable learning algorithms, which we analyze both theoretically and experimentally.
Distributionally Robust Reinforcement Learning
Our formulation results in a efficient algorithm that accounts for a simple re-weighting of policy actions in the standard policy iteration scheme.
Deep Determinantal Point Processes
While DPPs have substantial expressive power, they are fundamentally limited by the parameterization of the kernel matrix and their inability to capture nonlinear interactions between items within sets.
Generalized No Free Lunch Theorem for Adversarial Robustness
This manuscript presents some new impossibility results on adversarial robustness in machine learning, a very important yet largely open problem.
Learning brain regions via large-scale online structured sparse dictionary learning
We propose a multivariate online dictionary-learning method for obtaining decompositions of brain images with structured and sparse components (aka atoms).
Continuation of Nesterov's Smoothing for Regression with Structured Sparsity in High-Dimensional Neuroimaging
Nesterov's smoothing technique can be used to minimize a large number of non-smooth convex structured penalties but reasonable precision requires a small smoothing parameter, which slows down the convergence speed.
FAASTA: A fast solver for total-variation regularization of ill-conditioned problems with application to brain imaging
The total variation (TV) penalty, as many other analysis-sparsity problems, does not lead to separable factors or a proximal operatorwith a closed-form expression, such as soft thresholding for the $\ell\_1$ penalty.
Region segmentation for sparse decompositions: better brain parcellations from rest fMRI
Functional Magnetic Resonance Images acquired during resting-state provide information about the functional organization of the brain through measuring correlations between brain areas.
