Search Results for author:
Found 89 papers, 10 papers with code
Streaming Submodular Maximization under a k-Set System Constraint
Moreover, we propose the first streaming algorithms for monotone submodular maximization subject to $k$-extendible and $k$-system constraints.
Replicable Clustering
In this paper, we design replicable algorithms in the context of statistical clustering under the recently introduced notion of replicability.
KDEformer: Accelerating Transformers via Kernel Density Estimation
Dot-product attention mechanism plays a crucial role in modern deep architectures (e. g., Transformer) for sequence modeling, however, na\"ive exact computation of this model incurs quadratic time and memory complexities in sequence length, hindering the training of long-sequence models.
The Impossibility of Parallelizing Boosting
The aim of boosting is to convert a sequence of weak learners into a strong learner.
Exact Gradient Computation for Spiking Neural Networks Through Forward Propagation
their weights, and (2) we propose a novel training algorithm, called \emph{forward propagation} (FP), that computes exact gradients for SNN.
On Optimal Learning Under Targeted Data Poisoning
In this work we aim to characterize the smallest achievable error $\epsilon=\epsilon(\eta)$ by the learner in the presence of such an adversary in both realizable and agnostic settings.
Multiclass Learnability Beyond the PAC Framework: Universal Rates and Partial Concept Classes
Second, we consider the problem of multiclass classification with structured data (such as data lying on a low dimensional manifold or satisfying margin conditions), a setting which is captured by partial concept classes (Alon, Hanneke, Holzman and Moran, FOCS '21).
Replicable Bandits
Similarly, for stochastic linear bandits (with finitely and infinitely many arms) we develop replicable policies that achieve the best-known problem-independent regret bounds with an optimal dependency on the replicability parameter.
Fast Neural Kernel Embeddings for General Activations
Moreover, most prior works on neural kernels have focused on the ReLU activation, mainly due to its popularity but also due to the difficulty of computing such kernels for general activations.
Scalable MCMC Sampling for Nonsymmetric Determinantal Point Processes
In this work, we develop a scalable MCMC sampling algorithm for $k$-NDPPs with low-rank kernels, thus enabling runtime that is sublinear in $n$.
Self-Consistency of the Fokker-Planck Equation
In this paper, we exploit this concept to design a potential function of the hypothesis velocity fields, and prove that, if such a function diminishes to zero during the training procedure, the trajectory of the densities generated by the hypothesis velocity fields converges to the solution of the FPE in the Wasserstein-2 sense.
Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD
For nonconvex smooth losses, we prove that full-batch GD efficiently generalizes close to any stationary point at termination, and recovers the generalization error guarantees of stochastic algorithms with fewer assumptions.
Learning Distributionally Robust Models at Scale via Composite Optimization
To train machine learning models that are robust to distribution shifts in the data, distributionally robust optimization (DRO) has been proven very effective.
The Best of Both Worlds: Reinforcement Learning with Logarithmic Regret and Policy Switches
In this paper, we study the problem of regret minimization for episodic Reinforcement Learning (RL) both in the model-free and the model-based setting.
What Functions Can Graph Neural Networks Generate?
We prove that: (i) a GNN, as a graph function, is necessarily permutation compatible; (ii) conversely, any permutation compatible function, when restricted on input graphs with distinct node features, can be generated by a GNN; (iii) for arbitrary node features (not necessarily distinct), a simple feature augmentation scheme suffices to generate a permutation compatible function by a GNN; (iv) permutation compatibility can be verified by checking only quadratically many functional constraints, rather than an exhaustive search over all the permutations; (v) GNNs can generate \textit{any} graph function once we augment the node features with node identities, thus going beyond graph isomorphism and permutation compatibility.
Black-Box Generalization: Stability of Zeroth-Order Learning
These bounds coincide with those for SGD, and rather surprisingly are independent of $d$, $K$ and the batch size $m$, under appropriate choices of a slightly decreased learning rate.
Scalable Sampling for Nonsymmetric Determinantal Point Processes
However, existing work leaves open the question of scalable NDPP sampling.
Data-driven mapping between functional connectomes using optimal transport
Being able to map connectomes and derived results between different atlases without additional pre-processing is a crucial step in improving interpretation and generalization between studies that use different atlases.
An Exponential Improvement on the Memorization Capacity of Deep Threshold Networks
Recently, Vershynin (2020) settled a long standing question by Baum (1988), proving that \emph{deep threshold} networks can memorize $n$ points in $d$ dimensions using $\widetilde{\mathcal{O}}(e^{1/\delta^2}+\sqrt{n})$ neurons and $\widetilde{\mathcal{O}}(e^{1/\delta^2}(d+\sqrt{n})+n)$ weights, where $\delta$ is the minimum distance between the points.
Submodular + Concave
It has been well established that first order optimization methods can converge to the maximal objective value of concave functions and provide constant factor approximation guarantees for (non-convex/non-concave) continuous submodular functions.
Parallelizing Thompson Sampling
How can we make use of information parallelism in online decision making problems while efficiently balancing the exploration-exploitation trade-off?
Learning and Certification under Instance-targeted Poisoning
In this paper, we study PAC learnability and certification of predictions under instance-targeted poisoning attacks, where the adversary who knows the test instance may change a fraction of the training set with the goal of fooling the learner at the test instance.
The Power of Subsampling in Submodular Maximization
We propose subsampling as a unified algorithmic technique for submodular maximization in centralized and online settings.
Federated Functional Gradient Boosting
First, in the semi-heterogeneous setting, when the marginal distributions of the feature vectors on client machines are identical, we develop the federated functional gradient boosting (FFGB) method that provably converges to the global minimum.
Batched Neural Bandits
In many sequential decision-making problems, the individuals are split into several batches and the decision-maker is only allowed to change her policy at the end of batches.
Online MAP Inference of Determinantal Point Processes
In this paper, we provide an efficient approximation algorithm for finding the most likelihood configuration (MAP) of size $k$ for Determinantal Point Processes (DPP) in the online setting where the data points arrive in an arbitrary order and the algorithm cannot discard the selected elements from its local memory.
How Do You Want Your Greedy: Simultaneous or Repeated?
We also present SubmodularGreedy. jl, a Julia package which implements these algorithms and may be downloaded at https://github. com/crharshaw/SubmodularGreedy. jl .
Multiple Descent: Design Your Own Generalization Curve
This paper explores the generalization loss of linear regression in variably parameterized families of models, both under-parameterized and over-parameterized.
Safe Learning under Uncertain Objectives and Constraints
More precisely, by assuming that Reliable-FW has access to a (stochastic) gradient oracle of the objective function and a noisy feasibility oracle of the safety polytope, it finds an $\epsilon$-approximate first-order stationary point with the optimal ${\mathcal{O}}({1}/{\epsilon^2})$ gradient oracle complexity (resp.
Continuous Submodular Maximization: Beyond DR-Submodularity
We first prove that a simple variant of the vanilla coordinate ascent, called Coordinate-Ascent+, achieves a $(\frac{e-1}{2e-1}-\varepsilon)$-approximation guarantee while performing $O(n/\varepsilon)$ iterations, where the computational complexity of each iteration is roughly $O(n/\sqrt{\varepsilon}+n\log n)$ (here, $n$ denotes the dimension of the optimization problem).
Meta Learning in the Continuous Time Limit
In this paper, we establish the ordinary differential equation (ODE) that underlies the training dynamics of Model-Agnostic Meta-Learning (MAML).
Submodularity in Action: From Machine Learning to Signal Processing Applications
We introduce a variety of submodular-friendly applications, and elucidate the relation of submodularity to convexity and concavity which enables efficient optimization.
Submodular Maximization in Clean Linear Time
In this paper, we provide the first deterministic algorithm that achieves the tight $1-1/e$ approximation guarantee for submodular maximization under a cardinality (size) constraint while making a number of queries that scales only linearly with the size of the ground set $n$.
The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization
In the medium adversary regime, with more training data, the generalization loss exhibits a double descent curve, which implies the existence of an intermediate stage where more training data hurts the generalization.
More Data Can Expand the Generalization Gap Between Adversarially Robust and Standard Models
Despite remarkable success in practice, modern machine learning models have been found to be susceptible to adversarial attacks that make human-imperceptible perturbations to the data, but result in serious and potentially dangerous prediction errors.
Regularized Submodular Maximization at Scale
In this paper, we propose scalable methods for maximizing a regularized submodular function $f = g - \ell$ expressed as the difference between a monotone submodular function $g$ and a modular function $\ell$.
Submodular Maximization Through Barrier Functions
In this paper, we introduce a novel technique for constrained submodular maximization, inspired by barrier functions in continuous optimization.
Streaming Submodular Maximization under a $k$-Set System Constraint
In this paper, we propose a novel framework that converts streaming algorithms for monotone submodular maximization into streaming algorithms for non-monotone submodular maximization.
Stochastic Continuous Greedy ++: When Upper and Lower Bounds Match
Concretely, for a monotone and continuous DR-submodular function, \SCGPP achieves a tight $[(1-1/e)\OPT -\epsilon]$ solution while using $O(1/\epsilon^2)$ stochastic gradients and $O(1/\epsilon)$ calls to the linear optimization oracle.
Adaptivity in Adaptive Submodularity
We propose an efficient semi adaptive policy that with $O(\log n \times\log k)$ adaptive rounds of observations can achieve an almost tight $1-1/e-\epsilon$ approximation guarantee with respect to an optimal policy that carries out $k$ actions in a fully sequential manner.
Online Continuous Submodular Maximization: From Full-Information to Bandit Feedback
In this paper, we propose three online algorithms for submodular maximisation.
Minimax Regret of Switching-Constrained Online Convex Optimization: No Phase Transition
To establish the dimension-independent upper bound, we next show that a mini-batching algorithm provides an $ O(\frac{T}{\sqrt{K}}) $ upper bound, and therefore conclude that the minimax regret of switching-constrained OCO is $ \Theta(\frac{T}{\sqrt{K}}) $ for any $K$.
Regret Bounds for Batched Bandits
We present simple and efficient algorithms for the batched stochastic multi-armed bandit and batched stochastic linear bandit problems.
One Sample Stochastic Frank-Wolfe
One of the beauties of the projected gradient descent method lies in its rather simple mechanism and yet stable behavior with inexact, stochastic gradients, which has led to its wide-spread use in many machine learning applications.
Submodular Streaming in All its Glory: Tight Approximation, Minimum Memory and Low Adaptive Complexity
We show how one can achieve the tight $(1/2)$-approximation guarantee with $O(k)$ shared memory while minimizing not only the required rounds of computations but also the total number of communicated bits.
Submodular Maximization Beyond Non-negativity: Guarantees, Fast Algorithms, and Applications
It is generally believed that submodular functions -- and the more general class of $\gamma$-weakly submodular functions -- may only be optimized under the non-negativity assumption $f(S) \geq 0$.
Stochastic Conditional Gradient++
It is known that this rate is optimal in terms of stochastic gradient evaluations.
Quantized Frank-Wolfe: Faster Optimization, Lower Communication, and Projection Free
How can we efficiently mitigate the overhead of gradient communications in distributed optimization?
Adaptive Sequence Submodularity
In many machine learning applications, one needs to interactively select a sequence of items (e. g., recommending movies based on a user's feedback) or make sequential decisions in a certain order (e. g., guiding an agent through a series of states).
Black Box Submodular Maximization: Discrete and Continuous Settings
In this paper, we consider the problem of black box continuous submodular maximization where we only have access to the function values and no information about the derivatives is provided.
Unconstrained Submodular Maximization with Constant Adaptive Complexity
In this paper, we consider the unconstrained submodular maximization problem.
Eliminating Latent Discrimination: Train Then Mask
How can we control for latent discrimination in predictive models?
Scalable Deletion-Robust Submodular Maximization: Data Summarization with Privacy and Fairness Constraints
Can we efficiently extract useful information from a large user-generated dataset while protecting the privacy of the users and/or ensuring fairness in representation?
Data Summarization at Scale: A Two-Stage Submodular Approach
The sheer scale of modern datasets has resulted in a dire need for summarization techniques that identify representative elements in a dataset.
Projection-Free Bandit Convex Optimization
In this paper, we propose the first computationally efficient projection-free algorithm for bandit convex optimization (BCO).
Stochastic Conditional Gradient Methods: From Convex Minimization to Submodular Maximization
Further, for a monotone and continuous DR-submodular function and subject to a general convex body constraint, we prove that our proposed method achieves a $((1-1/e)OPT-\eps)$ guarantee with $O(1/\eps^3)$ stochastic gradient computations.
Projection-Free Online Optimization with Stochastic Gradient: From Convexity to Submodularity
We also propose One-Shot Frank-Wolfe, a simpler algorithm which requires only a single stochastic gradient estimate in each round and achieves an $O(T^{2/3})$ stochastic regret bound for convex and continuous submodular optimization.
Do Less, Get More: Streaming Submodular Maximization with Subsampling
In this paper, we develop the first one-pass streaming algorithm for submodular maximization that does not evaluate the entire stream even once.
Comparison Based Learning from Weak Oracles
More specifically, we aim at devising efficient algorithms to locate a target object in a database equipped with a dissimilarity metric via invocation of the weak comparison oracle.
Online Continuous Submodular Maximization
For such settings, we then propose an online stochastic gradient ascent algorithm that also achieves a regret bound of $O(\sqrt{T})$ regret, albeit against a weaker $1/2$-approximation to the best feasible solution in hindsight.
Interactive Submodular Bandit
We then receive a noisy feedback about the utility of the action (e. g., ratings) which we model as a submodular function over the context-action space.
Deletion-Robust Submodular Maximization at Scale
Can we efficiently extract useful information from a large user-generated dataset while protecting the privacy of the users and/or ensuring fairness in representation.
Conditional Gradient Method for Stochastic Submodular Maximization: Closing the Gap
More precisely, for a monotone and continuous DR-submodular function and subject to a \textit{general} convex body constraint, we prove that \alg achieves a $[(1-1/e)\text{OPT} -\eps]$ guarantee (in expectation) with $\mathcal{O}{(1/\eps^3)}$ stochastic gradient computations.
Gradient Methods for Submodular Maximization
Despite the apparent lack of convexity in such functions, we prove that stochastic projected gradient methods can provide strong approximation guarantees for maximizing continuous submodular functions with convex constraints.
Differentially Private Submodular Maximization: Data Summarization in Disguise
Many data summarization applications are captured by the general framework of submodular maximization.
Deletion-Robust Submodular Maximization: Data Summarization with "the Right to be Forgotten"
How can we summarize a dynamic data stream when elements selected for the summary can be deleted at any time?
Probabilistic Submodular Maximization in Sub-Linear Time
As a remedy, we introduce the problem of sublinear time probabilistic submodular maximization: Given training examples of functions (e. g., via user feature vectors), we seek to reduce the ground set so that optimizing new functions drawn from the same distribution will provide almost as much value when restricted to the reduced ground set as when using the full set.
Weakly Submodular Maximization Beyond Cardinality Constraints: Does Randomization Help Greedy?
In this paper, we prove that a randomized version of the greedy algorithm (previously used by Buchbinder et al. (2014) for a different problem) achieves an approximation ratio of $(1 + 1/\gamma)^{-2}$ for the maximization of a weakly submodular function subject to a general matroid constraint, where $\gamma$ is a parameter measuring the distance of the function from submodularity.
Greed is Good: Near-Optimal Submodular Maximization via Greedy Optimization
Sample Greedy achieves $(k + 3)$-approximation with only $O(nr/k)$ function evaluations.
Streaming Weak Submodularity: Interpreting Neural Networks on the Fly
In many machine learning applications, it is important to explain the predictions of a black-box classifier.
Fast Distributed Submodular Cover: Public-Private Data Summarization
The goal is to provide a succinct summary of massive dataset, ideally as small as possible, from which customized summaries can be built for each user, i. e. it can contain elements from the public data (for diversity) and users' private data (for personalization).
Estimating the Size of a Large Network and its Communities from a Random Sample
In this paper we consider a population random graph G = (V;E) from the stochastic block model (SBM) with K communities/blocks.
Tradeoffs for Space, Time, Data and Risk in Unsupervised Learning
Using k-means clustering as a prototypical unsupervised learning problem, we show how we can strategically summarize the data (control space) in order to trade off risk and time when data is generated by a probabilistic model.
Submodular Variational Inference for Network Reconstruction
In real-world and online social networks, individuals receive and transmit information in real time.
Near-Optimal Active Learning of Halfspaces via Query Synthesis in the Noisy Setting
This problem has recently gained a lot of interest in automated science and adversarial reverse engineering for which only heuristic algorithms are known.
Distributed Submodular Cover: Succinctly Summarizing Massive Data
In this paper, we formalize this challenge as a submodular cover problem.
Seeing the Unseen Network: Inferring Hidden Social Ties from Respondent-Driven Sampling
Learning about the social structure of hidden and hard-to-reach populations --- such as drug users and sex workers --- is a major goal of epidemiological and public health research on risk behaviors and disease prevention.
Fast Mixing for Discrete Point Processes
We show that if the set function (not necessarily submodular) displays a natural notion of decay of correlation, then, for $\beta$ small enough, it is possible to design fast mixing Markov chain Monte Carlo methods that yield error bounds on marginal approximations that do not depend on the size of the set $V$.
Distributed Submodular Maximization
Such problems can often be reduced to maximizing a submodular set function subject to various constraints.
Lazier Than Lazy Greedy
Is it possible to maximize a monotone submodular function faster than the widely used lazy greedy algorithm (also known as accelerated greedy), both in theory and practice?
Convolutional Neural Associative Memories: Massive Capacity with Noise Tolerance
The resulting network has a retrieval capacity that is exponential in the size of the network.
Noise Facilitation in Associative Memories of Exponential Capacity
More surprisingly, we show that internal noise actually improves the performance of the recall phase while the pattern retrieval capacity remains intact, i. e., the number of stored patterns does not reduce with noise (up to a threshold).
Near Optimal Bayesian Active Learning for Decision Making
Instead of minimizing uncertainty per se, we consider a set of overlapping decision regions of these hypotheses.
Near-Optimally Teaching the Crowd to Classify
How should we present training examples to learners to teach them classification rules?
Noise-Enhanced Associative Memories
More surprisingly, we show that internal noise actually improves the performance of the recall phase.
Distributed Submodular Maximization: Identifying Representative Elements in Massive Data
Such problems can often be reduced to maximizing a submodular set function subject to cardinality constraints.
Neural Networks Built from Unreliable Components
Recent advances in associative memory design through strutured pattern sets and graph-based inference algorithms have allowed the reliable learning and retrieval of an exponential number of patterns.
Coupled Neural Associative Memories
We propose a novel architecture to design a neural associative memory that is capable of learning a large number of patterns and recalling them later in presence of noise.
From Small-World Networks to Comparison-Based Search
In short, a user searching for a target object navigates through a database in the following manner: the user is asked to select the object most similar to her target from a small list of objects.
