Search Results for author:
Found 100 papers, 12 papers with code
Fast Neural Kernel Embeddings for General Activations
Moreover, most prior works on neural kernels have focused on the ReLU activation, mainly due to its popularity but also due to the difficulty of computing such kernels for general activations.
Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
In this work, we present Tree of Attacks with Pruning (TAP), an automated method for generating jailbreaks that only requires black-box access to the target LLM.
HyperAttention: Long-context Attention in Near-Linear Time
Recent work suggests that in the worst-case scenario, quadratic time is necessary unless the entries of the attention matrix are bounded or the matrix has low stable rank.
Streaming Weak Submodularity: Interpreting Neural Networks on the Fly
In many machine learning applications, it is important to explain the predictions of a black-box classifier.
Adaptive Sequence Submodularity
In many machine learning applications, one needs to interactively select a sequence of items (e. g., recommending movies based on a user's feedback) or make sequential decisions in a certain order (e. g., guiding an agent through a series of states).
Submodular Maximization Beyond Non-negativity: Guarantees, Fast Algorithms, and Applications
It is generally believed that submodular functions -- and the more general class of $\gamma$-weakly submodular functions -- may only be optimized under the non-negativity assumption $f(S) \geq 0$.
How Do You Want Your Greedy: Simultaneous or Repeated?
We also present SubmodularGreedy. jl, a Julia package which implements these algorithms and may be downloaded at https://github. com/crharshaw/SubmodularGreedy. jl .
Self-Consistency of the Fokker-Planck Equation
In this paper, we exploit this concept to design a potential function of the hypothesis velocity fields, and prove that, if such a function diminishes to zero during the training procedure, the trajectory of the densities generated by the hypothesis velocity fields converges to the solution of the FPE in the Wasserstein-2 sense.
KDEformer: Accelerating Transformers via Kernel Density Estimation
Dot-product attention mechanism plays a crucial role in modern deep architectures (e. g., Transformer) for sequence modeling, however, na\"ive exact computation of this model incurs quadratic time and memory complexities in sequence length, hindering the training of long-sequence models.
Streaming Submodular Maximization under a $k$-Set System Constraint
In this paper, we propose a novel framework that converts streaming algorithms for monotone submodular maximization into streaming algorithms for non-monotone submodular maximization.
Scalable Sampling for Nonsymmetric Determinantal Point Processes
However, existing work leaves open the question of scalable NDPP sampling.
Projection-Free Online Optimization with Stochastic Gradient: From Convexity to Submodularity
We also propose One-Shot Frank-Wolfe, a simpler algorithm which requires only a single stochastic gradient estimate in each round and achieves an $O(T^{2/3})$ stochastic regret bound for convex and continuous submodular optimization.
Data Summarization at Scale: A Two-Stage Submodular Approach
The sheer scale of modern datasets has resulted in a dire need for summarization techniques that identify representative elements in a dataset.
Projection-Free Bandit Convex Optimization
In this paper, we propose the first computationally efficient projection-free algorithm for bandit convex optimization (BCO).
Stochastic Conditional Gradient Methods: From Convex Minimization to Submodular Maximization
Further, for a monotone and continuous DR-submodular function and subject to a general convex body constraint, we prove that our proposed method achieves a $((1-1/e)OPT-\eps)$ guarantee with $O(1/\eps^3)$ stochastic gradient computations.
Do Less, Get More: Streaming Submodular Maximization with Subsampling
In this paper, we develop the first one-pass streaming algorithm for submodular maximization that does not evaluate the entire stream even once.
Comparison Based Learning from Weak Oracles
More specifically, we aim at devising efficient algorithms to locate a target object in a database equipped with a dissimilarity metric via invocation of the weak comparison oracle.
Online Continuous Submodular Maximization
For such settings, we then propose an online stochastic gradient ascent algorithm that also achieves a regret bound of $O(\sqrt{T})$ regret, albeit against a weaker $1/2$-approximation to the best feasible solution in hindsight.
Deletion-Robust Submodular Maximization at Scale
Can we efficiently extract useful information from a large user-generated dataset while protecting the privacy of the users and/or ensuring fairness in representation.
Conditional Gradient Method for Stochastic Submodular Maximization: Closing the Gap
More precisely, for a monotone and continuous DR-submodular function and subject to a \textit{general} convex body constraint, we prove that \alg achieves a $[(1-1/e)\text{OPT} -\eps]$ guarantee (in expectation) with $\mathcal{O}{(1/\eps^3)}$ stochastic gradient computations.
Gradient Methods for Submodular Maximization
Despite the apparent lack of convexity in such functions, we prove that stochastic projected gradient methods can provide strong approximation guarantees for maximizing continuous submodular functions with convex constraints.
Weakly Submodular Maximization Beyond Cardinality Constraints: Does Randomization Help Greedy?
In this paper, we prove that a randomized version of the greedy algorithm (previously used by Buchbinder et al. (2014) for a different problem) achieves an approximation ratio of $(1 + 1/\gamma)^{-2}$ for the maximization of a weakly submodular function subject to a general matroid constraint, where $\gamma$ is a parameter measuring the distance of the function from submodularity.
Submodular Variational Inference for Network Reconstruction
In real-world and online social networks, individuals receive and transmit information in real time.
Greed is Good: Near-Optimal Submodular Maximization via Greedy Optimization
Sample Greedy achieves $(k + 3)$-approximation with only $O(nr/k)$ function evaluations.
Near-Optimal Active Learning of Halfspaces via Query Synthesis in the Noisy Setting
This problem has recently gained a lot of interest in automated science and adversarial reverse engineering for which only heuristic algorithms are known.
Estimating the Size of a Large Network and its Communities from a Random Sample
In this paper we consider a population random graph G = (V;E) from the stochastic block model (SBM) with K communities/blocks.
Distributed Submodular Maximization
Such problems can often be reduced to maximizing a submodular set function subject to various constraints.
Tradeoffs for Space, Time, Data and Risk in Unsupervised Learning
Using k-means clustering as a prototypical unsupervised learning problem, we show how we can strategically summarize the data (control space) in order to trade off risk and time when data is generated by a probabilistic model.
Seeing the Unseen Network: Inferring Hidden Social Ties from Respondent-Driven Sampling
Learning about the social structure of hidden and hard-to-reach populations --- such as drug users and sex workers --- is a major goal of epidemiological and public health research on risk behaviors and disease prevention.
Fast Mixing for Discrete Point Processes
We show that if the set function (not necessarily submodular) displays a natural notion of decay of correlation, then, for $\beta$ small enough, it is possible to design fast mixing Markov chain Monte Carlo methods that yield error bounds on marginal approximations that do not depend on the size of the set $V$.
Lazier Than Lazy Greedy
Is it possible to maximize a monotone submodular function faster than the widely used lazy greedy algorithm (also known as accelerated greedy), both in theory and practice?
Convolutional Neural Associative Memories: Massive Capacity with Noise Tolerance
The resulting network has a retrieval capacity that is exponential in the size of the network.
Noise Facilitation in Associative Memories of Exponential Capacity
More surprisingly, we show that internal noise actually improves the performance of the recall phase while the pattern retrieval capacity remains intact, i. e., the number of stored patterns does not reduce with noise (up to a threshold).
Near-Optimally Teaching the Crowd to Classify
How should we present training examples to learners to teach them classification rules?
Near Optimal Bayesian Active Learning for Decision Making
Instead of minimizing uncertainty per se, we consider a set of overlapping decision regions of these hypotheses.
From Small-World Networks to Comparison-Based Search
In short, a user searching for a target object navigates through a database in the following manner: the user is asked to select the object most similar to her target from a small list of objects.
Coupled Neural Associative Memories
We propose a novel architecture to design a neural associative memory that is capable of learning a large number of patterns and recalling them later in presence of noise.
Neural Networks Built from Unreliable Components
Recent advances in associative memory design through strutured pattern sets and graph-based inference algorithms have allowed the reliable learning and retrieval of an exponential number of patterns.
Eliminating Latent Discrimination: Train Then Mask
How can we control for latent discrimination in predictive models?
Unconstrained Submodular Maximization with Constant Adaptive Complexity
In this paper, we consider the unconstrained submodular maximization problem.
Interactive Submodular Bandit
We then receive a noisy feedback about the utility of the action (e. g., ratings) which we model as a submodular function over the context-action space.
Fast Distributed Submodular Cover: Public-Private Data Summarization
The goal is to provide a succinct summary of massive dataset, ideally as small as possible, from which customized summaries can be built for each user, i. e. it can contain elements from the public data (for diversity) and users' private data (for personalization).
Distributed Submodular Cover: Succinctly Summarizing Massive Data
In this paper, we formalize this challenge as a submodular cover problem.
Noise-Enhanced Associative Memories
More surprisingly, we show that internal noise actually improves the performance of the recall phase.
Distributed Submodular Maximization: Identifying Representative Elements in Massive Data
Such problems can often be reduced to maximizing a submodular set function subject to cardinality constraints.
Differentially Private Submodular Maximization: Data Summarization in Disguise
Many data summarization applications are captured by the general framework of submodular maximization.
Probabilistic Submodular Maximization in Sub-Linear Time
As a remedy, we introduce the problem of sublinear time probabilistic submodular maximization: Given training examples of functions (e. g., via user feature vectors), we seek to reduce the ground set so that optimizing new functions drawn from the same distribution will provide almost as much value when restricted to the reduced ground set as when using the full set.
Deletion-Robust Submodular Maximization: Data Summarization with "the Right to be Forgotten"
How can we summarize a dynamic data stream when elements selected for the summary can be deleted at any time?
Scalable Deletion-Robust Submodular Maximization: Data Summarization with Privacy and Fairness Constraints
Can we efficiently extract useful information from a large user-generated dataset while protecting the privacy of the users and/or ensuring fairness in representation?
Black Box Submodular Maximization: Discrete and Continuous Settings
In this paper, we consider the problem of black box continuous submodular maximization where we only have access to the function values and no information about the derivatives is provided.
Quantized Frank-Wolfe: Faster Optimization, Lower Communication, and Projection Free
How can we efficiently mitigate the overhead of gradient communications in distributed optimization?
Stochastic Conditional Gradient++
It is known that this rate is optimal in terms of stochastic gradient evaluations.
Submodular Streaming in All its Glory: Tight Approximation, Minimum Memory and Low Adaptive Complexity
We show how one can achieve the tight $(1/2)$-approximation guarantee with $O(k)$ shared memory while minimizing not only the required rounds of computations but also the total number of communicated bits.
One Sample Stochastic Frank-Wolfe
One of the beauties of the projected gradient descent method lies in its rather simple mechanism and yet stable behavior with inexact, stochastic gradients, which has led to its wide-spread use in many machine learning applications.
Regret Bounds for Batched Bandits
We present simple and efficient algorithms for the batched stochastic multi-armed bandit and batched stochastic linear bandit problems.
Minimax Regret of Switching-Constrained Online Convex Optimization: No Phase Transition
To establish the dimension-independent upper bound, we next show that a mini-batching algorithm provides an $ O(\frac{T}{\sqrt{K}}) $ upper bound, and therefore conclude that the minimax regret of switching-constrained OCO is $ \Theta(\frac{T}{\sqrt{K}}) $ for any $K$.
Online Continuous Submodular Maximization: From Full-Information to Bandit Feedback
In this paper, we propose three online algorithms for submodular maximisation.
Adaptivity in Adaptive Submodularity
We propose an efficient semi adaptive policy that with $O(\log n \times\log k)$ adaptive rounds of observations can achieve an almost tight $1-1/e-\epsilon$ approximation guarantee with respect to an optimal policy that carries out $k$ actions in a fully sequential manner.
Stochastic Continuous Greedy ++: When Upper and Lower Bounds Match
Concretely, for a monotone and continuous DR-submodular function, \SCGPP achieves a tight $[(1-1/e)\OPT -\epsilon]$ solution while using $O(1/\epsilon^2)$ stochastic gradients and $O(1/\epsilon)$ calls to the linear optimization oracle.
Submodular Maximization Through Barrier Functions
In this paper, we introduce a novel technique for constrained submodular maximization, inspired by barrier functions in continuous optimization.
Regularized Submodular Maximization at Scale
In this paper, we propose scalable methods for maximizing a regularized submodular function $f = g - \ell$ expressed as the difference between a monotone submodular function $g$ and a modular function $\ell$.
More Data Can Expand the Generalization Gap Between Adversarially Robust and Standard Models
Despite remarkable success in practice, modern machine learning models have been found to be susceptible to adversarial attacks that make human-imperceptible perturbations to the data, but result in serious and potentially dangerous prediction errors.
The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization
In the medium adversary regime, with more training data, the generalization loss exhibits a double descent curve, which implies the existence of an intermediate stage where more training data hurts the generalization.
Submodular Maximization in Clean Linear Time
In this paper, we provide the first deterministic algorithm that achieves the tight $1-1/e$ approximation guarantee for submodular maximization under a cardinality (size) constraint while making a number of queries that scales only linearly with the size of the ground set $n$.
Meta Learning in the Continuous Time Limit
In this paper, we establish the ordinary differential equation (ODE) that underlies the training dynamics of Model-Agnostic Meta-Learning (MAML).
Continuous Submodular Maximization: Beyond DR-Submodularity
We first prove that a simple variant of the vanilla coordinate ascent, called Coordinate-Ascent+, achieves a $(\frac{e-1}{2e-1}-\varepsilon)$-approximation guarantee while performing $O(n/\varepsilon)$ iterations, where the computational complexity of each iteration is roughly $O(n/\sqrt{\varepsilon}+n\log n)$ (here, $n$ denotes the dimension of the optimization problem).
Safe Learning under Uncertain Objectives and Constraints
More precisely, by assuming that Reliable-FW has access to a (stochastic) gradient oracle of the objective function and a noisy feasibility oracle of the safety polytope, it finds an $\epsilon$-approximate first-order stationary point with the optimal ${\mathcal{O}}({1}/{\epsilon^2})$ gradient oracle complexity (resp.
Multiple Descent: Design Your Own Generalization Curve
This paper explores the generalization loss of linear regression in variably parameterized families of models, both under-parameterized and over-parameterized.
Streaming Submodular Maximization under a k-Set System Constraint
Moreover, we propose the first streaming algorithms for monotone submodular maximization subject to $k$-extendible and $k$-system constraints.
Online MAP Inference of Determinantal Point Processes
In this paper, we provide an efficient approximation algorithm for finding the most likelihood configuration (MAP) of size $k$ for Determinantal Point Processes (DPP) in the online setting where the data points arrive in an arbitrary order and the algorithm cannot discard the selected elements from its local memory.
Submodularity in Action: From Machine Learning to Signal Processing Applications
We introduce a variety of submodular-friendly applications, and elucidate the relation of submodularity to convexity and concavity which enables efficient optimization.
Batched Neural Bandits
In many sequential decision-making problems, the individuals are split into several batches and the decision-maker is only allowed to change her policy at the end of batches.
Federated Functional Gradient Boosting
First, in the semi-heterogeneous setting, when the marginal distributions of the feature vectors on client machines are identical, we develop the federated functional gradient boosting (FFGB) method that provably converges to the global minimum.
The Power of Subsampling in Submodular Maximization
We propose subsampling as a unified algorithmic technique for submodular maximization in centralized and online settings.
Learning and Certification under Instance-targeted Poisoning
In this paper, we study PAC learnability and certification of predictions under instance-targeted poisoning attacks, where the adversary who knows the test instance may change a fraction of the training set with the goal of fooling the learner at the test instance.
Parallelizing Thompson Sampling
How can we make use of information parallelism in online decision making problems while efficiently balancing the exploration-exploitation trade-off?
Submodular + Concave
It has been well established that first order optimization methods can converge to the maximal objective value of concave functions and provide constant factor approximation guarantees for (non-convex/non-concave) continuous submodular functions.
An Exponential Improvement on the Memorization Capacity of Deep Threshold Networks
Recently, Vershynin (2020) settled a long standing question by Baum (1988), proving that \emph{deep threshold} networks can memorize $n$ points in $d$ dimensions using $\widetilde{\mathcal{O}}(e^{1/\delta^2}+\sqrt{n})$ neurons and $\widetilde{\mathcal{O}}(e^{1/\delta^2}(d+\sqrt{n})+n)$ weights, where $\delta$ is the minimum distance between the points.
Data-driven mapping between functional connectomes using optimal transport
Being able to map connectomes and derived results between different atlases without additional pre-processing is a crucial step in improving interpretation and generalization between studies that use different atlases.
Black-Box Generalization: Stability of Zeroth-Order Learning
These bounds coincide with those for SGD, and rather surprisingly are independent of $d$, $K$ and the batch size $m$, under appropriate choices of a slightly decreased learning rate.
What Functions Can Graph Neural Networks Generate?
We prove that: (i) a GNN, as a graph function, is necessarily permutation compatible; (ii) conversely, any permutation compatible function, when restricted on input graphs with distinct node features, can be generated by a GNN; (iii) for arbitrary node features (not necessarily distinct), a simple feature augmentation scheme suffices to generate a permutation compatible function by a GNN; (iv) permutation compatibility can be verified by checking only quadratically many functional constraints, rather than an exhaustive search over all the permutations; (v) GNNs can generate \textit{any} graph function once we augment the node features with node identities, thus going beyond graph isomorphism and permutation compatibility.
The Best of Both Worlds: Reinforcement Learning with Logarithmic Regret and Policy Switches
In this paper, we study the problem of regret minimization for episodic Reinforcement Learning (RL) both in the model-free and the model-based setting.
Learning Distributionally Robust Models at Scale via Composite Optimization
To train machine learning models that are robust to distribution shifts in the data, distributionally robust optimization (DRO) has been proven very effective.
Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD
For nonconvex smooth losses, we prove that full-batch GD efficiently generalizes close to any stationary point at termination, and recovers the generalization error guarantees of stochastic algorithms with fewer assumptions.
Scalable MCMC Sampling for Nonsymmetric Determinantal Point Processes
In this work, we develop a scalable MCMC sampling algorithm for $k$-NDPPs with low-rank kernels, thus enabling runtime that is sublinear in $n$.
Multiclass Learnability Beyond the PAC Framework: Universal Rates and Partial Concept Classes
Second, we consider the problem of multiclass classification with structured data (such as data lying on a low dimensional manifold or satisfying margin conditions), a setting which is captured by partial concept classes (Alon, Hanneke, Holzman and Moran, FOCS '21).
Replicable Bandits
Similarly, for stochastic linear bandits (with finitely and infinitely many arms) we develop replicable policies that achieve the best-known problem-independent regret bounds with an optimal dependency on the replicability parameter.
On Optimal Learning Under Targeted Data Poisoning
In this work we aim to characterize the smallest achievable error $\epsilon=\epsilon(\eta)$ by the learner in the presence of such an adversary in both realizable and agnostic settings.
FIMP: Foundation Model-Informed Message Passing for Graph Neural Networks
Foundation models have revolutionized the landscape of Deep Learning (DL), serving as a versatile platform which can be adapted to a wide range of downstream tasks.
Exact Gradient Computation for Spiking Neural Networks Through Forward Propagation
their weights, and (2) we propose a novel training algorithm, called \emph{forward propagation} (FP), that computes exact gradients for SNN.
The Impossibility of Parallelizing Boosting
The aim of boosting is to convert a sequence of weak learners into a strong learner.
Select without Fear: Almost All Mini-Batch Schedules Generalize Optimally
We establish matching upper and lower generalization error bounds for mini-batch Gradient Descent (GD) training with either deterministic or stochastic, data-independent, but otherwise arbitrary batch selection rules.
Learning from Aggregated Data: Curated Bags versus Random Bags
Our method is based on the observation that the sum of the gradients of the loss function on individual data examples in a curated bag can be computed from the aggregate label without the need for individual labels.
Statistical Indistinguishability of Learning Algorithms
When two different parties use the same learning rule on their own data, how can we test whether the distributions of the two outcomes are similar?
Submodular Minimax Optimization: Finding Effective Sets
Despite the rich existing literature about minimax optimization in continuous settings, only very partial results of this kind have been obtained for combinatorial settings.
Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learning
Methods for carefully selecting or generating a small set of training data to learn from, i. e., data pruning, coreset selection, and data distillation, have been shown to be effective in reducing the ever-increasing cost of training neural networks.
Langevin Thompson Sampling with Logarithmic Communication: Bandits and Reinforcement Learning
Thompson sampling (TS) is widely used in sequential decision making due to its ease of use and appealing empirical performance.
Optimal Guarantees for Algorithmic Reproducibility and Gradient Complexity in Convex Optimization
Particularly, given the inexact initialization oracle, our regularization-based algorithms achieve the best of both worlds - optimal reproducibility and near-optimal gradient complexity - for minimization and minimax optimization.
SubGen: Token Generation in Sublinear Time and Memory
In this work, our focus is on developing an efficient compression technique for the KV cache.
Replicable Learning of Large-Margin Halfspaces
Departing from the requirement of polynomial time algorithms, using the DP-to-Replicability reduction of Bun, Gaboardi, Hopkins, Impagliazzo, Lei, Pitassi, Sorrell, and Sivakumar [STOC, 2023], we show how to obtain a replicable algorithm for large-margin halfspaces with improved sample complexity with respect to the margin parameter $\tau$, but running time doubly exponential in $1/\tau^2$ and worse sample complexity dependence on $\epsilon$ than one of our previous algorithms.
