Search Results for author:
Found 23 papers, 7 papers with code
It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization
Going beyond these objectives, we present a set of alternative attentional bias configurations along with their effective approximations to stabilize their training procedure.
Titans: Learning to Memorize at Test Time
Over more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention.
Addax: Utilizing Zeroth-Order Gradients to Improve Memory Efficiency and Performance of SGD for Fine-Tuning Language Models
In our experiments on the larger OPT-30B model, on average, Addax outperforms MeZO in terms of accuracy/F1 score by >16 and runs 30x faster on a single H100 GPU.
Retraining with Predicted Hard Labels Provably Increases Model Accuracy
In this paper, we theoretically analyze retraining in a linearly separable setting with randomly corrupted labels given to us and prove that retraining can improve the population accuracy obtained by initially training with the given (noisy) labels.
Perturb-and-Project: Differentially Private Similarities and Marginals
We revisit the input perturbations framework for differential privacy where noise is added to the input $A\in \mathcal{S}$ and the result is then projected back to the space of admissible datasets $\mathcal{S}$.
PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels
For context lengths of 32k and GPT-2 style models, our model achieves a 2. 5-4x speedup in training compared to FlashAttention, with no observed degradation in quality across our experiments.
Differentially Private Clustering in Data Streams
In this work, we provide the first differentially private streaming algorithms for $k$-means and $k$-median clustering of $d$-dimensional Euclidean data points over a stream with length at most $T$ using $poly(k, d,\log(T))$ space to achieve a constant multiplicative error and a $poly(k, d,\log(T))$ additive error.
Measuring Re-identification Risk
In this work, we present a new theoretical framework to measure re-identification risk in such user representations.
Stars: Tera-Scale Graph Building for Clustering and Graph Learning
We evaluate the performance of Stars for clustering and graph learning, and demonstrate 10~1000-fold improvements in pairwise similarity comparisons compared to different baselines, and 2~10-fold improvement in running time without quality loss.
Differentially Private Graph Learning via Sensitivity-Bounded Personalized PageRank
Personalized PageRank (PPR) is a fundamental tool in unsupervised learning of graph representations such as node ranking, labeling, and graph embedding.
Planning with General Objective Functions: Going Beyond Total Rewards
Standard sequential decision-making paradigms aim to maximize the cumulative reward when interacting with the unknown environment., i. e., maximize $\sum_{h = 1}^H r_h$ where $H$ is the planning horizon.
Average Case Column Subset Selection for Entrywise $\ell_1$-Norm Loss
entries drawn from any distribution $\mu$ for which the $(1+\gamma)$-th moment exists, for an arbitrarily small constant $\gamma > 0$, then it is possible to obtain a $(1+\epsilon)$-approximate column subset selection to the entrywise $\ell_1$-norm in nearly linear time.
Average Case Column Subset Selection for Entrywise \ell_1-Norm Loss
entries drawn from any distribution $\mu$ for which the $(1+\gamma)$-th moment exists, for an arbitrarily small constant $\gamma > 0$, then it is possible to obtain a $(1+\epsilon)$-approximate column subset selection to the entrywise $\ell_1$-norm in nearly linear time.
Efficient Symmetric Norm Regression via Linear Sketching
When the loss function is a general symmetric norm, our algorithm produces a $\sqrt{d} \cdot \mathrm{polylog} n \cdot \mathrm{mmc}(\ell)$-approximate solution in input-sparsity time, where $\mathrm{mmc}(\ell)$ is a quantity related to the symmetric norm under consideration.
Enhancing Adversarial Defense by k-Winners-Take-All
In all cases, the robustness of k-WTA networks outperforms that of traditional networks under white-box attacks.
Rethinking Generative Mode Coverage: A Pointwise Guaranteed Approach
The conventional wisdom to this end is by reducing through training a statistical distance (such as $f$-divergence) between the generated distribution and provided data distribution.
Towards a Zero-One Law for Column Subset Selection
Our approximation algorithms handle functions which are not even scale-invariant, such as the Huber loss function, which we show have very different structural properties than $\ell_p$-norms, e. g., one can show the lack of scale-invariance causes any column subset selection algorithm to provably require a $\sqrt{\log n}$ factor larger number of columns than $\ell_p$-norms; nevertheless we design the first efficient column subset selection algorithms for such error measures.
Subspace Embedding and Linear Regression with Orlicz Norm
An Orlicz norm is parameterized by a non-negative convex function $G:\mathbb{R}_+\rightarrow\mathbb{R}_+$ with $G(0)=0$: the Orlicz norm of a vector $x\in\mathbb{R}^n$ is defined as $ \|x\|_G=\inf\left\{\alpha>0\large\mid\sum_{i=1}^n G(|x_i|/\alpha)\leq 1\right\}.
BourGAN: Generative Networks with Metric Embeddings
This paper addresses the mode collapse for generative adversarial networks (GANs).
Nearly Optimal Dynamic $k$-Means Clustering for High-Dimensional Data
We consider the $k$-means clustering problem in the dynamic streaming setting, where points from a discrete Euclidean space $\{1, 2, \ldots, \Delta\}^d$ can be dynamically inserted to or deleted from the dataset.
Relative Error Tensor Low Rank Approximation
Despite the success on obtaining relative error low rank approximations for matrices, no such results were known for tensors.
Low Rank Approximation with Entrywise $\ell_1$-Norm Error
We give the first provable approximation algorithms for $\ell_1$-low rank approximation, showing that it is possible to achieve approximation factor $\alpha = (\log d) \cdot \mathrm{poly}(k)$ in $\mathrm{nnz}(A) + (n+d) \mathrm{poly}(k)$ time, where $\mathrm{nnz}(A)$ denotes the number of non-zero entries of $A$.
Distributed Low Rank Approximation of Implicit Functions of a Matrix
For example, each of $s$ servers may have an $n \times d$ matrix $A^t$, and we may be interested in computing a low rank approximation to $A = f(\sum_{t=1}^s A^t)$, where $f$ is a function which is applied entrywise to the matrix $\sum_{t=1}^s A^t$.
