Search Results for author:
Found 93 papers, 34 papers with code
InfoGAN-CR: Disentangling Generative Adversarial Networks with Contrastive Regularizers
This contrastive regularizer is inspired by a natural notion of disentanglement: latent traversal.
On the Convergence of Differentially-Private Fine-tuning: To Linearly Probe or to Fully Fine-tune?
We provide theoretical insights into the convergence of DP fine-tuning within an overparameterized neural network and establish a utility curve that determines the allocation of privacy budget between linear probing and full fine-tuning.
Privacy-Preserving Instructions for Aligning Large Language Models
Service providers of large language model (LLM) applications collect user instructions in the wild and use them in further aligning LLMs with users' intentions.
DeepPolar: Inventing Nonlinear Large-Kernel Polar Codes via Deep Learning
Polar codes, developed on the foundation of Arikan's polarization kernel, represent a breakthrough in coding theory and have emerged as the state-of-the-art error-correction-code in short-to-medium block length regimes.
DPZero: Private Fine-Tuning of Language Models without Backpropagation
The widespread practice of fine-tuning large language models (LLMs) on domain-specific data faces two major challenges in memory and privacy.
Profit: Benchmarking Personalization and Robustness Trade-off in Federated Prompt Tuning
In many applications of federated learning (FL), clients desire models that are personalized using their local data, yet are also robust in the sense that they retain general global knowledge.
Private Federated Learning with Autotuned Compression
We propose new techniques for reducing communication in private federated learning without the need for setting or tuning compression rates.
Can Public Large Language Models Help Private Cross-device Federated Learning?
We study (differentially) private federated learning (FL) of language models.
DataComp: In search of the next generation of multimodal datasets
Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms.
Why Is Public Pretraining Necessary for Private Model Training?
To explain this phenomenon, we hypothesize that the non-convex loss landscape of a model training necessitates an optimization algorithm to go through two phases.
One-shot Empirical Privacy Estimation for Federated Learning
Privacy estimation techniques for differentially private (DP) algorithms are useful for comparing against analytical bounds, or to empirically measure privacy loss in settings where known analytical bounds are not tight.
Near Optimal Private and Robust Linear Regression
Under label-corruption, this is the first efficient linear regression algorithm to guarantee both $(\varepsilon,\delta)$-DP and robustness.
Machine Learning-Aided Efficient Decoding of Reed-Muller Subcodes
Next, we derive the soft-decision based version of our algorithm, called soft-subRPA, that not only improves upon the performance of subRPA but also enables a differentiable decoding algorithm.
Sequence-to-sequence translation from mass spectra to peptides with a transformer model
A fundamental challenge for any mass spectrometry-based proteomics experiment is the identification of the peptide that generated each acquired tandem mass spectrum.
MAUVE Scores for Generative Models: Theory and Practice
We present MAUVE, a family of comparison measures between pairs of distributions such as those encountered in the generative modeling of text or images.
Learning to Generate Image Embeddings with User-level Differential Privacy
Small on-device models have been successfully trained with user-level differential privacy (DP) for next word prediction and image classification tasks in the past.
Zonotope Domains for Lagrangian Neural Network Verification
Neural network verification aims to provide provable bounds for the output of a neural network for a given input range.
Few-shot Backdoor Attacks via Neural Tangent Kernels
In a backdoor attack, an attacker injects corrupted examples into the training set.
Stochastic optimization on matrices and a graphon McKean-Vlasov limit
The limiting curve of graphons is characterized by a family of stochastic differential equations with reflections and can be thought of as an extension of the classical McKean-Vlasov limit for interacting diffusions.
CRISP: Curriculum based Sequential Neural Decoders for Polar Code Family
We design a principled curriculum, guided by information-theoretic insights, to train CRISP and show that it outperforms the successive-cancellation (SC) decoder and attains near-optimal reliability performance on the Polar(32, 16) and Polar(64, 22) codes.
Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP
Web-crawled datasets have enabled remarkable generalization capabilities in recent image-text models such as CLIP (Contrastive Language-Image pre-training) or Flamingo, but little is known about the dataset creation processes.
Bring Your Own Algorithm for Optimal Differentially Private Stochastic Minimax Optimization
We provide a general framework for solving differentially private stochastic minimax optimization (DP-SMO) problems, which enables the practitioners to bring their own base optimization algorithm and use it as a black-box to obtain the near-optimal privacy-loss trade-off.
DP-PCA: Statistically Optimal and Differentially Private PCA
For sub-Gaussian data, we provide nearly optimal statistical error rates even for $n=\tilde O(d)$.
Towards a Defense Against Federated Backdoor Attacks Under Continuous Training
We propose shadow learning, a framework for defending against backdoor attacks in the FL setting under long-range training.
MAML and ANIL Provably Learn Representations
Recent empirical evidence has driven conventional wisdom to believe that gradient-based meta-learning (GBML) methods perform well at few-shot learning because they learn an expressive data representation that is shared across tasks.
Lifted Primal-Dual Method for Bilinearly Coupled Smooth Minimax Optimization
We also provide a direct single-loop algorithm, using the LPD method, that achieves the iteration complexity of $O(\sqrt{\frac{L_x}{\varepsilon}} + \frac{\|A\|}{\sqrt{\mu_y \varepsilon}} + \sqrt{\frac{L_y}{\varepsilon}})$.
Statistically and Computationally Efficient Linear Meta-representation Learning
To cope with such data scarcity, meta-representation learning methods train across many related tasks to find a shared (lower-dimensional) representation of the data where all tasks can be solved accurately.
Gradient flows on graphons: existence, convergence, continuity equations
Wasserstein gradient flows on probability measures have found a host of applications in various optimization problems.
Differential privacy and robust statistics in high dimensions
The key insight is that if we design an exponential mechanism that accesses the data only via one-dimensional robust statistics, then the resulting local sensitivity can be dramatically reduced.
Gradient Inversion with Generative Image Prior
Federated Learning (FL) is a distributed learning framework, in which the local data never leaves clients devices to preserve privacy, and the server trains models on the data via accessing only the gradients of those local data.
KO codes: Inventing Nonlinear Encoding and Decoding for Reliable Wireless Communication via Deep-learning
In this paper, we construct KO codes, a computationaly efficient family of deep-learning driven (encoder, decoder) pairs that outperform the state-of-the-art reliability performance on the standardized AWGN channel.
FedChain: Chained Algorithms for Near-Optimal Communication Cost in Federated Learning
We propose FedChain, an algorithmic framework that combines the strengths of local methods and global methods to achieve fast convergence in terms of R while leveraging the similarity between clients.
Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals
The spectacular success of deep generative models calls for quantitative tools to measure their statistical performance.
Sample Efficient Linear Meta-Learning by Alternating Minimization
We show that, for a constant subspace dimension MLLAM obtains nearly-optimal estimation error, despite requiring only $\Omega(\log d)$ samples per task.
SPECTRE: Defending Against Backdoor Attacks Using Robust Statistics
There have been promising attempts to use the intermediate representations of such a model to separate corrupted examples from clean ones.
Robust and Differentially Private Mean Estimation
In statistical learning and analysis from shared data, which is increasingly widely adopted in platforms such as federated learning and meta-learning, there are two major concerns: privacy and robustness.
Efficient Algorithms for Federated Saddle Point Optimization
Our goal is to design an algorithm that can harness the benefit of similarity in the clients while recovering the Minibatch Mirror-prox performance under arbitrary heterogeneity (up to log factors).
Eliminating Sharp Minima from SGD with Truncated Heavy-tailed Noise
The empirical success of deep learning is often attributed to SGD's mysterious ability to avoid sharp local minima in the loss landscape, as sharp minima are known to lead to poor generalization.
Reed-Muller Subcodes: Machine Learning-Aided Design of Efficient Soft Recursive Decoding
To lower the complexity of our decoding algorithm, referred to as subRPA in this paper, we investigate different ways for pruning the projections.
Information Theory Information Theory
Projection Efficient Subgradient Method and Optimal Nonsmooth Frank-Wolfe Method
Further, instead of a PO if we only have a linear minimization oracle (LMO, a la Frank-Wolfe) to access the constraint set, an extension of our method, MOLES, finds a feasible $\epsilon$-suboptimal solution using $O(\epsilon^{-2})$ LMO calls and FO calls---both match known lower bounds, resolving a question left open since White (1993).
Deepcode and Modulo-SK are Designed for Different Settings
DeepCode is designed and evaluated for the AWGN channel with (potentially delayed) uncoded output feedback.
Robust Meta-learning for Mixed Linear Regression with Small Batches
Together, this approach is robust against outliers and achieves a graceful statistical trade-off; the lack of $\Omega(k^{1/2})$-size tasks can be compensated for with smaller tasks, which can now be as small as $O(\log k)$.
Meta-learning for mixed linear regression
In modern supervised learning, there are a large number of tasks, but many of them are associated with only a small amount of labeled data.
Minimax Optimal Estimation of Approximate Differential Privacy on Neighboring Databases
We pose it as a property estimation problem, and study the fundamental trade-offs involved in the accuracy in estimated privacy guarantees and the number of samples required.
Turbo Autoencoder: Deep learning based channel codes for point-to-point communication channels
Designing codes that combat the noise in a communication medium has remained a significant area of research in information theory as well as wireless communications.
Towards Principled Objectives for Contrastive Disentanglement
%In fact, contrastive disentanglement and unsupervised recovery are often combined in that we seek additional variations that exhibit salient factors/properties.
Optimal transport mapping via input convex neural networks
Building upon recent advances in the field of input convex neural networks, we propose a new framework where the gradient of one convex function represents the optimal transport mapping.
Efficient Algorithms for Smooth Minimax Optimization
This paper studies first order methods for solving smooth minimax optimization problems $\min_x \max_y g(x, y)$ where $g(\cdot,\cdot)$ is smooth and $g(x,\cdot)$ is concave for each $x$.
InfoGAN-CR and ModelCentrality: Self-supervised Model Training and Selection for Disentangling GANs
Disentangled generative models map a latent code vector to a target space, while enforcing that a subset of the learned latent codes are interpretable and associated with distinct properties of the target distribution.
Robust conditional GANs under missing or uncertain labels
Matching the performance of conditional Generative Adversarial Networks with little supervision is an important task, especially in venturing into new domains.
Learning in Gated Neural Networks
Gating is a key feature in modern neural networks including LSTMs, GRUs and sparsely-gated deep neural networks.
Minimax Rates of Estimating Approximate Differential Privacy
We pose it as a property estimation problem, and study the fundamental trade-offs involved in the accuracy in estimated privacy guarantees and the number of samples required.
Information Theory Information Theory
DeepTurbo: Deep Turbo Decoder
We focus on Turbo codes and propose DeepTurbo, a novel deep learning based architecture for Turbo decoding.
Number of Connected Components in a Graph: Estimation via Counting Patterns
This representation is crucial in introducing a novel estimator for the number of connected components for general graphs, under the knowledge of the spectral gap of the original graph.
LEARN Codes: Inventing Low-latency Codes via Recurrent Neural Networks
Designing channel codes under low-latency constraints is one of the most demanding requirements in 5G standards.
Robustness of Conditional GANs to Noisy Labels
When the distribution of the noise is known, we introduce a novel architecture which we call Robust Conditional GAN (RCGAN).
Rate Distortion For Model Compression: From Theory To Practice
Theoretically, we prove that the proposed scheme is optimal for compressing one-hidden-layer ReLU neural networks.
Learning One-hidden-layer Neural Networks under General Input Distributions
Significant advances have been made recently on training neural networks, where the main challenge is in solving an optimization problem with abundant critical points.
Deepcode: Feedback Codes via Deep Learning
The design of codes for communicating reliably over a statistically well defined channel is an important endeavor involving deep mathematical research and wide-ranging practical applications.
Communication Algorithms via Deep Learning
We show that creatively designed and trained RNN architectures can decode well known sequential codes such as the convolutional and turbo codes with close to optimal performance on the additive white Gaussian noise (AWGN) channel, which itself is achieved by breakthrough algorithms of our times (Viterbi and BCJR decoders, representing dynamic programing and forward-backward algorithms).
Attention-based Graph Neural Network for Semi-supervised Learning
Recently popularized graph neural networks achieve the state-of-the-art accuracy on a number of standard benchmark datasets for graph-based semi-supervised learning, improving significantly over existing approaches.
Ranked #10 on
Graph Regression
on Lipophilicity
Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms
Once the experts are known, the recovery of gating parameters still requires an EM algorithm; however, we show that the EM algorithm for this simplified problem, unlike the joint EM algorithm, converges to the true parameters.
PacGAN: The power of two samples in generative adversarial networks
Generative adversarial networks (GANs) are innovative techniques for learning generative models of complex data distributions from samples.
Optimal Sample Complexity of M-wise Data for Top-K Ranking
As our result, we characterize the minimax optimality on the sample size for top-K ranking.
Matrix Norm Estimation from a Few Entries
This paper focuses on the technical challenges in accurately estimating the Schatten norms from a sampling of a matrix.
Estimating Mutual Information for Discrete-Continuous Mixtures
We provide numerical experiments suggesting superiority of the proposed estimator compared to other heuristics of adding small continuous noise to all the samples and applying standard estimators tailored for purely continuous variables, and quantizing the samples and applying standard estimators tailored for purely discrete variables.
Discovering Potential Correlations via Hypercontractivity
Discovering a correlation from one variable to another variable is of fundamental scientific and practical interest.
Learning from Comparisons and Choices
This also allows one to compute similarities among users and items to be used for categorization and search.
Spectrum Estimation from a Few Entries
We propose first estimating the Schatten $k$-norms of a matrix, and then applying Chebyshev approximation to the spectral sum function or applying moment matching in Wasserstein distance to recover the singular values.
Iterative Bayesian Learning for Crowdsourced Regression
Crowdsourcing platforms emerged as popular venues for purchasing human intelligence at low cost for large volume of tasks.
Breaking the Bandwidth Barrier: Geometrical Adaptive Entropy Estimation
In this paper, we combine both these approaches to design new estimators of entropy and mutual information that outperform state of the art methods.
Computational and Statistical Tradeoffs in Learning to Rank
For massive and heterogeneous modern datasets, it is of fundamental interest to provide guarantees on the accuracy of estimation when computational resources are limited.
Demystifying Fixed k-Nearest Neighbor Information Estimators
In this paper we demonstrate that the estimator is consistent and also identify an upper bound on the rate of convergence of the bias as a function of number of samples.
Top-$K$ Ranking from Pairwise Comparisons: When Spectral Ranking is Optimal
First, in a general comparison model where item pairs to compare are given a priori, we attain an upper and lower bound on the sample size for reliable recovery of the top-$K$ ranked items.
Optimal Inference in Crowdsourced Classification via Belief Propagation
Crowdsourcing systems are popular for solving large-scale labelling tasks with low-paid workers.
Achieving Budget-optimality with Adaptive Schemes in Crowdsourcing
Under this generalized Dawid-Skene model, we characterize the fundamental trade-off between budget and accuracy.
Conditional Dependence via Shannon Capacity: Axioms, Estimators and Applications
We conduct an axiomatic study of the problem of estimating the strength of a known causal relationship between a pair of variables.
Data-driven Rank Breaking for Efficient Rank Aggregation
Rank aggregation systems collect ordinal preferences from individuals to produce a global ranking that represents the social preference.
Secure Multi-party Differential Privacy
In this setting, each party is interested in computing a function on its private bit and all the other parties' bits.
Collaboratively Learning Preferences from Ordinal Data
In order to predict the preferences, we want to learn the underlying model from noisy observations of the low-rank matrix, collected as revealed preferences in various forms of ordinal data.
Spy vs. Spy: Rumor Source Obfuscation
Whether for fear of judgment or personal endangerment, it is crucial to keep anonymous the identity of the user who initially posted a sensitive message.
Learning Mixed Multinomial Logit Model from Ordinal Data
In case of single MNL models (no mixture), computationally and statistically tractable learning from pair-wise comparisons is feasible.
Minimax-optimal Inference from Partial Rankings
For a given assignment of items to users, we first derive an oracle lower bound of the estimation error that holds even for the more general Thurstone models.
Provable Tensor Factorization with Missing Data
We show that under certain standard assumptions, our method can recover a three-mode $n\times n\times n$ dimensional rank-$r$ tensor exactly from $O(n^{3/2} r^5 \log^4 n)$ randomly sampled entries.
Learning Mixtures of Discrete Product Distributions using Spectral Decompositions
The main challenge in learning mixtures of discrete product distributions is that these low-rank tensors cannot be obtained directly from the sample moments.
The Composition Theorem for Differential Privacy
Sequential querying of differentially private mechanisms degrades the overall privacy level.
Data Structures and Algorithms Cryptography and Security Information Theory Information Theory
Iterative ranking from pair-wise comparisons
In most settings, in addition to obtaining ranking, finding ‘scores’ for each object (e. g. player’s rating) is of interest to understanding the intensity of the preferences.
Rank Centrality: Ranking from Pair-wise Comparisons
To study the efficacy of the algorithm, we consider the popular Bradley-Terry-Luce (BTL) model (equivalent to the Multinomial Logit (MNL) for pair-wise comparisons) in which each object has an associated score which determines the probabilistic outcomes of pair-wise comparisons between objects.
Iterative Learning for Reliable Crowdsourcing Systems
Crowdsourcing systems, in which tasks are electronically distributed to numerous ``information piece-workers'', have emerged as an effective paradigm for human-powered solving of large scale problems in domains such as image classification, data entry, optical character recognition, recommendation, and proofreading.
Budget-Optimal Task Allocation for Reliable Crowdsourcing Systems
Further, we compare our approach with a more general class of algorithms which can dynamically assign tasks.
A Gradient Descent Algorithm on the Grassman Manifold for Matrix Completion
We consider the problem of reconstructing a low-rank matrix from a small subset of its entries.
Matrix Completion from Noisy Entries
Given a matrix M of low-rank, we consider the problem of reconstructing it from noisy observations of a small, random subset of its entries.
Matrix Completion from a Few Entries
In the process of proving these statements, we obtain a generalization of a celebrated result by Friedman-Kahn-Szemeredi and Feige-Ofek on the spectrum of sparse random matrices.
