Search Results for author:
Found 38 papers, 17 papers with code
Modality-specific Distillation
In this paper, we propose modality-specific distillation (MSD) to effectively transfer knowledge from a teacher on multimodal datasets.
DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning
Text-to-image diffusion models have been shown to suffer from sample-level memorization, possibly reproducing near-perfect replica of images that they are trained on, which may be undesirable.
Differentially Private Representation Learning via Image Captioning
In this work, we show that effective DP representation learning can be done via image captioning and scaling up to internet-scale multimodal datasets.
Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human Expression
Evaluation results show our method improves visual quality by 14%, prompt alignment by 16. 2% and scene diversity by 15. 3%, compared to prompt engineering the base Emu model for stickers generation.
Resprompt: Residual Connection Prompting Advances Multi-Step Reasoning in Large Language Models
Breakdown analysis further highlights RESPROMPT particularly excels in complex multi-step reasoning: for questions demanding at least five reasoning steps, RESPROMPT outperforms the best CoT based benchmarks by a remarkable average improvement of 21. 1% on LLaMA-65B and 14. 3% on LLaMA2-70B.
EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methods
From both the human study and automated evaluation, we find that: (i) Instruct-Pix2Pix, Null-Text and SINE are the top-performing methods averaged across different edit types, however {\it only} Instruct-Pix2Pix and Null-Text are able to preserve original image properties; (ii) Most of the editing methods fail at edits involving spatial operations (e. g., changing the position of an object).
Identifying Interpretable Subspaces in Image Representations
Although many existing approaches interpret features independently, we observe in state-of-the-art self-supervised and supervised models, that less than 20% of the representation space can be explained by individual features.
Augmenting CLIP with Improved Visio-Linguistic Reasoning
On the challenging Winoground compositional reasoning benchmark, our method improves the absolute visio-linguistic performance of different CLIP models by up to 7%, while on the ARO dataset, our method improves the visio-linguistic performance by upto 3%.
ViP: A Differentially Private Foundation Model for Computer Vision
In this work, we propose as a mitigation measure a recipe to train foundation vision models with differential privacy (DP) guarantee.
Text-To-Concept (and Back) via Cross-Model Alignment
We observe that the mapping between an image's representation in one model to its representation in another can be learned surprisingly well with just a linear layer, even across diverse models.
Defending Against Patch-based Backdoor Attacks on Self-Supervised Learning
It was shown that an adversary can poison a small part of the unlabeled data so that when a victim trains an SSL model on it, the final model will have a backdoor that the adversary can exploit.
Analyzing Privacy Leakage in Machine Learning via Multiple Hypothesis Testing: A Lesson From Fano
Differential privacy (DP) is by far the most widely accepted framework for mitigating privacy risks in machine learning.
Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning
Surprisingly, we also find that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity.
COFFEE: Counterfactual Fairness for Personalized Text Generation in Explainable Recommendation
However, the bias inherent in user written text, often used for PTG model training, can inadvertently associate different levels of linguistic quality with users' protected attributes.
FRAME: Evaluating Rationale-Label Consistency Metrics for Free-Text Rationales
Following how humans communicate, free-text rationales aim to use natural language to explain neural language model (LM) behavior.
Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning
Surprisingly, we also find that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity.
ER-Test: Evaluating Explanation Regularization Methods for Language Models
to align with human rationales (Which input tokens would humans focus on?).
FedShuffle: Recipes for Better Use of Local Work in Federated Learning
The practice of applying several local updates before aggregation across clients has been empirically shown to be a successful approach to overcoming the communication bottleneck in Federated Learning (FL).
Detection, Disambiguation, Re-ranking: Autoregressive Entity Linking as a Multi-Task Problem
Without the use of a knowledge base or candidate sets, our model sets a new state of the art in two benchmark datasets of entity linking: COMETA in the biomedical domain, and AIDA-CoNLL in the news domain.
Federated Learning with Partial Model Personalization
We consider two federated learning algorithms for training partially personalized models, where the shared and personal parameters are updated either simultaneously or alternately on the devices.
Measuring Self-Supervised Representation Quality for Downstream Classification using Discriminative Features
Fine-tuning with Q-Score regularization can boost the linear probing accuracy of SSL models by up to 5. 8% on ImageNet-100 and 3. 7% on ImageNet-1K compared to their baselines.
BARACK: Partially Supervised Group Robustness With Guarantees
Theoretically, we provide generalization bounds for our approach in terms of the worst-group performance, which scale with respect to both the total number of training points and the number of training points with group labels.
A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision
In this paper, we focus on teasing out what parts of the language supervision are essential for training zero-shot image classification models.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
An extractive rationale explains a language model's (LM's) prediction on a given task instance by highlighting the text inputs that most influenced the prediction.
On Tilted Losses in Machine Learning: Theory and Applications
Finally, we demonstrate that TERM can be used for a multitude of applications in machine learning, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance.
MSD: Saliency-aware Knowledge Distillation for Multimodal Understanding
The idea aims at mimicking a teacher's modality-specific predictions by introducing auxiliary loss terms for each modality.
Alternating Direction Method of Multipliers for Quantization
For such optimization problems, we study the performance of the Alternating Direction Method of Multipliers for Quantization ($\texttt{ADMM-Q}$) algorithm, which is a variant of the widely-used ADMM method applied to our discrete optimization problem.
Tilted Empirical Risk Minimization
Empirical risk minimization (ERM) is typically designed to perform well on the average loss, which can result in estimators that are sensitive to outliers, generalize poorly, or treat subgroups unfairly.
Non-convex Min-Max Optimization: Applications, Challenges, and Recent Theoretical Advances
The min-max optimization problem, also known as the saddle point problem, is a classical optimization problem which is also studied in the context of zero-sum games.
FedDANE: A Federated Newton-Type Method
Federated learning aims to jointly learn statistical models over massively distributed remote devices.
When Does Non-Orthogonal Tensor Decomposition Have No Spurious Local Minima?
We study the optimization problem for decomposing $d$ dimensional fourth-order Tensors with $k$ non-orthogonal components.
Fair Resource Allocation in Federated Learning
Federated learning involves training statistical models in massive, heterogeneous networks.
Training generative networks using random discriminators
This design helps us to avoid the min-max formulation and leads to an optimization problem that is stable and could be solved efficiently.
Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methods
In this paper, we study the problem in the non-convex regime and show that an \varepsilon--first order stationary point of the game can be computed when one of the player's objective can be optimized to global optimality efficiently.
Federated Optimization in Heterogeneous Networks
Theoretically, we provide convergence guarantees for our framework when learning over data from non-identical distributions (statistical heterogeneity), and while adhering to device-level systems constraints by allowing each participating device to perform a variable amount of work (systems heterogeneity).
Solving Non-Convex Non-Concave Min-Max Games Under Polyak-Łojasiewicz Condition
In this short note, we consider the problem of solving a min-max zero-sum game.
On the Convergence and Robustness of Training GANs with Regularized Optimal Transport
A popular GAN formulation is based on the use of Wasserstein distance as a metric between probability distributions.
Federated Multi-Task Learning
Federated learning poses new statistical and systems challenges in training machine learning models over distributed networks of devices.
