Search Results for author:
Found 94 papers, 63 papers with code
Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models
Recent work computes non-vacuous compression-based generalization bounds for LLMs, but these bounds are vacuous for large models at the billion-parameter scale.
LiveBench: A Challenging, Contamination-Free LLM Benchmark
In this work, we introduce a new benchmark for LLMs designed to be immune to both test set contamination and the pitfalls of LLM judging and human crowdsourcing.
Just How Flexible are Neural Networks in Practice?
Our findings indicate that: (1) standard optimizers find minima where the model can only fit training sets with significantly fewer samples than it has parameters; (2) convolutional networks are more parameter-efficient than MLPs and ViTs, even on randomly labeled data; (3) while stochastic training is thought to have a regularizing effect, SGD actually finds minima that fit more training data than full-batch gradient descent; (4) the difference in capacity to fit correctly labeled and incorrectly labeled samples can be predictive of generalization; (5) ReLU activation functions result in finding minima that fit more data despite being designed to avoid vanishing and exploding gradients in deep architectures.
Large Language Models Must Be Taught to Know What They Don't Know
We show that a thousand graded examples are sufficient to outperform baseline methods and that training through the features of a model is necessary for good performance and tractable for large open-source models when using LoRA.
Compute Better Spent: Replacing Dense Layers with Structured Matrices
We show that different structures often require drastically different initialization scales and learning rates, which are crucial to performance, especially as models scale.
Adaptive Rentention & Correction for Continual Learning
A common problem in continual learning is the classification layer's bias towards the most recent task.
Measuring Style Similarity in Diffusion Models
We also propose a method to extract style descriptors that can be used to attribute style of a generated image to the images used in the training dataset of a text-to-image model.
Generating Potent Poisons and Backdoors from Scratch with Guided Diffusion
As a result, we may be able to craft more potent poisons by carefully choosing the base samples.
TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks
Similar to large language models, PFNs make use of pretraining and in-context learning to achieve strong performance on new tasks in a single forward pass.
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
Detecting text generated by modern large language models is thought to be hard, as both LLMs and humans can exhibit a wide range of complex behaviors.
Non-Vacuous Generalization Bounds for Large Language Models
Modern language models can contain billions of parameters, raising the question of whether they can generalize beyond the training data or simply parrot their training corpora.
Perspectives on the State and Future of Deep Learning -- 2023
The goal of this series is to chronicle opinions and issues in the field of machine learning as they stand today and as they change over time.
Simplifying Neural Network Training Under Class Imbalance
Real-world datasets are often highly class-imbalanced, which can adversely impact the performance of deep learning models.
A Simple and Efficient Baseline for Data Attribution on Images
Our approach serves as a simple and efficient baseline for data attribution on images.
Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks
Battle of the Backbones (BoB) makes this choice easier by benchmarking a diverse suite of pretrained models, including vision-language models, those trained via self-supervised learning, and the Stable Diffusion backbone, across a diverse set of computer vision tasks ranging from classification to object detection to OOD generalization and more.
NEFTune: Noisy Embeddings Improve Instruction Finetuning
We show that language model finetuning can be improved, sometimes dramatically, with a simple augmentation.
Baseline Defenses for Adversarial Attacks Against Aligned Language Models
We find that the weakness of existing discrete optimizers for text, combined with the relatively high costs of optimization, makes standard adaptive attacks more challenging for LLMs.
Seeing in Words: Learning to Classify through Language Bottlenecks
Neural networks for computer vision extract uninterpretable features despite achieving high accuracy on benchmarks.
Bring Your Own Data! Self-Supervised Evaluation for Large Language Models
With the rise of Large Language Models (LLMs) and their ubiquitous deployment in diverse domains, measuring language model behavior on realistic data is imperative.
On the Reliability of Watermarks for Large Language Models
We also consider a range of new detection schemes that are sensitive to short spans of watermarked text embedded inside a large document, and we compare the robustness of watermarking to other kinds of detectors.
Understanding and Mitigating Copying in Diffusion Models
While it is widely believed that duplicated images in the training set are responsible for content replication at inference time, we observe that the text conditioning of the model plays a similarly important role.
What Can We Learn from Unlearnable Datasets?
First, it is widely believed that neural networks trained on unlearnable datasets only learn shortcuts, simpler rules that are not useful for generalization.
When Do Neural Nets Outperform Boosted Trees on Tabular Data?
To this end, we conduct the largest tabular data analysis to date, comparing 19 algorithms across 176 datasets, and we find that the 'NN vs. GBDT' debate is overemphasized: for a surprisingly high number of datasets, either the performance difference between GBDTs and NNs is negligible, or light hyperparameter tuning on a GBDT is more important than choosing between NNs and GBDTs.
A Cookbook of Self-Supervised Learning
Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning.
The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning
No free lunch theorems for supervised learning state that no learner can solve all problems or that all learners achieve exactly the same accuracy on average over a uniform distribution on learning problems.
Universal Guidance for Diffusion Models
Typical diffusion models are trained to accept a particular form of conditioning, most commonly text, and cannot be conditioned on other modalities without retraining.
Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery
In the text-to-image setting, the method creates hard prompts for diffusion models, allowing API users to easily generate, discover, and mix and match image concepts without prior knowledge on how to prompt the model.
Exploring and Exploiting Decision Boundary Dynamics for Adversarial Robustness
However, it is unclear whether existing robust training methods effectively increase the margin for each vulnerable point during training.
What do Vision Transformers Learn? A Visual Exploration
In addition, we show that ViTs maintain spatial information in all layers except the final layer.
Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models
Cutting-edge diffusion models produce images with high quality and customizability, enabling them to be used for commercial art and graphic design purposes.
Chroma-VAE: Mitigating Shortcut Learning with Generative Classifiers
Deep neural networks are susceptible to shortcut learning, using simple features to achieve low training loss without discovering essential semantic structure.
PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization
While there has been progress in developing non-vacuous generalization bounds for deep neural networks, these bounds tend to be uninformative about why deep learning works.
K-SAM: Sharpness-Aware Minimization at the Speed of SGD
Sharpness-Aware Minimization (SAM) has recently emerged as a robust technique for improving the accuracy of deep neural networks.
Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queries
As industrial applications are increasingly automated by machine learning models, enforcing personal data ownership and intellectual property rights requires tracing training data back to their rightful owners.
Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition
Our search outputs a suite of models which Pareto-dominate all other high-performance architectures and existing bias mitigation methods in terms of accuracy and fairness, often by large margins, on the two most widely used datasets for face identification, CelebA and VGGFace2.
Thinking Two Moves Ahead: Anticipating Other Users Improves Backdoor Attacks in Federated Learning
Federated learning is particularly susceptible to model poisoning and backdoor attacks because individual users have direct control over the training data and model updates.
How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization
Despite the clear performance benefits of data augmentations, little is known about why they are so effective.
The Lie Derivative for Measuring Learned Equivariance
In order to better understand the role of equivariance in recent vision models, we introduce the Lie derivative, a method for measuring equivariance with strong mathematical foundations and minimal hyperparameters.
Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise
We observe that the generative behavior of diffusion models is not strongly dependent on the choice of image degradation, and in fact an entire family of generative models can be constructed by varying this choice.
Transfer Learning with Deep Tabular Models
In this work, we demonstrate that upstream data gives tabular neural networks a decisive advantage over widely used GBDT models.
Autoregressive Perturbations for Data Poisoning
Unfortunately, existing methods require knowledge of both the target architecture and the complete dataset so that a surrogate network can be trained, the parameters of which are used to generate the attack.
Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors
Deep learning is increasingly moving towards a transfer learning paradigm whereby large foundation models are fine-tuned on downstream tasks, starting from an initialization learned on the source task.
Poisons that are learned faster are more effective
We advocate for evaluating poisons in terms of peak test accuracy.
A Deep Dive into Dataset Imbalance and Bias in Face Identification
This is an unfortunate omission, as 'imbalance' is a more complex matter in identification; imbalance may arise in not only the training data, but also the testing data, and furthermore may affect the proportion of identities belonging to each demographic group or the number of images belonging to each identity.
Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective
We also use decision boundary methods to visualize double descent phenomena.
Bayesian Model Selection, the Marginal Likelihood, and Generalization
We provide a partial remedy through a conditional marginal likelihood, which we show is more aligned with generalization, and practically valuable for large-scale hyperparameter learning, such as in deep kernel learning.
End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking
Algorithmic extrapolation can be achieved through recurrent systems, which can be iterated many times to solve difficult reasoning problems.
Fishing for User Data in Large-Batch Federated Learning via Gradient Magnification
Federated learning (FL) has rapidly risen in popularity due to its promise of privacy and efficiency.
Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentations
Existing techniques for model inversion typically rely on hard-to-tune regularizers, such as total variation or feature regularization, which must be individually calibrated for each network in order to produce adequate images.
Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Models
A central tenet of Federated learning (FL), which trains models without centralizing user data, is privacy.
Active Learning at the ImageNet Scale
Active learning (AL) algorithms aim to identify an optimal subset of data for annotation, such that deep neural networks (DNN) can achieve better performance when trained on this labeled subset.
Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Models
Federated learning has quickly gained popularity with its promises of increased user privacy and efficiency.
Comparing Human and Machine Bias in Face Recognition
Much recent research has uncovered and discussed serious concerns of bias in facial analysis technologies, finding performance disparities between groups of people based on perceived gender, skin type, lighting condition, etc.
Identification of Attack-Specific Signatures in Adversarial Examples
The adversarial attack literature contains a myriad of algorithms for crafting perturbations which yield pathological behavior in neural networks.
Contrastive Learning is Just Meta-Learning
Contrastive learning has recently taken off as a paradigm for learning from unlabeled data.
Protecting Proprietary Data: Poisoning for Secure Dataset Release
These two behaviors can be in conflict as an organization wants to prevent competitors from using their own data to replicate the performance of their proprietary models.
A Flexible Measurement of Diversity in Datasets with Random Network Distillation
We validate and deploy this metric on both images and text.
A Closer Look at Distribution Shifts and Out-of-Distribution Generalization on Graphs
We observe that in most cases, we need both a suitable domain generalization algorithm and a strong GNN backbone model to optimize out-of-distribution test performance.
Thinking Deeper With Recurrent Networks: Logical Extrapolation Without Overthinking
Classical machine learning systems perform best when they are trained and tested on the same distribution, and they lack a mechanism to increase model power after training is complete.
An Investigation into the Role of Author Demographics in ICLR Participation and Review
As machine learning conferences grow rapidly, many are concerned that individuals will be left behind on the basis of traits such as gender and geography.
DP-InstaHide: Data Augmentations Provably Enhance Guarantees Against Dataset Manipulations
Data poisoning and backdoor attacks manipulate training data to induce security breaches in a victim model.
Stochastic Training is Not Necessary for Generalization
It is widely believed that the implicit regularization of SGD is fundamental to the impressive generalization behavior we observe in neural networks.
Towards Transferable Adversarial Attacks on Vision Transformers
We evaluate the transferability of attacks on state-of-the-art ViTs, CNNs and robustly trained CNNs.
Datasets for Studying Generalization from Easy to Hard Examples
We describe new datasets for studying generalization from easy to hard examples.
Where do Models go Wrong? Parameter-Space Saliency Maps for Explainability
We find that samples which cause similar parameters to malfunction are semantically similar.
Adversarial Examples Make Strong Poisons
The adversarial machine learning literature is largely partitioned into evasion attacks on testing data and poisoning attacks on training data.
MetaBalance: High-Performance Neural Networks for Class-Imbalanced Data
Class-imbalanced data, in which some classes contain far more samples than others, is ubiquitous in real-world applications.
Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch
In contrast, the Hidden Trigger Backdoor Attack achieves poisoning without placing a trigger into the training data at all.
Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks
In this work, we show that recurrent networks trained to solve simple problems with few recurrent steps can indeed solve much more complex problems simply by performing additional recurrences during inference.
SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
We devise a hybrid deep learning approach to solving tabular data problems.
The Intrinsic Dimension of Images and Its Impact on Learning
We find that common natural image datasets indeed have very low intrinsic dimension relative to the high number of pixels in the images.
DP-InstaHide: Provably Defusing Poisoning and Backdoor Attacks with Differentially Private Data Augmentations
The InstaHide method has recently been proposed as an alternative to DP training that leverages supposed privacy properties of the mixup augmentation, although without rigorous guarantees.
What Doesn't Kill You Makes You Robust(er): How to Adversarially Train against Data Poisoning
Data poisoning is a threat model in which a malicious actor tampers with training data to manipulate outcomes at inference time.
The Uncanny Similarity of Recurrence and Depth
It is widely believed that deep neural networks contain layer specialization, wherein neural networks extract hierarchical features representing edges and patterns in shallow layers and complete objects in deeper layers.
Preventing Unauthorized Use of Proprietary Data: Poisoning for Secure Dataset Release
Large organizations such as social media companies continually release data, for example user images.
Technical Challenges for Training Fair Neural Networks
As machine learning algorithms have been widely deployed across applications, many concerns have been raised over the fairness of their predictions, especially in high stakes settings (such as facial recognition and medical imaging).
LowKey: Leveraging Adversarial Attacks to Protect Social Media Users from Facial Recognition
Facial recognition systems are increasingly deployed by private corporations, government agencies, and contractors for consumer services and mass surveillance programs alike.
Just How Toxic is Data Poisoning? A Benchmark for Backdoor and Data Poisoning Attacks
Data poisoning and backdoor attacks manipulate training data in order to cause models to fail during inference.
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses
As machine learning systems grow in scale, so do their training data requirements, forcing practitioners to automate and outsource the curation of training data in order to achieve state-of-the-art performance.
Analyzing the Machine Learning Conference Review Process
Members of the machine learning community are likely to overhear allegations ranging from randomness of acceptance decisions to institutional bias.
Strong Data Augmentation Sanitizes Poisoning and Backdoor Attacks Without an Accuracy Tradeoff
Data poisoning and backdoor attacks manipulate victim models by maliciously modifying training data.
Data Augmentation for Meta-Learning
Conventional image classifiers are trained by randomly sampling mini-batches of images.
Random Network Distillation as a Diversity Metric for Both Image and Text Generation
We validate and deploy this metric on both images and text.
An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process
Members of the machine learning community are likely to overhear allegations ranging from randomness of acceptance decisions to institutional bias.
Prepare for the Worst: Generalizing across Domain Shifts with Adversarial Batch Normalization
Adversarial training is the industry standard for producing models that are robust to small adversarial perturbations.
Encoding Robustness to Image Style via Adversarial Feature Perturbations
We adapt adversarial training by directly perturbing feature statistics, rather than image pixels, to produce models that are robust to various unseen distributional shifts.
Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks
Data poisoning and backdoor attacks manipulate training data in order to cause models to fail during inference.
Adversarial Attacks on Machine Learning Systems for High-Frequency Trading
Algorithmic trading systems are often completely automated, and deep learning is increasingly receiving attention in this domain.
Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks
In doing so, we introduce and verify several hypotheses for why meta-learned models perform better.
WITCHcraft: Efficient PGD attacks with random step size
State-of-the-art adversarial attacks on neural networks use expensive iterative methods and numerous random restarts from different initial points.
Adversarially Robust Few-Shot Learning: A Meta-Learning Approach
Previous work on adversarially robust neural networks for image classification requires large training sets and computationally expensive training procedures.
Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike.
Understanding Generalization through Visualizations
The power of neural networks lies in their ability to generalize to unseen data, yet the underlying reasons for this phenomenon remain elusive.
Adversarially Robust Distillation
In addition to producing small models with high test accuracy like conventional distillation, ARD also passes the superior robustness of large networks onto the student.
