Search Results for author:
Found 42 papers, 16 papers with code
On Learning Representations for Tabular Data Distillation
Dataset distillation generates a small set of information-rich instances from a large dataset, resulting in reduced storage requirements, privacy or copyright risks, and computational costs for downstream modeling, though much of the research has focused on the image data modality.
Quantifying artificial intelligence through algebraic generalization
Thus, algebraic circuit complexity theory - the study of algebraic expressions as circuit models (i. e., directed acyclic graphs) - is a natural framework to study the complexity of symbolic computation.
Dense Associative Memory Through the Lens of Random Features
Dense Associative Memories are high storage capacity variants of the Hopfield networks that are capable of storing a large number of memory patterns in the weights of the network of a given size.
WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models
The need for effective unlearning mechanisms in large language models (LLMs) is increasingly urgent, driven by the necessity to adhere to data regulations and foster ethical generative AI practices.
Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning
Post-training of pre-trained LLMs, which typically consists of the supervised fine-tuning (SFT) stage and the preference learning (RLHF or DPO) stage, is crucial to effective and safe LLM applications.
Neural Reasoning Networks: Efficient Interpretable Neural Networks With Automatic Textual Explanations
Recent advances in machine learning have led to a surge in adoption of neural networks for various tasks, but lack of interpretability remains an issue for many others in which an understanding of the features influencing the prediction is necessary to ensure fairness, safety, and legal compliance.
On the Utility of Domain-Adjacent Fine-Tuned Model Ensembles for Few-shot Problems
Large Language Models (LLMs) have been observed to perform well on a wide range of downstream tasks when fine-tuned on domain-specific data.
Learning positional encodings in transformers depends on initialization
Critically, we find that the choice of initialization of a learnable PE greatly influences its ability to learn accurate PEs that lead to enhanced generalization.
What makes Models Compositional? A Theoretical View: With Supplement
We then show how various existing general and special purpose sequence processing models (such as recurrent, convolution and attention-based ones) fit this definition and use it to analyze their compositional complexity.
Enhancing In-context Learning via Linear Probe Calibration
However, applying ICL in real cases does not scale with the number of samples, and lacks robustness to different prompt templates and demonstration permutations.
Single-shot General Hyper-parameter Optimization for Federated Learning
We address the problem of hyper-parameter optimization (HPO) for federated learning (FL-HPO).
Compositional Program Generation for Few-Shot Systematic Generalization
Compositional generalization is a key ability of humans that enables us to learn new concepts from only a handful examples.
End-to-end Differentiable Clustering with Associative Memories
Clustering is a widely used unsupervised learning technique involving an intensive discrete optimization problem.
Model Sparsity Can Simplify Machine Unlearning
We show in both theory and practice that model sparsity can boost the multi-criteria unlearning performance of an approximate unlearner, closing the approximation gap, while continuing to be efficient.
What Is Missing in IRM Training and Evaluation? Challenges and Solutions
We propose a new IRM variant to address this limitation based on a novel viewpoint of ensemble IRM games as consensus-constrained bi-level optimization.
Toward Theoretical Guidance for Two Common Questions in Practical Cross-Validation based Hyperparameter Selection
We show, to our knowledge, the first theoretical treatments of two common questions in cross-validation based hyperparameter selection: (1) After selecting the best hyperparameter using a held-out set, we train the final model using {\em all} of the training data -- since this may or may not improve future generalization error, should one do this?
Navigating Ensemble Configurations for Algorithmic Fairness
Bias mitigators can improve algorithmic fairness in machine learning models, but their effect on fairness is often not stable across data splits.
Advancing Model Pruning via Bi-level Optimization
To reduce the computation overhead, various efficient 'one-shot' pruning methods have been developed, but these schemes are usually unable to find winning tickets as good as IMP.
Min-Max Bilevel Multi-objective Optimization with Applications in Machine Learning
We consider a generic min-max multi-objective bilevel optimization problem with applications in robust machine learning such as representation learning and hyperparameter optimization.
Single-shot Hyper-parameter Optimization for Federated Learning: A General Algorithm & Analysis
We address the relatively unexplored problem of hyper-parameter optimization (HPO) for federated learning (FL-HPO).
An Empirical Study of Modular Bias Mitigators and Ensembles
A popular approach to train more stable models is ensemble learning.
FLoRA: Single-shot Hyper-parameter Optimization for Federated Learning
We address the relatively unexplored problem of hyper-parameter optimization (HPO) for federated learning (FL-HPO).
Federated Nearest Neighbor Classification with a Colony of Fruit-Flies: With Supplement
The mathematical formalization of a neurological mechanism in the olfactory circuit of a fruit-fly as a locality sensitive hash (Flyhash) and bloom filter (FBF) has been recently proposed and "reprogrammed" for various machine learning tasks such as similarity search, outlier detection and text embeddings.
Pipeline Combinators for Gradual AutoML
Automated machine learning (AutoML) can make data scientists more productive.
Sign-MAML: Efficient Model-Agnostic Meta-Learning by SignSGD
The key enabling technique is to interpret MAML as a bilevel optimization (BLO) problem and leverage the sign-based SGD(signSGD) as a lower-level optimizer of BLO.
Designing Machine Learning Pipeline Toolkit for AutoML Surrogate Modeling Optimization
The pipeline optimization problem in machine learning requires simultaneous optimization of pipeline structures and parameter adaptation of their elements.
Bandit Limited Discrepancy Search and Application to Machine Learning Pipeline Optimization
Optimizing a machine learning (ML) pipeline has been an important topic of AI and ML.
Leveraging Theoretical Tradeoffs in Hyperparameter Selection for Improved Empirical Performance
The tradeoffs in the excess risk incurred from data-driven learning of a single model has been studied by decomposing the excess risk into approximation, estimation and optimization errors.
Overcoming Catastrophic Forgetting via Direction-Constrained Optimization
We study the principal directions of the trajectory of the optimizer after convergence and show that traveling along a few top principal directions can quickly bring the parameters outside the cone but this is not the case for the remaining directions.
Learning to Generate Image Source-Agnostic Universal Adversarial Perturbations
The resulting scheme for meta-learning a UAP generator (i) has better performance (50% higher ASR) than baselines such as Projected Gradient Descent, (ii) has better performance (37% faster) than the vanilla L2O and MAML frameworks (when applicable), and (iii) is able to simultaneously handle UAP generation for different victim models and image data sources.
Neural Neighborhood Encoding for Classification
Inspired by the fruit-fly olfactory circuit, the Fly Bloom Filter [Dasgupta et al., 2018] is able to efficiently summarize the data with a single pass and has been used for novelty detection.
Lale: Consistent Automated Machine Learning
Automated machine learning makes it easier for data scientists to develop pipelines by searching over possible choices for hyperparameters, algorithms, and even pipeline topologies.
Solving Constrained CASH Problems with ADMM
The CASH problem has been widely studied in the context of automated configurations of machine learning (ML) pipelines and various solvers and toolkits are available.
How can AI Automate End-to-End Data Science?
Data science is labor-intensive and human experts are scarce but heavily involved in every aspect of it.
Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
The rapid advancement of artificial intelligence (AI) is changing our lives in many ways.
Type-Driven Automated Learning with Lale
Machine-learning automation tools, ranging from humble grid-search to hyperopt, auto-sklearn, and TPOT, help explore large search spaces of possible pipelines.
An ADMM Based Framework for AutoML Pipeline Configuration
We study the AutoML problem of automatically configuring machine learning pipelines by jointly selecting algorithms and their appropriate hyper-parameters for all steps in supervised learning pipelines.
Plug-and-play dual-tree algorithm runtime analysis
In this paper, we present a problem-independent runtime guarantee for any dual-tree algorithm using the cover tree, separating out the problem-dependent and the problem-independent elements.
Which Space Partitioning Tree to Use for Search?
We consider the task of nearest-neighbor search with the class of binary-space-partitioning trees, which includes kd-trees, principal axis trees and random projection trees, and try to rigorously answer the question which tree to use for nearest-neighbor search?''
Maximum Inner-Product Search using Tree Data-structures
Finally we present a new data structure for increasing the efficiency of the dual-tree algorithm.
Rank-Approximate Nearest Neighbor Search: Retaining Meaning and Speed in High Dimensions
The long-standing problem of efficient nearest-neighbor (NN) search has ubiquitous applications ranging from astrophysics to MP3 fingerprinting to bioinformatics to movie recommendations.
Linear-time Algorithms for Pairwise Statistical Problems
Several key computational bottlenecks in machine learning involve pairwise distance computations, including all-nearest-neighbors (finding the nearest neighbor(s) for each point, e. g. in manifold learning) and kernel summations (e. g. in kernel density estimation or kernel machines).
