Search Results for author:
Found 22 papers, 8 papers with code
R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training
With as little as 0. 01% additional compute overhead, R&B matches or exceeds the performance of state-of-the-art data mixing strategies.
Compute Optimal Scaling of Skills: Knowledge vs Reasoning
Scaling laws are a critical component of the LLM development pipeline, most famously as a way to forecast training decisions such as 'compute-optimally' trading-off parameter count and dataset size, alongside a more recent growing list of other crucial decisions.
Tabby: Tabular Data Synthesis with Language Models
While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less attention.
MLGym: A New Framework and Benchmark for Advancing AI Research Agents
Our MLGym framework makes it easy to add new tasks, integrate and evaluate models or agents, generate synthetic data at scale, as well as develop new learning algorithms for training agents on AI research tasks.
Stronger Than You Think: Benchmarking Weak Supervision on Realistic Tasks
Weak supervision (WS) is a popular approach for label-efficient learning, leveraging diverse sources of noisy but inexpensive weak labels to automatically annotate training data.
MoRe Fine-Tuning with 10x Fewer Parameters
Parameter-efficient fine-tuning (PEFT) techniques have unlocked the potential to cheaply and easily specialize large pretrained models.
Pretrained Hybrids with MAD Skills
Manticore $\textit{automates the design of hybrid architectures}$ while reusing pretrained models to create $\textit{pretrained}$ hybrids.
Lifting Weak Supervision To Structured Prediction
Weak supervision (WS) is a rich set of techniques that produce pseudolabels by aggregating easily obtained but potentially noisy label estimates from a variety of sources.
AutoML for Climate Change: A Call to Action
The challenge that climate change poses to humanity has spurred a rapidly developing field of artificial intelligence research focused on climate change applications.
AutoWS-Bench-101: Benchmarking Automated Weak Supervision with 100 Labels
While it has been used successfully in many domains, weak supervision's application scope is limited by the difficulty of constructing labeling functions for domains with complex or high-dimensional features.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Generative Modeling Helps Weak Supervision (and Vice Versa)
The model outperforms baseline weak supervision label models on a number of multiclass image classification datasets, improves the quality of generated images, and further improves end-model performance through data augmentation with synthetic samples.
Universalizing Weak Supervision
We apply this technique to important problems previously not tackled by WS frameworks including learning to rank, regression, and learning in hyperbolic space.
NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks
This makes the performance of NAS approaches in more diverse areas poorly understood.
Rethinking Neural Operations for Diverse Tasks
An important goal of AutoML is to automate-away the design of neural networks on new tasks in under-explored domains.
Decoding and Diversity in Machine Translation
This makes human-level BLEU a misleading benchmark in that modern MT systems cannot approach human-level BLEU while simultaneously maintaining human-level translation diversity.
Model Weight Theft With Just Noise Inputs: The Curious Case of the Petulant Attacker
Bernoulli noise inputs.
Deep Connectomics Networks: Neural Network Architectures Inspired by Neuronal Networks
The interplay between inter-neuronal network topology and cognition has been studied deeply by connectomics researchers and network scientists, which is crucial towards understanding the remarkable efficacy of biological neural networks.
Grassmannian Packings in Neural Networks: Learning with Maximal Subspace Packings for Diversity and Anti-Sparsity
Kernel sparsity ("dying ReLUs") and lack of diversity are commonly observed in CNN kernels, which decreases model capacity.
Deep connectomics networks: Results from neural network architectures inspired from network neuroscience
Using the teachings from network neuroscience and connectomics, we show improvements over the ResNet architecture, we show a possible connection between early training and the spectral properties of the network, and we show the trainability of a DNN based on the neuronal network of C. Elegans.
Learning from discriminative feature feedback
We consider the problem of learning a multi-class classifier from labels as well as simple explanations that we call "discriminative features".
Using Deep Siamese Neural Networks to Speed up Natural Products Research
Computing a similarity score between 2D NMR spectra for a novel compound and a compound whose structure is known helps determine the structure of the novel compound.
