Search Results for author:
Found 51 papers, 22 papers with code
Data Acquisition: A New Frontier in Data-centric AI
As Machine Learning (ML) systems continue to grow, the demand for relevant and comprehensive datasets becomes imperative.
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
Evaluating retrieval-augmented generation (RAG) systems traditionally relies on hand annotations for input queries, passages to retrieve, and responses to generate.
Exploration with Principles for Diverse AI Supervision
Training large transformers using next-token prediction has given rise to groundbreaking advancements in AI.
Reinforcement Learning (RL)
Unsupervised Reinforcement Learning
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
The ML community is rapidly exploring techniques for prompting language models (LMs) and for stacking them into pipelines that solve complex tasks.
Ring Attention with Blockwise Transformers for Near-Infinite Context
Transformers have emerged as the architecture of choice for many state-of-the-art AI models, showcasing exceptional performance across a wide range of AI applications.
How is ChatGPT's behavior changing over time?
We find that the performance and behavior of both GPT-3. 5 and GPT-4 can vary greatly over time.
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
There is a rapidly growing number of large language models (LLMs) that users can query for a fee.
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks
Recent advances in instruction-following large language models (LLMs) have led to dramatic improvements in a range of NLP tasks.
Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM).
Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking
Neural information retrieval (IR) systems have progressed rapidly in recent years, in large part due to the release of publicly available benchmarking tasks.
MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
We present MegaBlocks, a system for efficient Mixture-of-Experts (MoE) training on GPUs.
HAPI: A Large-scale Longitudinal Dataset of Commercial ML API Predictions
HAPI is the first large-scale dataset of ML API usages and is a unique resource for studying ML-as-a-service (MLaaS).
Estimating and Explaining Model Performance When Both Covariates and Labels Shift
We further propose SEES, an algorithmic framework to characterize the distribution shift under SJS and to estimate a model's performance on new data without any labels.
PLAID: An Efficient Engine for Late Interaction Retrieval
PLAID uses centroid interaction as well as centroid pruning, a mechanism for sparsifying the bag of centroids, within a highly-optimized engine to reduce late interaction search latency by up to 7$\times$ on a GPU and 45$\times$ on a CPU against vanilla ColBERTv2, while continuing to deliver state-of-the-art retrieval quality.
What can Data-Centric AI Learn from Data and ML Engineering?
Data-centric AI is a new and exciting research topic in the AI community, but many organizations already build and maintain various "data-centric" applications whose goal is to produce high quality data.
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
Neural information retrieval (IR) has greatly advanced search and other knowledge-intensive language tasks.
Ranked #6 on
Zero-shot Text Search
on BEIR
Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression
This paper investigates deep neural network (DNN) compression from the perspective of compactly representing and storing trained parameters.
DistIR: An Intermediate Representation and Simulator for Efficient Neural Network Distribution
The rapidly growing size of deep neural network (DNN) models and datasets has given rise to a variety of distribution strategies such as data, tensor-model, pipeline parallelism, and hybrid combinations thereof.
Hindsight: Posterior-guided training of retrievers for improved open-ended generation
Many text generation systems benefit from using a retriever to retrieve passages from a textual knowledge corpus (e. g., Wikipedia) which are then provided as additional context to the generator.
How Did the Model Change? Efficiently Assessing Machine Learning API Shifts
ML prediction APIs from providers like Amazon and Google have made it simple to use ML in applications.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Did the Model Change? Efficiently Assessing Machine Learning API Shifts
This motivated us to formulate the API shift assessment problem at a more fine-grained level as estimating how the API model's confusion matrix changes over time when the data distribution is constant.
Proof: Accelerating Approximate Aggregation Queries with Expensive Predicates
Given a dataset $\mathcal{D}$, we are interested in computing the mean of a subset of $\mathcal{D}$ which matches a predicate.
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
In this paper, we show how different types of parallelism methods (tensor, pipeline, and data parallelism) can be composed to scale to thousands of GPUs and models with trillions of parameters.
Efficient Online ML API Selection for Multi-Label Classification Tasks
In this work, we propose FrugalMCT, a principled framework that adaptively selects the APIs to use for different data in an online fashion while respecting user's budget.
Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval
Multi-hop reasoning (i. e., reasoning across two or more documents) is a key ingredient for NLP models that leverage large corpora to exhibit broad knowledge.
Jointly Optimizing Preprocessing and Inference for DNN-based Visual Analytics
This runtime engine a) efficiently pipelines preprocessing and DNN execution for inference, b) places preprocessing operations on the CPU or GPU in a hardware- and input-aware manner, and c) efficiently manages memory and threading for high throughput execution.
Relevance-guided Supervision for OpenQA with ColBERT
In much recent work, the retriever is a learned component that uses coarse-grained vector representations of questions and passages.
Similarity Search for Efficient Active Learning and Search of Rare Concepts
Many active learning and search approaches are intractable for large-scale industrial settings with billions of unlabeled examples.
Sparse GPU Kernels for Deep Learning
In this work, we study sparse matrices from deep learning applications and identify favorable properties that can be exploited to accelerate computation.
Memory-Efficient Pipeline-Parallel DNN Training
Many state-of-the-art ML results have been obtained by scaling up the number of parameters in existing models.
FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply
Prediction APIs offered for a fee are a fast-growing industry and an important part of machine learning as a service.
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
ColBERT introduces a late interaction architecture that independently encodes the query and the document using BERT and then employs a cheap yet powerful interaction step that models their fine-grained similarity.
Model Assertions for Monitoring and Improving ML Models
We propose methods of using model assertions at all stages of ML system deployment, including runtime monitoring, validating labels, and continuously improving ML models.
Express: Lowering the Cost of Metadata-hiding Communication with Cryptographic Privacy
Existing systems for metadata-hiding messaging that provide cryptographic privacy properties have either high communication costs, high computation costs, or both.
Cryptography and Security
MLPerf Training Benchmark
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML.
LIT: Learned Intermediate Representation Training for Model Compression
In this work, we introduce Learned Intermediate representation Training (LIT), a novel model compression technique that outperforms a range of recent model compression techniques by leveraging the highly repetitive structure of modern DNNs (e. g., ResNet).
Selection via Proxy: Efficient Data Selection for Deep Learning
By removing hidden layers from the target model, using smaller architectures, and training for fewer epochs, we create proxies that are an order of magnitude faster to train.
Willump: A Statistically-Aware End-to-end Optimizer for Machine Learning Inference
First, Willump automatically cascades feature computation for classification queries: Willump classifies most data inputs using only high-value, low-cost features selected through empirical observations of ML model performance, improving query performance by up to 5x without statistically significant accuracy loss.
Select Via Proxy: Efficient Data Selection For Training Deep Networks
In our approach, we first train a small proxy model quickly, which we then use to estimate the utility of individual training data points, and then select the most informative ones for training the large target model.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
LIT: Block-wise Intermediate Representation Training for Model Compression
Knowledge distillation (KD) is a popular method for reducing the computational overhead of deep network inference, in which the output of a teacher model is used to train a smaller, faster student model.
Beyond Data and Model Parallelism for Deep Neural Networks
We also propose FlexFlow, a deep learning framework that uses guided randomized search of the SOAP space to find a fast parallelization strategy for a specific parallel machine.
Distributed, Parallel, and Cluster Computing
Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark
In this work, we analyze the entries from DAWNBench, which received optimized submissions from multiple industrial groups, to investigate the behavior of TTA as a metric as well as trends in the best-performing entries.
BlazeIt: Optimizing Declarative Aggregation and Limit Queries for Neural Network-Based Video Analytics
We introduce two new query optimization techniques in BlazeIt that are not supported by prior work.
Databases
Model Specialization for Inference Via End-to-End Distillation, Pruning, and Cascades
Thus, simply fine-tuning or transfer learn- ing from a general-purpose network inherits a large computational cost that may not be necessary for a given task.
To Index or Not to Index: Optimizing Exact Maximum Inner Product Search
The brute-force approach to solving exact MIPS is computationally expensive, thus spurring recent development of novel indexes and pruning techniques for this task.
Infrastructure for Usable Machine Learning: The Stanford DAWN Project
Despite incredible recent advances in machine learning, building machine learning applications remains prohibitively time-consuming and expensive for all but the best-trained, best-funded engineering organizations.
NoScope: Optimizing Neural Network Queries over Video at Scale
Given a target video, object to detect, and reference neural network, NoScope automatically searches for and trains a sequence, or cascade, of models that preserves the accuracy of the reference network but is specialized to the target video and are therefore far less computationally expensive.
Yggdrasil: An Optimized System for Training Deep Decision Trees at Scale
Deep distributed decision trees and tree ensembles have grown in importance due to the need to model increasingly large datasets.
MLlib: Machine Learning in Apache Spark
Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks.
