Search Results for author:
Found 58 papers, 27 papers with code
Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM).
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
The ML community is rapidly exploring techniques for prompting language models (LMs) and for stacking them into pipelines that solve complex tasks.
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
We integrate our constructs into the recent DSPy programming model for LMs, and present new strategies that allow DSPy to compile programs with LM Assertions into more reliable and accurate systems.
RAFT: Adapting Language Model to Domain Specific RAG
In this paper, we present Retrieval Augmented FineTuning (RAFT), a training recipe that improves the model's ability to answer questions in a "open-book" in-domain settings.
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
In this paper, we show how different types of parallelism methods (tensor, pipeline, and data parallelism) can be composed to scale to thousands of GPUs and models with trillions of parameters.
World Model on Million-Length Video And Language With Blockwise RingAttention
To address these challenges, we curate a large dataset of diverse videos and books, utilize the Blockwise RingAttention technique to scalably train on long sequences, and gradually increase context size from 4K to 1M tokens.
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
ColBERT introduces a late interaction architecture that independently encodes the query and the document using BERT and then employs a cheap yet powerful interaction step that models their fine-grained similarity.
Relevance-guided Supervision for OpenQA with ColBERT
In much recent work, the retriever is a learned component that uses coarse-grained vector representations of questions and passages.
Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval
Multi-hop reasoning (i. e., reasoning across two or more documents) is a key ingredient for NLP models that leverage large corpora to exhibit broad knowledge.
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
Neural information retrieval (IR) has greatly advanced search and other knowledge-intensive language tasks.
Ranked #6 on
Zero-shot Text Search
on BEIR
PLAID: An Efficient Engine for Late Interaction Retrieval
PLAID uses centroid interaction as well as centroid pruning, a mechanism for sparsifying the bag of centroids, within a highly-optimized engine to reduce late interaction search latency by up to 7$\times$ on a GPU and 45$\times$ on a CPU against vanilla ColBERTv2, while continuing to deliver state-of-the-art retrieval quality.
How is ChatGPT's behavior changing over time?
We find that the performance and behavior of both GPT-3. 5 and GPT-4 can vary greatly over time.
MLPerf Training Benchmark
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML.
MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
We present MegaBlocks, a system for efficient Mixture-of-Experts (MoE) training on GPUs.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Similarity Search for Efficient Active Learning and Search of Rare Concepts
Many active learning and search approaches are intractable for large-scale industrial settings with billions of unlabeled examples.
Ring Attention with Blockwise Transformers for Near-Infinite Context
Transformers have emerged as the architecture of choice for many state-of-the-art AI models, showcasing exceptional performance across a wide range of AI applications.
NoScope: Optimizing Neural Network Queries over Video at Scale
Given a target video, object to detect, and reference neural network, NoScope automatically searches for and trains a sequence, or cascade, of models that preserves the accuracy of the reference network but is specialized to the target video and are therefore far less computationally expensive.
Memory-Efficient Pipeline-Parallel DNN Training
Many state-of-the-art ML results have been obtained by scaling up the number of parameters in existing models.
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
Evaluating retrieval-augmented generation (RAG) systems traditionally relies on hand annotations for input queries, passages to retrieve, and responses to generate.
Sparse GPU Kernels for Deep Learning
In this work, we study sparse matrices from deep learning applications and identify favorable properties that can be exploited to accelerate computation.
Selection via Proxy: Efficient Data Selection for Deep Learning
By removing hidden layers from the target model, using smaller architectures, and training for fewer epochs, we create proxies that are an order of magnitude faster to train.
To Index or Not to Index: Optimizing Exact Maximum Inner Product Search
The brute-force approach to solving exact MIPS is computationally expensive, thus spurring recent development of novel indexes and pruning techniques for this task.
LIT: Learned Intermediate Representation Training for Model Compression
In this work, we introduce Learned Intermediate representation Training (LIT), a novel model compression technique that outperforms a range of recent model compression techniques by leveraging the highly repetitive structure of modern DNNs (e. g., ResNet).
Model Assertions for Monitoring and Improving ML Models
We propose methods of using model assertions at all stages of ML system deployment, including runtime monitoring, validating labels, and continuously improving ML models.
Express: Lowering the Cost of Metadata-hiding Communication with Cryptographic Privacy
Existing systems for metadata-hiding messaging that provide cryptographic privacy properties have either high communication costs, high computation costs, or both.
Cryptography and Security
HAPI: A Large-scale Longitudinal Dataset of Commercial ML API Predictions
HAPI is the first large-scale dataset of ML API usages and is a unique resource for studying ML-as-a-service (MLaaS).
Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark
In this work, we analyze the entries from DAWNBench, which received optimized submissions from multiple industrial groups, to investigate the behavior of TTA as a metric as well as trends in the best-performing entries.
Infrastructure for Usable Machine Learning: The Stanford DAWN Project
Despite incredible recent advances in machine learning, building machine learning applications remains prohibitively time-consuming and expensive for all but the best-trained, best-funded engineering organizations.
MLlib: Machine Learning in Apache Spark
Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks.
LIT: Block-wise Intermediate Representation Training for Model Compression
Knowledge distillation (KD) is a popular method for reducing the computational overhead of deep network inference, in which the output of a teacher model is used to train a smaller, faster student model.
Yggdrasil: An Optimized System for Training Deep Decision Trees at Scale
Deep distributed decision trees and tree ensembles have grown in importance due to the need to model increasingly large datasets.
Select Via Proxy: Efficient Data Selection For Training Deep Networks
In our approach, we first train a small proxy model quickly, which we then use to estimate the utility of individual training data points, and then select the most informative ones for training the large target model.
Model Specialization for Inference Via End-to-End Distillation, Pruning, and Cascades
Thus, simply fine-tuning or transfer learn- ing from a general-purpose network inherits a large computational cost that may not be necessary for a given task.
Beyond Data and Model Parallelism for Deep Neural Networks
We also propose FlexFlow, a deep learning framework that uses guided randomized search of the SOAP space to find a fast parallelization strategy for a specific parallel machine.
Distributed, Parallel, and Cluster Computing
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
Willump: A Statistically-Aware End-to-end Optimizer for Machine Learning Inference
First, Willump automatically cascades feature computation for classification queries: Willump classifies most data inputs using only high-value, low-cost features selected through empirical observations of ML model performance, improving query performance by up to 5x without statistically significant accuracy loss.
BlazeIt: Optimizing Declarative Aggregation and Limit Queries for Neural Network-Based Video Analytics
We introduce two new query optimization techniques in BlazeIt that are not supported by prior work.
Databases
FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply
Prediction APIs offered for a fee are a fast-growing industry and an important part of machine learning as a service.
Jointly Optimizing Preprocessing and Inference for DNN-based Visual Analytics
This runtime engine a) efficiently pipelines preprocessing and DNN execution for inference, b) places preprocessing operations on the CPU or GPU in a hardware- and input-aware manner, and c) efficiently manages memory and threading for high throughput execution.
Efficient Online ML API Selection for Multi-Label Classification Tasks
In this work, we propose FrugalMCT, a principled framework that adaptively selects the APIs to use for different data in an online fashion while respecting user's budget.
Proof: Accelerating Approximate Aggregation Queries with Expensive Predicates
Given a dataset $\mathcal{D}$, we are interested in computing the mean of a subset of $\mathcal{D}$ which matches a predicate.
Did the Model Change? Efficiently Assessing Machine Learning API Shifts
This motivated us to formulate the API shift assessment problem at a more fine-grained level as estimating how the API model's confusion matrix changes over time when the data distribution is constant.
How Did the Model Change? Efficiently Assessing Machine Learning API Shifts
ML prediction APIs from providers like Amazon and Google have made it simple to use ML in applications.
Hindsight: Posterior-guided training of retrievers for improved open-ended generation
Many text generation systems benefit from using a retriever to retrieve passages from a textual knowledge corpus (e. g., Wikipedia) which are then provided as additional context to the generator.
DistIR: An Intermediate Representation and Simulator for Efficient Neural Network Distribution
The rapidly growing size of deep neural network (DNN) models and datasets has given rise to a variety of distribution strategies such as data, tensor-model, pipeline parallelism, and hybrid combinations thereof.
Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression
This paper investigates deep neural network (DNN) compression from the perspective of compactly representing and storing trained parameters.
What can Data-Centric AI Learn from Data and ML Engineering?
Data-centric AI is a new and exciting research topic in the AI community, but many organizations already build and maintain various "data-centric" applications whose goal is to produce high quality data.
Estimating and Explaining Model Performance When Both Covariates and Labels Shift
We further propose SEES, an algorithmic framework to characterize the distribution shift under SJS and to estimate a model's performance on new data without any labels.
Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking
Neural information retrieval (IR) systems have progressed rapidly in recent years, in large part due to the release of publicly available benchmarking tasks.
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks
Recent advances in instruction-following large language models (LLMs) have led to dramatic improvements in a range of NLP tasks.
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
There is a rapidly growing number of large language models (LLMs) that users can query for a fee.
Exploration with Principles for Diverse AI Supervision
Training large transformers using next-token prediction has given rise to groundbreaking advancements in AI.
Reinforcement Learning (RL)
Unsupervised Reinforcement Learning
Data Acquisition: A New Frontier in Data-centric AI
As Machine Learning (ML) systems continue to grow, the demand for relevant and comprehensive datasets becomes imperative.
Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems
We find empirically that across multiple language tasks, surprisingly, Voting Inference Systems' performance first increases but then decreases as a function of the number of LLM calls.
ALTO: An Efficient Network Orchestrator for Compound AI Systems
We present ALTO, a network orchestrator for efficiently serving compound AI systems such as pipelines of language models.
ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Data
Applications increasingly leverage mixed-modality data, and must jointly search over vector data, such as embedded images, text and video, as well as structured data, such as attributes and keywords.
Optimizing LLM Queries in Relational Workloads
In this paper, we explore how to optimize LLM inference for analytical workloads that invoke LLMs within relational queries.
