Search Results for author: Matei Zaharia

Found 51 papers, 22 papers with code

Data Acquisition: A New Frontier in Data-centric AI

no code implementations22 Nov 2023 Lingjiao Chen, Bilge Acun, Newsha Ardalani, Yifan Sun, Feiyang Kang, Hanrui Lyu, Yongchan Kwon, Ruoxi Jia, Carole-Jean Wu, Matei Zaharia, James Zou

As Machine Learning (ML) systems continue to grow, the demand for relevant and comprehensive datasets becomes imperative.

ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems

1 code implementation16 Nov 2023 Jon Saad-Falcon, Omar Khattab, Christopher Potts, Matei Zaharia

Evaluating retrieval-augmented generation (RAG) systems traditionally relies on hand annotations for input queries, passages to retrieve, and responses to generate.


Exploration with Principles for Diverse AI Supervision

no code implementations13 Oct 2023 Hao liu, Matei Zaharia, Pieter Abbeel

Training large transformers using next-token prediction has given rise to groundbreaking advancements in AI.

Reinforcement Learning (RL) Unsupervised Reinforcement Learning

Ring Attention with Blockwise Transformers for Near-Infinite Context

3 code implementations3 Oct 2023 Hao liu, Matei Zaharia, Pieter Abbeel

Transformers have emerged as the architecture of choice for many state-of-the-art AI models, showcasing exceptional performance across a wide range of AI applications.

Language Modelling

How is ChatGPT's behavior changing over time?

3 code implementations18 Jul 2023 Lingjiao Chen, Matei Zaharia, James Zou

We find that the performance and behavior of both GPT-3. 5 and GPT-4 can vary greatly over time.

Code Generation Language Modelling +3

FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

no code implementations9 May 2023 Lingjiao Chen, Matei Zaharia, James Zou

There is a rapidly growing number of large language models (LLMs) that users can query for a fee.

Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP

2 code implementations28 Dec 2022 Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, Matei Zaharia

Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM).

Language Modelling Question Answering +1

Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking

no code implementations2 Dec 2022 Keshav Santhanam, Jon Saad-Falcon, Martin Franz, Omar Khattab, Avirup Sil, Radu Florian, Md Arafat Sultan, Salim Roukos, Matei Zaharia, Christopher Potts

Neural information retrieval (IR) systems have progressed rapidly in recent years, in large part due to the release of publicly available benchmarking tasks.

Benchmarking Information Retrieval +1

MegaBlocks: Efficient Sparse Training with Mixture-of-Experts

no code implementations29 Nov 2022 Trevor Gale, Deepak Narayanan, Cliff Young, Matei Zaharia

We present MegaBlocks, a system for efficient Mixture-of-Experts (MoE) training on GPUs.

HAPI: A Large-scale Longitudinal Dataset of Commercial ML API Predictions

1 code implementation18 Sep 2022 Lingjiao Chen, Zhihua Jin, Sabri Eyuboglu, Christopher Ré, Matei Zaharia, James Zou

HAPI is the first large-scale dataset of ML API usages and is a unique resource for studying ML-as-a-service (MLaaS).

object-detection Object Detection +4

Estimating and Explaining Model Performance When Both Covariates and Labels Shift

no code implementations18 Sep 2022 Lingjiao Chen, Matei Zaharia, James Zou

We further propose SEES, an algorithmic framework to characterize the distribution shift under SJS and to estimate a model's performance on new data without any labels.

PLAID: An Efficient Engine for Late Interaction Retrieval

1 code implementation19 May 2022 Keshav Santhanam, Omar Khattab, Christopher Potts, Matei Zaharia

PLAID uses centroid interaction as well as centroid pruning, a mechanism for sparsifying the bag of centroids, within a highly-optimized engine to reduce late interaction search latency by up to 7$\times$ on a GPU and 45$\times$ on a CPU against vanilla ColBERTv2, while continuing to deliver state-of-the-art retrieval quality.

Information Retrieval Retrieval

What can Data-Centric AI Learn from Data and ML Engineering?

no code implementations13 Dec 2021 Neoklis Polyzotis, Matei Zaharia

Data-centric AI is a new and exciting research topic in the AI community, but many organizations already build and maintain various "data-centric" applications whose goal is to produce high quality data.

Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression

no code implementations19 Nov 2021 Yuezhou Sun, Wenlong Zhao, Lijun Zhang, Xiao Liu, Hui Guan, Matei Zaharia

This paper investigates deep neural network (DNN) compression from the perspective of compactly representing and storing trained parameters.

Neural Network Compression Quantization

DistIR: An Intermediate Representation and Simulator for Efficient Neural Network Distribution

no code implementations9 Nov 2021 Keshav Santhanam, Siddharth Krishna, Ryota Tomioka, Tim Harris, Matei Zaharia

The rapidly growing size of deep neural network (DNN) models and datasets has given rise to a variety of distribution strategies such as data, tensor-model, pipeline parallelism, and hybrid combinations thereof.

Efficient Neural Network

Hindsight: Posterior-guided training of retrievers for improved open-ended generation

no code implementations ICLR 2022 Ashwin Paranjape, Omar Khattab, Christopher Potts, Matei Zaharia, Christopher D. Manning

Many text generation systems benefit from using a retriever to retrieve passages from a textual knowledge corpus (e. g., Wikipedia) which are then provided as additional context to the generator.

Text Generation

On the Opportunities and Risks of Foundation Models

3 code implementations16 Aug 2021 Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, aditi raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, Percy Liang

AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.

Transfer Learning

Did the Model Change? Efficiently Assessing Machine Learning API Shifts

no code implementations29 Jul 2021 Lingjiao Chen, Tracy Cai, Matei Zaharia, James Zou

This motivated us to formulate the API shift assessment problem at a more fine-grained level as estimating how the API model's confusion matrix changes over time when the data distribution is constant.

BIG-bench Machine Learning

Proof: Accelerating Approximate Aggregation Queries with Expensive Predicates

no code implementations27 Jul 2021 Daniel Kang, John Guibas, Peter Bailis, Tatsunori Hashimoto, Yi Sun, Matei Zaharia

Given a dataset $\mathcal{D}$, we are interested in computing the mean of a subset of $\mathcal{D}$ which matches a predicate.

Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM

1 code implementation9 Apr 2021 Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick Legresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, Matei Zaharia

In this paper, we show how different types of parallelism methods (tensor, pipeline, and data parallelism) can be composed to scale to thousands of GPUs and models with trillions of parameters.

Language Modelling

Efficient Online ML API Selection for Multi-Label Classification Tasks

no code implementations18 Feb 2021 Lingjiao Chen, Matei Zaharia, James Zou

In this work, we propose FrugalMCT, a principled framework that adaptively selects the APIs to use for different data in an online fashion while respecting user's budget.

General Classification Multi-Label Classification +7

Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval

2 code implementations NeurIPS 2021 Omar Khattab, Christopher Potts, Matei Zaharia

Multi-hop reasoning (i. e., reasoning across two or more documents) is a key ingredient for NLP models that leverage large corpora to exhibit broad knowledge.

Claim Verification Question Answering +1

Jointly Optimizing Preprocessing and Inference for DNN-based Visual Analytics

no code implementations25 Jul 2020 Daniel Kang, Ankit Mathur, Teja Veeramacheneni, Peter Bailis, Matei Zaharia

This runtime engine a) efficiently pipelines preprocessing and DNN execution for inference, b) places preprocessing operations on the CPU or GPU in a hardware- and input-aware manner, and c) efficiently manages memory and threading for high throughput execution.

Relevance-guided Supervision for OpenQA with ColBERT

5 code implementations1 Jul 2020 Omar Khattab, Christopher Potts, Matei Zaharia

In much recent work, the retriever is a learned component that uses coarse-grained vector representations of questions and passages.

Natural Questions Open-Domain Question Answering +2

Sparse GPU Kernels for Deep Learning

1 code implementation18 Jun 2020 Trevor Gale, Matei Zaharia, Cliff Young, Erich Elsen

In this work, we study sparse matrices from deep learning applications and identify favorable properties that can be exploited to accelerate computation.

Memory-Efficient Pipeline-Parallel DNN Training

1 code implementation16 Jun 2020 Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, Matei Zaharia

Many state-of-the-art ML results have been obtained by scaling up the number of parameters in existing models.

ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT

7 code implementations27 Apr 2020 Omar Khattab, Matei Zaharia

ColBERT introduces a late interaction architecture that independently encodes the query and the document using BERT and then employs a cheap yet powerful interaction step that models their fine-grained similarity.

Document Ranking Information Retrieval +3

Model Assertions for Monitoring and Improving ML Models

no code implementations3 Mar 2020 Daniel Kang, Deepti Raghavan, Peter Bailis, Matei Zaharia

We propose methods of using model assertions at all stages of ML system deployment, including runtime monitoring, validating labels, and continuously improving ML models.

Active Learning

Express: Lowering the Cost of Metadata-hiding Communication with Cryptographic Privacy

1 code implementation20 Nov 2019 Saba Eskandarian, Henry Corrigan-Gibbs, Matei Zaharia, Dan Boneh

Existing systems for metadata-hiding messaging that provide cryptographic privacy properties have either high communication costs, high computation costs, or both.

Cryptography and Security

LIT: Learned Intermediate Representation Training for Model Compression

1 code implementation4 Sep 2019 Animesh Koratana, Daniel Kang, Peter Bailis, Matei Zaharia

In this work, we introduce Learned Intermediate representation Training (LIT), a novel model compression technique that outperforms a range of recent model compression techniques by leveraging the highly repetitive structure of modern DNNs (e. g., ResNet).

Image Classification Model Compression +2

Selection via Proxy: Efficient Data Selection for Deep Learning

1 code implementation ICLR 2020 Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, Matei Zaharia

By removing hidden layers from the target model, using smaller architectures, and training for fewer epochs, we create proxies that are an order of magnitude faster to train.

Active Learning

Willump: A Statistically-Aware End-to-end Optimizer for Machine Learning Inference

no code implementations3 Jun 2019 Peter Kraft, Daniel Kang, Deepak Narayanan, Shoumik Palkar, Peter Bailis, Matei Zaharia

First, Willump automatically cascades feature computation for classification queries: Willump classifies most data inputs using only high-value, low-cost features selected through empirical observations of ML model performance, improving query performance by up to 5x without statistically significant accuracy loss.

BIG-bench Machine Learning

Select Via Proxy: Efficient Data Selection For Training Deep Networks

no code implementations ICLR 2019 Cody Coleman, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, Matei Zaharia

In our approach, we first train a small proxy model quickly, which we then use to estimate the utility of individual training data points, and then select the most informative ones for training the large target model.

BIG-bench Machine Learning Image Classification +1

LIT: Block-wise Intermediate Representation Training for Model Compression

no code implementations ICLR 2019 Animesh Koratana, Daniel Kang, Peter Bailis, Matei Zaharia

Knowledge distillation (KD) is a popular method for reducing the computational overhead of deep network inference, in which the output of a teacher model is used to train a smaller, faster student model.

Knowledge Distillation Model Compression

Beyond Data and Model Parallelism for Deep Neural Networks

no code implementations14 Jul 2018 Zhihao Jia, Matei Zaharia, Alex Aiken

We also propose FlexFlow, a deep learning framework that uses guided randomized search of the SOAP space to find a fast parallelization strategy for a specific parallel machine.

Distributed, Parallel, and Cluster Computing

Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark

no code implementations4 Jun 2018 Cody Coleman, Daniel Kang, Deepak Narayanan, Luigi Nardi, Tian Zhao, Jian Zhang, Peter Bailis, Kunle Olukotun, Chris Re, Matei Zaharia

In this work, we analyze the entries from DAWNBench, which received optimized submissions from multiple industrial groups, to investigate the behavior of TTA as a metric as well as trends in the best-performing entries.

Benchmarking BIG-bench Machine Learning

BlazeIt: Optimizing Declarative Aggregation and Limit Queries for Neural Network-Based Video Analytics

no code implementations2 May 2018 Daniel Kang, Peter Bailis, Matei Zaharia

We introduce two new query optimization techniques in BlazeIt that are not supported by prior work.


Model Specialization for Inference Via End-to-End Distillation, Pruning, and Cascades

no code implementations ICLR 2018 Daniel Kang, Karey Shi, Thao Ngyuen, Stephanie Mallard, Peter Bailis, Matei Zaharia

Thus, simply fine-tuning or transfer learn- ing from a general-purpose network inherits a large computational cost that may not be necessary for a given task.

General Classification Image Classification

To Index or Not to Index: Optimizing Exact Maximum Inner Product Search

1 code implementation5 Jun 2017 Firas Abuzaid, Geet Sethi, Peter Bailis, Matei Zaharia

The brute-force approach to solving exact MIPS is computationally expensive, thus spurring recent development of novel indexes and pruning techniques for this task.

Recommendation Systems

Infrastructure for Usable Machine Learning: The Stanford DAWN Project

no code implementations22 May 2017 Peter Bailis, Kunle Olukotun, Christopher Re, Matei Zaharia

Despite incredible recent advances in machine learning, building machine learning applications remains prohibitively time-consuming and expensive for all but the best-trained, best-funded engineering organizations.

BIG-bench Machine Learning

NoScope: Optimizing Neural Network Queries over Video at Scale

1 code implementation7 Mar 2017 Daniel Kang, John Emmons, Firas Abuzaid, Peter Bailis, Matei Zaharia

Given a target video, object to detect, and reference neural network, NoScope automatically searches for and trains a sequence, or cascade, of models that preserves the accuracy of the reference network but is specialized to the target video and are therefore far less computationally expensive.

Binary Classification

Yggdrasil: An Optimized System for Training Deep Decision Trees at Scale

no code implementations NeurIPS 2016 Firas Abuzaid, Joseph K. Bradley, Feynman T. Liang, Andrew Feng, Lee Yang, Matei Zaharia, Ameet S. Talwalkar

Deep distributed decision trees and tree ensembles have grown in importance due to the need to model increasingly large datasets.

Cannot find the paper you are looking for? You can Submit a new open access paper.