Search Results for author:
Found 104 papers, 55 papers with code
Communication-Efficient Federated Learning with Sketching
A key insight in the design of FedSketchedSGD is that, because the Count Sketch is linear, momentum and error accumulation can both be carried out within the sketch.
Efficient LLM Scheduling by Learning to Rank
In this paper, we reexamine this assumption -- we show that, although predicting the exact generation length of each request is infeasible, it is possible to predict the relative ranks of output lengths in a batch of requests, using learning to rank.
Post-Training Sparse Attention with Double Sparsity
Our key insight is that the pattern of channel sparsity is relatively static, allowing us to use offline calibration to make it efficient at runtime, thereby enabling accurate and efficient identification of important tokens.
MPC-Minimized Secure LLM Inference
Many inference services based on large language models (LLMs) pose a privacy concern, either revealing user prompts to the service or the proprietary weights to the user.
Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Design
This work aims to inform future research and practice in the design of compound AI systems.
RouteLLM: Learning to Route LLMs with Preference Data
Large language models (LLMs) exhibit impressive capabilities across a wide range of tasks, yet the choice of which model to use often involves a trade-off between performance and cost.
Optimizing Speculative Decoding for Serving Large Language Models Using Goodput
SmartSpec dynamically determines the best speculation length for each request (from 0, i. e., no speculation, to many tokens) -- hence the associated speculative execution costs -- based on a new metric called goodput, which characterizes the current observed load of the entire system and the speculation accuracy.
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
Under these principles, we developed BenchBuilder, a living benchmark that filters high-quality prompts from live data sources to enable offline evaluation on fresh, challenging prompts.
OR-Bench: An Over-Refusal Benchmark for Large Language Models
OR-Bench comprises 80, 000 seemingly toxic prompts across 10 common rejection categories, a subset of around 1, 000 hard prompts that are challenging even for state-of-the-art LLMs, and an additional 600 toxic prompts to prevent indiscriminate responses.
Crafting Interpretable Embeddings by Asking LLMs Questions
Large language models (LLMs) have rapidly improved text embeddings for a growing array of natural-language processing tasks.
Stylus: Automatic Adapter Selection for Diffusion Models
This paper explores the problem of matching the prompt to a set of relevant adapters, built on recent work that highlight the performance gains of composing adapters.
Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity
Based on this analysis, we introduce M\'elange, a GPU allocation framework that navigates these diverse LLM service characteristics and heterogeneous GPU option space to automatically and efficiently derive the minimal-cost GPU allocation for a given LLM service.
GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications
We believe this is critical to unlock the potential for LLM agents to interact with applications and services with limited (post-facto) human involvement.
Trustless Audits without Revealing Data or Models
In this work, we show that it is possible to simultaneously allow model providers to keep their model weights (but not architecture) and data secret while allowing other parties to trustlessly audit model and data properties.
RAFT: Adapting Language Model to Domain Specific RAG
In this paper, we present Retrieval Augmented FineTuning (RAFT), a training recipe that improves the model's ability to answer questions in a "open-book" in-domain settings.
depyf: Open the Opaque Box of PyTorch Compiler for Machine Learning Researchers
PyTorch \texttt{2. x} introduces a compiler designed to accelerate deep learning programs.
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry.
Optimizing LLM Queries in Relational Workloads
In this paper, we explore how to optimize LLM inference for analytical workloads that invoke LLMs within relational queries.
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
To address this issue, we introduce Chatbot Arena, an open platform for evaluating LLMs based on human preferences.
Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems
Many recent state-of-the-art results in language tasks were achieved using compound systems that perform multiple Language Model (LM) calls and aggregate their responses.
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
Autoregressive decoding of large language models (LLMs) is memory bandwidth bounded, resulting in high latency and significant wastes of the parallel processing power of modern accelerators.
Fairness in Serving Large Language Models
High-demand LLM inference services (e. g., ChatGPT and BARD) support a wide range of requests from short chat conversations to long document reading.
SuperServe: Fine-Grained Inference Serving for Unpredictable Workloads
Serving models under such conditions requires these systems to strike a careful balance between the latency and accuracy requirements of the application and the overall efficiency of utilization of scarce resources.
CodeScholar: Growing Idiomatic Code Examples
A tool that could generate realistic, idiomatic, and contextual usage examples for one or more APIs would be immensely beneficial to developers.
SGLang: Efficient Execution of Structured Language Model Programs
SGLang consists of a frontend language and a runtime.
LLM-Assisted Code Cleaning For Training Accurate Code Generators
In this work, we investigate data quality for code and find that making the code more structured and readable leads to improved code generation performance of the system.
Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
Many have raised concerns about the trustworthiness of public benchmarks due to potential contamination in pre-training or fine-tuning datasets.
S-LoRA: Serving Thousands of Concurrent LoRA Adapters
To capitalize on these opportunities, we present S-LoRA, a system designed for the scalable serving of many LoRA adapters.
MemGPT: Towards LLMs as Operating Systems
Large language models (LLMs) have revolutionized AI, but are constrained by limited context windows, hindering their utility in tasks like extended conversations and document analysis.
Online Speculative Decoding
Adapting to query distribution mitigates the shifts between the training distribution of the draft model and the query distribution, enabling the draft model to more accurately predict the target model's outputs.
DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training
FlashAttention (Dao, 2023) effectively reduces the quadratic peak memory usage to linear in training transformer-based large language models (LLMs) on a single GPU.
Efficient Memory Management for Large Language Model Serving with PagedAttention
On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences.
Ranked #3 on
Long-Context Understanding
on Ada-LEval (TSort)
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144.
AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
Model parallelism is conventionally viewed as a method to scale a single large deep learning model beyond the memory limits of a single device.
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks
Recent advances in instruction-following large language models (LLMs) have led to dramatic improvements in a range of NLP tasks.
On Optimizing the Communication of Model Parallelism
This pattern emerges when the two paradigms of model parallelism - intra-operator and inter-operator parallelism - are combined to support large models on large clusters.
CLUTR: Curriculum Learning via Unsupervised Task Representation Learning
Recently, Unsupervised Environment Design (UED) emerged as a new paradigm for zero-shot generalization by simultaneously learning a task distribution and agent policies on the generated tasks.
Context-Aware Streaming Perception in Dynamic Environments
These works evaluate accuracy offline, one image at a time.
POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Paging
We demonstrate that it is possible to fine-tune both ResNet-18 and BERT within the memory constraints of a Cortex-M class embedded device while outperforming current edge training methods in energy efficiency.
NumS: Scalable Array Programming for the Cloud
LSHS is a local search method which optimizes operator placement by minimizing maximum memory and network load on any given node within a distributed system.
Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
Existing model-parallel training systems either require users to manually create a parallelization plan or automatically generate one from a limited space of model parallelism configurations.
Representing Long-Range Context for Graph Neural Networks with Global Attention
Inspired by recent computer vision results that find position-invariant attention performant in learning long-range relationships, our method, which we call GraphTrans, applies a permutation-invariant Transformer module after a standard GNN module.
Balsa: Learning a Query Optimizer Without Expert Demonstrations
Query optimizers are a performance-critical component in every database system.
Composing MPC with LQR and Neural Network for Amortized Efficiency and Stable Control
The proposed scheme does not require pre-computation and can improve the amortized running time of the composed MPC with a well-trained neural network.
MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance
Safe exploration is critical for using reinforcement learning (RL) in risk-sensitive environments.
Grounded Graph Decoding Improves Compositional Generalization in Question Answering
Grounding enables the model to retain syntax information from the input in thereby significantly improving generalization over complex inputs.
Accelerating Quadratic Optimization with Reinforcement Learning
First-order methods for quadratic optimization such as OSQP are widely used for large-scale machine learning and embedded optimal control, where many related problems must be rapidly solved.
Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments
To showcase the benefits, we interfaced SCENIC to an existing RTS environment Google Research Football(GRF) simulator and introduced a benchmark consisting of 32 realistic scenarios, encoded in SCENIC, to train RL agents and testing their generalization capabilities.
Learning Competitive Equilibria in Exchange Economies with Bandit Feedback
The allocations at a CE are Pareto efficient and fair.
PAC Best Arm Identification Under a Deadline
In this work, the decision-maker is given a deadline of $T$ rounds, where, on each round, it can adaptively choose which arms to pull and how many times to pull them; this distinguishes the number of decisions made (i. e., time or number of rounds) from the number of samples acquired (cost).
ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training
On all these tasks, ActNN compresses the activation to 2 bits on average, with negligible accuracy loss.
Solution: Matchmaker Paxos: A Reconfigurable Consensus Protocol
State machine replication protocols, like MultiPaxos and Raft, are at the heart of numerous distributed systems.
SoK: A Generalized Multi-Leader State Machine Replication Tutorial
MultiPaxos and Raft are the two most popular and widely deployed state machine replication protocols.
TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models
With this key idea, we design TeraPipe, a high-performance token-level pipeline parallel algorithm for synchronous model-parallel training of Transformer-based language models.
Scaling Replicated State Machines with Compartmentalization [Technical Report]
In this paper, we introduce compartmentalization, the first comprehensive technique to eliminate state machine replication bottlenecks.
Distributed, Parallel, and Cluster Computing
Ekya: Continuous Learning of Video Analytics Models on Edge Compute Servers
Compressed models that are deployed on the edge servers for inference suffer from data drift, where the live video data diverges from the training data.
Online Learning Demands in Max-min Fairness
We describe mechanisms for the allocation of a scarce resource among multiple users in a way that is efficient, fair, and strategy-proof, but when users do not know their resource requirements.
RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem
Researchers and practitioners in the field of reinforcement learning (RL) frequently leverage parallel computation, which has led to a plethora of new algorithms and systems in the last few years.
Resource Allocation in Multi-armed Bandit Exploration: Overcoming Sublinear Scaling with Adaptive Parallelism
Second, we present an algorithm for a fixed deadline setting, where we are given a time deadline and need to maximize the probability of finding the best arm.
FetchSGD: Communication-Efficient Federated Learning with Sketching
A key insight in the design of FetchSGD is that, because the Count Sketch is linear, momentum and error accumulation can both be carried out within the sketch.
Variable Skipping for Autoregressive Range Density Estimation
In this paper, we explore a technique, variable skipping, for accelerating range density estimation over deep autoregressive models.
Contrastive Code Representation Learning
Recent work learns contextual representations of source code by reconstructing tokens from their context.
Ranked #1 on
Method name prediction
on CodeSearchNet
NeuroCard: One Cardinality Estimator for All Tables
Query optimizers rely on accurate cardinality estimates to produce good execution plans.
Ansor: Generating High-Performance Tensor Programs for Deep Learning
Ansor can find high-performance programs that are outside the search space of existing state-of-the-art approaches.
ProTuner: Tuning Programs with Monte Carlo Tree Search
We explore applying the Monte Carlo Tree Search (MCTS) algorithm in a notoriously difficult task: tuning programs for high-performance deep learning and image processing.
VCG Mechanism Design with Unknown Agent Values under Stochastic Bandit Feedback
To that end, we first define three notions of regret for the welfare, the individual utilities of each agent and that of the mechanism.
AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning
We compare the performance of AutoPhase to state-of-the-art algorithms that address the phase-ordering problem.
Hoplite: Efficient and Fault-Tolerant Collective Communication for Task-Based Distributed Systems
Task-based distributed frameworks (e. g., Ray, Dask, Hydro) have become increasingly popular for distributed applications that contain asynchronous and dynamic workloads, including asynchronous gradient descent, reinforcement learning, and model serving.
HyperSched: Dynamic Resource Reallocation for Model Development on a Deadline
Prior research in resource scheduling for machine learning training workloads has largely focused on minimizing job completion times.
Hierarchical Variational Imitation Learning of Control Programs
Autonomous agents can learn by imitating teacher demonstrations of the intended behavior.
IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks
To address this, we propose a new distributed reinforcement learning algorithm, IMPACT.
Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integration
DNN accelerators are often developed and evaluated in isolation without considering the cross-stack, system-level effects in real-world environments.
Blink: Fast and Generic Collectives for Distributed ML
Model parameter synchronization across GPUs introduces high overheads for data-parallel training at scale.
Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization
We formalize the problem of trading-off DNN training time and memory requirements as the tensor rematerialization optimization problem, a generalization of prior checkpointing strategies.
NeuroVectorizer: End-to-End Vectorization with Deep Reinforcement Learning
However, these models are unable to capture the data dependency, the computation graph, or the organization of instructions.
Distributed, Parallel, and Cluster Computing Performance Programming Languages
A View on Deep Reinforcement Learning in System Optimization
We propose a set of essential metrics to guide future works in evaluating the efficacy of using deep reinforcement learning in system optimization.
Helen: Maliciously Secure Coopetitive Learning for Linear Models
Many organizations wish to collaboratively train machine learning models on their combined datasets for a common benefit (e. g., better medical research, or fraud detection).
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
A key challenge in leveraging data augmentation for neural network training is choosing an effective augmentation policy from a large search space of candidate operations.
Ranked #4 on
Image Classification
on SVHN
Deep Unsupervised Cardinality Estimation
To produce a truly usable estimator, we develop a Monte Carlo integration scheme on top of autoregressive models that can efficiently handle range queries with dozens of dimensions or more.
Harmonia: Near-Linear Scalability for Replicated Storage with In-Network Conflict Detection
Experimental results show that Harmonia improves the throughput of these protocols by up to 10X for a replication factor of 10, providing near-linear scalability up to the limit of our testbed.
Distributed, Parallel, and Cluster Computing
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
Communication-efficient distributed SGD with Sketching
Large-scale distributed training of neural networks is often limited by network bandwidth, wherein the communication time overwhelms the local computation time.
Constrained Thompson Sampling for Wireless Link Optimization
We model rate selection as a stochastic multi-armed bandit (MAB) problem, where a finite set of transmission rates are modeled as independent bandit arms.
Neural Packet Classification
First, many of the existing solutions are iteratively building a decision tree by splitting nodes in the tree.
Cloud Programming Simplified: A Berkeley View on Serverless Computing
Serverless cloud computing handles virtually all the system administration operations needed to make it easier for programmers to use the cloud.
Operating Systems
The OoO VLIW JIT Compiler for GPU Inference
Current trends in Machine Learning~(ML) inference on hardware accelerated devices (e. g., GPUs, TPUs) point to alarmingly low utilization.
AutoPhase: Compiler Phase-Ordering for High Level Synthesis with Deep Reinforcement Learning
We implement a framework in the context of the LLVM compiler to optimize the ordering for HLS programs and compare the performance of deep reinforcement learning to state-of-the-art algorithms that address the phase-ordering problem.
InferLine: ML Inference Pipeline Composition Framework
The dominant cost in production machine learning workloads is not training individual models but serving predictions from increasingly complex prediction pipelines spanning multiple models, machine learning frameworks, and parallel hardware accelerators.
Distributed, Parallel, and Cluster Computing
Learning to Optimize Join Queries With Deep Reinforcement Learning
Exhaustive enumeration of all possible join orders is often avoided, and most optimizers leverage heuristics to prune the search space.
Databases
Tune: A Research Platform for Distributed Model Selection and Training
We show that this interface meets the requirements for a broad range of hyperparameter search algorithms, allows straightforward scaling of search to large clusters, and simplifies algorithm implementation.
Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
By enabling wider use of learned dynamics models within a model-free reinforcement learning algorithm, we improve value estimation, which, in turn, reduces the sample complexity of learning.
Parametrized Hierarchical Procedures for Neural Programming
Neural programs are highly accurate and structured policies that perform algorithmic tasks by controlling the behavior of a computation mechanism.
RLlib: Abstractions for Distributed Reinforcement Learning
Reinforcement learning (RL) algorithms involve the deep nesting of highly irregular computation patterns, each of which typically exhibits opportunities for distributed computation.
Ray: A Distributed Framework for Emerging AI Applications
To meet the performance requirements, Ray employs a distributed scheduler and a distributed and fault-tolerant store to manage the system's control state.
A Berkeley View of Systems Challenges for AI
With the increasing commoditization of computer vision, speech recognition and machine translation systems and the widespread deployment of learning-based back-end technologies such as digital advertising and intelligent infrastructures, AI (Artificial Intelligence) has moved from research labs to production.
Multi-Level Discovery of Deep Options
Augmenting an agent's control with useful higher-level behaviors called options can greatly reduce the sample complexity of reinforcement learning, but manually designing options is infeasible in high-dimensional and abstract state spaces.
Real-Time Machine Learning: The Missing Pieces
Machine learning applications are increasingly deployed not only to serve predictions using static models, but also as tightly-integrated components of feedback loops involving dynamic, real-time decision making.
Clipper: A Low-Latency Online Prediction Serving System
In this paper, we introduce Clipper, a general-purpose low-latency prediction serving system.
Fast and Accurate Performance Analysis of LTE Radio Access Networks
Our choice of this domain is influenced by its commonalities with several other domains that produce real-time data, our access to a large live dataset, and their real-time nature and dimensionality which makes it a natural fit for a popular analysis technique, machine learning (ML).
SparkNet: Training Deep Networks in Spark
We introduce SparkNet, a framework for training deep networks in Spark.
Highly Available Transactions: Virtues and Limitations (Extended Version)
To minimize network latency and remain online during server failures and network partitions, many modern distributed data storage systems eschew transactional functionality, which provides strong semantic guarantees for groups of multiple operations over multiple data items.
Databases
