Search Results for author:
Found 21 papers, 7 papers with code
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
However, batching multiple requests leads to an interleaving of prefill and decode iterations which makes it challenging to achieve both high throughput and low latency.
SuperServe: Fine-Grained Inference Serving for Unpredictable Workloads
Serving models under such conditions requires these systems to strike a careful balance between the latency and accuracy requirements of the application and the overall efficiency of utilization of scarce resources.
Signed Binarization: Unlocking Efficiency Through Repetition-Sparsity Trade-Off
Efficient inference of Deep Neural Networks (DNNs) on resource-constrained edge devices is essential.
ABKD: Graph Neural Network Compression with Attention-Based Knowledge Distillation
To address this shortcoming, we propose a novel KD approach to GNN compression that we call Attention-Based Knowledge Distillation (ABKD).
Ethosight: A Reasoning-Guided Iterative Learning System for Nuanced Perception based on Joint-Embedding & Contextual Label Affinity
Traditional computer vision models often necessitate extensive data acquisition, annotation, and validation.
Pareto-Secure Machine Learning (PSML): Fingerprinting and Securing Inference Serving Systems
Second, we counter the proposed attack with a noise-based defense mechanism that thwarts fingerprinting by adding noise to the specified performance metrics.
Subgraph Stationary Hardware-Software Inference Co-Design
For the stream of queries, SUSHI yields up to 25% improvement in latency, 0. 98% increase in served accuracy.
DynaQuant: Compressing Deep Learning Training Checkpoints via Dynamic Quantization
With the increase in the scale of Deep Learning (DL) training workloads in terms of compute resources and time consumption, the likelihood of encountering in-training failures rises substantially, leading to lost work and resource wastage.
SuperFed: Weight Shared Federated Learning
We propose SuperFed - an architectural framework that incurs $O(1)$ cost to co-train a large family of models in a federated fashion by leveraging weight-shared learning.
Signed Binary Weight Networks
Efficient inference of Deep Neural Networks (DNNs) is essential to making AI ubiquitous.
UnfoldML: Cost-Aware and Uncertainty-Based Dynamic 2D Prediction for Multi-Stage Classification
It achieves within 0. 1% accuracy from the highest-performing multi-class baseline, while saving close to 20X on spatio-temporal cost of inference and earlier (3. 5hrs) disease onset prediction.
CompOFA: Compound Once-For-All Networks for Faster Multi-Platform Deployment
The emergence of CNNs in mainstream deployment has necessitated methods to design and train efficient architectures tailored to maximize the accuracy under diverse hardware & latency constraints.
CompOFA – Compound Once-For-All Networks for Faster Multi-Platform Deployment
The emergence of CNNs in mainstream deployment has necessitated methods to design and train efficient architectures tailored to maximize the accuracy under diverse hardware & latency constrains.
HOLMES: Health OnLine Model Ensemble Serving for Deep Learning Models in Intensive Care Units
HOLMES is tested on risk prediction task on pediatric cardio ICU data with above 95% prediction accuracy and sub-second latency on 64-bed simulation.
HyperSched: Dynamic Resource Reallocation for Model Development on a Deadline
Prior research in resource scheduling for machine learning training workloads has largely focused on minimizing job completion times.
The OoO VLIW JIT Compiler for GPU Inference
Current trends in Machine Learning~(ML) inference on hardware accelerated devices (e. g., GPUs, TPUs) point to alarmingly low utilization.
Serverless Computing: One Step Forward, Two Steps Back
Serverless computing offers the potential to program the cloud in an autoscaling, pay-as-you go manner.
Distributed, Parallel, and Cluster Computing Databases
InferLine: ML Inference Pipeline Composition Framework
The dominant cost in production machine learning workloads is not training individual models but serving predictions from increasingly complex prediction pipelines spanning multiple models, machine learning frameworks, and parallel hardware accelerators.
Distributed, Parallel, and Cluster Computing
Ray: A Distributed Framework for Emerging AI Applications
To meet the performance requirements, Ray employs a distributed scheduler and a distributed and fault-tolerant store to manage the system's control state.
IDK Cascades: Fast Deep Learning by Learning not to Overthink
Advances in deep learning have led to substantial increases in prediction accuracy but have been accompanied by increases in the cost of rendering predictions.
Real-Time Machine Learning: The Missing Pieces
Machine learning applications are increasingly deployed not only to serve predictions using static models, but also as tightly-integrated components of feedback loops involving dynamic, real-time decision making.
