Search Results for author:
Found 14 papers, 3 papers with code
RAGServe: Fast Quality-Aware RAG Systems with Configuration Adaptation
RAG (Retrieval Augmented Generation) allows LLMs (large language models) to generate better responses with external knowledge, but using more external knowledge often improves generation quality at the expense of response delay.
Distributed AI Platform for the 6G RAN
Cellular Radio Access Networks (RANs) are rapidly evolving towards 6G, driven by the need to reduce costs and introduce new revenue streams for operators and enterprises.
EdgeSight: Enabling Modeless and Cost-Efficient Inference at the Edge
Traditional ML inference is evolving toward modeless inference, which abstracts the complexity of model selection from users, allowing the system to automatically choose the most appropriate model for each request based on accuracy and resource requirements.
CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving
Compared to the recent systems that reuse the KV cache, CacheGen reduces the KV cache size by 3. 5-4. 3x and the total delay in fetching and processing contexts by 3. 2-3. 7x with negligible impact on the LLM response quality.
OneAdapt: Fast Configuration Adaptation for Video Analytics Applications via Backpropagation
While the high demand for network bandwidth and GPU resources could be substantially reduced by optimally adapting the configuration knobs, such as video resolution and frame rate, current adaptation techniques fail to meet three requirements simultaneously: adapt configurations (i) with minimum extra GPU or bandwidth overhead; (ii) to reach near-optimal decisions based on how the data affects the final DNN's accuracy, and (iii) do so for a range of configuration knobs.
GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edge
Video analytics pipelines have steadily shifted to edge deployments to reduce bandwidth overheads and privacy violations, but in doing so, face an ever-growing resource tension.
Ekya: Continuous Learning of Video Analytics Models on Edge Compute Servers
Compressed models that are deployed on the edge servers for inference suffer from data drift, where the live video data diverges from the training data.
Visor: Privacy-Preserving Video Analytics as a Cloud Service
Video-analytics-as-a-service is becoming an important offering for cloud providers.
Machine Learning at the Network Edge: A Survey
To address this issue, efforts have been made to place additional computing devices at the edge of the network, i. e close to the IoT devices where the data is generated.
Collage Inference: Achieving low tail latency during distributed image classification using coded redundancy models
In this work, we argue that MLaaS platforms also provide unique opportunities to cut the cost of redundancy.
Collage Inference: Using Coded Redundancy for Low Variance Distributed Image Classification
Deploying the collage-cnn models in the cloud, we demonstrate that the 99th percentile tail latency of inference can be reduced by 1. 2x to 2x compared to replication based approaches while providing high accuracy.
ReXCam: Resource-Efficient, Cross-Camera Video Analytics at Scale
Enterprises are increasingly deploying large camera networks for video analytics.
Scaling Video Analytics Systems to Large Camera Deployments
Driven by advances in computer vision and the falling costs of camera hardware, organizations are deploying video cameras en masse for the spatial monitoring of their physical premises.
Focus: Querying Large Video Datasets with Low Latency and Low Cost
Focus handles the lower accuracy of the cheap CNNs by judiciously leveraging expensive CNNs at query-time.
