Search Results for author:
Found 20 papers, 11 papers with code
Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
In this work, we propose a different design approach: we run LLM "workers" in parallel , allowing them to synchronize via a concurrently-updated attention cache and prompt these workers to decide how best to collaborate.
Scale-wise Distillation of Diffusion Models
We present SwD, a scale-wise distillation framework for diffusion models (DMs), which effectively employs next-scale prediction ideas for diffusion-based few-step generators.
Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Efficient real-world deployments of large language models (LLMs) rely on Key-Value (KV) caching for processing and generating long outputs, reducing the need for repetitive computation.
Label Privacy in Split Learning for Large Models with Parameter-Efficient Training
Using this analysis, we propose P$^3$EFT, a multi-party split learning algorithm that takes advantage of existing PEFT properties to maintain privacy at a lower performance overhead.
Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesis
This work presents Switti, a scale-wise transformer for text-to-image generation.
EvoPress: Towards Optimal Dynamic Model Compression via Evolutionary Search
Yet, current methods rely on heuristics for identifying the "importance" of a given layer towards the loss, based on assumptions such as \emph{error monotonicity}, i. e. that the end-to-end model compression error is proportional to the sum of layer-wise errors.
Accurate Compression of Text-to-Image Diffusion Models via Vector Quantization
Specifically, we tailor vector-based PTQ methods to recent billion-scale text-to-image models (SDXL and SDXL-Turbo), and show that the diffusion models of 2B+ parameters compressed to around 3 bits using VQ exhibit the similar image quality and textual alignment as previous 4-bit compression techniques.
The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information
The rising footprint of machine learning has led to a focus on imposing \emph{model sparsity} as a means of reducing computational and memory costs.
Does Diffusion Beat GAN in Image Super Resolution?
There is a prevalent opinion that diffusion-based models outperform GAN-based counterparts in the Image Super Resolution (ISR) problem.
PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression
In this work, we question the use of STE for extreme LLM compression, showing that it can be sub-optimal, and perform a systematic study of quantization-aware fine-tuning strategies for LLMs.
YaART: Yet Another ART Rendering Technology
In the rapidly progressing field of generative models, the development of efficient and high-fidelity text-to-image diffusion systems represents a significant frontier.
Extreme Compression of Large Language Models via Additive Quantization
The emergence of accurate open large language models (LLMs) has led to a race towards performant quantization techniques which can enable their execution on end-user devices.
Sparse Fine-tuning for Inference Acceleration of Large Language Models
While the standard approach is to leverage sparsity for computational reduction, we observe that in the case of memory-bound LLMs sparsity can also be leveraged for reducing memory bandwidth.
Accurate Neural Network Pruning Requires Rethinking Sparse Optimization
Obtaining versions of deep neural networks that are both highly-accurate and highly-sparse is one of the main challenges in the area of model compression, and several high-performance pruning techniques have been investigated by the community.
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
Recent advances in large language model (LLM) pretraining have led to high-quality LLMs with impressive abilities.
Vision Models Can Be Efficiently Specialized via Few-Shot Task-Aware Compression
To address this, we ask: can we quickly compress large generalist models into accurate and efficient specialists?
A critical look at the evaluation of GNNs under heterophily: Are we really making progress?
Graphs without this property are called heterophilous, and it is typically assumed that specialized methods are required to achieve strong performance on such graphs.
CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Models
To further showcase CAP's accuracy and scalability, we use it to show for the first time that extremely-accurate large vision models, trained via self-supervised techniques, can also be pruned to moderate sparsities, with negligible accuracy loss.
Characterizing Graph Datasets for Node Classification: Homophily-Heterophily Dichotomy and Beyond
For this, we formalize desirable properties for a proper homophily measure and verify which measures satisfy which properties.
A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momenta
Mini-batch SGD with momentum is a fundamental algorithm for learning large predictive models.
