Search Results for author:
Found 44 papers, 16 papers with code
TurboAttention: Efficient Attention Approximation For High Throughputs LLMs
Our solution introduces two key innovations: FlashQ, a headwise attention quantization technique that enables both compression of KV cache and quantized execution of activation-activation multiplication, and Sparsity-based Softmax Approximation (SAS), which eliminates the need for dequantization to FP32 during exponentiation operation in attention.
Low-Rank Correction for Quantized LLMs
We consider the problem of model compression for Large Language Models (LLMs) at post-training time, where the task is to compress a well-trained model using only a small set of calibration input data.
Pyramid Vector Quantization for LLMs
In this work, we aim to further exploit this spherical geometry of the weights when performing quantization by considering Pyramid Vector Quantization (PVQ) for large language models.
KBLaM: Knowledge Base augmented Language Model
In this paper, we propose Knowledge Base augmented Language Model (KBLaM), a new method for augmenting Large Language Models (LLMs) with external knowledge.
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits.
Learning to Extract Structured Entities Using Language Models
Recent advances in machine learning have significantly impacted the field of information extraction, with Language Models (LMs) playing a pivotal role in extracting structured information from unstructured text.
SliceGPT: Compress Large Language Models by Deleting Rows and Columns
Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources.
DiSK: A Diffusion Model for Structured Knowledge
We introduce Diffusion Models of Structured Knowledge (DiSK) - a new architecture and training approach specialized for structured data.
Sparse Gaussian Processes with Spherical Harmonic Features Revisited
Further, we introduce sparseness in the eigenbasis by variational learning of the spherical harmonic phases.
Additive Gaussian Processes Revisited
Gaussian Process (GP) models are a class of flexible non-parametric models that have rich representational power.
Improved Inverse-Free Variational Bounds for Sparse Gaussian Processes
The need for matrix decompositions (inverses) is often named as a major impediment to scaling Gaussian process (GP) models, even in efficient approximations.
Deep Neural Networks as Point Estimates for Deep Gaussian Processes
This results in models that can either be seen as neural networks with improved uncertainty prediction or deep Gaussian processes with increased prediction accuracy.
GPflux: A Library for Deep Gaussian Processes
GPflux is compatible with and built on top of the Keras deep learning eco-system.
Sparse Gaussian Processes with Spherical Harmonic Features
We introduce a new class of inter-domain variational Gaussian processes (GP) where data is mapped onto the unit hypersphere in order to use spherical harmonic representations.
Amortized variance reduction for doubly stochastic objectives
Approximate inference in complex probabilistic models such as deep Gaussian processes requires the optimisation of doubly stochastic objective functions.
A Framework for Interdomain and Multioutput Gaussian Processes
One obstacle to the use of Gaussian processes (GPs) in large-scale problems, and as a component in deep learning system, is the need for bespoke derivations and implementations for small variations in the model or inference.
Doubly Sparse Variational Gaussian Processes
The use of Gaussian process models is typically limited to datasets with a few tens of thousands of observations due to their complexity and memory footprint.
Variational Gaussian Process Models without Matrix Inverses
We present a variational approximation for a wide range of GP models that does not require a matrix inverse to be performed at each optimisation step.
Overcoming Mean-Field Approximations in Recurrent Gaussian Process Models
As we demonstrate in our experiments, the factorisation between latent system states and transition function can lead to a miscalibrated posterior and to learning unnecessarily large noise terms.
Deep Gaussian Processes with Importance-Weighted Variational Inference
Deep Gaussian processes (DGPs) can model complex marginal densities as well as complex mappings.
Banded Matrix Operators for Gaussian Markov Models in the Automatic Differentiation Era
Banded matrices can be used as precision matrices in several models including linear state-space models, some Gaussian processes, and Gaussian Markov random fields.
Bayesian Image Classification with Deep Convolutional Gaussian Processes
We also demonstrate that our fully Bayesian approach improves on dropout-based Bayesian deep learning methods in terms of uncertainty and marginal likelihood estimates.
Non-Factorised Variational Inference in Dynamical Systems
We focus on variational inference in dynamical systems where the discrete time transition function (or evolution rule) is modelled by a Gaussian process.
Infinite-Horizon Gaussian Processes
The complexity is still cubic in the state dimension $m$ which is an impediment to practical application.
Gaussian Process Conditional Density Estimation
Conditional Density Estimation (CDE) models deal with estimating conditional distributions.
Learning Invariances using the Marginal Likelihood
Generalising well in supervised learning tasks relies on correctly extrapolating the training data to a large region of the input space.
Large-Scale Cox Process Inference using Variational Fourier Features
This allows us to formulate a grid-free approximation that scales well with the number of data points and the size of the domain.
Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models
The natural gradient method has been used effectively in conjugate Gaussian process models, but the non-conjugate case has been largely unexplored.
Convolutional Gaussian Processes
We present a practical way of introducing convolutional structure into Gaussian processes, making them more suited to high-dimensional inputs like images.
Pseudo-extended Markov chain Monte Carlo
In this paper, we introduce the pseudo-extended MCMC method as a simple approach for improving the mixing of the MCMC sampler for multi-modal posterior distributions.
Scalable Joint Models for Reliable Uncertainty-Aware Event Prediction
Alternatively, state-of-the-art joint modeling techniques can be used for jointly modeling the longitudinal and event data and compute event probabilities conditioned on the longitudinal observations.
Identification of Gaussian Process State Space Models
To address this challenge, we impose a structured Gaussian variational posterior distribution over the latent states, which is parameterised by a recognition model in the form of a bi-directional recurrent neural network.
Variational Fourier features for Gaussian processes
This work brings together two powerful concepts in Gaussian processes: the variational approach to sparse approximation and the spectral representation of Gaussian processes.
GPflow: A Gaussian process library using TensorFlow
GPflow is a Gaussian process library that uses TensorFlow for its core computations and Python for its front end.
Chained Gaussian Processes
Gaussian process models are flexible, Bayesian non-parametric approaches to regression.
MCMC for Variationally Sparse Gaussian Processes
This paper simultaneously addresses these, using a variational approximation to the posterior which is sparse in support of the function but otherwise free-form.
Spike and Slab Gaussian Process Latent Variable Models
The Gaussian process latent variable model (GP-LVM) is a popular approach to non-linear probabilistic dimensionality reduction.
On Sparse variational methods and the Kullback-Leibler divergence between stochastic processes
We then discuss augmented index sets and show that, contrary to previous works, marginal consistency of augmentation is not enough to guarantee consistency of variational inference with the original model.
Nested Variational Compression in Deep Gaussian Processes
Deep Gaussian processes provide a flexible approach to probabilistic modelling of data using either supervised or unsupervised learning.
Scalable Variational Gaussian Process Classification
Gaussian process classification is a popular method with a number of appealing properties.
Gaussian Process Models with Parallelization and GPU acceleration
In this work, we present an extension of Gaussian process (GP) models with sophisticated parallelization and GPU acceleration.
Fast nonparametric clustering of structured time-series
In this publication, we combine two Bayesian non-parametric models: the Gaussian Process (GP) and the Dirichlet Process (DP).
Gaussian Processes for Big Data
We introduce stochastic variational inference for Gaussian process models.
Fast Variational Inference in the Conjugate Exponential Family
We present a general method for deriving collapsed variational inference algorithms for probabilistic models in the conjugate exponential family.
