Search Results for author:
Found 12 papers, 7 papers with code
Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression
Autoregressive language models rely on a Key-Value (KV) Cache, which avoids re-computing past hidden states during generation, making it faster.
Mixture-of-Experts Graph Transformers for Interpretable Particle Collision Detection
In this paper, we propose a novel approach that combines a Graph Transformer model with Mixture-of-Expert layers to achieve high predictive performance while embedding interpretability into the architecture.
Goal-oriented Communications based on Recursive Early Exit Neural Networks
This paper presents a novel framework for goal-oriented semantic communications leveraging recursive early exit models.
Analysing the Residual Stream of Language Models Under Knowledge Conflicts
Through probing tasks, we find that LLMs can internally register the signal of knowledge conflict in the residual stream, which can be accurately detected by probing the intermediate model activations.
Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering
In this work, we propose \textsc{SpARE}, a \emph{training-free} representation engineering method that uses pre-trained sparse auto-encoders (SAEs) to control the knowledge selection behaviour of LLMs.
Adaptive Layer Selection for Efficient Vision Transformer Fine-Tuning
To this end, in this paper we introduce an efficient fine-tuning method for ViTs called $\textbf{ALaST}$ ($\textit{Adaptive Layer Selection Fine-Tuning for Vision Transformers}$) to speed up the fine-tuning process while reducing computational cost, memory load, and training time.
A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression
Existing approaches to reduce the KV cache size involve either fine-tuning the model to learn a compression strategy or leveraging attention scores to reduce the sequence length.
Are We Done with MMLU?
For example, we find that 57% of the analysed questions in the Virology subset contain errors.
Adaptive Semantic Token Selection for AI-native Goal-oriented Communications
In this paper, we propose a novel design for AI-native goal-oriented communications, exploiting transformer neural networks under dynamic inference constraints on bandwidth and computation.
Conditional computation in neural networks: principles and research trends
This article summarizes principles and ideas from the emerging area of applying \textit{conditional computation} methods to the design of neural networks.
Class incremental learning with probability dampening and cascaded gated classifier
The latter is a gated incremental classifier, helping the model modify past predictions without directly interfering with them.
Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference
While transformer models have been highly successful, they are computationally inefficient.
