FastAttention: Extend FlashAttention2 to NPUs and Low-resource GPUs

no code implementations • 22 Oct 2024 • Haoran Lin, Xianzhi Yu, Kang Zhao, Lu Hou, Zongyuan Zhan, Stanislav Kamenev, Han Bao, Ting Hu, Mingkai Wang, Qixin Chang, Siyue Sui, Weihao Sun, Jiaxin Hu, Jun Yao, Zekun Yin, Cheng Qian, Ying Zhang, Yinfei Pan, Yu Yang, Weiguo Liu

In this work, we propose FastAttention which pioneers the adaptation of FlashAttention series for NPUs and low-resource GPUs to boost LLM inference efficiency.

Scalable Infomin Learning

1 code implementation • 21 Feb 2023 • Yanzhi Chen, Weihao Sun, Yingzhen Li, Adrian Weller

The task of infomin learning aims to learn a representation with high utility while being uninformative about a specified target, with the latter achieved by minimising the mutual information between the representation and the target.

Domain Adaptation Fairness +1

SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition

4 code implementations • ICLR 2020 • Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, Sungjin Ahn

Previous approaches for unsupervised object-oriented scene representation learning are either based on spatial-attention or scene-mixture approaches and limited in scalability which is a main obstacle towards modeling real-world scenes.

Object Representation Learning