Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence

4 code implementations • 8 Apr 2024 • Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartłomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr., Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Stanisław Woźniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, Rui-Jie Zhu

We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon the RWKV (RWKV-4) architecture.

RWKV-TS: Beyond Traditional Recurrent Neural Network for Time Series Tasks

1 code implementation • 17 Jan 2024 • Haowen Hou, F. Richard Yu

The combination of competitive performance, low latency, and efficient memory usage positions RWKV-TS as a promising avenue for future research in time series tasks.

Computational Efficiency Time Series +1

Revisit Human-Scene Interaction via Space Occupancy

no code implementations • 5 Dec 2023 • Xinpeng Liu, Haowen Hou, Yanchao Yang, Yong-Lu Li, Cewu Lu

Human-scene Interaction (HSI) generation is a challenging task and crucial for various downstream tasks.

RWKV: Reinventing RNNs for the Transformer Era

5 code implementations • 22 May 2023 • Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, Rui-Jie Zhu

This work presents a significant step towards reconciling trade-offs between computational efficiency and model performance in sequence processing tasks.

Computational Efficiency Natural Language Inference

BagFormer: Better Cross-Modal Retrieval via bag-wise interaction

no code implementations • 29 Dec 2022 • Haowen Hou, Xiaopeng Yan, Yigeng Zhang, Fengzong Lian, Zhanhui Kang

In the field of cross-modal retrieval, single encoder models tend to perform better than dual encoder models, but they suffer from high latency and low throughput.

Cross-Modal Retrieval Retrieval

Sams-Net: A Sliced Attention-based Neural Network for Music Source Separation

1 code implementation • 12 Sep 2019 • Tingle Li, Jia-Wei Chen, Haowen Hou, Ming Li

Convolutional Neural Network (CNN) or Long short-term memory (LSTM) based models with the input of spectrogram or waveforms are commonly used for deep learning based audio source separation.

Audio Source Separation Music Source Separation