no code implementations • 27 Jan 2025 • Mude Hui, Rui-Jie Zhu, Songlin Yang, Yu Zhang, ZiRui Wang, Yuyin Zhou, Jason Eshraghian, Cihang Xie
Flow models are effective at progressively generating realistic images, but they generally struggle to capture long-range dependencies during the generation process as they compress all the information from previous time steps into a single corrupted image.
1 code implementation • 23 Jan 2025 • Xuerui Qiu, Jieyuan Zhang, Wenjie Wei, Honglin Cao, Junsheng Guo, Rui-Jie Zhu, Yimeng Shan, Yang Yang, Malu Zhang, Haizhou Li
To mitigate this issue, we take inspiration from mutual information entropy and propose a bi-level optimization strategy to rectify the information distribution in Q-SDSA.
no code implementations • 19 Oct 2024 • Skye Gunasekaran, Assel Kembay, Hugo Ladret, Rui-Jie Zhu, Laurent Perrinet, Omid Kavehei, Jason Eshraghian
By incorporating a predictive feedback mechanism that adapts to data distribution drift, Future-Guided Learning offers a promising avenue for advancing time-series forecasting with deep learning.
no code implementations • 12 Oct 2024 • Ruhai Lin, Rui-Jie Zhu, Jason K. Eshraghian
Through this research, we can determine the trade-off between data transfer volume and model performance, enabling the identification of a balanced point that achieves good performance while minimizing data transfer volume.
no code implementations • 6 Oct 2024 • Qichao Ma, Rui-Jie Zhu, Peiye Liu, Renye Yan, Fahong Zhang, Ling Liang, Meng Li, Zhaofei Yu, Zongwei Wang, Yimao Cai, Tiejun Huang
Thus, InnerProbe performs sub-dataset contribution analysis using a lightweight LSTM-based network trained on MHA results in a supervised manner.
1 code implementation • 5 Jun 2024 • Xuerui Qiu, Zheng Luan, Zhaorui Wang, Rui-Jie Zhu
Furthermore, our ALIF neuron model achieves remarkable classification accuracy on MNIST (99. 78\%) and CIFAR-10 (93. 89\%) datasets, demonstrating the effectiveness of learning adaptive thresholds for spiking neurons.
1 code implementation • 4 Jun 2024 • Rui-Jie Zhu, Yu Zhang, Ethan Sifferman, Tyler Sheaves, Yiqiao Wang, Dustin Richmond, Peng Zhou, Jason K. Eshraghian
Our experiments show that our proposed MatMul-free models achieve performance on-par with state-of-the-art Transformers that require far more memory during inference at a scale up to at least 2. 7B parameters.
1 code implementation • 30 May 2024 • Rui-Jie Zhu, Ziqing Wang, Leilani Gilpin, Jason K. Eshraghian
Autonomous driving demands an integrated approach that encompasses perception, prediction, and planning, all while operating under strict energy constraints to enhance scalability and environmental sustainability.
no code implementations • 22 May 2024 • Yimeng Shan, Malu Zhang, Rui-Jie Zhu, Xuerui Qiu, Jason K. Eshraghian, Haicheng Qu
To address this issue, we have designed a Spiking Multiscale Attention (SMA) module that captures multiscale spatiotemporal interaction information.
6 code implementations • 8 Apr 2024 • Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartłomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr., Jiaju Lin, Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Cahya Wirawan, Stanisław Woźniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, Rui-Jie Zhu
We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon the RWKV (RWKV-4) architecture.
no code implementations • 2 Dec 2023 • Souvik Kundu, Rui-Jie Zhu, Akhilesh Jaiswal, Peter A. Beerel
Neuromorphic computing and, in particular, spiking neural networks (SNNs) have become an attractive alternative to deep neural networks for a broad range of signal processing applications, processing static and/or temporal inputs from different sensory modalities, including audio and vision sensors.
1 code implementation • 11 Nov 2023 • Yimeng Shan, Xuerui Qiu, Rui-Jie Zhu, Jason K. Eshraghian, Malu Zhang, Haicheng Qu
As the demand for heightened performance in SNNs surges, the trend towards training deeper networks becomes imperative, while residual learning stands as a pivotal method for training deep neural networks.
1 code implementation • 12 Aug 2023 • Xuerui Qiu, Rui-Jie Zhu, Yuhong Chou, Zhaorui Wang, Liang-Jian Deng, Guoqi Li
Experiments on CIFAR10/100 and ImageNet datasets demonstrate that GAC achieves state-of-the-art accuracy with remarkable efficiency.
Ranked #111 on
Image Classification
on CIFAR-10
14 code implementations • 22 May 2023 • Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, Rui-Jie Zhu
This work presents a significant step towards reconciling trade-offs between computational efficiency and model performance in sequence processing tasks.
Ranked #22 on
Natural Language Inference
on WNLI
no code implementations • 5 Apr 2023 • Qihang Zhao, Rui-Jie Zhu, Liu Yang, He Yongming, Bo Zhou, Luo Cheng
In the realm of search systems, multi-stage cascade architecture is a prevalent method, typically consisting of sequential modules such as matching, pre-ranking, and ranking.
1 code implementation • 27 Feb 2023 • Rui-Jie Zhu, Qihang Zhao, Guoqi Li, Jason K. Eshraghian
As a result, their performance lags behind modern deep learning, and we are yet to see the effectiveness of SNNs in language generation.
1 code implementation • 21 Jun 2022 • Rui-Jie Zhu, Malu Zhang, Qihang Zhao, Haoyu Deng, Yule Duan, Liang-Jian Deng
Given the critical role of attention mechanisms in enhancing neural network performance, the integration of SNNs and attention mechanisms exhibits potential to deliver energy-efficient and high-performance computing paradigms.
no code implementations • 30 Mar 2022 • Cheng Jin, Rui-Jie Zhu, Xiao Wu, Liang-Jian Deng
Spiking Neural Networks (SNNs) have piqued researchers' interest because of their capacity to process temporal information and low power consumption.