Liquid Structural State-Space Models

A proper parametrization of state transition matrices of linear state-space models (SSMs) followed by standard nonlinearities enables them to efficiently learn representations from sequential data, establishing the state-of-the-art on a large series of long-range sequence modeling benchmarks. In this paper, we show that we can improve further when the structural SSM such as S4 is given by a linear liquid time-constant (LTC) state-space model. LTC neural networks are causal continuous-time neural networks with an input-dependent state transition module, which makes them learn to adapt to incoming inputs at inference. We show that by using a diagonal plus low-rank decomposition of the state transition matrix introduced in S4, and a few simplifications, the LTC-based structural state-space model, dubbed Liquid-S4, achieves the new state-of-the-art generalization across sequence modeling tasks with long-term dependencies such as image, text, audio, and medical time-series, with an average performance of 87.32% on the Long-Range Arena benchmark. On the full raw Speech Command recognition, dataset Liquid-S4 achieves 96.78% accuracy with a 30% reduction in parameter counts compared to S4. The additional gain in performance is the direct result of the Liquid-S4's kernel structure that takes into account the similarities of the input sequence samples during training and inference.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
SpO2 estimation BIDMC Liquid-S4 MAE [bpm, session-wise] 0.066 # 1
Heart rate estimation BIDMC Liquid-S4 MAE [bpm, session-wise] 0.303 # 1
Long-range modeling LRA Liquid-S4 ListOps 62.75 # 2
Text 89.02 # 6
Retrieval 91.20 # 4
Image 89.50 # 4
Pathfinder 94.8 # 6
Avg 87.32 # 5
Pathfinder-X 96.66 # 6
Speech Recognition Speech Commands Liquid-S4 Accuracy (%) 98.51 # 1

Methods


No methods listed for this paper. Add relevant methods here