Seq2Tens: An Efficient Representation of Sequences by Low-Rank Tensor Projections
Sequential data such as time series, video, or text can be challenging to analyse as the ordered structure gives rise to complex dependencies. At the heart of this is non-commutativity, in the sense that reordering the elements of a sequence can completely change its meaning. We use a classical mathematical object -- the tensor algebra -- to capture such dependencies. To address the innate computational complexity of high degree tensors, we use compositions of low-rank tensor projections. This yields modular and scalable building blocks for neural networks that give state-of-the-art performance on standard benchmarks such as multivariate time series classification and generative models for video.
PDF Abstract ICLR 2021 PDF ICLR 2021 AbstractCode
Datasets
SHAPES
Sprites
PhysioNet Challenge 2012
PenDigits
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
|---|---|---|---|---|---|---|
| Time Series Classification | ArabicDigits | FCN-SNLST | Accuracy | 0.993 | # 3 | |
| Time Series Classification | ArabicDigits | SNLST | Accuracy | 0.968 | # 10 | |
| Time Series Classification | AUSLAN | FCN-SNLST | Accuracy | 0.993 | # 1 | |
| Time Series Classification | AUSLAN | SNLST | Accuracy | 0.969 | # 4 | |
| Time Series Classification | CharacterTrajectories | FCN-SNLST | Accuracy | 0.994 | # 2 | |
| Time Series Classification | CharacterTrajectories | SNLST | Accuracy | 0.957 | # 5 | |
| Time Series Classification | CMUsubject16 | FCN-SNLST | Accuracy | 1 | # 1 | |
| Time Series Classification | CMUsubject16 | SNLST | Accuracy | 1 | # 1 | |
| Time Series Classification | DigitShapes | SNLST | Accuracy | 1 | # 1 | |
| Time Series Classification | DigitShapes | FCN-SNLST | Accuracy | 1 | # 1 | |
| Time Series Classification | ECG | FCN-SNLST | Accuracy | 0.860 | # 1 | |
| Time Series Classification | ECG | SNLST | Accuracy | 0.842 | # 4 | |
| Imputation | HMNIST | GP-VAE (B-NLST) | NLL | 0.251 | # 1 | |
| MSE | 0.092 | # 1 | ||||
| AUROC | 0.962 | # 1 | ||||
| Time Series Classification | JapaneseVowels | SNLST | Accuracy | 0.979 | # 10 | |
| Time Series Classification | JapaneseVowels | FCN-SNLST | Accuracy | 0.980 | # 9 | |
| Time Series Classification | KickvsPunch | FCN-SNLST | Accuracy | 1 | # 1 | |
| Time Series Classification | KickvsPunch | SNLST | Accuracy | 1 | # 1 | |
| Time Series Classification | Libras | SNLST | Accuracy | 0.773 | # 7 | |
| Time Series Classification | Libras | FCN-SNLST | Accuracy | 0.957 | # 2 | |
| Time Series Classification | NetFlow | SNLST | Accuracy | 0.793 | # 9 | |
| Time Series Classification | NetFlow | FCN-SNLST | Accuracy | 0.960 | # 1 | |
| Time Series Classification | PEMS | FCN-SNLST | Accuracy | 0.857 | # 1 | |
| Time Series Classification | PEMS | SNLST | Accuracy | 0.747 | # 7 | |
| Time Series Classification | PenDigits | FCN-SNLST | Accuracy | 0.953 | # 3 | |
| Time Series Classification | PenDigits | SNLST | Accuracy | 0.954 | # 2 | |
| Imputation | PhysioNet Challenge 2012 | GP-VAE (B-NLST) | AUROC | 0.743 | # 1 | |
| Time Series Classification | SHAPES | FCN-SNLST | Accuracy | 1 | # 1 | |
| Time Series Classification | SHAPES | SNLST | Accuracy | 1 | # 1 | |
| Imputation | Sprites | GP-VAE (B-NLST) | MSE | 0.002 | # 1 | |
| Time Series Classification | UWave | SNLST | Accuracy | 0.938 | # 7 | |
| Time Series Classification | UWave | FCN-SNLST | Accuracy | 0.969 | # 3 | |
| Time Series Classification | Wafer | SNLST | Accuracy | 0.981 | # 7 | |
| Time Series Classification | Wafer | FCN-SNLST | Accuracy | 0.989 | # 4 | |
| Time Series Classification | WalkvsRun | FCN-SNLST | Accuracy | 1 | # 1 | |
| Time Series Classification | WalkvsRun | SNLST | Accuracy | 1 | # 1 |
