Attention

General • 125 methods

Attention is a technique for attending to different parts of an input vector to capture long-term dependencies. Within the context of NLP, traditional sequence-to-sequence models compressed the input sequence to a fixed-length context vector, which hindered their ability to remember long inputs such as sentences. In contrast, attention creates shortcuts between the context vector and the entire source input. Below you will find a continuously updating list of attention based building blocks used in deep learning.

Subcategories

Method Year Papers
2017 18585
2017 18464
2019 1480
2019 1479
2017 273
2017 234
2015 223
2014 198
2017 182
2018 181
2021 167
2018 136
2022 128
2014 109
2021 87
2020 86
2020 79
2020 78
2020 75
2020 75
2018 73
2019 71
2020 69
2019 52
2019 49
2021 48
2020 45
2017 41
2018 40
2020 40
2020 38
2019 37
2015 33
2015 33
2014 32
2021 32
2020 25
2015 24
2018 24
2021 24
2019 23
2022 23
2021 22
2018 21
2019 20
2020 20
2015 20
2020 18
2020 18
2018 16
2020 15
2022 14
2023 12
2020 12
2021 11
2019 11
2019 10
2017 9
2018 9
2019 9
2021 8
2017 8
2020 7
2019 7
2019 7
2020 7
2021 6
2015 6
2019 6
2019 6
2018 6
2019 5
2021 4
2018 4
2020 4
2020 4
2021 4
2021 3
2018 3
2016 3
2020 3
2020 3
2021 3
2020 3
2015 3
2018 3
2020 2
2018 2
2
2021 2
2020 2
2021 2
2017 2
2021 2
2017 2
2016 2
2021 2
2019 2
2020 2
2021 2
2021 2
2019 2
2020 1
2020 1
2018 1
2018 1
2020 1
2022 1
2016 1
2022 1
2020 1
2021 1
2021 1
2023 1
2020 1
2021 1
2022 1
2020 1
2019 1
2021 1
2020 1
2019 1
2020 1
2021 1
2020 1
2020 1
2000 0