Dot-Product Attention

Introduced by Luong et al. in Effective Approaches to Attention-based Neural Machine Translation

Dot-Product Attention is an attention mechanism where the alignment score function is calculated as:

$$f_{att}\left(\textbf{h}_{i}, \textbf{s}_{j}\right) = h_{i}^{T}s_{j}$$

It is equivalent to multiplicative attention (without a trainable weight matrix, assuming this is instead an identity matrix). Here $\textbf{h}$ refers to the hidden states for the encoder, and $\textbf{s}$ is the hidden states for the decoder. The function above is thus a type of alignment score function.

Within a neural network, once we have the alignment scores, we calculate the final scores/weights using a softmax function of these alignment scores (ensuring it sums to 1).

Source: Effective Approaches to Attention-based Neural Machine Translation

Latest Papers

PAPER DATE
Conditioning Trick for Training Stable GANs
Mohammad EsmaeilpourRaymel Alfonso SalloOlivier St-GeorgesPatrick CardinalAlessandro Lameiras Koerich
2020-10-12
SMYRF: Efficient Attention using Asymmetric Clustering
| Giannis DarasNikita KitaevAugustus OdenaAlexandros G. Dimakis
2020-10-11
LOGAN: Local Group Bias Detection by Clustering
Jieyu ZhaoKai-Wei Chang
2020-10-06
Learning Hard Retrieval Cross Attention for Transformer
Hongfei XuQiuhui Liu
2020-09-30
TinyGAN: Distilling BigGAN for Conditional Image Generation
| Ting-Yun ChangChi-Jen Lu
2020-09-29
not-so-BigGAN: Generating High-Fidelity Images on a Small Compute Budget
Seungwook HanAkash SrivastavaCole HurwitzPrasanna SattigeriDavid D. Cox
2020-09-09
Multi-Attention-Network for Semantic Segmentation of High-Resolution Remote Sensing Images
Rui LiShunyi ZhengChenxi DuanJianlin Su
2020-09-03
Neural Crossbreed: Neural Based Image Metamorphosis
Sanghun ParkKwanggyoon SeoJunyong Noh
2020-09-02
Instance Selection for GANs
Terrance DeVriesMichal DrozdzalGraham W. Taylor
2020-07-30
Linear Attention Mechanism: An Efficient Attention for Semantic Segmentation
| Rui LiJianlin SuChenxi DuanShunyi Zheng
2020-07-29
TensorCoder: Dimension-Wise Attention via Tensor Representation for Natural Language Modeling
Shuai ZhangPeng ZhangXindian MaJunqiu Weiningning WangQun Liu
2020-07-28
BAKSA at SemEval-2020 Task 9: Bolstering CNN with Self-Attention for Sentiment Analysis of Code Mixed Text
| Ayush KumarHarsh AgarwalKeshav BansalAshutosh Modi
2020-07-21
Interpolating GANs to Scaffold Autotelic Creativity
Ziv EpsteinOcéane BoulaisSkylar GordonMatt Groh
2020-07-21
Relational-Grid-World: A Novel Relational Reasoning Environment and An Agent Model for Relational Information Extraction
Faruk KucuksubasiElif Surer
2020-07-12
Differentiable Augmentation for Data-Efficient GAN Training
| Shengyu ZhaoZhijian LiuJi LinJun-Yan ZhuSong Han
2020-06-18
Training Generative Adversarial Networks with Limited Data
| Tero KarrasMiika AittalaJanne HellstenSamuli LaineJaakko LehtinenTimo Aila
2020-06-11
Learning disconnected manifolds: a no GANs land
Ugo TanielianThibaut IssenhuthElvis DohmatobJeremie Mary
2020-06-08
Big GANs Are Watching You: Towards Unsupervised Object Segmentation with Off-the-Shelf Generative Models
| Andrey VoynovStanislav MorozovArtem Babenko
2020-06-08
A U-Net Based Discriminator for Generative Adversarial Networks
Edgar Schonfeld Bernt Schiele Anna Khoreva
2020-06-01
Permutation Matters: Anisotropic Convolutional Layer for Learning on Point Clouds
| Zhongpai GaoGuangtao ZhaiJunchi YanXiaokang Yang
2020-05-27
Network Fusion for Content Creation with Conditional INNs
Robin RombachPatrick EsserBjörn Ommer
2020-05-27
Exploring Self-attention for Image Recognition
| Hengshuang ZhaoJiaya JiaVladlen Koltun
2020-04-28
Pyramid Attention Networks for Image Restoration
| Yiqun MeiYuchen FanYulun ZhangJiahui YuYuqian ZhouDing LiuYun FuThomas S. HuangHumphrey Shi
2020-04-28
GANSpace: Discovering Interpretable GAN Controls
| Erik HärkönenAaron HertzmannJaakko LehtinenSylvain Paris
2020-04-06
Evolving Normalization-Activation Layers
| Hanxiao LiuAndrew BrockKaren SimonyanQuoc V. Le
2020-04-06
Feature Quantization Improves GAN Training
| Yang ZhaoChunyuan LiPing YuJianfeng GaoChangyou Chen
2020-04-05
BigGAN-based Bayesian reconstruction of natural images from human brain activity
Kai QiaoJian ChenLinyuan WangChi ZhangLi TongBin Yan
2020-03-13
A U-Net Based Discriminator for Generative Adversarial Networks
| Edgar SchönfeldBernt SchieleAnna Khoreva
2020-02-28
Multimodal Transformer with Pointer Network for the DSTC8 AVSD Challenge
Hung LeNancy F. Chen
2020-02-25
Improved Consistency Regularization for GANs
Zhengli ZhaoSameer SinghHonglak LeeZizhao ZhangAugustus OdenaHan Zhang
2020-02-11
Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN
Milad MozafariLeila ReddyRufin VanRullen
2020-01-31
Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures
Mohamed El Amine SeddikCosme LouartMohamed TamaazoustiRomain Couillet
2020-01-21
CNN-generated images are surprisingly easy to spot... for now
| Sheng-Yu WangOliver WangRichard ZhangAndrew OwensAlexei A. Efros
2019-12-23
LOGAN: Latent Optimisation for Generative Adversarial Networks
| Yan WuJeff DonahueDavid BalduzziKaren SimonyanTimothy Lillicrap
2019-12-02
Detecting GAN generated errors
Xiru ZhuFengdi CheTianzi YangTzuyang YuDavid MegerGregory Dudek
2019-12-02
Your Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative Models
| Giannis DarasAugustus OdenaHan ZhangAlexandros G. Dimakis
2019-11-27
Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis
| Ceyuan YangYujun ShenBolei Zhou
2019-11-21
Personalizing Graph Neural Networks with Attention Mechanism for Session-based Recommendation
| Shu WuMengqi ZhangXin JiangXu KeLiang Wang
2019-10-20
Improving sample diversity of a pre-trained, class-conditional GAN by changing its class embeddings
| Qi LiLong MaiMichael A. AlcornAnh Nguyen
2019-10-10
Attribute Manipulation Generative Adversarial Networks for Fashion Images
Kenan E. Ak Joo Hwee Lim Jo Yew Tham Ashraf A. Kassim
2019-10-01
Adversarial Video Generation on Complex Datasets
Aidan ClarkJeff DonahueKaren Simonyan
2019-07-15
Large Scale Adversarial Representation Learning
| Jeff DonahueKaren Simonyan
2019-07-04
Attention over Heads: A Multi-Hop Attention for Neural Machine Translation
Shohei IidaRyuichiro KimuraHongyi CuiPo-Hsuan HungTakehito UtsuroMasaaki Nagata
2019-07-01
Sharing Attention Weights for Fast Transformer
Tong XiaoYinqiao LiJingbo ZhuZhengtao YuTongran Liu
2019-06-26
Spatial Group-wise Enhance: Improving Semantic Feature Learning in Convolutional Networks
| Xiang LiXiaolin HuJian Yang
2019-05-23
Improved Precision and Recall Metric for Assessing Generative Models
| Tuomas KynkäänniemiTero KarrasSamuli LaineJaakko LehtinenTimo Aila
2019-04-15
High-Fidelity Image Generation With Fewer Labels
| Mario LucicMichael TschannenMarvin RitterXiaohua ZhaiOlivier BachemSylvain Gelly
2019-03-06
Link Prediction with Mutual Attention for Text-Attributed Networks
Robin BrochierAdrien GuilleJulien Velcin
2019-02-28
Unsupervised Image-to-Image Translation with Self-Attention Networks
| Taewon KangKwang Hee Lee
2019-01-24
Efficient Attention: Attention with Linear Complexities
| Zhuoran ShenMingyuan ZhangHaiyu ZhaoShuai YiHongsheng Li
2018-12-04
Discriminator Rejection Sampling
| Samaneh AzadiCatherine OlssonTrevor DarrellIan GoodfellowAugustus Odena
2018-10-16
Metropolis-Hastings view on variational inference and adversarial training
Kirill NeklyudovEvgenii EgorovPavel ShvechikovDmitry Vetrov
2018-10-16
Large Scale GAN Training for High Fidelity Natural Image Synthesis
| Andrew BrockJeff DonahueKaren Simonyan
2018-09-28
Generative Adversarial Network with Spatial Attention for Face Attribute Editing
| Gang ZhangMeina KanShiguang ShanXilin Chen
2018-09-01
Self-Attention Generative Adversarial Networks
| Han ZhangIan GoodfellowDimitris MetaxasAugustus Odena
2018-05-21
Phase Conductor on Multi-layered Attentions for Machine Comprehension
Rui LiuWei WeiWeiguang MaoMaria Chikina
2017-10-28
Effective Approaches to Attention-based Neural Machine Translation
| Minh-Thang LuongHieu PhamChristopher D. Manning
2015-08-17

Components

COMPONENT TYPE
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign

Categories