Temporal Convolutions

Lightweight Convolution

Introduced by Wu et al. in Pay Less Attention with Lightweight and Dynamic Convolutions

LightConv is a type of depthwise convolution for sequential modelling which shares certain output channels and whose weights are normalized across the temporal dimension using a softmax. Compared to self-attention, LightConv has a fixed context window and it determines the importance of context elements with a set of weights that do not change over time steps. LightConv computes the following for the $i$-th element in the sequence and output channel $c$:

$$ \text{LightConv}\left(X, W_{\text{ceil}\left(\frac{cH}{d}\right),:}, i, c\right) = \text{DepthwiseConv}\left(X,\text{softmax}\left(W_{\text{ceil}\left(\frac{cH}{d}\right),:}\right), i, c\right) $$

Source: Pay Less Attention with Lightweight and Dynamic Convolutions

Papers


Paper Code Results Date Stars

Tasks


Task Papers Share
Abstractive Text Summarization 1 33.33%
Language Modelling 1 33.33%
Machine Translation 1 33.33%

Categories