One difficulty that arises with optimization of deep neural networks is that large parameter gradients can lead an SGD optimizer to update the parameters strongly into a region where the loss function is much greater, effectively undoing much of the work that was needed to get to the current solution.
Gradient Clipping clips the size of the gradients to ensure optimization performs more reasonably near sharp areas of the loss surface. It can be performed in a number of ways. One option is to simply clip the parameter gradient elementwise before a parameter update. Another option is to clip the norm $\textbf{g}$ of the gradient $\textbf{g}$ before a parameter update:
$$\text{ if } \textbf{g} > v \text{ then } \textbf{g} \leftarrow \frac{\textbf{g}{v}}{\textbf{g}}$$
where $v$ is a norm threshold.
Source: Deep Learning, Goodfellow et al
Image Source: Pascanu et al
Paper  Code  Results  Date  Stars 

Task  Papers  Share 

Language Modelling  9  7.63% 
Reinforcement Learning (RL)  8  6.78% 
Image Classification  7  5.93% 
Federated Learning  7  5.93% 
Text Generation  4  3.39% 
Recommendation Systems  3  2.54% 
SelfSupervised Learning  3  2.54% 
Object Detection  3  2.54% 
General Classification  3  2.54% 
Component  Type 


🤖 No Components Found  You can add them if they exist; e.g. Mask RCNN uses RoIAlign 