Regularization

Label Smoothing

Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the likelihood of $\log{p}\left(y\mid{x}\right)$ directly can be harmful. Assume for a small constant $\epsilon$, the training set label $y$ is correct with probability $1-\epsilon$ and incorrect otherwise. Label Smoothing regularizes a model based on a softmax with $k$ output values by replacing the hard $0$ and $1$ classification targets with targets of $\frac{\epsilon}{k-1}$ and $1-\epsilon$ respectively.

Source: Deep Learning, Goodfellow et al

Image Source: When Does Label Smoothing Help?

Papers


Paper Code Results Date Stars

Tasks


Task Papers Share
Language Modelling 46 5.79%
Semantic Segmentation 31 3.90%
Large Language Model 30 3.77%
Object Detection 23 2.89%
Decision Making 18 2.26%
Question Answering 14 1.76%
Autonomous Driving 13 1.64%
Retrieval 13 1.64%
Denoising 11 1.38%

Components


Component Type
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign

Categories