Prioritized Experience Replay is a type of experience replay in reinforcement learning where we more frequently replay transitions with high expected learning progress, as measured by the magnitude of their temporaldifference (TD) error. This prioritization can lead to a loss of diversity, which is alleviated with stochastic prioritization, and introduce bias, which can be corrected with importance sampling.
The stochastic sampling method interpolates between pure greedy prioritization and uniform random sampling. The probability of being sampled is ensured to be monotonic in a transition's priority, while guaranteeing a nonzero probability even for the lowestpriority transition. Concretely, define the probability of sampling transition $i$ as
$$P(i) = \frac{p_i^{\alpha}}{\sum_k p_k^{\alpha}}$$
where $p_i > 0$ is the priority of transition $i$. The exponent $\alpha$ determines how much prioritization is used, with $\alpha=0$ corresponding to the uniform case.
Prioritized replay introduces bias because it changes this distribution in an uncontrolled fashion, and therefore changes the solution that the estimates will converge to. We can correct this bias by using importancesampling (IS) weights:
$$ w_{i} = \left(\frac{1}{N}\cdot\frac{1}{P\left(i\right)}\right)^{\beta} $$
that fully compensates for the nonuniform probabilities $P\left(i\right)$ if $\beta = 1$. These weights can be folded into the Qlearning update by using $w_{i}\delta_{i}$ instead of $\delta_{i}$  weighted IS rather than ordinary IS. For stability reasons, we always normalize weights by $1/\max_{i}w_{i}$ so that they only scale the update downwards.
The two types of prioritization are proportional based, where $p_{i} = \delta_{i} + \epsilon$ and rankbased, where $p_{i} = \frac{1}{\text{rank}\left(i\right)}$, the latter where $\text{rank}\left(i\right)$ is the rank of transition $i$ when the replay memory is sorted according to $\delta_{i}$, For proportional based, hyperparameters used were $\alpha = 0.7$, $\beta_{0} = 0.5$. For the rankbased variant, hyperparameters used were $\alpha = 0.6$, $\beta_{0} = 0.4$.
Source: Prioritized Experience ReplayPaper  Code  Results  Date  Stars 

Task  Papers  Share 

Reinforcement Learning (RL)  65  34.39% 
Modelbased Reinforcement Learning  18  9.52% 
Continuous Control  14  7.41% 
Atari Games  13  6.88% 
Decision Making  8  4.23% 
OpenAI Gym  7  3.70% 
Starcraft  5  2.65% 
Game of Go  5  2.65% 
Offline RL  4  2.12% 
Component  Type 


🤖 No Components Found  You can add them if they exist; e.g. Mask RCNN uses RoIAlign 