In a Replacing Eligibility Trace, each time the state is revisited, the trace is reset to $1$ regardless of the presence of a prior trace.. For the memory vector $\textbf{e}_{t} \in \mathbb{R}^{b} \geq \textbf{0}$:
$$\mathbf{e_{0}} = \textbf{0}$$
$$\textbf{e}_{t} = \gamma\lambda{e}_{t-1}\left(s\right) \text{ if } s \neq s_{t}$$
$$\textbf{e}_{t} = 1 \text{ if } s = s_{t}$$
They can be seen as crude approximation to dutch traces, which have largely superseded them as they perform better than replacing traces and have a clearer theoretical basis. Accumulating traces remain of interest for nonlinear function approximations where dutch traces are not available.
Source: Sutton and Barto, Reinforcement Learning, 2nd Edition
Paper | Code | Results | Date | Stars |
---|
Component | Type |
|
---|---|---|
🤖 No Components Found | You can add them if they exist; e.g. Mask R-CNN uses RoIAlign |