Notes

Deep Reinforcement (Q) Learning Paper: "Human-Level Control Through Deep Reinforcement Learning".

In the previous sections, distribution were always treated as discrete tables. This is not possible for large state/action spaces. Therefore, functional approximations to these functions must be found. We have already seen that deep neural networks (DNN) are function approximators in their nature. One can parametrize a policy or Q-function and use DNN to estimate its parameters directly from state space. For optimization purposes, we define a loss function. This loss follows from moving Q inside of expectation.

$$L = \frac{1}{2}\left\lbrack r + max_{a^{'}}Q\left( s^{'},a^{'} \right) - Q(s,a) \right\rbrack^{2}$$

We find the optimal action-value function $Q^{*}$ by parametrizing it with $\theta$ (make it DNN compatible).

$$Q^{*}(s,a) = max_{\pi}\mathbb{E\lbrack}r_{t} + \gamma r_{t + 1} + \gamma^{2}r_{t + 2} + \ldots|s_{t} = s,a_{t} = a,\pi\rbrack$$

and then minimize the loss w.r.t $\theta$. Comment: Often, we approximate $\mathbb{E}$ by sampling. The following represents the loss function to update the Q-learning rule:

$$L_{i}\left( \theta_{i} \right) = \mathbb{E}_{s,a,r,s^{'}\sim U(D)}\left\lbrack \left( r + \gamma max_{a^{'}}Q\left( s^{'},a^{'} \middle| \theta_{i}^{-} \right) - Q\left( s,a \middle| \theta_{i} \right) \right)^{2} \right\rbrack$$

This is again the temporal difference rule.

In the picture above we have a deep RL system trained directly from input (Atari game video output) to actions of the controlling joystick.

On-policy methods estimate the value of a policy while using it for control. In Off-policy methods, the policy used to generate behavior, called the behavior policy, may be unrelated to the policy that is evaluated and improved, called the estimation policy.

From the Sutton book: "The on-policy approach in the preceding section is actually a compromise - it learns action values not for the optimal policy, but for a near-optimal policy that still explores. A more straightforward approach is to use two policies, one that is learned about and that becomes the optimal policy, and one that is more exploratory and is used to generate behavior. The policy being learned about is called the target policy, and the policy used to generate behavior is called the behavior policy. In this case we say that learning is from data "$o \hookleftarrow$" the target policy, and the overall process is termed "$o \hookleftarrow$-policy learning".