MacOS 26.1 Tahoe Upgrade

损失函数对比

PPO v.s. DPO v.s GRPO

$$\begin{aligned} \mathcal L_{\text{DPO}}(\pi_{\theta};\pi_{ref})&=-\mathbb E_{(x,y_w,y_l)\sim \mathcal D}[\log\sigma(\beta\log\frac{\pi_{\theta}(y_w|x)}{\pi_{ref}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_l|x)}{\pi_{ref}(y_l|x)})]\\&=-\log\sigma(\beta\log\frac{\pi_{\theta}(y_w|x)}{\pi_{ref}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_l|x)}{\pi_{ref}(y_l|x)}) \end{aligned}$$ $$\begin{aligned} \mathcal L_{\text{PPO}}&=\mathcal L_{\text{policy gradient}}+c_v\cdot \mathcal L_{value}-c_{kl}\cdot \mathcal L_{entropy} \\ &=\mathbb E[\min(r_t(\theta)A_t,\text{clip}(r_t(\theta),1-\epsilon,1+\epsilon)A_t)]+\mathbb{E}_t[(V_\theta(s_t)-R_t)^2]+\mathbb{E}[KL_{t}(\pi_{\theta}(·|s_t) || \pi_{ref}(·|s_t))] \end{aligned}$$ $$\mathcal{L}_{\text{GRPO}}(\theta) = -\mathbb{E}\left[ q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{old}}(O|q) \right] \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left\{ \min\left[ \frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|q,o_{i,<t})} \hat{A}_{i,t}, \ \text{clip}\left( \frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|q,o_{i,<t})}, 1-\varepsilon, 1+\varepsilon \right) \hat{A}_{i,t} \right] - \beta \cdot \mathbb{D}_{KL}\left[ \pi_{\theta} \parallel \pi_{ref} \right] \right\}$$

---

转载请注明来源 goldandrabbit.github.io