一般範數與兩次反傳Dual norms and the two-pass update

IDL115 · CH03 · KP13 · 35/44

算擾動、再在擾動點算更新梯度。 / Compute a perturbation, then a gradient at the perturbed point.

對偶範數Dual norm

1<p<∞、1/p+1/q=1；p=1 或 ∞ 用極限或對應解。 / Handle endpoint norms separately.

\hat\epsilon=\rho\frac{\operatorname{sign}(g)\odot|g|^{q-1}}{\|g\|_q^{q-1}}\quad\text{(51)}

實作通常將擾動視為常數，不回傳穿過擾動計算。 / Stop gradients through the perturbation.

g_{\mathrm{SAM}}\approx\nabla\mathcal L(w+\operatorname{stopgrad}(\hat\epsilon))\quad\text{(53)}

暫加擾動 → 第二次反傳 → 移除擾動 → 用新梯度更新原權重。 / Restore original weights before the optimizer step.

w^+=w-\eta g_{\mathrm{SAM}}