反向傳播與優化
Backpropagation & Optimization

IDL115 · 第三章 · KP1–KP16 · 44 張獨立 16:9 雙語 HTML5＋Canvas 資訊圖表。每張均可離線開啟、操作與嵌入 iframe。公式已預先排版，無需網路載入字型或套件。

建議使用 1280×720 或較大視窗。小視窗會將整張 16:9 頁面同比縮放；閱讀細節時請放大視窗。曲線若未標為實算，均為概念示意，非實驗結果。

KP1

訓練的三個問題Three questions behind training 一階與二階：資訊與成本First and second order: information and cost

KP2

梯度下降：方向乘上步長Gradient descent: direction times step size L-smooth：步長為何重要L-smoothness explains safe steps 凸與非凸：能保證什麼？Convex and nonconvex guarantees

KP3

什麼時候停止？四種訊號Four stopping signals 梯度為零，為何不是谷底？Zero gradient does not imply a minimum

KP4

線搜索：太大就減半Backtracking: halve an oversized step 充分下降與精確線搜索Armijo and exact line search

KP5

動量：把歷史方向帶進下一步Momentum carries past directions 動量的向量合成Vector addition behind momentum

KP6

前向：先保存中間計算Forward pass: save intermediate values δ 是什麼？權重梯度怎麼來？Delta and the weight gradient 隱藏層：沿所有下游路徑加總Hidden-layer delta sums downstream paths 完整回傳：算完，再更新Complete backprop: compute, then update

KP7

SGD：用一筆估計全體SGD estimates the full gradient 學習率排程：大步到小步Learning-rate schedules

KP8

Batch 與 Epoch：一小批與一整輪Batch and epoch Mini-batch：平均一小批梯度Mini-batch gradient averaging 抽樣方差：為何大批次更穩？Sampling variance and batch size

KP9

全梯度：凸平滑下的收斂界Full-gradient convergence under convexity SGD：先說清楚衡量什麼SGD rates need a stated metric 噪聲地板：上界不是下限Noise floor: an upper bound is not a lower bound SGD 上界：從一步下降到加總Deriving an averaged stationarity bound

KP10

AdaGrad：每個維度一把尺AdaGrad uses a separate scale per coordinate AdaGrad 的記憶：只加不減AdaGrad never forgets

KP11

RMSProp：記住近期，忘掉遠期RMSProp emphasizes recent gradients Adam：方向記憶加逐維步幅Adam combines direction and scale 偏差修正：補回從零開始的低估Bias correction for zero initialization

KP12

同樣低的點，不同寬度的谷Equal minima, different neighborhood widths 平坦與泛化：動機不是保證Flatness motivates, but does not guarantee

KP13

SAM：先找附近最壞，再降低它SAM minimizes the worst nearby loss SAM 損失 = 原損失 + 銳度SAM loss equals loss plus sharpness 一階近似：往上坡推一小步First-order adversarial perturbation 一般範數與兩次反傳Dual norms and the two-pass update

KP14

ESAM：只擾動一部分權重ESAM stochastic weight perturbation SDS：挑損失上升較多的樣本Select sharpness-sensitive samples ASAM：讓鄰域跟著權重尺度ASAM adapts the neighborhood scale

KP15

過參數化：訓得動的理論Overparameterization and optimization 命題 4：局部谷底與鞍點Proposition 4: minima and saddles 命題 5：深網收斂與條件Proposition 5: deep-network convergence 深網定理：寬度、步長、高機率Width, step size and high probability

KP16

延伸理論：三個研究方向Three directions beyond the core theorems 全章收網：算梯度、走一步、驗結果Compute gradients, update, then evaluate

公式修正、前提與稽核紀錄 / Audit notes