神经网络——反向传播算法

神经网络的损失函数为

[Jleft( Theta  ight) =  - frac{1}{m}left[ {sumlimits_{i = 1}^m {sumlimits_{k = 1}^k {y_k^{left( i ight)}log {{left( {{h_Theta }left( {{x^{left( i ight)}}} ight)} ight)}_k} + left( {1 - y_k^{left( i ight)}} ight)log left( {1 - {{left( {{h_Theta }left( {{x^{left( i ight)}}} ight)} ight)}_k}} ight)} } } ight] + frac{lambda }{{2m}}sumlimits_{l = 1}^{L - 1} {sumlimits_{i = 1}^{{s_l}} {sumlimits_{j = 1}^{{s_{l + 1}}} {{{left( {Theta _{ji}^{left( l ight)}} ight)}^2}} } } ]

我们想要最小化J(Θ)

[underbrace {min }_Theta Jleft( Theta  ight)]

需要计算

[Jleft( Theta  ight)]

[frac{partial }{{partial _{ij}^{left( l ight)}}}Jleft( Theta  ight)]

---

问题的关键是计算

[frac{partial }{{partial _{ij}^{left( l ight)}}}Jleft( Theta  ight)]

---

有如下神经网络（省略中间的连线）

以一个样本为例（x, y）

先计算向前传播

[egin{array}{l}
{a^{left( 1 ight)}} = x\
{z^{left( 2 ight)}} = {Theta ^{left( 1 ight)}}{a^{left( 1 ight)}}\
{a^{left( 2 ight)}} = gleft( {{z^{left( 2 ight)}}} ight)left( { + a_0^{left( 2 ight)}} ight)\
{z^{left( 3 ight)}} = {Theta ^{left( 2 ight)}}{a^{left( 2 ight)}}\
{a^{left( 3 ight)}} = gleft( {{z^{left( 3 ight)}}} ight)left( { + a_0^{left( 3 ight)}} ight)\
{z^{left( 4 ight)}} = {Theta ^{left( 3 ight)}}{a^{left( 3 ight)}}\
{a^{left( 3 ight)}} = {h_Theta }left( x ight) = gleft( {{z^{left( 4 ight)}}} ight)
end{array}]

 反响传播算法

定义

[delta _j^{left( l ight)} = “error” of node j in layer l.]

[delta _j^{left( l ight)} = 第 l 层的第 j 个节点的“偏差”]

因此，对于每个输出层单元（L=4）

[delta _j^{left( 4 ight)} = a_J^{left( 4 ight)} - {y_j}]

y_j是真实值。

接下来计算前面几层的“偏差”

[egin{array}{l}
{delta ^{left( 3 ight)}} = {left( {{Theta ^{left( 3 ight)}}} ight)^T}{delta ^{left( 4 ight)}}. * g'left( {{z^{left( 3 ight)}}} ight)\
{delta ^{left( 2 ight)}} = {left( {{Theta ^{left( 2 ight)}}} ight)^T}{delta ^{left( 3 ight)}}. * g'left( {{z^{left( 2 ight)}}} ight)
end{array}]

第一层没有“偏差”

又可以证明（我没有证明）

[egin{array}{l}
{delta ^{left( 3 ight)}} = {left( {{Theta ^{left( 3 ight)}}} ight)^T}{delta ^{left( 4 ight)}}. * left( {{a^{left( 3 ight)}}. * left( {1 - {a^{left( 3 ight)}}} ight)} ight)\
{delta ^{left( 2 ight)}} = {left( {{Theta ^{left( 2 ight)}}} ight)^T}{delta ^{left( 3 ight)}}. * left( {{a^{left( 2 ight)}}. * left( {1 - {a^{left( 2 ight)}}} ight)} ight)
end{array}]

又有

[frac{partial }{{partial Theta _{ij}^{left( l ight)}}}Jleft( Theta  ight) = a_j^{left( l ight)}delta _i^{left( {l + 1} ight)}]

---

总结反响传播算法

有训练集 

[left{ {left( {{x^{left( 1 ight)}},{y^{left( 1 ight)}}} ight),...,left( {{x^{left( m ight)}},{y^{left( m ight)}}} ight)} ight}]

1，令

[Delta _{ij}^{left( l ight)} = 0]

[Delta 是delta的大写 ]

2，计算

For i = 1 to m {

　　Set a⁽¹⁾ = x⁽ⁱ⁾

　　Perform forward propagation to compute a^(l) for l = 2, 3,.., L

　　Using y⁽ⁱ⁾, compute δ^(L) = a^(L) - y⁽ⁱ⁾

　　Compute δ^(L-1), δ^(L-2),...,δ⁽²⁾

　　[Delta _{ij}^{left( l ight)}: = Delta _{ij}^{left( l ight)} + a_j^{left( l ight)}delta _i^{left( {l + 1} ight)}]

 }

3，计算

if j ≠ 0

[D_{ij}^{left( l ight)}: = frac{1}{m}Delta _{ij}^{left( l ight)} + lambda Theta _{ij}^{left( l ight)}]

if j = 0

[D_{ij}^{left( l ight)}: = frac{1}{m}Delta _{ij}^{left( l ight)}]

这里

[D_{ij}^{left( l ight)} = frac{partial }{{partial Theta _{ij}^{left( l ight)}}}Jleft( Theta  ight)]