梯度下降算法与Normal equation

Normal equation: Method to solve for θ analytically

正规方程：分析求解θ的方法

对于损失函数

[Jleft( {{ heta _0},{ heta _1},...,{ heta _n}} ight) = frac{1}{{2m}}sumlimits_{i = 1}^m {{{left( {{h_ heta }left( {{x^{left( i ight)}}} ight) - {y^{left( i ight)}}} ight)}^2}} ]

只要满足

[frac{partial }{{partial { heta _1}}}Jleft( heta  ight) = frac{partial }{{partial { heta _2}}}Jleft( heta  ight) =  cdot  cdot  cdot  = frac{partial }{{partial { heta _n}}}Jleft( heta  ight) = 0]

就可以直接得到所有的参数

[{{ heta _0},{ heta _1},...,{ heta _n}}]

而满足上面的连续等式的解是

[ heta  = {left( {{X^T}X} ight)^{ - 1}}{X^T}y]

其中 

[X = left[ {egin{array}{*{20}{c}}
{egin{array}{*{20}{c}}
{1,x_1^{left( 1 ight)},x_2^{left( 1 ight)},...,x_n^{left( 1 ight)}}\
{1,x_1^{left( 2 ight)},x_2^{left( 2 ight)},...,x_n^{left( 2 ight)}}\
egin{array}{l}
cdot \
cdot
end{array}
end{array}}\
{1,x_1^{left( m ight)},x_2^{left( m ight)},...,x_n^{left( m ight)}}
end{array}} ight]]

是变量的矩阵；

[y = left[ {egin{array}{*{20}{c}}
{egin{array}{*{20}{c}}
{{y^{left( 1 ight)}}}\
{{y^{left( 2 ight)}}}\
egin{array}{l}
cdot \
cdot
end{array}
end{array}}\
{{y^{left( m ight)}}}
end{array}} ight]]

是对应的输出值

Gradient Descent	Normal Equation
Need to choose α	No need to choose α
Needs many iterations	Don't need to iterate
Works well even when n is large	O(n3)Need to compute ()
O(kn2)	Slow if n is very large

---

 如果矩阵不可逆，可以计算伪逆矩阵。