2.x ESL第二章习题2.5

题目

     

描述

• $y_i=x_i^Teta+epsilon_i$
  $epsilon_isim N(0,sigma^2)$

• 已有训练集$ au$，其中$X:n imes p,y:n imes 1,epsilon:n imes 1$
  使用最小二乘得到$hat{eta}=left(X^TX ight)^{-1}X^Ty$
  $y=Xeta+epsilon$

• 需要预测点$x_0$的値$y_0$

题2.7

     

• 准备

  • $E(y_0)=E(x_0^Teta+epsilon_0)=E(x_0^Teta)+E(epsilon_0)=x_0^Teta+0$
  • $E[left(y_0-E(y_0) ight)^2]=E[left(x_0^Teta+epsilon_0-x_0^Teta ight)^2]=E[epsilon_0^2]=sigma^2$
  • $hat{y_0}=x_0^That{eta}=x_0^Tleft(X^TX ight)^{-1}X^Tleft(Xeta+epsilon ight)\ =x_0^Tleft(X^TX ight)^{-1}X^TXeta+x_0^Tleft(X^TX ight)^{-1}X^Tepsilon\ =x_0^Teta+x_0^Tleft(X^TX ight)^{-1}X^Tepsilon\ =x_0^Teta+sum_i a_iepsilon_i$ <br>其中$a_i=left[x_0^Tleft(X^TX ight)^{-1}X^T ight]_i$
  • $E(hat{y_0})=E(x_0^Teta+sum_i a_iepsilon_i)=x_0^Teta+sum_i E(a_i)E(epsilon_i)=x_0^Teta$
    这里由于$X$由某分布产生，所以$E(a_i)$不是简单常数

• 题解

  $EPE(x_0)=intintleft(y_0-hat{y_0} ight)^2p(y_0)p(hat{y_0})mathrm{d} y_0mathrm{d}hat{y_0}\ =intintleft[hat{y_0}-E(hat{y_0})+E(hat{y_0})-y_0 ight]^2p(y_0)p(hat{y_0})mathrm{d} y_0mathrm{d}hat{y_0}\ =intleft[hat{y_0}-E(hat{y_0}) ight]^2p(hat{y_0})mathrm{d}hat{y_0}+intintleft[E(hat{y_0})-y_0 ight]^2p(y_0)p(hat{y_0})mathrm{d} y_0mathrm{d}hat{y_0}+2 imes 0\ ={Var}_ au(hat{y_0})+intintleft[y_0-E(y_0)+E(y_0)-E(hat{y_0}) ight]^2p(y_0)p(hat{y_0})mathrm{d} y_0mathrm{d}hat{y_0}\ ={Var}_ au(hat{y_0})+intleft[y_0-E(y_0) ight]^2p(y_0)mathrm{d} y_0+left[E(y_0)-E(hat{y_0} ight]^2+2 imes 0\ ={Var}_ au(hat{y_0})+sigma^2+0^2$
  ${Var}_ au(hat{y_0})=Eleft[hat{y_0}-E(hat{y_0}) ight]^2\ =Eleft[x_0^Teta+sum_i a_iepsilon_i-x_0^Teta ight]^2=Eleft[sum_isum_j a_ia_jepsilon_iepsilon_j ight]\ =Eleft[ sum_i a_i^2epsilon_i^2 ight]+Eleft[sum_isum_{j:j eq i} a_ia_jepsilon_iepsilon_j ight]\ =sum_iE(a_i^2)E(epsilon_i^2)+sum_isum_{j:j eq i} E(a_ia_j)E(epsilon_i)E(epsilon_j)\ =sigma^2E(sum_i a_i^2)+0=sigma^2Eleft(x_0^Tleft(X^TX ight)^{-1}X^TXleft(X^TX ight)^{-1}x_0 ight)\ =sigma^2Eleft(x_0^Tleft(X^TX ight)^{-1}x_0 ight)$

题2.8

     

• 准备

  • 假设$E(x^{(i)})=0,i=1...p$，即每个维度的期望都为0
    $X^TX$得到$p imes p$的矩阵
    $X_{:i}$表示$X$的第$i$列，即训练集输入部分的第i个维度
    $X_{:i}^TX_{:i}=sum_j^N {x_j^{(i)}}^2=N frac{1}{N}sum_j^N (x_j^{(i)}-E(x^{(i)}))^2=Nhat{Var}(x^{(i)})$得到对角元素
    $X_{:i}^TX_{:j}=sum_t^N x_t^{(i)}x_t^{(j)} = N frac{1}{N} (x_t^{(i)}-E(x^{(i)}))(x_t^{(j)}-E(x^{(j)}))=Nhat{Cov}(x^{(i)},x^{(j)})$
    当$N o infty $，$X^TX o NCov(x)$
  • $K:p imes p,b:p imes 1$
    $trace Kbb^T=sum_i {[Kbb^T]}_{ii}=sum_i sum_j K_{ij}{[bb^T]}_{ji}=sum_i sum_j K_{ij}b_ib_j$
    $b^TKb=sum_i {b^T}_{1i}{[Kb]}_{i1}=sum_i sum_j b_iK_{ij}b_j$
    $trace Kbb^T=b^TKb$

• 题解

  $Eleft(x_0^Tleft(X^TX ight)^{-1}x_0 ight)sim Eleft(x_0^T{Cov(x)}^{-1}x_0 ight)/N\ =Eleft(trace {Cov(x)}^{-1}x_0x_0^T ight)/N\ =trace {Cov(x)}^{-1}E(x_0x_0^T)/N=trace {Cov(x)}^{-1}Cov(x)/N\ =trace I/N=p/N$
  $EPE(x_0)=(p/N+1)sigma^2$