zoukankan      html  css  js  c++  java
  • 证明最优化loss function+penalty等价于最优化带限制条件的loss function

    Equivalence of constrained and unconstrained form for lasso

    Problem 1 The unconstrained form of lasso

    [operatorname{min}_{eta}|y-X eta|_{2}^{2}+lambda|eta|_{1} ag{1} ]

    Suppose we solve Problem 1 for a given (lambda) and obtain its solution (eta_{ ext{problem1}}^*(lambda)).

    Problem 2 The constrained form of lasso

    [operatorname{min}_{eta}|Y-X eta|_{2}^{2} ]

    [s.t. {|eta|_{1} leq s} ]

    We can rewrite the constrained form into unconstrained form using Lagrangian mutiplier method.

    The unconstrained form for the problem is given by:

    [operatorname{min}_{eta,v}|Y-X eta|_{2}^{2}+vleft(|eta|_{1}-s ight) ag{2} ]

    Since the objective and the constraints are convex, so we have the pair ((eta^*,v^*)) is primal-dual optimal if and only if it is a saddle-point of the Lagrangian.

    [operatorname{min}_{eta}operatorname{max}_{v}|Y-X eta|_{2}^{2}+vleft(|eta|_{1}-s ight) =operatorname{max}_{v}operatorname{min}_{eta}|Y-X eta|_{2}^{2}+vleft(|eta|_{1}-s ight) ]

    First we solve the

    [operatorname{min}_{eta}|Y-X eta|_{2}^{2}+vleft(|eta|_{1}-s ight) ]

    The form is the same with eq(1), so we have the same solution with Problem 1, i.e. (eta_{ ext{problem2}}^*=eta_{ ext{problem1}}^*(v))

    Then we solve

    [operatorname{max}_{v}|Y-X eta^*(v)|_{2}^{2}+vleft(|eta^*(v)|_{1}-s ight) ]

    The solution is (v^*).

    Finally, we have that (eta^*_{ ext{problem2}}=eta_{ ext{problem1}}^*(v^*))

    Therefore,if we let (lambda) in Problem 1 be (v^*), the solution in Problem 1 is (eta_{ ext{problem1}}^*=eta_{ ext{problem1}}^*(lambda)=eta_{ ext{problem1}}^*(v^*)), this is the same with solution in Problem 2.

    So the two forms are equivalent.

    Equivalence of constrained and unconstrained form for Ridge Regression

    Problem 1 The unconstrained form of ridge regression

    [operatorname{min}_{eta}|y-X eta|_{2}^{2}+lambda|eta|_{2}^{2} ag{3} ]

    Suppose we solve Problem 3 using F.O.C for a given (lambda) and obtain its solution (eta^*(lambda)).

    Problem 2 The constrained form of ridge regression

    [operatorname{min}_{eta}|Y-X eta|_{2}^{2} ]

    [s.t. {|eta|_{2}^2 leq s} ]

    We can rewrite the constrained form into unconstrained form using Lagrangian mutiplier method.

    The unconstrained form for the problem is given by:

    [operatorname{min}_{eta,v}|Y-X eta|_{2}^{2}+vleft(|eta|_{2}^{2}-s ight) ag{4} ]

    The first KKT condition (stationarity) says that the gradient with respect to (eta) of the lagrangian equals to 0. Since s is independent on (eta), so solving for the derivative of eq (3) is thus equivalent to solving for the derivate of eq (4) when (lambda=v) .

    The second KKT condition (complementarity) says that

    [vleft(|eta|_{2}^{2}-s ight)=0 ]

    Let (s=|eta^*(lambda)|^2), then we can find that (v^*=lambda) and (eta^*=eta^*(lambda)) satisfy the KKT conditions for Problem 2, so they are the solution of Problem 2, which is the same as the solution in Problem 1.

    So the two forms are equivalent.

    本文为跑得飞快的凤凰花原创,如需转载,请标明出处~
  • 相关阅读:
    爬虫 比较好的视频推荐
    Scrapy爬取某装修网站部分装修效果图
    scrapy爬取阳光电影网全站资源
    爬虫练手实例
    Scrapy框架详解
    淘宝商品信息定向爬虫实例介绍
    Python基础第20天
    Python基础第19天
    Python基础第18天
    Python基础第17天
  • 原文地址:https://www.cnblogs.com/zzqingwenn/p/12668264.html
Copyright © 2011-2022 走看看