zoukankan      html  css  js  c++  java
  • 关于subGradent descent和Proximal gradient descent的迭代速度

    clc;clear;
    D=1000;N=10000;thre=10e-8;zeroRatio=0.6;
    X = randn(N,D);
    r=rand(1,D);
    r=sign(1-2*r).*(2+2*r);
    perm=randperm(D);r(perm(1:floor(D*zeroRatio)))=0;
    Y = X*r' + randn(N,1)*.1; % small added noise
    lamda=1;stepsize=10e-5;
    %%% y=x*beta'
    %%% Loss=0.5*(y-x*beta')_2++lamda|beta|
    
    
    %%%% GD
    %%% al_y/al_beta=sigma(x_i*(x_i*beta'-y_i)+fabs(lamda))
    beta=zeros(size(r));
    
    pre_error=inf;new_error=0;
    count=1;tic;
    while abs(pre_error-new_error)>thre
        pre_error=new_error;
        tmp=0;
        for j=1:length(Y)
            tmp=tmp+X(j,:)*(X(j,:)*beta'-Y(j,:));
        end
        beta=beta-stepsize*(tmp+lamda);    
        new_error=lamda*norm(beta,1);
        for j=1:length(Y)
            new_error=new_error+(Y(j,:)-X(j,:)*beta')*(Y(j,:)-X(j,:)*beta');
        end 
        fprintf('%d..%f
    ',count,new_error);
        count=count+1;
    end
    toc;
    
    % %%%% Proximal GD
    % Loss=0.5*(y-x*beta')_2++lamda|beta|=g(x)+h(x)
    % 左边可导 x_{t+1}=x_{t}-stepsize*sigma(x_i*(x_i*beta'-y_i)
    % X_{t+1}=prox_{l1-norm ball}(x_{t+1})=
    
    disp('pgd')
    beta_pgd=zeros(size(r));
    pre_error=inf;new_error=0;count=1;tic;
    while abs(pre_error-new_error)>thre
        pre_error=new_error;
        tmp=0;
        for j=1:length(Y)
            tmp=tmp+X(j,:)*(X(j,:)*beta_pgd'-Y(j,:));
        end
        newbeta=beta_pgd-stepsize*(tmp+lamda); add=stepsize*lamda;
        pidx=newbeta>add;beta_pgd(pidx)=newbeta(pidx)-add;
        zeroidx=newbeta<abs(add);beta_pgd(zeroidx)=0;
        nidx=newbeta+add<0;beta_pgd(nidx)=newbeta(nidx)+add;
        
        new_error=lamda*norm(beta_pgd,1);
        for j=1:length(Y)
            new_error=new_error+(Y(j,:)-X(j,:)*beta_pgd')*(Y(j,:)-X(j,:)*beta_pgd');
        end 
         fprintf('%d..%f
    ',count,new_error);
         count=count+1;
    end
    toc;
    

      

    PGD的代码说明见下图

    PGD主要是projection那一步有解析解,速度快

    subGradent收敛速度O(1/sqrt(T))

    速度提升不明显可能是因为步长的原因。。。

  • 相关阅读:
    转char varchar nvarchar区别
    NHibernate和Spring.Net框架介绍(一)
    ASP.NET面试题(一)
    存储过程编写经验和优化措施
    软件工程师不可不知的10个概念
    优化数据库前问自己的10个问题
    ZOJ 1610 Count the Colors (线段树)
    POJ 3667 Hotel (线段树)
    HDU Best Reward (扩展KMP)
    POJ 3277 City Horizon (线段树)
  • 原文地址:https://www.cnblogs.com/sylar120/p/4379378.html
Copyright © 2011-2022 走看看