zoukankan      html  css  js  c++  java
  • Deep Learning 学习随记(四)自学习和非监督特征学习

    接着看讲义,接下来这章应该是Self-Taught Learning and Unsupervised Feature Learning。 

    含义:

    从字面上不难理解其意思。这里的self-taught learning指的是用非监督的方法提取特征,然后用监督方法进行分类。比如用稀疏自编码+softmax regression。

    对于非监督特征学习,有两种类型,一类是self-taught learning,一类是semi-supervised learning。看他们的定义不如看讲义中给出的那个简单的例子:

    假定有一个计算机视觉方面的任务,目标是区分汽车和摩托车图像;也即训练样本里面要么是汽车的图像,要么是摩托车的图像。哪里获取大量的无类标数据呢?最简单的方式可能是到互联网上下载一些随机的图像数据集,这这些数据上训练出一个稀疏自编码神经网络,从中得到有用的特征。这个例子里,无类标数据完全来自于一个和带类标数据不同的分布(无类标数据集中,或许其中一些图像包含汽车或者摩托车,但是不是所有的图像都如此)。这种情形被称为自学习。

    相反,如果有大量的无类标图像数据,要么是汽车图像,要么是摩托车图像,仅仅是缺失了类标(没有标注每张图片到底是汽车还是摩托车)。也可以用这些无类标数据来学习特征。这种方式,即要求无类标样本和带类标样本服从相同的分布,有时候被称为半监督学习。在实践中,常常无法找到满足这种要求的无类标数据(到哪里找到一个每张图像不是汽车就是摩托车,只是丢失了类标的图像数据库?)因此,自学习被广泛的应用于从无类标数据集中学习特征。

    练习:

    下面是讲义中的练习,要解决的还是MNIST手写库的识别问题,主要过程就是稀疏自编码提取特征然后用softmax regression分类。

    一开始用一台32位的机器跑,出现内存不够的情况,后来换了台64位的机器才好。主要代码如下:

    stlExercise.m:

    %% CS294A/CS294W Self-taught Learning Exercise
    
    %  Instructions
    %  ------------
    % 
    %  This file contains code that helps you get started on the
    %  self-taught learning. You will need to complete code in feedForwardAutoencoder.m
    %  You will also need to have implemented sparseAutoencoderCost.m and 
    %  softmaxCost.m from previous exercises.
    %
    %% ======================================================================
    %  STEP 0: Here we provide the relevant parameters values that will
    %  allow your sparse autoencoder to get good filters; you do not need to 
    %  change the parameters below.
    
    inputSize  = 28 * 28;
    numLabels  = 5;
    hiddenSize = 200;
    sparsityParam = 0.1; % desired average activation of the hidden units.
                         % (This was denoted by the Greek alphabet rho, which looks like a lower-case "p",
                         %  in the lecture notes). 
    lambda = 3e-3;       % weight decay parameter       
    beta = 3;            % weight of sparsity penalty term   
    maxIter = 400;
    
    %% ======================================================================
    %  STEP 1: Load data from the MNIST database
    %
    %  This loads our training and test data from the MNIST database files.
    %  We have sorted the data for you in this so that you will not have to
    %  change it.
    
    % Load MNIST database files
    mnistData   = loadMNISTImages('mnist/train-images-idx3-ubyte');
    mnistLabels = loadMNISTLabels('mnist/train-labels-idx1-ubyte');
    
    % Set Unlabeled Set (All Images)
    
    % Simulate a Labeled and Unlabeled set
    labeledSet   = find(mnistLabels >= 0 & mnistLabels <= 4);
    unlabeledSet = find(mnistLabels >= 5);
    
    numTrain = round(numel(labeledSet)/2);
    trainSet = labeledSet(1:numTrain);
    testSet  = labeledSet(numTrain+1:end);
    
    unlabeledData = mnistData(:, unlabeledSet);
    
    trainData   = mnistData(:, trainSet);
    trainLabels = mnistLabels(trainSet)' + 1; % Shift Labels to the Range 1-5
    
    testData   = mnistData(:, testSet);
    testLabels = mnistLabels(testSet)' + 1;   % Shift Labels to the Range 1-5
    
    % Output Some Statistics
    fprintf('# examples in unlabeled set: %d
    ', size(unlabeledData, 2));
    fprintf('# examples in supervised training set: %d
    
    ', size(trainData, 2));
    fprintf('# examples in supervised testing set: %d
    
    ', size(testData, 2));
    
    %% ======================================================================
    %  STEP 2: Train the sparse autoencoder
    %  This trains the sparse autoencoder on the unlabeled training
    %  images. 
    
    %  Randomly initialize the parameters
    theta = initializeParameters(hiddenSize, inputSize);
    
    %% ----------------- YOUR CODE HERE ----------------------
    %  Find opttheta by running the sparse autoencoder on
    %  unlabeledTrainingImages
    
    opttheta = theta; 
    %  Use minFunc to minimize the function
    addpath minFunc/
    options.Method = 'lbfgs'; % Here, we use L-BFGS to optimize our cost
                              % function. Generally, for minFunc to work, you
                              % need a function pointer with two outputs: the
                              % function value and the gradient. In our problem,
                              % sparseAutoencoderCost.m satisfies this.
    options.maxIter = 400;      % Maximum number of iterations of L-BFGS to run 
    options.display = 'on';
    
    
    [opttheta, cost] = minFunc( @(p) sparseAutoencoderCost(p, ...
                                        inputSize, hiddenSize, ...
                                        lambda, sparsityParam, ...
                                        beta, unlabeledData), ...
                                    theta, options);
    
    %% -----------------------------------------------------
                              
    % Visualize weights
    W1 = reshape(opttheta(1:hiddenSize * inputSize), hiddenSize, inputSize);
    display_network(W1');
    
    %======================================================================
    %% STEP 3: Extract Features from the Supervised Dataset
    %  
    %  You need to complete the code in feedForwardAutoencoder.m so that the 
    %  following command will extract features from the data.
    
    trainFeatures = feedForwardAutoencoder(opttheta, hiddenSize, inputSize, ...
                                           trainData);
    
    testFeatures = feedForwardAutoencoder(opttheta, hiddenSize, inputSize, ...
                                           testData);
    
    %======================================================================
    %% STEP 4: Train the softmax classifier
    
    softmaxModel = struct;  
    %% ----------------- YOUR CODE HERE ----------------------
    %  Use softmaxTrain.m from the previous exercise to train a multi-class
    %  classifier. 
    
    %  Use lambda = 1e-4 for the weight regularization for softmax
    
    % You need to compute softmaxModel using softmaxTrain on trainFeatures and
    % trainLabels
    options.maxIter = 100;
    softmax_lambda = 1e-4;
    inputSize = 200;              %features的维度与data的维度不一样了
    softmaxModel = softmaxTrain(inputSize, numLabels, softmax_lambda, ...
                                trainFeatures, trainLabels, options);
    
    
    %% -----------------------------------------------------
    
    
    %%======================================================================
    %% STEP 5: Testing 
    
    %% ----------------- YOUR CODE HERE ----------------------
    % Compute Predictions on the test set (testFeatures) using softmaxPredict
    % and softmaxModel
    [pred] = softmaxPredict(softmaxModel, testFeatures);
    acc = mean(testLabels(:) == pred(:));
    fprintf('Accuracy: %0.3f%%
    ', acc * 100);
    
    %% -----------------------------------------------------
    
    % Classification Score
    fprintf('Test Accuracy: %f%%
    ', 100*mean(pred(:) == testLabels(:)));
    
    % (note that we shift the labels by 1, so that digit 0 now corresponds to
    %  label 1)
    %
    % Accuracy is the proportion of correctly classified images
    % The results for our implementation was:
    %
    % Accuracy: 98.3%
    %
    % 

     feedForwardAutoencoder.m:

    function [activation] = feedForwardAutoencoder(theta, hiddenSize, visibleSize, data)
    
    % theta: trained weights from the autoencoder
    % visibleSize: the number of input units (probably 64) 
    % hiddenSize: the number of hidden units (probably 25) 
    % data: Our matrix containing the training data as columns.  So, data(:,i) is the i-th training example. 
      
    % We first convert theta to the (W1, W2, b1, b2) matrix/vector format, so that this 
    % follows the notation convention of the lecture notes. 
    
    W1 = reshape(theta(1:hiddenSize*visibleSize), hiddenSize, visibleSize);
    b1 = theta(2*hiddenSize*visibleSize+1:2*hiddenSize*visibleSize+hiddenSize);
    
    %% ---------- YOUR CODE HERE --------------------------------------
    %  Instructions: Compute the activation of the hidden layer for the Sparse Autoencoder.
    activation = W1*data+repmat(b1,[1,size(data,2)]);
    activation = sigmoid(activation);
    
    %-------------------------------------------------------------------
    
    end
    
    %-------------------------------------------------------------------
    % Here's an implementation of the sigmoid function, which you may find useful
    % in your computation of the costs and the gradients.  This inputs a (row or
    % column) vector (say (z1, z2, z3)) and returns (f(z1), f(z2), f(z3)). 
    
    function sigm = sigmoid(x)
        sigm = 1 ./ (1 + exp(-x));
    end

     实验结果如下:

    最终的正确率:

    讲义和代码中提到正确率在98.3%,基本差不多。

  • 相关阅读:
    PGsql(PostgreSQL)的本地连接和远程连接的问题
    CreateFolder文件夹操作【Create】
    给Windows Phone Application 换开发环境!
    C#函数,PadLeft(),填充指定数量的空格。
    sql中格式化字符串或时间,遇到多少,写多少,持续记录。
    C# 将字符串转换成GB2312很蛋疼的一个Class
    asp.net获取客户端IP Class
    MessageBox HelperClass
    Cookie帮助类
    今何していますか
  • 原文地址:https://www.cnblogs.com/bzjia-blog/p/3373452.html
Copyright © 2011-2022 走看看