一：单变量线性回归（Linear regression with one variable）
背景：在某城市开办饭馆，我们有这样的数据集ex1data1.txt，第一列代表某个城市的人口，第二列代表在该城市开办饭馆的利润。
我们将数据集显示在可视图，可以看出跟某个线性方程有关，而此数据只有单个变量（某城市人口），故接下来我们就使用单变量线性回归拟合出一条近似满足于上数据的直线。
1，单变量的脚本ex1.m:
%% Machine Learning Online Class - Exercise 1: Linear Regression
% Instructions
% ------------
%
% This file contains code that helps you get started on the
% linear exercise. You will need to complete the following functions
% in this exericse:
%
% warmUpExercise.m
% plotData.m
% gradientDescent.m
% computeCost.m
% gradientDescentMulti.m
% computeCostMulti.m
% featureNormalize.m
% normalEqn.m
%
% For this exercise, you will not need to change any code in this file,
% or any other files other than those mentioned above.
%
% x refers to the population size in 10,000s
% y refers to the profit in $10,000s
%
%% Initialization
clear ; close all; clc
%% ==================== Part 1: Basic Function ====================
% Complete warmUpExercise.m
fprintf('Running warmUpExercise ...
');
fprintf('5x5 Identity Matrix:
');
warmUpExercise()
fprintf('Program paused. Press enter to continue.
');
pause;
%% ======================= Part 2: Plotting =======================
fprintf('Plotting Data ...
')
data = load('ex1data1.txt');
X = data(:, 1); y = data(:, 2);
m = length(y); % number of training examples
% Plot Data
% Note: You have to complete the code in plotData.m
plotData(X, y);
fprintf('Program paused. Press enter to continue.
');
pause;
%% =================== Part 3: Cost and Gradient descent ===================
X = [ones(m, 1), data(:,1)]; % Add a column of ones to x
theta = zeros(2, 1); % initialize fitting parameters
% Some gradient descent settings
iterations = 1500;
alpha = 0.01;
fprintf('
Testing the cost function ...
')
% compute and display initial cost
J = computeCost(X, y, theta);
fprintf('With theta = [0 ; 0]
Cost computed = %f
', J);
fprintf('Expected cost value (approx) 32.07
');
% further testing of the cost function
J = computeCost(X, y, [-1 ; 2]);
fprintf('
With theta = [-1 ; 2]
Cost computed = %f
', J);
fprintf('Expected cost value (approx) 54.24
');
fprintf('Program paused. Press enter to continue.
');
pause;
fprintf('
Running Gradient Descent ...
')
% run gradient descent
theta = gradientDescent(X, y, theta, alpha, iterations);
% print theta to screen
fprintf('Theta found by gradient descent:
');
fprintf('%f
', theta);
fprintf('Expected theta values (approx)
');
fprintf(' -3.6303
1.1664
');
% Plot the linear fit
hold on; % keep previous plot visible
plot(X(:,2), X*theta, '-')
legend('Training data', 'Linear regression')
hold off % don't overlay any more plots on this figure
% Predict values for population sizes of 35,000 and 70,000
predict1 = [1, 3.5] *theta;
fprintf('For population = 35,000, we predict a profit of %f
',...
predict1*10000);
predict2 = [1, 7] * theta;
fprintf('For population = 70,000, we predict a profit of %f
',...
predict2*10000);
fprintf('Program paused. Press enter to continue.
');
pause;
%% ============= Part 4: Visualizing J(theta_0, theta_1) =============
fprintf('Visualizing J(theta_0, theta_1) ...
')
% Grid over which we will calculate J
theta0_vals = linspace(-10, 10, 100);
theta1_vals = linspace(-1, 4, 100);
% initialize J_vals to a matrix of 0's
J_vals = zeros(length(theta0_vals), length(theta1_vals));
% Fill out J_vals
for i = 1:length(theta0_vals)
for j = 1:length(theta1_vals)
t = [theta0_vals(i); theta1_vals(j)];
J_vals(i,j) = computeCost(X, y, t);
end
end
% Because of the way meshgrids work in the surf command, we need to
% transpose J_vals before calling surf, or else the axes will be flipped
J_vals = J_vals';
% Surface plot
figure;
surf(theta0_vals, theta1_vals, J_vals)
xlabel(' heta_0'); ylabel(' heta_1');
% Contour plot
figure;
% Plot J_vals as 15 contours spaced logarithmically between 0.01 and 100
contour(theta0_vals, theta1_vals, J_vals, logspace(-2, 3, 20))
xlabel(' heta_0'); ylabel(' heta_1');
hold on;
plot(theta(1), theta(2), 'rx', 'MarkerSize', 10, 'LineWidth', 2);
2，单变量代价函数（cost function）:
$J( heta_0, heta_1)=J( heta)=frac{1}{2m}sum_{i=1}^{m}(h_ heta(x^{(i)})-y^{(i)})^2$
其中假设(预测)函数：$ h_{ heta}(x)= heta^T * x= heta_0*x_0+ heta_1*x_1$，其中$x_0=1$
function J = computeCost(X, y, theta)
m = length(y);
J = 0;
ans=X*theta; %X*theta计算hθ(x)
ans=(ans-y).^2; %计算平方差成本函数
J=sum(ans)/(2*m); %计算所有样本m的代价
end
3，单变量梯度下降（Gradient descent）：
$ heta_j:= heta_j- frac{alpha}{m}sum_{i=1}^{m}[(h_ heta(x^{(i)})-y^{(i)})x^{(i)}_j]$ (同时更新$large heta_j$(all j))，其中$x^{(0)}=1$
function [theta, J_history] = gradientDescent(X, y, theta, alpha, num_iters)
%GRADIENTDESCENT Performs gradient descent to learn theta
% theta = GRADIENTDESCENT(X, y, theta, alpha, num_iters) updates theta by
% taking num_iters gradient steps with learning rate alpha
% Initialize some useful values
m = length(y); % number of training examples
J_history = zeros(num_iters, 1);
for iter = 1:num_iters
% ====================== YOUR CODE HERE ======================
% Instructions: Perform a single gradient step on the parameter vector
% theta.
%
% Hint: While debugging, it can be useful to print out the values
% of the cost function (computeCost) and gradient here.
%
% ans1=theta(1)-sum((sum(X*theta,2)-y).*X(:,1))*alpha/m;
% ans2=theta(2)-sum((sum(X*theta,2)-y).*X(:,2))*alpha/m;
% theta(1)=ans1;
% theta(2)=ans2;
%梯度下降，X为(m,2)，hθ(x)-y为(m,1)，先将X转置
theta=theta-((X')*(X*theta-y)).*(alpha/m);
% ============================================================
% Save the cost J in every iteration
J_history(iter) = computeCost(X, y, theta);
end
end
梯度下降算出来theta参数值后，我们就可以预测了。
假设我们想预测城市人口为35000，在该城市开办饭馆预计能获得多少利润。
利润=（[1,3.5]*theta)*10000。(我们将数据进行了特征缩放10000)。
二：多变量线性回归（Linear regression with multiple variables）
背景：预测房价，现在有一些数据集ex1data2.txt，第一列为房子大小(平方英尺)，第二列为该房子卧室数量，第三列为该房子的价值。
我们观察该数据集的可视图后，用多变量线性回归去拟合该数据集，注意此处的数据集ex1data2.txt并没有进行特征缩放，故我们首先对该数据集进行特征缩放。
1，多变量的脚本ex1_multi.m:
%% Machine Learning Online Class
% Exercise 1: Linear regression with multiple variables
%
% Instructions
% ------------
%
% This file contains code that helps you get started on the
% linear regression exercise.
%
% You will need to complete the following functions in this
% exericse:
%
% warmUpExercise.m
% plotData.m
% gradientDescent.m
% computeCost.m
% gradientDescentMulti.m
% computeCostMulti.m
% featureNormalize.m
% normalEqn.m
%
% For this part of the exercise, you will need to change some
% parts of the code below for various experiments (e.g., changing
% learning rates).
%
%% Initialization
%% ================ Part 1: Feature Normalization ================
%% Clear and Close Figures
clear ; close all; clc
fprintf('Loading data ...
');
%% Load Data
data = load('ex1data2.txt');
X = data(:, 1:2);
y = data(:, 3);
m = length(y);
% Print out some data points
fprintf('First 10 examples from the dataset:
');
fprintf(' x = [%.0f %.0f], y = %.0f
', [X(1:10,:) y(1:10,:)]');
fprintf('Program paused. Press enter to continue.
');
pause;
% Scale features and set them to zero mean
fprintf('Normalizing Features ...
');
[X mu sigma] = featureNormalize(X);
% Add intercept term to X
X = [ones(m, 1) X]; %而外增加一列截距项为1的数据列
%% ================ Part 2: Gradient Descent ================
% ====================== YOUR CODE HERE ======================
% Instructions: We have provided you with the following starter
% code that runs gradient descent with a particular
% learning rate (alpha).
%
% Your task is to first make sure that your functions -
% computeCost and gradientDescent already work with
% this starter code and support multiple variables.
%
% After that, try running gradient descent with
% different values of alpha and see which one gives
% you the best result.
%
% Finally, you should complete the code at the end
% to predict the price of a 1650 sq-ft, 3 br house.
%
% Hint: By using the 'hold on' command, you can plot multiple
% graphs on the same figure.
%
% Hint: At prediction, make sure you do the same feature normalization.
%
fprintf('Running gradient descent ...
');
% Choose some alpha value
alpha = 0.01;
num_iters = 400;
% Init Theta and Run Gradient Descent
theta = zeros(3, 1);
[theta, J_history] = gradientDescentMulti(X, y, theta, alpha, num_iters);
##alpha2 = 0.1;
##theta2 = zeros(3, 1);
##[theta2, J2] = gradientDescentMulti(X, y, theta2, alpha2, num_iters);
##alpha3 = 0.9;
##theta3 = zeros(3, 1);
##[theta3, J3] = gradientDescentMulti(X, y, theta3, alpha3, num_iters);
% Plot the convergence graph
figure;
plot(1:numel(J_history), J_history, '-b', 'LineWidth', 2);
##plot(1:50, J_history(1:50), '-b', 'LineWidth', 2);
xlabel('Number of iterations');
ylabel('Cost J');
##hold on;
##plot(1:50,J2(1:50),'r');
##
##hold on;
##plot(1:50,J3(1:50),'k');
% Display gradient descent's result
fprintf('Theta computed from gradient descent:
');
fprintf(' %f
', theta);
fprintf('
');
% Estimate the price of a 1650 sq-ft, 3 br house
% ====================== YOUR CODE HERE ======================
% Recall that the first column of X is all-ones. Thus, it does
% not need to be normalized.
%预测房子1650平方，3个房间的价格
predict1=[1650 3];
predict1=(predict1.-mu)./sigma; %用测试数据的特征缩放的平均值与标准差的值
predict1=[ones(1,1) predict1];
fprintf(' %f
', predict1);
price = predict1 *theta; % 预测价格
% ============================================================
fprintf(['Predicted price of a 1650 sq-ft, 3 br house ' ...
'(using gradient descent):
$%f
'], price);
fprintf('Program paused. Press enter to continue.
');
pause;
%% ================ Part 3: Normal Equations ================
fprintf('Solving with normal equations...
');
% ====================== YOUR CODE HERE ======================
% Instructions: The following code computes the closed form
% solution for linear regression using the normal
% equations. You should complete the code in
% normalEqn.m
%
% After doing so, you should complete this code
% to predict the price of a 1650 sq-ft, 3 br house.
%
%% Load Data
data = csvread('ex1data2.txt');
X = data(:, 1:2);
y = data(:, 3);
m = length(y);
% Add intercept term to X
X = [ones(m, 1) X];
% Calculate the parameters from the normal equation
theta = normalEqn(X, y);
% Display normal equation's result
fprintf('Theta computed from the normal equations:
');
fprintf(' %f
', theta);
fprintf('
');
% Estimate the price of a 1650 sq-ft, 3 br house
% ====================== YOUR CODE HERE ======================
predict1=[1 1650 3];
price = predict1*theta; % 预测价格
% ============================================================
fprintf(['Predicted price of a 1650 sq-ft, 3 br house ' ...
'(using normal equations):
$%f
'], price);
2，特征缩放：X进行缩放前，不需添加截距项，缩放后添加
function [X_norm, mu, sigma] = featureNormalize(X)
%FEATURENORMALIZE Normalizes the features in X
% FEATURENORMALIZE(X) returns a normalized version of X where
% the mean value of each feature is 0 and the standard deviation
% is 1. This is often a good preprocessing step to do when
% working with learning algorithms.
% You need to set these values correctly
X_norm = X;
mu = zeros(1, size(X, 2));
sigma = zeros(1, size(X, 2));
% ====================== YOUR CODE HERE ======================
% Instructions: First, for each feature dimension, compute the mean
% of the feature and subtract it from the dataset,
% storing the mean value in mu. Next, compute the
% standard deviation of each feature and divide
% each feature by it's standard deviation, storing
% the standard deviation in sigma.
%
% Note that X is a matrix where each column is a
% feature and each row is an example. You need
% to perform the normalization separately for
% each feature.
%
% Hint: You might find the 'mean' and 'std' functions useful.
%
mu=mean(X); %计算X每列的平均值
sigma=std(X); %计算X每列的标准差
X_norm=(X.-mu)./sigma; %特征缩放
% ============================================================
end
3，多变量代价函数（cost function）:
$J( heta_0, heta_1,..., heta_n)=J( heta)=frac{1}{2m}sum_{i=1}^{m}(h_ heta(x^{(i)})-y^{(i)})^2$
其中假设(预测)函数： $h_{ heta}(x)= heta^T * x= heta_0*x_0+ heta_1*x_1+...+ heta_n*x_n$ ，其中$x_0=1$。
function J = computeCostMulti(X, y, theta) %COMPUTECOSTMULTI Compute cost for linear regression with multiple variables % J = COMPUTECOSTMULTI(X, y, theta) computes the cost of using theta as the % parameter for linear regression to fit the data points in X and y % Initialize some useful values m = length(y); % number of training examples % You need to return the following variables correctly J = 0; % ====================== YOUR CODE HERE ====================== % Instructions: Compute the cost of a particular choice of theta % You should set J to the cost. ans=X*theta-y; %计算hθ(x)-y ans=(ans')*ans; %计算(hθ(x)-y)^2 % ans=ans.^2; %或者是这样计算 J=sum(ans)./(2*m); %计算代价函数 % ========================================================================= end
4，多变量梯度下降（Gradient descent）：
$ heta_j:= heta_j- frac{alpha}{m}sum_{i=1}^{m}[(h_ heta(x^{(i)})-y^{(i)})x^{(i)}_j]$ (同时更新$large heta_j$(all j))，其中$x^{(0)}=1$
function [theta, J_history] = gradientDescentMulti(X, y, theta, alpha, num_iters)
%GRADIENTDESCENTMULTI Performs gradient descent to learn theta
% theta = GRADIENTDESCENTMULTI(x, y, theta, alpha, num_iters) updates theta by
% taking num_iters gradient steps with learning rate alpha
% Initialize some useful values
m = length(y); % number of training examples
J_history = zeros(num_iters, 1);
for iter = 1:num_iters
% ====================== YOUR CODE HERE ======================
% Instructions: Perform a single gradient step on the parameter vector
% theta.
%
% Hint: While debugging, it can be useful to print out the values
% of the cost function (computeCostMulti) and gradient here.
%
theta=theta-((X')*(X*theta-y)).*(alpha/m);
% ============================================================
% Save the cost J in every iteration
%每一次迭代计算代价函数保存，以后可作为可视化图
J_history(iter) = computeCostMulti(X, y, theta);
end
end
梯度下降算出来theta参数值后，我们就可以预测了。
假设我们想预测某所房子房屋面积为1650平方英尺，3个房间，该所房子的价值大约是多少？
predict1=([1650 3]-mu)./sigma，先进行缩放，使用训练集的平均值与标准差。
predict1=[1 predict1] ，增加一个截距项1。
价值=predict1*theta。
我们可以选择不同的学习速率alpha，试试那个学习速率收敛得更快，一般我们每次选择为前一个选择的3倍，例如：0.01,0.03,0.1,0.3,0.9......
三：正规方程(Normal Equations)求解线性回归：
用正规方程求解线性回归方程更便捷，它不需要一直迭代，只需求一个式子就行了：
$ heta=(X^TX)^{-1}X^Ty$ $X$为该数据集的变量(已添加了截距项)，$y$为该数据集的结果。
最后预测值就为：predict1*theta。
总结：
梯度下降与正规方程的比较：
1，需要选择$alpha$ 1，不需要选择$alpha$
2，需要迭代次数 2，不需要迭代
3，当n很大时，也能很好的运行 3，只需计算一次$(X^TX)^{-1}$
4，当n很大时，会很慢(因为要逆运算),一般n不大于10000
我的便签：做个有情怀的程序员。