深度学习笔记

Python中有很多软件包可用于构建深度神经网络。

最直观的一个是scikit-neuralnetwork软件包，sknn.mlp模块即来于此

---

正则化可以提高神经网路的泛化，避免过度拟合。

对代价函数要进行改写，添加正则项。

添加衰减参数来惩罚较大的权重来实现

回归例子

from sklearn import datasets

boston = datasets.load_boston()
x, y = boston.data, boston.target
print(x, y)
from sklearn import preprocessing

x_MinMax = preprocessing.MinMaxScaler()
y_MinMax = preprocessing.MinMaxScaler()
import numpy as np

y = np.array(y).reshape((len(y), 1))
x = x_MinMax.fit_transform(x)
y = y_MinMax.fit_transform(y)
x.mean(axis=0)  # x的均值置为0
print(y_MinMax.scale_)

import random
from sklearn.model_selection import train_test_split

np.random.seed(2016)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
print(len(x_train))

from sknn.mlp import Regressor, Layer

# ====================================================================
# ====================================================================
fit1 = Regressor(
    layers=[
        Layer('Sigmoid', units=6),
        Layer('Sigmoid', units=14),
        Layer('Linear')],
    learning_rate=0.02,  # 学习速率设置，要适中
    random_state=2016,
    n_iter=100)

print('fitting model right now')
fit1.fit(x_train, y_train)

pred1_train = fit1.predict(x_train)
from sklearn.metrics import mean_squared_error

mse_1 = mean_squared_error(pred1_train, y_train)

print('Sigmoid train error: ', mse_1)

# ====================================================================
# ====================================================================
# 换成ReLU激活函数

fit2 = Regressor(
    layers=[
        Layer('Rectifier', units=6),
        Layer('Rectifier', units=14),
        Layer('Linear')],
    learning_rate=0.02,  # 学习速率设置，要适中
    random_state=2016,
    n_iter=100)

print('fitting model right now')
fit2.fit(x_train, y_train)

pred2_train = fit2.predict(x_train)
from sklearn.metrics import mean_squared_error

mse_2 = mean_squared_error(pred2_train, y_train)

print('ReLU train error: ', mse_2)

# ====================================================================
# ====================================================================
# 换成ReLU激活函数,添加正则化，减少过度拟合

fit3 = Regressor(
    layers=[
        Layer('Rectifier', units=6),
        Layer('Rectifier', units=14),
        Layer('Linear')],
    learning_rate=0.02,  # 学习速率设置，要适中
    random_state=2016,
    regularize='L2',
    weight_decay=0.001,
    n_iter=100)

print('fitting model right now')
fit3.fit(x_train, y_train)

pred3_train = fit3.predict(x_train)
from sklearn.metrics import mean_squared_error

mse_3 = mean_squared_error(pred3_train, y_train)

print('train error: ', mse_3)

# 查看在测试样本上的表现
pred3_test = fit3.predict(x_test)
mse3_test = mean_squared_error(pred3_test, y_test)
print('test error: ', mse3_test)
for i in range(len(y_test)):
    print(y_test[i], pred3_test[i])

plt.scatter(y_test, pred3_test)

x = np.linspace(0, 1)
y = x
plt.plot(x, y, color='red', label='ideal case')
plt.xlabel("observed value")
plt.ylabel("predicted value")
# plt.title("PyPlot First Example")
# plt.ylim(-1.2,1.2)
plt.legend()

plt.show()

RESULT:

fitting model right now
Sigmoid train error: 0.0110050767582
fitting model right now
ReLU train error: 0.00583105752773
fitting model right now
train error: 0.00738282398554
test error: 0.00919026590683
[ 0.39333333] [ 0.31815418]
[ 0.26222222] [ 0.29489496]
[ 0.41111111] [ 0.48599465]
[ 0.28] [ 0.24576025]
[ 0.38666667] [ 0.32436206]
[ 0.21777778] [ 0.2463605]
[ 0.5] [ 0.35651292]
[ 0.38666667] [ 0.39062989]
[ 0.36] [ 0.30109043]

.......

测试集上性能的评估：

打印出相关性和R平方

from scipy.stats.stats import pearsonr

correl = pearsonr(pred3_test, y_test)
print('Test correlation is ', correl[0])
print('Test R^2 is ', correl[0] * correl[0])

Test correlation is [ 0.9073899]
Test R^2 is [ 0.82335643]

 

手写字母的分类

rmsdrop算法：对每个更新向量分量使用不同的学习速率

算法	正确率