【482】Keras 实现 LSTM & BiLSTM

参考：Keras 实现 LSTM

参考：Keras-递归层Recurrent官方说明

参考：GitHub - Keras LSTM

参考：GitHub - Keras BiLSTM

---

　　LSTM 是优秀的循环神经网络 (RNN) 结构，而 LSTM 在结构上也比较复杂，对 RNN 和 LSTM 还稍有疑问的朋友可以参考：Recurrent Neural Networks vs LSTM【参考李宏毅老师的讲课PPT内容】

　　这里我们将要使用 Keras 搭建 LSTM.Keras 封装了一些优秀的深度学习框架的底层实现，使用起来相当简洁，甚至不需要深度学习的理论知识，你都可以轻松快速的搭建你的深度学习网络，强烈推荐给刚入门深度学习的同学使用，当然我也是还没入门的那个。Keras：https://keras.io/，keras的backend有，theano，TensorFlow、CNTk，这里我使用的是 TensorFlow。

　　下面我们就开始搭建 LSTM & BiLSTM，实现 mnist 数据的分类。

一、加载包和定义参数

　　mnist 的 image 是 28*28 的 shape，我们定义 LSTM 的 input 为 (28,)，将 image 一行一行地输入到 LSTM 的 cell 中，这样 time_step 就是 28，表示一个 image 有 28 行，LSTM 的 output 是 30 个。

from tensorflow import keras
import mnist
from keras.layers import Dense, LSTM, Bidirectional
from keras.utils import to_categorical
from keras.models import Sequential

# parameters for LSTM
nb_lstm_outputs = 30    # 输出神经元个数
nb_time_steps = 28    # 时间序列的长度
nb_input_vectors = 28    # 每个输入序列的向量维度

二、数据预处理

　　特别注意 label 要使用 one_hot encoding，x_train 的 shape 为 (60000, 28,28）

# data preprocessing
x_train = mnist.train_images()
y_train = mnist.train_labels()
x_test = mnist.test_images()
y_test = mnist.test_labels()

# Nomalize the images
x_train = (x_train / 255) - 0.5
x_test = (x_test / 255) - 0.5

# one_hot encoding
y_train = to_categorical(y_train, num_classes=10)
y_test = to_categorical(y_test, num_classes=10)

三、搭建模型 (LSTM, BiLSTM)

　　keras 搭建模型相当简单，只需要在 Sequential 容器中不断 add 新的 layer 就可以了。

# building model
model = Sequential()
model.add(LSTM(units=nb_lstm_outputs, input_shape=(nb_time_steps, nb_input_vectors)))
model.add(Dense(10, activation='softmax'))

　　BiLSTM 模型搭建如下：具体实现方法差别不大

# building model
model = Sequential()

model.add(
    Bidirectional(
        LSTM(
            units=nb_lstm_outputs, 
            return_sequences=True
        ), 
        input_shape=(nb_time_steps, nb_input_vectors)
    )
)

model.add(
    Bidirectional(
        LSTM(units=nb_lstm_outputs)
    )
)

model.add(
    Dense(
        10, 
        activation='softmax'
    )
)

四、compile

　　模型 compile，指定 loss function，optimizer，metrics

# compile:loss, optimizer, metrics
model.compile(
    loss='categorical_crossentropy',
    optimizer='adam',
    metrics=['accuracy']
)

五、summary

　　可以使用 model.summary() 来查看你的神经网络的架构和参数量等信息。

model.summary()

output:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm_1 (LSTM)                (None, 30)                7080      
_________________________________________________________________
dense_1 (Dense)              (None, 10)                310       
=================================================================
Total params: 7,390
Trainable params: 7,390
Non-trainable params: 0
_________________________________________________________________

　　BiLSTM 结果如下：多了一层 layer

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
bidirectional_1 (Bidirection (None, 28, 60)            14160     
_________________________________________________________________
bidirectional_2 (Bidirection (None, 60)                21840     
_________________________________________________________________
dense_2 (Dense)              (None, 10)                610       
=================================================================
Total params: 36,610
Trainable params: 36,610
Non-trainable params: 0
_________________________________________________________________

六、train

　　模型训练，需要指定，epochs 训练的轮次数，batch_size。

model.fit(
    x_train,
    y_train,
    epochs=20,
    batch_size=128,
    verbose=1
)

output:

Epoch 1/20
60000/60000 [==============================] - 11s 184us/step - loss: 0.9702 - acc: 0.6919
Epoch 2/20
60000/60000 [==============================] - 9s 152us/step - loss: 0.3681 - acc: 0.8921
Epoch 3/20
60000/60000 [==============================] - 9s 143us/step - loss: 0.2505 - acc: 0.9263
Epoch 4/20
60000/60000 [==============================] - 9s 147us/step - loss: 0.1985 - acc: 0.9411
Epoch 5/20
60000/60000 [==============================] - 9s 156us/step - loss: 0.1673 - acc: 0.9508
Epoch 6/20
60000/60000 [==============================] - 10s 163us/step - loss: 0.1473 - acc: 0.9563
Epoch 7/20
60000/60000 [==============================] - 10s 162us/step - loss: 0.1311 - acc: 0.9605
Epoch 8/20
60000/60000 [==============================] - 10s 162us/step - loss: 0.1176 - acc: 0.9650
Epoch 9/20
60000/60000 [==============================] - 10s 167us/step - loss: 0.1054 - acc: 0.9688
Epoch 10/20
60000/60000 [==============================] - 10s 165us/step - loss: 0.0991 - acc: 0.9702
Epoch 11/20
60000/60000 [==============================] - 10s 164us/step - loss: 0.0899 - acc: 0.9730
Epoch 12/20
60000/60000 [==============================] - 10s 169us/step - loss: 0.0857 - acc: 0.9741
Epoch 13/20
60000/60000 [==============================] - 10s 166us/step - loss: 0.0781 - acc: 0.9758
Epoch 14/20
60000/60000 [==============================] - 10s 167us/step - loss: 0.0740 - acc: 0.9776
Epoch 15/20
60000/60000 [==============================] - 10s 172us/step - loss: 0.0697 - acc: 0.9786
Epoch 16/20
60000/60000 [==============================] - 10s 171us/step - loss: 0.0678 - acc: 0.9795
Epoch 17/20
60000/60000 [==============================] - 10s 170us/step - loss: 0.0639 - acc: 0.9798
Epoch 18/20
60000/60000 [==============================] - 10s 169us/step - loss: 0.0589 - acc: 0.9817
Epoch 19/20
60000/60000 [==============================] - 10s 172us/step - loss: 0.0597 - acc: 0.9817
Epoch 20/20
60000/60000 [==============================] - 10s 168us/step - loss: 0.0558 - acc: 0.9825

　　BiLSTM 结果如下：结果更好

Epoch 1/20
60000/60000 [==============================] - 46s 767us/step - loss: 0.6845 - acc: 0.7782
Epoch 2/20
60000/60000 [==============================] - 48s 799us/step - loss: 0.1843 - acc: 0.9435
Epoch 3/20
60000/60000 [==============================] - 45s 751us/step - loss: 0.1241 - acc: 0.9627
Epoch 4/20
60000/60000 [==============================] - 45s 747us/step - loss: 0.0956 - acc: 0.9712
Epoch 5/20
60000/60000 [==============================] - 46s 766us/step - loss: 0.0806 - acc: 0.9754
Epoch 6/20
60000/60000 [==============================] - 46s 771us/step - loss: 0.0667 - acc: 0.9793
Epoch 7/20
60000/60000 [==============================] - 45s 754us/step - loss: 0.0584 - acc: 0.9820
Epoch 8/20
60000/60000 [==============================] - 44s 741us/step - loss: 0.0513 - acc: 0.9835
Epoch 9/20
60000/60000 [==============================] - 45s 742us/step - loss: 0.0445 - acc: 0.9863
Epoch 10/20
60000/60000 [==============================] - 46s 767us/step - loss: 0.0419 - acc: 0.9874
Epoch 11/20
60000/60000 [==============================] - 45s 755us/step - loss: 0.0378 - acc: 0.9885
Epoch 12/20
60000/60000 [==============================] - 46s 758us/step - loss: 0.0332 - acc: 0.9894
Epoch 13/20
60000/60000 [==============================] - 45s 750us/step - loss: 0.0318 - acc: 0.9894
Epoch 14/20
60000/60000 [==============================] - 45s 756us/step - loss: 0.0279 - acc: 0.9911
Epoch 15/20
60000/60000 [==============================] - 45s 745us/step - loss: 0.0262 - acc: 0.9917
Epoch 16/20
60000/60000 [==============================] - 45s 758us/step - loss: 0.0258 - acc: 0.9916
Epoch 17/20
60000/60000 [==============================] - 47s 791us/step - loss: 0.0226 - acc: 0.9923
Epoch 18/20
60000/60000 [==============================] - 47s 791us/step - loss: 0.0223 - acc: 0.9930
Epoch 19/20
60000/60000 [==============================] - 46s 773us/step - loss: 0.0179 - acc: 0.9943
Epoch 20/20
60000/60000 [==============================] - 45s 747us/step - loss: 0.0199 - acc: 0.9935

七、evaluate

　　通过 model.evaluate() 来实现。

score = model.evaluate(x_test, y_test, batch_size=128, verbose=1)
print(score)

output:

10000/10000 [==============================] - 0s 49us/step
[0.06827456439994276, 0.9802]

　　BiLSTM 结果：更好

10000/10000 [==============================] - 2s 250us/step
[0.055307343754824254, 0.9838]