zoukankan      html  css  js  c++  java
  • bert fine tuning方法

    1、使用很小的学习率进行学习,且:

    for l in bert_model.layers:

      l.trainable = True

    2、由于bert模型巨大,我们每次训练只能取batch=4进行训练,而训练4个epoch之后,可以freeze bert模型,单独训练softmax

    for l in bert_model.layers:

      l.trainable = False

    __________________________________________________________________________________________________
    Layer (type) Output Shape Param # Connected to
    ==================================================================================================
    input_1 (InputLayer) (None, None) 0
    __________________________________________________________________________________________________
    input_2 (InputLayer) (None, None) 0
    __________________________________________________________________________________________________
    model_2 (Model) multiple 101677056 input_1[0][0]
    input_2[0][0]
    __________________________________________________________________________________________________
    lambda_1 (Lambda) (None, 768) 0 model_2[1][0]
    __________________________________________________________________________________________________
    dense_1 (Dense) (None, 50) 38450 lambda_1[0][0]
    ==================================================================================================
    Total params: 101,715,506
    Trainable params: 38,450
    Non-trainable params: 101,677,056

    调整学习率,单独训练几个epoch

    尤其是我们有一个类是其他,这种类别不确定的分类问题的时候,单独训练softmax有很大的帮助

  • 相关阅读:
    003_硬件基础电路_LM2596
    002_硬件基础电路_7805电路
    007_软件安装之_串口屏软件
    006_软件安装之_Proteus 8.6 SP2 Professional
    jquery最快速入门文档
    CSS的属性与选择器
    mysql的使用
    python协程的使用
    python线程与进程
    python爬虫学习第六章
  • 原文地址:https://www.cnblogs.com/yjybupt/p/11881689.html
Copyright © 2011-2022 走看看