zoukankan      html  css  js  c++  java
  • bert fine tuning方法

    1、使用很小的学习率进行学习,且:

    for l in bert_model.layers:

      l.trainable = True

    2、由于bert模型巨大,我们每次训练只能取batch=4进行训练,而训练4个epoch之后,可以freeze bert模型,单独训练softmax

    for l in bert_model.layers:

      l.trainable = False

    __________________________________________________________________________________________________
    Layer (type) Output Shape Param # Connected to
    ==================================================================================================
    input_1 (InputLayer) (None, None) 0
    __________________________________________________________________________________________________
    input_2 (InputLayer) (None, None) 0
    __________________________________________________________________________________________________
    model_2 (Model) multiple 101677056 input_1[0][0]
    input_2[0][0]
    __________________________________________________________________________________________________
    lambda_1 (Lambda) (None, 768) 0 model_2[1][0]
    __________________________________________________________________________________________________
    dense_1 (Dense) (None, 50) 38450 lambda_1[0][0]
    ==================================================================================================
    Total params: 101,715,506
    Trainable params: 38,450
    Non-trainable params: 101,677,056

    调整学习率,单独训练几个epoch

    尤其是我们有一个类是其他,这种类别不确定的分类问题的时候,单独训练softmax有很大的帮助

  • 相关阅读:
    10 期末大作业
    09 spark连接mysql数据库
    08 学生课程分数的Spark SQL分析
    07 从RDD创建DataFrame
    06 Spark SQL 及其DataFrame的基本操作
    05 RDD编程
    05 RDD练习:词频统计
    04 RDD编程练习
    Spark RDD编程
    Spark架构与运行流程
  • 原文地址:https://www.cnblogs.com/yjybupt/p/11881689.html
Copyright © 2011-2022 走看看