zoukankan      html  css  js  c++  java
  • day-18 滑动平均模型测试样例

            为了使训练模型在测试数据上有更好的效果,可以引入一种新的方法:滑动平均模型。通过维护一个影子变量,来代替最终训练参数,进行训练模型的验证。

          在tensorflow中提供了ExponentialMovingAverage来实行滑动平均模型,模型会维护一个影子变量,其计算公式为:

    shadow_variable = decay * shadow_variable + (1 - decay) * variable

            当训练模型时,维护训练参数的滑动平均值是有好处的。相比较最终训练值,验证时使用滑动平均值有时能产生更好的结果。
    apply()函数方法会添加一个影子拷贝到训练变量中,然后在他们影子副本上维护训练参数的滑动平均值计算操作。这个操作通常在一轮训练完之后进行。
    average()和average_name()函数提供了访问影子变量和他们名字的方法。这在构建一个评估模型或者从checkpoint文件中重载模型时非常有用。在验证时,可以帮助使用滑动平均值替换最后训练值。要使用这个模型,需要有3个步骤:

           1、  创建一个滑动平均模型对象

    step = tf.Variable(initial_value=0,dtype=tf.float32,trainable=False)
    ema = tf.train.ExponentialMovingAverage(decay=0.99,num_updates=step)

            decay就是前面公式里面的衰减因此,合理的decay值可以是接近1.0,例如0.999,0.9999等多个9中变换。num_updates为一个可选的参数,decay值由如下公式决定:

    min(decay, (1 + num_updates) / (10 + num_updates))。目的是使影子变量在刚开始训练的时候,更新的更快。 因此num_updates通常可以传入一个递增的训练步数变量。

           2、  加入训练参数列表到模型中进行维护

           新建两个训练参数,并将其加入滑动平均模型对象中维护,apply()函数接受一个参数列表。

    var0 = tf.Variable(initial_value=0,dtype=tf.float32,trainable=False)
    var1 = tf.Variable(initial_value=0,dtype=tf.float32,trainable=False)
    maintain_averages_op = ema.apply([var0,var1])

           3、  训练完成以后,更新滑动平均模型中各个影子变量的值

    sess.run(maintain_averages_op)
    print(sess.run([var0,ema.average(var0),var1,ema.average(var1)]))  # 输出[10,4.555,10,9.01]

             完整的滑动平均模型测试样例如下:

    # 导入tensorflow库
    import tensorflow as tf
    
    # 创建一个滑动平均模型对象
    step = tf.Variable(initial_value=0,dtype=tf.float32,trainable=False)
    ema = tf.train.ExponentialMovingAverage(decay=0.99,num_updates=step)
    
    # 创建两个训练参数,并将其加入滑动平均模型对象中,对象会为这两个训练参数创建两个影子变量
    # 影子变量shadow_variable = decay * shadow_variable + (1 - decay) * variable
    # 如果滑动平均模型对象创建时,指定了num_updates,则decay = min{decay,(1 + num_updates)/(10 + num_updates)}
    var0 = tf.Variable(initial_value=0,dtype=tf.float32,trainable=False)
    var1 = tf.Variable(initial_value=0,dtype=tf.float32,trainable=False)
    maintain_averages_op = ema.apply([var0,var1])
    
    # 测试更新影子变量值
    with tf.Session() as sess:
        init = tf.global_variables_initializer()
        sess.run(init)
    
        # 第一次初始滑动平均
        sess.run(maintain_averages_op)
        # decay = min(0.99,0.1) = 0.1
        # 初始时:
        # shadow_variable_var0 = var0 = 0
        # shadow_variable_var1 = var1 = 0
        print(sess.run([var0,ema.average(var0),var1,ema.average(var1)]))
    
        # 第二次更新滑动平均
        sess.run(tf.assign(var0,5.0))
        sess.run(tf.assign(var1, 10.0))
        # decay = min(0.99,(1+0)/(10+0)) = 0.1
        # shadow_variable_var0 = decay * shadow_variable + (1 - decay) * variable = 0.1*0 + (1-0.1)*5 = 4.5
        # shadow_variable_var1 = 9.0
        sess.run(maintain_averages_op)
        print(sess.run([var0,ema.average(var0),var1,ema.average(var1)]))  # 输出[5.0,4.5,10,9.0]
    
        # 第三次更新滑动平均
        sess.run(tf.assign(step,10000))
        sess.run(tf.assign(var0,10))
        # decay = min(0.99,(1+10000)/(10+10000)) = 0.99
        # shadow_variable_var0 = decay * shadow_variable + (1 - decay) * variable = 0.99*4.5 + (1-0.99)*10 = 4.555
        # shadow_variable_var1 = 0.99*9.0+(1-0.99)*10 = 9.01
        sess.run(maintain_averages_op)
        print(sess.run([var0,ema.average(var0),var1,ema.average(var1)]))  # 输出[10,4.555,10,9.01]
    
        # 第四次更新滑动平均
        # decay = min(0.99,(1+10000)/(10+10000)) = 0.99
        # shadow_variable_var0 = decay * shadow_variable + (1 - decay) * variable = 0.99*4.555 + (1-0.99)*10 = 4.60945
        # shadow_variable_var1 = 0.99*9.01+(1-0.99)*10 = 9.0199
        sess.run(maintain_averages_op)
        print(sess.run([var0, ema.average(var0), var1, ema.average(var1)]))  # 输出[10,4.60945,10,9.0199]

            下面是tensorflow官方给出的两种滑动模型使用场景:

      Example usage when creating a training model:
    
      ```python
      # Create variables.
      var0 = tf.Variable(...)
      var1 = tf.Variable(...)
      # ... use the variables to build a training model...
      ...
      # Create an op that applies the optimizer.  This is what we usually
      # would use as a training op.
      opt_op = opt.minimize(my_loss, [var0, var1])
    
      # Create an ExponentialMovingAverage object
      ema = tf.train.ExponentialMovingAverage(decay=0.9999)
    
      with tf.control_dependencies([opt_op]):
          # Create the shadow variables, and add ops to maintain moving averages
          # of var0 and var1. This also creates an op that will update the moving
          # averages after each training step.  This is what we will use in place
          # of the usual training op.
          training_op = ema.apply([var0, var1])
    
      ...train the model by running training_op...
      ```
    
    
      There are two ways to use the moving averages for evaluations:
    
      *  Build a model that uses the shadow variables instead of the variables.
         For this, use the `average()` method which returns the shadow variable
         for a given variable.
      *  Build a model normally but load the checkpoint files to evaluate by using
         the shadow variable names.  For this use the `average_name()` method.  See
         the @{tf.train.Saver} for more
         information on restoring saved variables.
    
      Example of restoring the shadow variable values:
    
      ```python
      # Create a Saver that loads variables from their saved shadow values.
      shadow_var0_name = ema.average_name(var0)
      shadow_var1_name = ema.average_name(var1)
      saver = tf.train.Saver({shadow_var0_name: var0, shadow_var1_name: var1})
      saver.restore(...checkpoint filename...)
      # var0 and var1 now hold the moving average values
      ```
      """
  • 相关阅读:
    PAT顶级 1024 Currency Exchange Centers (35分)(最小生成树)
    Codeforces 1282B2 K for the Price of One (Hard Version)
    1023 Have Fun with Numbers (20)
    1005 Spell It Right (20)
    1092 To Buy or Not to Buy (20)
    1118 Birds in Forest (25)
    1130 Infix Expression (25)
    1085 Perfect Sequence (25)
    1109 Group Photo (25)
    1073 Scientific Notation (20)
  • 原文地址:https://www.cnblogs.com/python-frog/p/9418183.html
Copyright © 2011-2022 走看看