zoukankan      html  css  js  c++  java
  • 张量限幅

    Outline

    • clip_by_value

    • relu

    • clip_by_norm

    • gradient clipping

    clip_by_value

    import tensorflow as tf
    
    a = tf.range(10)
    a
    
    <tf.Tensor: id=3, shape=(10,), dtype=int32, numpy=array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int32)>
    
    # a中小于2的元素值为2
    tf.maximum(a, 2)
    
    <tf.Tensor: id=6, shape=(10,), dtype=int32, numpy=array([2, 2, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int32)>
    
    # a中大于8的元素值为8
    tf.minimum(a, 8)
    
    <tf.Tensor: id=9, shape=(10,), dtype=int32, numpy=array([0, 1, 2, 3, 4, 5, 6, 7, 8, 8], dtype=int32)>
    
    # a中的元素值限制在[2,8]区间内
    tf.clip_by_value(a, 2, 8)
    
    <tf.Tensor: id=14, shape=(10,), dtype=int32, numpy=array([2, 2, 2, 3, 4, 5, 6, 7, 8, 8], dtype=int32)>
    

    relu

    a = a - 5
    a
    
    <tf.Tensor: id=17, shape=(10,), dtype=int32, numpy=array([-5, -4, -3, -2, -1,  0,  1,  2,  3,  4], dtype=int32)>
    
    tf.nn.relu(a)
    
    <tf.Tensor: id=19, shape=(10,), dtype=int32, numpy=array([0, 0, 0, 0, 0, 0, 1, 2, 3, 4], dtype=int32)>
    
    tf.maximum(a, 0)
    
    <tf.Tensor: id=22, shape=(10,), dtype=int32, numpy=array([0, 0, 0, 0, 0, 0, 1, 2, 3, 4], dtype=int32)>
    

    clip_by_norm

    • 缩放时不改变梯度方向
    a = tf.random.normal([2, 2], mean=10)
    a
    
    <tf.Tensor: id=35, shape=(2, 2), dtype=float32, numpy=
    array([[ 8.630464, 10.737844],
           [ 9.764073, 10.382202]], dtype=float32)>
    
    tf.norm(a)
    
    <tf.Tensor: id=41, shape=(), dtype=float32, numpy=19.822044>
    
    # 等比例的放缩a, norm为15
    aa = tf.clip_by_norm(a, 15)
    aa
    
    <tf.Tensor: id=58, shape=(2, 2), dtype=float32, numpy=
    array([[6.5309587, 8.125684 ],
           [7.388799 , 7.8565574]], dtype=float32)>
    
    tf.norm(aa)
    
    <tf.Tensor: id=64, shape=(), dtype=float32, numpy=15.0>
    

    gradient clipping

    • Gradient Exploding or vanishing

    • set lr=1

    • new_grads,total_norm = tf.clip_by_global_norm(grads,25)

    • 裁剪所有向量,但是所有向量的梯度方向都不变化

  • 相关阅读:
    BASE64Encoder/BASE64Decoder(转)
    对象转化为json
    Intent
    RecyclerView的单击和长按事件(转)
    selector的例子
    修改app工程名 Android Studio
    The number of method references in a .dex file cannot exceed 64K.(转)
    $(function() {})
    jQuery插件开发的两种方法及$.fn.extend的详解(转)
    myeclipse2014安装aptana3.4.0插件(转)
  • 原文地址:https://www.cnblogs.com/abdm-989/p/14123252.html
Copyright © 2011-2022 走看看