zoukankan      html  css  js  c++  java
  • cs224n

    Vanishing Gradients and Fancy RNNs

    Vanishing gradient problem

    图片名称

    Why is vanishing gradient a problem?

    图片名称

    Another explanation: Gradient can be viewed as a measure of
    the effect of the past on the future. If the gradient becomes vanishingly small over longer distances (step t to step t+n), then we can’t tell whether:

    • There’s no dependency between step t and t+n in the data
    • We have wrong parameters to capture the true dependency between t and t+n

    Effect of vanishing gradient on RNN-LM

    图片名称

    Why is exploding gradient a problem?

    If the gradient becomes too big, then the SGD update step
    becomes too big:

    [ heta^{ ext {new}}= heta^{ ext { old }}-alpha abla_{ heta} J( heta) ]

    This can cause bad updates: we take too large a step and reach
    a bad parameter configuration (with large loss). In the worst case, this will result in Inf or NaN in your network (then you have to restart training from an earlier checkpoint).

    Gradient clipping: solution for exploding gradient

    Gradient clipping: if the norm of the gradient is greater than
    some threshold, scale it down before applying SGD update

    Intuition: take a step in the same direction, but a smaller step

    图片名称

    How to fix vanishing gradient problem?

    The main problem is that it’s too difficult for the RNN to learn to preserve information over many timesteps. In a vanilla RNN, the hidden state is constantly being rewritten:

    [oldsymbol{h}^{(t)}=sigmaleft(oldsymbol{W}_{h} oldsymbol{h}^{(t-1)}+oldsymbol{W}_{x} oldsymbol{x}^{(t)}+oldsymbol{b} ight) ]

    Long Short-Term Memory (LSTM)

    图片名称
    图片名称

    How does LSTM solve vanishing gradients?

    • The LSTM architecture makes it easier for the RNN to preserve information over many timesteps
    • LSTM doesn’t guarantee that there is no vanishing/exploding gradient, but it does provide an easier way for the model to learn long-distance dependencies

    Gated Recurrent Units (GRU)

    图片名称

    LSTM vs GRU

    • The biggest difference is that GRU is quicker to compute and has fewer parameters
    • There is no conclusive evidence that one consistently performs better than the other
    • LSTM is a good default choice (especially if your data has particularly long dependencies, or you have lots of training data)
    • Rule of thumb: start with LSTM, but switch to GRU if you want something more efficient

    Conclusion: Though vanishing/exploding gradients are a general
    problem, RNNs are particularly unstable due to the repeated
    multiplication by the same weight matrix

    Bidirectional RNNs

    图片名称

    Multi-layer RNNs

    图片名称
  • 相关阅读:
    spring-mvc----数据库数据到页面错误--tomcat启动不了
    springmvc注解@RequestMapping
    数据库到jsp页面报错(一)
    IDEA秒退或者一直让填写激活码问题
    spting Boot 创建一个springBoot项目
    SSM框架整合(实现从数据库到页面展示)
    asp.net dbproviderfactory(提供程序工厂模型)多数据库访问
    connectionStrings 中的 providerName 属性
    IDisposable
    利用提示引导语句运行
  • 原文地址:https://www.cnblogs.com/curtisxiao/p/10666150.html
Copyright © 2011-2022 走看看