zoukankan      html  css  js  c++  java
  • CH08 Advanced Sequence Modeling for Natural Language Processing

    CH08 Advanced Sequence Modeling for Natural Language Processing

    摘要:这部分主要介绍了seq2seq中的attention机制,之前对attention机制一知半解。

    The mapping between the output and the input is called an alignment.

    Capturing More from a Sequence: Bidirectional Recurrent Models

    When modeling a sequence, it is useful to observe not just the words in the past but also the words that appear in the future. Consider the following sentence:

    The man who hunts ducks out on the weekends

    If the model were to observe only from left to right, its representation for “ducks” would be different from that of a model that had also observed the words from right to left.

    Taken together, information from the past and the future will be able to robustly represent the meaning of a word in a sequence.

    Capturing More from a Sequence: Attention

    • One problem is the S2S model encodes the input inyo one single vector and use it to generate the output. Although this might work with very short sentences, for long sentences such models fail to capture the information in the entire input.
    • Another problem with long inputs is that the gradients vanish when back-propagating through time, making the training difficult.

    Attention in Deep Neural Networks

    Recall that in a typical S2S model, each time step produces a hidden state representation, denoted as ϕ , specific to that time step in the encoder. To incorporate attention, we consider not only the final hidden state of the encoder, but also the hidden states for each of the intermediate steps.

    These encoder hidden states are, somewhat uninformatively, called values (or in some situations, keys). Attention also depends on the previous hidden state of the decoder, called the query.

    Attention is represented by a vector with the same dimension as the number of values it is attending to. This is called the attention vector, or attention weights, or sometimes alignment.

    The attention weights are combined with the encoder states (“values”) to generate a context vector that’s sometimes also known as a glimpse.

    computational graph

    software attention : The attention weights are typically floating-point values between 0 and 1.

    hard attention : It is possible to learn a binary 0/1 vector for attention. This is called hard attention.

    global attention : The attention mechanism illustrated in depends on the encoder states for all the time steps in the input.

    local attention : You could devise an attention mechanism that depended only on a window of the input around the current time step.

  • 相关阅读:
    lcd驱动解析(二)
    php参数引用
    Curl来做冒烟测试
    TIB自动化测试工作室QTP脚本汇总比较有价值
    使用QTP自动化 DZ消息复选框选择实例!
    正则表达式30分钟
    sqlserver查询数据库中有多少个表
    怎样获取页面上所有链接的名称和url
    Curl 来做自动跟踪重定向
    sqlserver2008 删除指定表
  • 原文地址:https://www.cnblogs.com/curtisxiao/p/10711545.html
Copyright © 2011-2022 走看看