zoukankan      html  css  js  c++  java
  • three supported reliability levels: * End-to-end * Store on failure * Best effort

    https://github.com/cloudera/flume/blob/master/flume-docs/src/docs/UserGuide/Introduction

    === Reliability
       
      Reliability, the ability to continue delivering events in the face of
      failures without losing data, is a vital feature of Flume. Large
      distributed systems can and do suffer partial failures in many ways -
      physical hardware can fail, resources such as network bandwidth or
      memory can become scarce, or software can crash or run slowly. Flume
      emphasizes fault-tolerance as a core design principle and keeps
      running and collecting data even when many components have failed.
       
      Flume can guarantee that all data received by an agent node will
      eventually make it to the collector at the end of its flow as long as
      the agent node keeps running. That is, data can be *reliably*
      delivered to its eventual destination.
       
      However, reliable delivery can be very resource intensive and is often
      a stronger guarantee than some data sources require. Therefore, Flume
      allows the user to specify, on a per-flow basis, the level of
      reliability required. There are three supported reliability levels:
       
      * End-to-end
      * Store on failure
      * Best effort
       
      .A Note About Reliability
      ******************
      Although Flume is extremely tolerant to machine, network, and software
      failures, there is never any such thing as '100% reliability'. If all
      the machines in a Flume installation were irrevocably destroyed in
      some terrible data center incident, all copies of Flume's data would
      be lost and there would be no way to recover them. Therefore all of
      Flume's reliability levels make guarantees about data delivery 'until
      some maximum number of failures have occurred'. Flume's failure modes
      - in terms of what can fail and what will keep running if they do -
      are described in detail later in this guide.
      ******************
       
      The *end-to-end* reliability level guarantees that once Flume accepts
      an event, that event will make it to the endpoint - as long as the
      agent that accepted the event remains live long enough. The first
      thing the agent does in this setting is write the event to disk in a
      ''write-ahead log'' (WAL) so that, if the agent crashes and restarts,
      knowledge of the event is not lost. After the event has successfully
      made its way to the end of its flow, an acknowledgment is sent back to
      the originating agent so that it knows it no longer needs to store the
      event on disk. This reliability level can withstand any number of
      failures downstream of the initial agent.
       
      The *store on failure* reliability level causes nodes to only require
      an acknowledgement from the node one hop downstream. If the sending
      node detects a failure, it will store data on its local disk until the
      downstream node is repaired, or an alternate downstream destination
      can be selected. While this is effective, data can be lost if a
      compound or silent failure occurs.
       
      The *best-effort* reliability level sends data to the next hop with no
      attempts to confirm or retry delivery. If nodes fail, any data that
      they were in the process of transmitting or receiving can be
      lost. This is the weakest reliability level, but also the most
      lightweight.
    === Reliability
       
      Reliability, the ability to continue delivering events in the face of
      failures without losing data, is a vital feature of Flume. Large
      distributed systems can and do suffer partial failures in many ways -
      physical hardware can fail, resources such as network bandwidth or
      memory can become scarce, or software can crash or run slowly. Flume
      emphasizes fault-tolerance as a core design principle and keeps
      running and collecting data even when many components have failed.
       
      Flume can guarantee that all data received by an agent node will
      eventually make it to the collector at the end of its flow as long as
      the agent node keeps running. That is, data can be *reliably*
      delivered to its eventual destination.
       
      However, reliable delivery can be very resource intensive and is often
      a stronger guarantee than some data sources require. Therefore, Flume
      allows the user to specify, on a per-flow basis, the level of
      reliability required. There are three supported reliability levels:
       
      * End-to-end
      * Store on failure
      * Best effort
       
      .A Note About Reliability
      ******************
      Although Flume is extremely tolerant to machine, network, and software
      failures, there is never any such thing as '100% reliability'. If all
      the machines in a Flume installation were irrevocably destroyed in
      some terrible data center incident, all copies of Flume's data would
      be lost and there would be no way to recover them. Therefore all of
      Flume's reliability levels make guarantees about data delivery 'until
      some maximum number of failures have occurred'. Flume's failure modes
      - in terms of what can fail and what will keep running if they do -
      are described in detail later in this guide.
      ******************
       
      The *end-to-end* reliability level guarantees that once Flume accepts
      an event, that event will make it to the endpoint - as long as the
      agent that accepted the event remains live long enough. The first
      thing the agent does in this setting is write the event to disk in a
      ''write-ahead log'' (WAL) so that, if the agent crashes and restarts,
      knowledge of the event is not lost. After the event has successfully
      made its way to the end of its flow, an acknowledgment is sent back to
      the originating agent so that it knows it no longer needs to store the
      event on disk. This reliability level can withstand any number of
      failures downstream of the initial agent.
       
      The *store on failure* reliability level causes nodes to only require
      an acknowledgement from the node one hop downstream. If the sending
      node detects a failure, it will store data on its local disk until the
      downstream node is repaired, or an alternate downstream destination
      can be selected. While this is effective, data can be lost if a
      compound or silent failure occurs.
       
      The *best-effort* reliability level sends data to the next hop with no
      attempts to confirm or retry delivery. If nodes fail, any data that
      they were in the process of transmitting or receiving can be
      lost. This is the weakest reliability level, but also the most
      lightweight.
  • 相关阅读:
    几款网络测试工具总结
    Linux安装telnet
    Linux下iptables 禁止端口和开放端口
    mysql创建某个数据库中的某张表 只读用户
    查看nginx版本号的几种方法
    Ngxtop-Nginx日志实时分析利器
    Nginx监控运维
    oracle经典书籍推荐
    华为典型局域网组网案例介绍(1)
    技术说明 路由器是如何工作的呢? 一个简单的解释
  • 原文地址:https://www.cnblogs.com/rsapaper/p/7817972.html
Copyright © 2011-2022 走看看