zoukankan      html  css  js  c++  java

  • Redhat cluster 的心跳机制

    Redhat cluster软件的架构分为两层：

    • 底层的cluster messaging layer（集群信息层）负责在所有节点之间传递集群信息包括心跳信息
      RHEL6及之前是CMAN，从RHEL7开始是Corosync。
    • 上层的cluster resource manager（集群资源管理器）管理划归集群所属的资源，包括IP地址、逻辑卷、文件系统等
      RHEL6及之前是RGManager，从RHEL7开始是Pacemaker。

    心跳机制使用的是Totem协议，Totem是一个巨复杂的协议，心跳只是其中一个很小的部分，可以这么理解：Totem协议负责处理集群内部所有节点之间的通信问题，包含四个主要组件：Total Ordering Protocol, Membership Protocol, Recovery Protocol, Flow Control Mechanism。这里只讲与心跳有关的部分。

    Totem的通信是两种方式互相配合进行的：multicast（多播）和token（令牌）。[注意multicast与broadcast(广播)不同，它相当于限定范围的广播，只有分组内的成员能收到。] 一个节点只有在持有token的时候才能发言，通过multicast给集群内所有节点发送信息，每一条multicast信息都附有顺序号，发送完成之后就把顺序号记录到token中，然后把token传给下一个节点，token的传递方式是一对一的，就像首尾相连的环形接力，从IP地址最小的节点依次传给下一个IP地址更大的节点，每个节点收到token之后，根据token里记录的顺序号检查自己是否漏收了multicast信息，如果漏收了，就在token里添加重传申请，然后把token传给下一个节点，下一个节点除了检查自己有没有漏收multicast之外，还会根据token里记录的重传申请把相应的multicast信息重新发送一次（当然前提是该节点已经收到了这条multicast信息）。

    totem

    集群通过两种机制检测节点的健康状态：

    • Token超时

    如果token中断的时间超过了指定的期限，节点就会触发membership protocol，重组cluster，这个期限通过以下参数设置：<totem token=”XXX”/>
    注：XXX以毫秒为单位。

    在上述期限内，token会尝试重传，重传的次数是以下参数指定的，缺省值是4：
    <totem token_retransmits_before_loss_const=”X”/>

    • Multicast retransmit(重传)超过阈值

    如果某个节点能收到token，但收不到multicast，那么经过若干次token循环之后，也会触发membership protocol，重组cluster，这个次数可以通过参数 fail_recv_const 设置，缺省值是2500次。

    Totem协议有4种状态，代表集群运行的不同阶段，不同的状态下运行的子协议也不同：

    • GATHER
    • COMMIT
      Gather和Commit状态出现在集群组建的阶段，可以比喻为集合、报数的过程。运行的是membership protocol。
    • RECOVERY
      Recovery状态出现在集群发生问题、进行恢复的阶段。运行的是recovery protocol。
    • OPERATIONAL
      Operational状态表示集群正常工作的阶段。运行的是Totem Ordering Protocol。

    totem_status

    有了这些基本概念，我们就可以大致看懂集群的日志了：

    // 本来正常的集群突然发生token丢失（心跳信号丢失）： Apr 26 01:47:26 node1 openais[4252]: [TOTEM] The token was lost in the OPERATIONAL state. Apr 26 01:47:26 node1 openais[4252]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes). Apr 26 01:47:26 node1 openais[4252]: [TOTEM] Transmit multicast socket send buffer size (320000 bytes). // 集群开始重组，进入Gather状态，召集各节点： Apr 26 01:47:26 node1 openais[4252]: [TOTEM] entering GATHER state from 2. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] entering GATHER state from 0. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Creating commit token because I am the rep. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Storing new sequence id for ring 798 // 进入Commit状态： Apr 26 01:47:28 node1 openais[4252]: [TOTEM] entering COMMIT state. // 进入Recovery状态： Apr 26 01:47:28 node1 openais[4252]: [TOTEM] entering RECOVERY state. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] position [0] member 10.23.201.30: Apr 26 01:47:28 node1 openais[4252]: [TOTEM] previous ring seq 1940 rep 10.23.201.30 Apr 26 01:47:28 node1 openais[4252]: [TOTEM] aru 2b high delivered 2b received flag 1 Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Did not need to originate any messages in recovery. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Sending initial ORF token Apr 26 01:47:28 node1 openais[4252]: [CLM ] CLM CONFIGURATION CHANGE Apr 26 01:47:28 node1 openais[4252]: [CLM ] New Configuration: Apr 26 01:47:28 node1 openais[4252]: [CLM ] r(0) ip(10.23.201.30) Apr 26 01:47:28 node1 kernel: dlm: closing connection to node 2 Apr 26 01:47:28 node1 openais[4252]: [CLM ] Members Left: Apr 26 01:47:28 node1 openais[4252]: [CLM ] r(0) ip(10.23.201.31) Apr 26 01:47:28 node1 openais[4252]: [CLM ] Members Joined: Apr 26 01:47:28 node1 openais[4252]: [CLM ] CLM CONFIGURATION CHANGE Apr 26 01:47:28 node1 openais[4252]: [CLM ] New Configuration: Apr 26 01:47:28 node1 openais[4252]: [CLM ] r(0) ip(10.23.201.30) Apr 26 01:47:28 node1 openais[4252]: [CLM ] Members Left: Apr 26 01:47:29 node1 openais[4252]: [CLM ] Members Joined: Apr 26 01:47:29 node1 openais[4252]: [SYNC ] This node is within the primary component and will provide service. // 集群重组完成，进入Operational状态，只剩下一个节点： Apr 26 01:47:29 node1 openais[4252]: [TOTEM] entering OPERATIONAL state. Apr 26 01:47:29 node1 openais[4252]: [CLM ] got nodejoin message 10.23.201.30 Apr 26 01:47:29 node1 openais[4252]: [CPG ] got joinlist message from node 1 // 把不正常的节点fence掉： Apr 26 01:47:58 node1 fenced[4302]: node2 not a cluster member after 30 sec post_fail_delay Apr 26 01:47:58 node1 fenced[4302]: fencing node "node2" Apr 26 01:48:10 node1 fenced[4302]: fence "node2" success
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    // 本来正常的集群突然发生token丢失（心跳信号丢失）：
    Apr 26 01:47:26 node1 openais[4252]: [TOTEM] The token was lost in the OPERATIONAL state.
    Apr 26 01:47:26 node1 openais[4252]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes).
    Apr 26 01:47:26 node1 openais[4252]: [TOTEM] Transmit multicast socket send buffer size (320000 bytes).
    // 集群开始重组，进入Gather状态，召集各节点：
    Apr 26 01:47:26 node1 openais[4252]: [TOTEM] entering GATHER state from 2.
    Apr 26 01:47:28 node1 openais[4252]: [TOTEM] entering GATHER state from 0.
    Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Creating commit token because I am the rep.
    Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Storing new sequence id for ring 798
    // 进入Commit状态：
    Apr 26 01:47:28 node1 openais[4252]: [TOTEM] entering COMMIT state.
    // 进入Recovery状态：
    Apr 26 01:47:28 node1 openais[4252]: [TOTEM] entering RECOVERY state.
    Apr 26 01:47:28 node1 openais[4252]: [TOTEM] position [0] member 10.23.201.30:
    Apr 26 01:47:28 node1 openais[4252]: [TOTEM] previous ring seq 1940 rep 10.23.201.30
    Apr 26 01:47:28 node1 openais[4252]: [TOTEM] aru 2b high delivered 2b received flag 1
    Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Did not need to originate any messages in recovery.
    Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Sending initial ORF token
    Apr 26 01:47:28 node1 openais[4252]: [CLM  ] CLM CONFIGURATION CHANGE
    Apr 26 01:47:28 node1 openais[4252]: [CLM  ] New Configuration:
    Apr 26 01:47:28 node1 openais[4252]: [CLM  ]  r(0) ip(10.23.201.30)  
    Apr 26 01:47:28 node1 kernel: dlm: closing connection to node 2
    Apr 26 01:47:28 node1 openais[4252]: [CLM  ] Members Left:
    Apr 26 01:47:28 node1 openais[4252]: [CLM  ]  r(0) ip(10.23.201.31)  
    Apr 26 01:47:28 node1 openais[4252]: [CLM  ] Members Joined:
    Apr 26 01:47:28 node1 openais[4252]: [CLM  ] CLM CONFIGURATION CHANGE
    Apr 26 01:47:28 node1 openais[4252]: [CLM  ] New Configuration:
    Apr 26 01:47:28 node1 openais[4252]: [CLM  ]  r(0) ip(10.23.201.30)  
    Apr 26 01:47:28 node1 openais[4252]: [CLM  ] Members Left:
    Apr 26 01:47:29 node1 openais[4252]: [CLM  ] Members Joined:
    Apr 26 01:47:29 node1 openais[4252]: [SYNC ] This node is within the primary component and will provide service.
    // 集群重组完成，进入Operational状态，只剩下一个节点：
    Apr 26 01:47:29 node1 openais[4252]: [TOTEM] entering OPERATIONAL state.
    Apr 26 01:47:29 node1 openais[4252]: [CLM  ] got nodejoin message 10.23.201.30
    Apr 26 01:47:29 node1 openais[4252]: [CPG  ] got joinlist message from node 1
    // 把不正常的节点fence掉：
    Apr 26 01:47:58 node1 fenced[4302]: node2 not a cluster member after 30 sec post_fail_delay
    Apr 26 01:47:58 node1 fenced[4302]: fencing node "node2"
    Apr 26 01:48:10 node1 fenced[4302]: fence "node2" success
    来源 http://linuxperf.com/?p=120
  • 相关阅读:
    git和github的使用
    C语言 -- 链表操作
    数据结构之表达式求值(C++版)
    数据结构之一元多项式加法(C++版)
    数据结构之邻接矩阵(C++版)
    数据结构之二叉树，赫夫曼树(C++版)
    数据结构之循环队列(C++版)
    数据结构之顺序栈(C++版)
    数据结构之单链表(C++版)
    数据结构之顺序表(C++版)
  • 原文地址：https://www.cnblogs.com/jonathanyue/p/9301146.html
Copyright © 2011-2022 走看看