zoukankan      html  css  js  c++  java
  • 构建高可靠hadoop集群之2-机栈

    本文主要参考 http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/RackAwareness.html

    hadoop组件是机栈敏感(译注rack,机栈,可以简单理解为节点的摆放)。

    例如,HDFS块的分布会利用机栈敏感来做容错,方式是把复制放在不同的机栈。这样即便网络切换导致故障或者其它断开导致的故障,也会有一些数据是可用的。

    译注:在重要的系统中,还有完全有必要考虑这个的,网络设备也会出现故障,毕竟这些网路设备的负载也很繁重的。

    haoop主守护程序可以获得集群从属节点(数据节点)的机栈id,方式是激活一个完毕脚本或者是配置文件制定的java类。无论使用哪种方式,它们的输出都必须符合java的org.apache.hadoop.net.DNSToSwitchMapping接口。

    这个接口要求一一对应,拓扑信息(机器摆放信息)必须形如'/myrack/myhost',其中‘/'是拓扑分隔符,’myrack'是rack的识别符,'myhost'是主机名称。假定一个rack有24个子网,那么其中一个就可以使用'/192.168.100.0/192.168.100.5'.

    如果要使用java类做拓扑映射,那么类的名称必须是通过配置文件的net.topology.node.switch.mapping.impl来设定。例如,networkTopology.java,已经包含在hadoop发布程序中,管理员可以配置。

    使用java类的好处是,当一个新的数据节点加入的时候,hadoop不需要调用外部进程(这样可以更高效一些)。

    如果使用外部脚本,那么必须使用参数net.topology.script.file.name来配置。不同于java类,完毕拓扑脚本并没有包含在hadoop发布程序中,必须由管理员提供。当hadoop调用这些脚本的时候,会发送多个ip地址给ARGV.需要发送给脚本的ip地址个数,是

    net.topology.script.number.args控制的,默认值是100.如果net.topology.script.number.args设置为1,那么数据节点或者几点管理器每提交一个ip地址,脚本就要被调用一次。

    如果net.topology.script.file.name 或者 net.topology.node.switch.mapping.impl的值没有设置,那么rack id '/default-rack'就任意ip的返回值。然而,这样的结果看起来一点也不理想,它可能会导致块同步问题(译注:大家一个rack,写入那里无所谓,所以可能性能和容错都会存在一些问题)。

    译注:以上几个参数都是在core-site.xml中配置的

    原文,给出了两个例子,一个是python,一个是bash

    ------------------------

    -- python

    ------------------------

    #!/usr/bin/python
    # this script makes assumptions about the physical environment.
    #  1) each rack is its own layer 3 network with a /24 subnet, which
    # could be typical where each rack has its own
    #     switch with uplinks to a central core router.
    #
    #             +-----------+
    #             |core router|
    #             +-----------+
    #            /            
    #   +-----------+        +-----------+
    #   |rack switch|        |rack switch|
    #   +-----------+        +-----------+
    #   | data node |        | data node |
    #   +-----------+        +-----------+
    #   | data node |        | data node |
    #   +-----------+        +-----------+
    #
    # 2) topology script gets list of IP's as input, calculates network address, and prints '/network_address/ip'.

    import netaddr
    import sys
    sys.argv.pop(0)                                                  # discard name of topology script from argv list as we just want IP addresses

    netmask = '255.255.255.0'                                        # set netmask to what's being used in your environment.  The example uses a /24

    for ip in sys.argv:                                              # loop over list of datanode IP's
    address = '{0}/{1}'.format(ip, netmask)                      # format address string so it looks like 'ip/netmask' to make netaddr work
    try:
       network_address = netaddr.IPNetwork(address).network     # calculate and print network address
       print "/{0}".format(network_address)
    except:
       print "/rack-unknown"                                    # print catch-all value if unable to calculate network address

    ------------------------

    -- bash

    ------------------------

    #!/bin/bash
    # Here's a bash example to show just how simple these scripts can be
    # Assuming we have flat network with everything on a single switch, we can fake a rack topology.
    # This could occur in a lab environment where we have limited nodes,like 2-8 physical machines on a unmanaged switch.
    # This may also apply to multiple virtual machines running on the same physical hardware.
    # The number of machines isn't important, but that we are trying to fake a network topology when there isn't one.
    #
    #       +----------+    +--------+
    #       |jobtracker|    |datanode|
    #       +----------+    +--------+
    #                      /
    #  +--------+  +--------+  +--------+
    #  |datanode|--| switch |--|datanode|
    #  +--------+  +--------+  +--------+
    #              /       
    #       +--------+    +--------+
    #       |datanode|    |namenode|
    #       +--------+    +--------+
    #
    # With this network topology, we are treating each host as a rack.  This is being done by taking the last octet
    # in the datanode's IP and prepending it with the word '/rack-'.  The advantage for doing this is so HDFS
    # can create its 'off-rack' block copy.
    # 1) 'echo $@' will echo all ARGV values to xargs.
    # 2) 'xargs' will enforce that we print a single argv value per line
    # 3) 'awk' will split fields on dots and append the last field to the string '/rack-'. If awk
    #    fails to split on four dots, it will still print '/rack-' last field value

    echo $@ | xargs -n 1 | awk -F '.' '{print "/rack-"$NF}'

    总结:

    网络拓扑如何设计,无论在哪个集群中,都是需要密切关注的

  • 相关阅读:
    HDU 3848 CC On The Tree 树形DP
    编程求取直线一般式表达式,两直线交点
    向外国学者所要论文源代码--英语模版
    找出该树中第二小的值--思路及算法实现
    不使用额外空间交换2个数据的源代码
    华为2018软件岗笔试题解题思路和源代码分享
    华为笔试题--LISP括号匹配 解析及源码实现
    Linux 快捷键汇总(偏基础)
    快速排序算法思路分析和C++源代码(递归和非递归)
    Python读取SQLite文件数据
  • 原文地址:https://www.cnblogs.com/lzfhope/p/7000456.html
Copyright © 2011-2022 走看看