zoukankan      html  css  js  c++  java
  • [hadoop][基本原理]zookeeper场景使用

    代码:https://github.com/xufeng79x/ZkClientTest

    1. 简介

      zookeeper的特性决定他适用到某些场景非常合适,比如典型的应用场景:

      1.集群管理(Group Membership)

      2.统一命名服务(Name Service)

      3.配置管理(Configuration Management)

      4.共享锁(Locks)

      5.队列管理

    2.集群管理                                                                                                                                                                                                                                  

       在hadoop中主备节点的概念大家都应该不默认,比如HBase中,我们可以启动多个master节点,但是在某时刻只有一个主master,当这个主master节点退出后

    其他的背master将会去争取成为主master。一般流程为:

      如上,一开始的时候多个master都会去zookeeper服务上创建相同的临时路径,其中某个master会节点会成功创建,那么这个节点就是主master了,其他创建不成功的将会观察这个临时路径。

    当主master由于宕机或者网络原因失去和zookeeper的连接(session)那么这个临时节点就会被zookeeper删除,随后其他备master将会感知到,再去争抢这个临时路径的创建权。

      如下代码利用多线程来模拟多个master的场景:

    public class Master implements Runnable {
        // master节点路径
        private String masterPath = null;
        
        // 当前master的信息
        private MasterInfo myinfo = null;
        
        // 当前主master的信息
        private MasterInfo activeMasterInfo = null;
        
        // 控制当前master的运行时间
        private long runningTime = 0;
        
        // 监控handler
        private IZkDataListener masterPathListener = null;
        
        // 当前session
        private ZkClient zc = new ZkClient("xufeng-1:2181,xufeng-2:2181,xufeng-3:2181", 10000, 10000, new SerializableSerializer());
        public Master(MasterInfo info, long runningTime, String masterPath)
        {
            this.myinfo = info;
            this.runningTime = runningTime;
            this.masterPath = masterPath;
            this.masterPathListener = new IZkDataListener() {
                // 当节点被删除的时候触发此方法
                public void handleDataDeleted(String dataPath) throws Exception {
                    attackMaster();
                }
                
                public void handleDataChange(String dataPath, Object data) throws Exception {
                    // do nothing
                    
                }
            };
            
            // 订阅节点
            zc.subscribeDataChanges(masterPath, masterPathListener);
        }
    
        public void run() {
            attackMaster();
            try {
                Thread.sleep(runningTime);
            } catch (InterruptedException e) {
                // do nothing
            }
        }
        
        // 去争抢这个节点创建(注册)
        private void attackMaster()
        {
            try {
                // 注册节点
                zc.create(masterPath, myinfo, CreateMode.EPHEMERAL);
                // 如果当前注册成功了,那么他就是主master
                activeMasterInfo = myinfo;
                System.out.println("the active master is : " + activeMasterInfo);
            }
            catch (ZkNodeExistsException e)
            {
                // 当节点已经被其他master注册了
                activeMasterInfo = zc.readData(masterPath);
                // 当无法读取到节点信息则认为其他master可能宕机了,再去抢注
                if (null == activeMasterInfo)
                {
                    attackMaster();
                }
                else{
                    System.out.println(activeMasterInfo + " has become active! " + myinfo +  " wait for next time to be active!");
                }
            }
            catch (Exception e)
            {
                // 当发生其他错误的时候,不去例会
            }
            
        }
    }

      当我们建立多个线程去启动的时候,多个master就会去抢注/master节点:

    the active master is : MasterInfo [id=C, name=masterC]
    MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=G, name=masterG] wait for next time to be active!
    MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=F, name=masterF] wait for next time to be active!
    MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=B, name=masterB] wait for next time to be active!
    MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=E, name=masterE] wait for next time to be active!
    MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=A, name=masterA] wait for next time to be active!
    MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=H, name=masterH] wait for next time to be active!
    MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=D, name=masterD] wait for next time to be active!

      这个时候我们模拟网络问题,手动地去删除/master节点多次的时候,各个master感知到节点删除会再次抢注:

    the active master is : MasterInfo [id=C, name=masterC]
    MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=G, name=masterG] wait for next time to be active!
    MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=F, name=masterF] wait for next time to be active!
    MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=B, name=masterB] wait for next time to be active!
    MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=E, name=masterE] wait for next time to be active!
    MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=A, name=masterA] wait for next time to be active!
    MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=H, name=masterH] wait for next time to be active!
    MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=D, name=masterD] wait for next time to be active!
    the active master is : MasterInfo [id=B, name=masterB]
    MasterInfo [id=B, name=masterB] has become active! MasterInfo [id=F, name=masterF] wait for next time to be active!
    MasterInfo [id=B, name=masterB] has become active! MasterInfo [id=H, name=masterH] wait for next time to be active!
    MasterInfo [id=B, name=masterB] has become active! MasterInfo [id=G, name=masterG] wait for next time to be active!
    MasterInfo [id=B, name=masterB] has become active! MasterInfo [id=E, name=masterE] wait for next time to be active!
    MasterInfo [id=B, name=masterB] has become active! MasterInfo [id=C, name=masterC] wait for next time to be active!
    MasterInfo [id=B, name=masterB] has become active! MasterInfo [id=A, name=masterA] wait for next time to be active!
    MasterInfo [id=B, name=masterB] has become active! MasterInfo [id=D, name=masterD] wait for next time to be active!
    the active master is : MasterInfo [id=E, name=masterE]
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=C, name=masterC] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=H, name=masterH] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=F, name=masterF] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=A, name=masterA] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=D, name=masterD] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=B, name=masterB] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=G, name=masterG] wait for next time to be active!

      我们可以看到每一次抢注成功的master都不一样,这样如果是由于网络问题而不是当前主master真的宕机了,那么会造成不必要的主备切换。所以说我们还可以进行如下的程序优化:  

      如上图,当由于网络原因原来的主master其实并没有宕机,那么为了减小由于主备切换带来的集群抖动,可以让其他备master延迟一定时间去争抢,而当前的主master则马上去争抢。所以说即使

    成为了主master也要观察这个临时路径。

      所以在程序中我们可以在抢注的时候判断如果当前master并不是之前的主master,则延迟一定时间去抢注,使得当前主master能够再次成功的抢的节点,具体优化代码如下:

                // 当节点被删除的时候触发此方法
                public void handleDataDeleted(String dataPath) throws Exception {
                    if (null != activeMasterInfo && activeMasterInfo.equals(myinfo))
                    {
                        // 如果当前master就是主master的时候,直接去抢注
                        attackMaster();
                    }
                    else
                    {
                        // 如果不是则延迟5秒去抢注,给原先的主master一个机会
                        delayExector.schedule(new Runnable() {
                            
                            public void run() {
                                attackMaster();
                                
                            }
                        }, 5, TimeUnit.SECONDS);
                    }
                    
                }

      手动的去删除/master节点,结果:masterE始终很稳定的抢注到了/master节点。

    the active master is : MasterInfo [id=E, name=masterE]
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=H, name=masterH] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=C, name=masterC] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=A, name=masterA] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=B, name=masterB] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=D, name=masterD] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=F, name=masterF] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=G, name=masterG] wait for next time to be active!
    the active master is : MasterInfo [id=E, name=masterE]
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=B, name=masterB] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=G, name=masterG] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=F, name=masterF] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=D, name=masterD] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=H, name=masterH] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=C, name=masterC] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=A, name=masterA] wait for next time to be active!
    the active master is : MasterInfo [id=E, name=masterE]
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=F, name=masterF] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=H, name=masterH] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=C, name=masterC] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=A, name=masterA] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=D, name=masterD] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=B, name=masterB] wait for next time to be active!
    MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=G, name=masterG] wait for next time to be active!

    小结:

      除了以上抢注某个临时节点的方式去进行主备切换实现外,我们也可以让每一个master在某个永久节点下各自注册自己的临时节点(CreateMode.EPHEMERAL_SEQUENTIAL

    方式,当观察到这个永久节点下znode有变动的时候,查看自己是不是后缀最小的一个,是,则将变成主master。

      除了主备切换场景外,集群管理中的节点发现,任务分发,也同样可以有zookeeper来处理,这种灵活的使用方式可以解决很多分布式场景的问题。

    3.其他                                                                                                                                                                                                                               

    // 略

  • 相关阅读:
    android 属性动画
    android EventBus
    android gson使用
    Date、String、Calendar类型之间的转化
    2020-08-26:裸写算法:树的非递归先序遍历。
    2020-08-25:BloomFilter的原理以及Zset的实现原理。
    2020-08-24:什么是小文件?很多小文件会有什么问题?很多小文件怎么解决?(大数据)
    2020-08-23:描述HTTPS和HTTP的区别。
    2020-08-22:I/O多路复用中select/poll/epoll的区别?
    2020-08-21:网络IO模型有哪些?
  • 原文地址:https://www.cnblogs.com/ios123/p/5735034.html
Copyright © 2011-2022 走看看