1. 简要概述
Zookeeper是一个开源的分布式的,为分布式应用提供协调服务的框架。
Zookeeper从设计模式角度来理解:是一个基于观察者模式设计的分布式服务管理框架它负责存储和管理大家都关心的数据,然后接受观察者的注册,一但这些数据的状态发生变化, Zookeeper就将负责通知已经在 Zookeeper上注册的那些观察者做出相应的反应,从而实现集群中类似 Master/ Slave管理模式
Zookeeper=文件系统+通知机制
2. 特点
- Zookeeper:一个领导者( leader),多个跟随者( follower)组成的集群
- Leader负责进行投票的发起和决议,更新系统状态
- Follower用于接收客户请求并向客户端返回结果,在选举 Leader过程中参与投票
4)集群中只要有半数以上节点存活, Zookeeper集群就能正常服务。
5)全局数据一致:每个 serer保存一份相同的数据副本, client无论连接到哪个 serer,数据都是一致的。
6)更新请求顺序进行,来自同一个 client的更新请求按其发送顺序依次执行
7)数据更新原子性,一次数据更新要么成功,要么失敗。
8)实时性,在一定时间范围内, client能读到最新数据。
3. 数据结构
Zookeeper数据模型的结枃与Unix文件系统很类似,整体上可以看作是一裸树,每个节点叫做一个 Znode,很显然 zookeeper集群自身推护了一套数据结构・这个存储结构是个树形结构,其上的每一个节点,我们称之为" znode",每一个 znode默认能够存儲IMB的数据,每个 Znode都可以通过其路径唯一标识
4. 应用场景
提供的服务包括:分布式消息同步和协调机制、服务器节点动态上下线、统一配置管理负载均衡、集群管理等
1)分布式环境下,配置文件管理和同步是一个常见问题
一个集群中,所有节点的配置信思是一致的,比如 Hadoop集群。
对配置文件修改后,希望能够快速同步到各个节点上。
2)配置管理可交由 Zookeeper实现。
可将配置信息写入 ZooKeeper上的一个 Znode.
各个节点监听这个 Znode。
一且 Znode中的数据被修改, Zookeeper将通知各个节点。
image.png
还有好多场景后续再研究补充吧
5. 下载安装
打开下面的地址
http://zookeeper.apache.org/releases.html
点击download
image.png
5.1 选择一个地址下载
image.png
5.2选一个版本点击会看到tar.gz
image.png
5.3 上传到我们的102机器
image.png
5.4 可以看到上传成功
[shaozhiqi@hadoop102 ~]$ cd /opt/software/ [shaozhiqi@hadoop102 software]$ ll total 776464 -rw-rw-r--. 1 shaozhiqi shaozhiqi 332433589 Jun 23 19:59 hadoop-3.1.2.tar.gz -rw-rw-r--. 1 shaozhiqi shaozhiqi 194990602 Jun 23 19:59 jdk-8u211-linux-x64.tar.gz -rw-rw-r--. 1 shaozhiqi shaozhiqi 229988313 Jun 30 17:46 spark-2.4.3-bin-hadoop2.7.tgz -rw-rw-r--. 1 shaozhiqi shaozhiqi 37676320 Jul 2 21:50 zookeeper-3.4.14.tar.gz [shaozhiqi@hadoop102 software]$
5.5解压jar
[shaozhiqi@hadoop102 software]$ tar -zxvf zookeeper-3.4.14.tar.gz -C /opt/module/
进入目录修改配置
[shaozhiqi@hadoop102 zookeeper-3.4.14]$ ll total 1684 drwxr-xr-x. 2 shaozhiqi shaozhiqi 4096 Mar 7 01:09 bin //我们常用的命令 -rw-rw-r--. 1 shaozhiqi shaozhiqi 97426 Mar 7 00:50 build.xml drwxr-xr-x. 2 shaozhiqi shaozhiqi 74 Mar 7 01:09 conf //我们要配置配置文件 drwxr-xr-x. 2 shaozhiqi shaozhiqi 4096 Mar 7 01:10 dist-maven -rw-rw-r--. 1 shaozhiqi shaozhiqi 1709 Mar 7 00:50 ivysettings.xml -rw-rw-r--. 1 shaozhiqi shaozhiqi 10742 Mar 7 00:50 ivy.xml drwxr-xr-x. 4 shaozhiqi shaozhiqi 4096 Mar 7 01:09 lib //依赖的jar -rw-rw-r--. 1 shaozhiqi shaozhiqi 11970 Mar 7 00:50 LICENSE.txt -rw-rw-r--. 1 shaozhiqi shaozhiqi 3132 Mar 7 00:50 NOTICE.txt -rw-rw-r--. 1 shaozhiqi shaozhiqi 31622 Mar 7 00:50 pom.xml -rw-rw-r--. 1 shaozhiqi shaozhiqi 1765 Mar 7 00:50 README.md -rw-rw-r--. 1 shaozhiqi shaozhiqi 1770 Mar 7 00:50 README_packaging.txt drwxr-xr-x. 3 shaozhiqi shaozhiqi 21 Mar 7 00:50 src -rw-rw-r--. 1 shaozhiqi shaozhiqi 1515359 Mar 7 00:50 zookeeper-3.4.14.jar -rw-rw-r--. 1 shaozhiqi shaozhiqi 836 Mar 7 01:10 zookeeper-3.4.14.jar.asc -rw-rw-r--. 1 shaozhiqi shaozhiqi 33 Mar 7 00:50 zookeeper-3.4.14.jar.md5 -rw-rw-r--. 1 shaozhiqi shaozhiqi 41 Mar 7 00:50 zookeeper-3.4.14.jar.sha1 drwxr-xr-x. 3 shaozhiqi shaozhiqi 45 Mar 7 01:09 zookeeper-client drwxr-xr-x. 12 shaozhiqi shaozhiqi 4096 Mar 7 01:09 zookeeper-contrib drwxr-xr-x. 7 shaozhiqi shaozhiqi 4096 Mar 7 01:09 zookeeper-docs drwxr-xr-x. 3 shaozhiqi shaozhiqi 33 Mar 7 01:09 zookeeper-it drwxr-xr-x. 4 shaozhiqi shaozhiqi 43 Mar 7 01:09 zookeeper-jute drwxr-xr-x. 5 shaozhiqi shaozhiqi 4096 Mar 7 01:09 zookeeper-recipes drwxr-xr-x. 3 shaozhiqi shaozhiqi 30 Mar 7 01:09 zookeeper-server [shaozhiqi@hadoop102 zookeeper-3.4.14]$
5.6改名zoo_sample.cfg
[shaozhiqi@hadoop102 conf]$ mv zoo_sample.cfg zoo.cfg
修改文件存储的地址为我们的地址dataDir=/opt/module/zookeeper-3.4.14/zkdata
其他的先不改
[shaozhiqi@hadoop102 conf]$ vim zoo.cfg # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/opt/module/zookeeper-3.4.14/zkdata # the port at which the clients will connect clientPort=2181 # the maximum number of client connections.
6. 单节点启动
6.1启动服务端：
[shaozhiqi@hadoop102 zookeeper-3.4.14]$ cd bin/ [shaozhiqi@hadoop102 bin]$ pwd /opt/module/zookeeper-3.4.14/bin [shaozhiqi@hadoop102 bin]$ ll total 44 -rwxr-xr-x. 1 shaozhiqi shaozhiqi 232 Mar 7 00:50 README.txt -rwxr-xr-x. 1 shaozhiqi shaozhiqi 1937 Mar 7 00:50 zkCleanup.sh -rwxr-xr-x. 1 shaozhiqi shaozhiqi 1056 Mar 7 00:50 zkCli.cmd -rwxr-xr-x. 1 shaozhiqi shaozhiqi 1534 Mar 7 00:50 zkCli.sh -rwxr-xr-x. 1 shaozhiqi shaozhiqi 1759 Mar 7 00:50 zkEnv.cmd -rwxr-xr-x. 1 shaozhiqi shaozhiqi 2919 Mar 7 00:50 zkEnv.sh -rwxr-xr-x. 1 shaozhiqi shaozhiqi 1089 Mar 7 00:50 zkServer.cmd -rwxr-xr-x. 1 shaozhiqi shaozhiqi 6773 Mar 7 00:50 zkServer.sh -rwxr-xr-x. 1 shaozhiqi shaozhiqi 996 Mar 7 00:50 zkTxnLogToolkit.cmd -rwxr-xr-x. 1 shaozhiqi shaozhiqi 1385 Mar 7 00:50 zkTxnLogToolkit.sh [shaozhiqi@hadoop102 bin]$ ./zkServer.sh start ZooKeeper JMX enabled by default Using config: /opt/module/zookeeper-3.4.14/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [shaozhiqi@hadoop102 bin]$
验证是否启动成功：QuorumPeerMain
[shaozhiqi@hadoop102 bin]$ jps 20067 Jps 20036 QuorumPeerMain [shaozhiqi@hadoop102 bin]$
6.2启动客户端，可以看到日志链接到了2181
[shaozhiqi@hadoop102 bin]$ ./zkCli.sh Connecting to localhost:2181 2019-07-02 08:00:02,854 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
单节点可以启动，证明我们的zookeeper是好用的
7. 集群启动
停掉zookeeper服务端：
shaozhiqi@hadoop102 bin]$ ./zkServer.sh stop
7.1修改zoo.cfg配置文件
添加配置，2888代表leader的端口，3888是当2888挂了，3888替换
2,3,4标示集群模式下的myid里面的id号，根据下面的配置用来判断是那个server
[shaozhiqi@hadoop102 conf]$ vim zoo.cfg server.2=hadoop102:2888:3888 server.3=hadoop103:2888:3888 server.4=hadoop104:2888:3888
7.2配置102的myid
[shaozhiqi@hadoop102 zookeeper-3.4.14]$ cd zkdata/ [shaozhiqi@hadoop102 zkdata]$ ll total 0 drwxrwxr-x. 2 shaozhiqi shaozhiqi 18 Jul 2 23:00 version-2 [shaozhiqi@hadoop102 zkdata]$ vim myid [shaozhiqi@hadoop102 zkdata]$ cat myid 2 [shaozhiqi@hadoop102 zkdata]$
7.3同步我们的zookeeper到我们的103 104机器
[shaozhiqi@hadoop102 module]$ testxsync zookeeper-3.4.14/
修改103和104的myid
[shaozhiqi@hadoop103 zookeeper-3.4.14]$ cd zkdata/ [shaozhiqi@hadoop103 zkdata]$ vim myid [shaozhiqi@hadoop103 zkdata]$ cat myid 3 [shaozhiqi@hadoop103 zkdata]$ [shaozhiqi@hadoop104 zkdata]$ vim myid [shaozhiqi@hadoop104 zkdata]$ cat myid 4 [shaozhiqi@hadoop104 zkdata]$
7.4启动集群
7.4.1 Zookeeper的选举机制
那么我们配置了三台机器，启动时他怎么选举leader呢个？
1)半数机制( Paxos协议):集群中半数以上机器存活,集群可用。所以zookeeper适合装在奇数位机器
2)Zookeeper虽然在配置文件中并没有指定 master和 slave。但是, zookeeper工作时是有一个节点为 leader.其他则为 follower. Leader是通过内部的选举机制临时产生的
3)以一个简单的例子来说明整个选举的过程,,假设有五台服务器组成的 zookeeper集群,它们的d从1-5,同时它们都是最新启动的也就是没有历史数据,在存放数据量这一点上,都是一样的,假设这些服务器依序启动,来看看发生什么。
leader产生说明
(1)服务器1启动,此时只有它一台服务器启动了,它发出去的报没有任何响应,所以它的选举状态一直是 LOOKING状态。
(2)服务器2启动,它与最开始启动的服务器1进行通信,互相交换自己的选举结果由于两者都没有历史数据,所以d 值较大的服务器2胜出,但是由于没有达到超过半数以上的服务器都同意选举它(这个例子中的半数以上是3,所以服务器1、2还是继铁保持LOOKING状态。
(3)服务器3启动,根据前面的理论分析,服务器3成为服务器1、2、3中的老大而与上面不同的是,此时有三台服务器选举了它,所以它成为了这次选举的 leader,
(4)服务器4启动,根据前面的分析,理论上服务器4应该是服务器1、2、3、4中最大的,但是由于前面已经有半数以上的服务器选举了服务器3,所以它只能接收当小弟的命了
(5)服务器5启动,同个一样当小弟
按照这个理论我们的三台机器应该是103
7.4.2启动102
[shaozhiqi@hadoop102 bin]$ ./zkServer.sh start ZooKeeper JMX enabled by default Using config: /opt/module/zookeeper-3.4.14/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [shaozhiqi@hadoop102 bin]$
当值起了一个服务时查看状态，error ，服务数大于一半时才能提供服务
[shaozhiqi@hadoop102 bin]$ ./zkServer.sh status ZooKeeper JMX enabled by default Using config: /opt/module/zookeeper-3.4.14/bin/../conf/zoo.cfg Error contacting service. It is probably not running. [shaozhiqi@hadoop102 bin]$
7.4.3启动103，查看状态发现103是leader
[shaozhiqi@hadoop103 bin]$ ./zkServer.sh start ZooKeeper JMX enabled by default Using config: /opt/module/zookeeper-3.4.14/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [shaozhiqi@hadoop103 bin]$ ./zkServer.sh status ZooKeeper JMX enabled by default Using config: /opt/module/zookeeper-3.4.14/bin/../conf/zoo.cfg Mode: leader [shaozhiqi@hadoop103 bin]$
7.4.4启动104，查看104状态follower，所以103是leader
[shaozhiqi@hadoop104 zookeeper-3.4.14]$ bin/zkServer.sh start ZooKeeper JMX enabled by default Using config: /opt/module/zookeeper-3.4.14/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [shaozhiqi@hadoop104 zookeeper-3.4.14]$ bin/zkServer.sh status ZooKeeper JMX enabled by default Using config: /opt/module/zookeeper-3.4.14/bin/../conf/zoo.cfg Mode: follower [shaozhiqi@hadoop104 zookeeper-3.4.14]$
后面持续更新中~~~