zoukankan      html  css  js  c++  java
  • Zookeeper与HBase的安装

      一、Zookeeper的安装

      1.http://www-us.apache.org/dist/zookeeper/stable/下载Zookeeper安装包,并将zookeeper-3.4.12.tar.gz复制到/home/jun下并解压

    [jun@master ~]$ cd zookeeper-3.4.12/
    [jun@master zookeeper-3.4.12]$ ls -l
    total 1604
    drwxr-xr-x.  2 jun jun     149 Mar 27 12:32 bin
    -rw-rw-r--.  1 jun jun   87945 Mar 27 12:32 build.xml
    drwxr-xr-x.  2 jun jun      77 Mar 27 12:32 conf
    drwxr-xr-x. 10 jun jun     130 Mar 27 12:32 contrib
    drwxr-xr-x.  2 jun jun    4096 Mar 27 12:37 dist-maven
    drwxr-xr-x.  6 jun jun    4096 Mar 27 12:32 docs
    -rw-rw-r--.  1 jun jun    1709 Mar 27 12:32 ivysettings.xml
    -rw-rw-r--.  1 jun jun    8197 Mar 27 12:32 ivy.xml
    drwxr-xr-x.  4 jun jun    4096 Mar 27 12:32 lib
    -rw-rw-r--.  1 jun jun   11938 Mar 27 12:32 LICENSE.txt
    -rw-rw-r--.  1 jun jun    3132 Mar 27 12:32 NOTICE.txt
    -rw-rw-r--.  1 jun jun    1585 Mar 27 12:32 README.md
    -rw-rw-r--.  1 jun jun    1770 Mar 27 12:32 README_packaging.txt
    drwxr-xr-x.  5 jun jun      47 Mar 27 12:32 recipes
    drwxr-xr-x.  8 jun jun     211 Mar 27 12:32 src
    -rw-rw-r--.  1 jun jun 1483366 Mar 27 12:32 zookeeper-3.4.12.jar
    -rw-rw-r--.  1 jun jun     819 Mar 27 12:36 zookeeper-3.4.12.jar.asc
    -rw-rw-r--.  1 jun jun      33 Mar 27 12:32 zookeeper-3.4.12.jar.md5
    -rw-rw-r--.  1 jun jun      41 Mar 27 12:32 zookeeper-3.4.12.jar.sha1

      2.使用gedit编辑zoo.cfg配置文件

    [jun@master zookeeper-3.4.12]$ cp /home/jun/zookeeper-3.4.12/conf/zoo_sample.cfg  /home/jun/zookeeper-3.4.12/conf/zoo.cfg
    [jun@master zookeeper-3.4.12]$ gedit /home/jun/zookeeper-3.4.12/conf/zoo.cfg 

      将下面的配置增加到最后面,配置的意思是“服务器编号、服务器地址、LF通信端口(Leader与Follower)、选举端口”

    server.1=master:2888:3888
    server.2=slave0:2888:3888
    server.3=slave1:2888:3888

      3.创建节点标志文件  

      由于在zoo.cfg当中配置了三台服务器,每一台服务器本地也需要一个myid文件,里面包含服务器的ID号,需要在三台机器上都创建myid文件。以master为例:

      首先递归创建目录(-p)

    [jun@master zookeeper-3.4.12]$ mkdir -p /tmp/zookeeper
    [jun@master zookeeper-3.4.12]$ gedit /tmp/zookeeper/myid

      然后新建myid文件并输入对应的id号,只输入一个1即可。

    [jun@master zookeeper-3.4.12]$ cat /tmp/zookeeper/myid 
    1

      另外两台机器同理。

      4.将Zookeeper安装文件从master复制到slave上(-r的意思是拷贝所有的文件和文件夹)

    [jun@master zookeeper-3.4.12]$ scp -r /home/jun/zookeeper-3.4.12/   slave0:~
    [jun@master zookeeper-3.4.12]$ scp -r /home/jun/zookeeper-3.4.12/   slave1:~

      5.Zookeeper集群的启动

      (1)需要在三个节点上分别执行启动操作

    [jun@master zookeeper-3.4.12]$ /home/jun/zookeeper-3.4.12/bin/zkServer.sh start
    ZooKeeper JMX enabled by default
    Using config: /home/jun/zookeeper-3.4.12/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED

    [jun@slave0 ~]$ /home/jun/zookeeper-3.4.12/bin/zkServer.sh start
    ZooKeeper JMX enabled by default
    Using config: /home/jun/zookeeper-3.4.12/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED

    [jun@slave1 ~]$ /home/jun/zookeeper-3.4.12/bin/zkServer.sh start
    ZooKeeper JMX enabled by default
    Using config: /home/jun/zookeeper-3.4.12/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED

      (2)至少启动两台服务器后,集群才会开始选举leader节点,可以在三台服务器上分别查看集群的状态,可以看到slave0节点是leader节点。

    [jun@master ~]$ /home/jun/zookeeper-3.4.12/bin/zkServer.sh status
    ZooKeeper JMX enabled by default
    Using config: /home/jun/zookeeper-3.4.12/bin/../conf/zoo.cfg
    Mode: follower
    
    [jun@slave0 ~]$ /home/jun/zookeeper-3.4.12/bin/zkServer.sh status
    ZooKeeper JMX enabled by default
    Using config: /home/jun/zookeeper-3.4.12/bin/../conf/zoo.cfg
    Mode: leader
    
    [jun@slave1 ~]$ /home/jun/zookeeper-3.4.12/bin/zkServer.sh status
    ZooKeeper JMX enabled by default
    Using config: /home/jun/zookeeper-3.4.12/bin/../conf/zoo.cfg
    Mode: follower

      (3)Zookeeper客户端交互式操作

    [jun@master ~]$ /home/jun/zookeeper-3.4.12/bin/zkCli.sh -server master:2181
    Connecting to master:2181
    2018-07-23 14:06:51,047 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.12-e5259e437540f349646870ea94dc2658c4e44b3b, built on 03/27/2018 03:55 GMT
    2018-07-23 14:06:51,049 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=master
    2018-07-23 14:06:51,050 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_171
    2018-07-23 14:06:51,051 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
    2018-07-23 14:06:51,051 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.8.0_171/jre
    2018-07-23 14:06:51,051 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/home/jun/zookeeper-3.4.12/bin/../build/classes:/home/jun/zookeeper-3.4.12/bin/../build/lib/*.jar:/home/jun/zookeeper-3.4.12/bin/../lib/slf4j-log4j12-1.7.25.jar:/home/jun/zookeeper-3.4.12/bin/../lib/slf4j-api-1.7.25.jar:/home/jun/zookeeper-3.4.12/bin/../lib/netty-3.10.6.Final.jar:/home/jun/zookeeper-3.4.12/bin/../lib/log4j-1.2.17.jar:/home/jun/zookeeper-3.4.12/bin/../lib/jline-0.9.94.jar:/home/jun/zookeeper-3.4.12/bin/../lib/audience-annotations-0.5.0.jar:/home/jun/zookeeper-3.4.12/bin/../zookeeper-3.4.12.jar:/home/jun/zookeeper-3.4.12/bin/../src/java/lib/*.jar:/home/jun/zookeeper-3.4.12/bin/../conf:
    2018-07-23 14:06:51,051 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2018-07-23 14:06:51,051 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
    2018-07-23 14:06:51,051 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
    2018-07-23 14:06:51,052 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
    2018-07-23 14:06:51,052 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
    2018-07-23 14:06:51,052 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-862.el7.x86_64
    2018-07-23 14:06:51,052 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=jun
    2018-07-23 14:06:51,052 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/home/jun
    2018-07-23 14:06:51,052 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/jun
    2018-07-23 14:06:51,053 [myid:] - INFO  [main:ZooKeeper@441] - Initiating client connection, connectString=master:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@799f7e29
    Welcome to ZooKeeper!
    2018-07-23 14:06:51,111 [myid:] - INFO  [main-SendThread(master:2181):ClientCnxn$SendThread@1028] - Opening socket connection to server master/192.168.1.100:2181. Will not attempt to authenticate using SASL (unknown error)
    JLine support is enabled
    2018-07-23 14:06:51,191 [myid:] - INFO  [main-SendThread(master:2181):ClientCnxn$SendThread@878] - Socket connection established to master/192.168.1.100:2181, initiating session
    [zk: master:2181(CONNECTING) 0] 2018-07-23 14:06:51,269 [myid:] - INFO  [main-SendThread(master:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server master/192.168.1.100:2181, sessionid = 0x1000005d8900000, negotiated timeout = 30000
    
    WATCHER::
    
    WatchedEvent state:SyncConnected type:None path:null
    
    [zk: master:2181(CONNECTED) 0] help
    ZooKeeper -server host:port cmd args
        stat path [watch]
        set path data [version]
        ls path [watch]
        delquota [-n|-b] path
        ls2 path [watch]
        setAcl path acl
        setquota -n|-b val path
        history 
        redo cmdno
        printwatches on|off
        delete path [version]
        sync path
        listquota path
        rmr path
        get path [watch]
        create [-s] [-e] path data acl
        addauth scheme auth
        quit 
        getAcl path
        close 
        connect host:port
    [zk: master:2181(CONNECTED) 1] quit
    Quitting...
    2018-07-23 14:07:36,575 [myid:] - INFO  [main:ZooKeeper@687] - Session: 0x1000005d8900000 closed
    2018-07-23 14:07:36,577 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@521] - EventThread shut down for session: 0x1000005d8900000

      

      二、HBase的安装

      1.从http://apache.fayea.com/hbase/stable/下载hbase-1.2.6.1-bin.tar.gz并赋值到/home/jun下并解压。、

    [jun@master ~]$ cd hbase-1.2.6.1/
    [jun@master hbase-1.2.6.1]$ ls -l
    total 340
    drwxr-xr-x.  4 jun jun   4096 Jun  4 09:43 bin
    -rw-r--r--.  1 jun jun 129753 Jun  4 09:43 CHANGES.txt
    drwxr-xr-x.  2 jun jun    178 Jun  4 09:43 conf
    drwxr-xr-x. 12 jun jun   4096 Jun  4 12:34 docs
    drwxr-xr-x.  7 jun jun     80 Jun  4 12:19 hbase-webapps
    -rw-rw-r--.  1 jun jun    261 Jun  4 12:36 LEGAL
    drwxrwxr-x.  3 jun jun   8192 Jul 23 14:11 lib
    -rw-rw-r--.  1 jun jun 143082 Jun  4 12:36 LICENSE.txt
    -rw-rw-r--.  1 jun jun  42141 Jun  4 12:36 NOTICE.txt
    -rw-r--r--.  1 jun jun   1477 Jun  4 09:43 README.txt

      2.修改conf文件夹下的环境变量hbase-env.sh

    # The java implementation to use.  Java 1.7+ required.
    export JAVA_HOME=/usr/java/jdk1.8.0_171/

      由于使用的jdk1.8,因此还需要将下面两行注释掉:

    # Configure PermSize. Only needed in JDK7. You can safely remove it for JDK8+
    #export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"
    #export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"

      3.修改conf文件夹下的配置文件hbase-site.xml

    <configuration>
    
        <property>
            <name>hbase.cluster.distributed</name>
            <value>true</value>
        </property>
    
        <property>
            <name>hbase.rootdir</name>
            <value>hdfs://master:9000/hbase</value>
        </property>
    
        <property>
            <name>hbase.zookeeper.quorum</name>
            <value>master</value>
        </property>
    
        <property>
            <name>hbase.master.info.port</name>
            <value>60010</value>
        </property>
    
    </configuration>

      4.设置conf文件夹下的regionservers

      regionservers文件类似Hadoop的slaves文件,其中保存了rigionserver的列表,启动HBase的时候,系统将根据该文件建立HBase集群。需要先删除localhost并添加两条配置

    slave0
    slave1

      5.设置linux环境变量并source生效

    #hbase
    export HBASE_HOME=/home/jun/hbase-1.2.6.1/
    export PATH=$HBASE_HOME/bin:$PATH
    export HADOOP_CLASSPATH=$HBASE_HOME/lib/*

      6.将HBase安装文件复制到slave节点

    [jun@master conf]$ scp -r /home/jun/hbase-1.2.6.1/  slave0:~/
    [jun@master conf]$ scp -r /home/jun/hbase-1.2.6.1/  slave1:~/

      7.启动并验证HBase

      (1)执行start-hbase.sh启动HBase,可以看到,系统首先启动HBase自带的ZooKeeper,然后启动HBase的HMaster,接着分别启动slave节点上的regionservers。

    [jun@master conf]$ start-hbase.sh 
    master: starting zookeeper, logging to /home/jun/hbase-1.2.6.1/bin/../logs/hbase-jun-zookeeper-master.out
    starting master, logging to /home/jun/hbase-1.2.6.1//logs/hbase-jun-master-master.out
    slave0: starting regionserver, logging to /home/jun/hbase-1.2.6.1/bin/../logs/hbase-jun-regionserver-slave0.out
    slave1: starting regionserver, logging to /home/jun/hbase-1.2.6.1/bin/../logs/hbase-jun-regionserver-slave1.out

      (2)查看各结点的jps,其中HMaster是HBase的主控节点进程,HQuorumPeer是HBase的ZooKeeper进程(即HBase内置的ZooKeeper)。

    [jun@master ~]$ jps
    6273 Jps
    3587 SecondaryNameNode
    3796 ResourceManager
    5989 HMaster
    5912 HQuorumPeer
    3357 NameNode
    
    [jun@slave0 ~]$ jps
    3441 NodeManager
    4177 HRegionServer
    3621 QuorumPeerMain
    3271 DataNode
    4375 Jps
    
    [jun@slave1 ~]$ jps
    3232 DataNode
    4370 Jps
    3606 QuorumPeerMain
    4169 HRegionServer
    3407 NodeManager

      (3)使用Web方式查看HBase系统的运行状态,访问http://master:60010

      三、HBase的Shell命令

      执行hbase shell即可进入HBase Shell

      1.创建表,其中第一个参数是表名,f1和f12是列名

    hbase(main):001:0> create "test1", {NAME => 'f1', VERSION => 5}
    Unknown argument ignored for column family f1: 1.8.7
    0 row(s) in 2.6300 seconds
    
    => Hbase::Table - test1
    hbase(main):002:0> create 'test2', 'f12'
    0 row(s) in 2.2360 seconds
    
    => Hbase::Table - test2

      2.查看列表

    hbase(main):003:0> list
    TABLE                                                                                                                                             
    test1                                                                                                                                             
    test2                                                                                                                                             
    2 row(s) in 0.0270 seconds
    
    => ["test1", "test2"]

      3.插入数据,其中test1是表名,id001是行键(row key),f1是列族名,uid是属性名,001是属性值

    hbase(main):004:0> put 'test1', 'id001', 'f1:uid', '001'
    0 row(s) in 0.2300 seconds

      4.扫描查询数据

    hbase(main):005:0> scan 'test1'
    ROW                                   COLUMN+CELL                                                                                                 
     id001                                column=f1:uid, timestamp=1532330789700, value=001                                                           
    1 row(s) in 0.0750 seconds

      5.查看表结构

    hbase(main):007:0> describe 'test1'
    Table test1 is ENABLED                                                                                                                            
    test1                                                                                                                                             
    COLUMN FAMILIES DESCRIPTION                                                                                                                       
    {NAME => 'f1', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => '
    FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}                       
    1 row(s) in 0.0310 seconds
    
    hbase(main):008:0> describe 'test2'
    Table test2 is ENABLED                                                                                                                            
    test2                                                                                                                                             
    COLUMN FAMILIES DESCRIPTION                                                                                                                       
    {NAME => 'f12', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 
    'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}                      
    1 row(s) in 0.0140 seconds

      6.删除表

    hbase(main):009:0> list
    TABLE                                                                                                                                             
    test1                                                                                                                                             
    test2                                                                                                                                             
    2 row(s) in 0.0100 seconds
    
    => ["test1", "test2"]
    hbase(main):010:0> disable 'test2'
    0 row(s) in 2.2890 seconds
    
    hbase(main):011:0> drop 'test2'
    0 row(s) in 1.2660 seconds
    
    hbase(main):012:0> list
    TABLE                                                                                                                                             
    test1                                                                                                                                             
    1 row(s) in 0.0110 seconds
    
    => ["test1"]

      7.退出HBase Shell

    hbase(main):013:0> exit

      四、HBase和ZooKeeper的关系

      如果用户在使用HBase自带的ZooKeeper的时候出现异常,导致HBase不能正常工作,这时可以停止使用自带的Zookeeper,转而启动独立安装的ZooKeeper,并重启HBase,可以快速解决上述问题。

      要想让HBase使用独立安装的Zookeeper,需要对HBase进行一些配置上的修改。

      1.修改conf/hbase-env.sh文件,添加如下代码(true表示使用自带的ZooKeeper)

    export HBASE_MANAGES_ZK=false

      2.将Zookeeper配置文件复制到HBase的CLASSPATH

    cp /home/jun/zookeeper-3.4.12/conf/zoo.cfg  /home/jun/hbase-1.2.6.1/conf

      3.重启HBase即可

  • 相关阅读:
    软件工程 团队开发(2)
    软件工程 团队开发(1)
    大道至简阅读笔记01
    小工具集合用户模板和用户场景
    人月神话阅读笔记03
    人月神话阅读笔记02
    本周java学习
    本周学习总结
    本周java 学习进度报告
    《构建之法》读后感
  • 原文地址:https://www.cnblogs.com/BigJunOba/p/9353872.html
Copyright © 2011-2022 走看看