zoukankan      html  css  js  c++  java
  • 分布式协调服务Zookeeper集群之ACL篇

                  分布式协调服务Zookeeper集群之ACL篇

                                               作者:尹正杰

    版权声明:原创作品,谢绝转载!否则将追究法律责任。

    一.zookeeper ACL相关知识概览

    1>.zookeeper官方文档(http://zookeeper.apache.org/doc/r3.4.14/zookeeperProgrammers.html#sc_ZooKeeperAccessControl

    2>.使用ACL的ZooKeeper访问控制的特别说明

    ZooKeeper使用ACL来控制对其znode(ZooKeeper数据树的数据节点)的访问。ACL实现与UNIX文件访问权限非常相似:它使用权限位来允许/禁止针对节点的各种操作以及位应用的范围。与标准UNIX权限不同,ZooKeeper节点不受用户(文件所有者),组和世界(其他)的三个标准范围的限制。ZooKeeper没有znode所有者的概念。相反,ACL指定与这些ID关联的ID和权限集。
    
    另请注意,ACL仅适用于特定的znode。特别是它不适用于儿童。例如,如果/ app只能通过ip读取:172.16.16.1和/ app / status是世界可读的,那么任何人都可以读取/ app / status ; ACL不是递归的。
    
    ZooKeeper支持可插拔的身份验证方案。使用表单scheme:expression指定ID ,其中scheme是id对应的认证方案。有效表达式集由方案定义。例如,ip:172.16.16.1是使用ip方案的地址为172.16.16.1的主机的id ,而digest:bob:password是使用摘要方案的名称为bob的用户的id 。
    
    当客户端连接到ZooKeeper并对其自身进行身份验证时,ZooKeeper会将与客户端对应的所有ID与客户端连接相关联。当客户端尝试访问节点时,将根据znode的ACL检查这些ID。ACL由(scheme:expression,perms)对组成。表达式的格式特定于该方案。例如,该对(ip:19.22.0.0/16,READ)为任何IP地址以19.22开头的客户端提供READ权限。

    3>.ACL权限说明

    ZooKeeper支持以下权限:
    CREATE:
        您可以创建子节点
    READ:
      您可以从节点获取数据并列出其子节点。 WRITE:
      您可以为节点设置数据 DELETE:
      您可以删除子节点 ADMIN:
      您可以设置权限,由于ZooKeeper没有文件所有者的概念,因此存在ADMIN权限。在某种意义上,ADMIN权限将实体指定为所有者。

    4>.内置ACL认证方案

    ZooKeeeper具有以下内置方案:
    world:
      有一个id,任何人,代表任何人。 auth:
      是一种特殊方案,它忽略任何提供的表达式,而是使用当前用户,凭据和方案。任何表达(无论用户喜欢用SASL认证或用户:密码等与DIGEST认证)提供由动物园管理员服务器持续的ACL时忽略。但是,仍必须在ACL中提供表达式,因为ACL必须与表单scheme:expression:perms匹配。提供此方案是为了方便,因为它是用户创建znode然后将该znode的访问权限仅限于该用户的常见用例。如果没有经过身份验证的用户,则使用auth方案设置ACL将失败。 digest:
      使用username:password字符串生成MD5哈希,然后将其用作ACL ID标识。通过以明文形式发送用户名:密码来完成身份验证。在ACL中使用时,表达式将是用户名:base64 编码的SHA1 密码摘要。 ip:
      使用客户端主机IP作为ACL ID标识。的ACL表达的形式的地址
    /位,其中最显著位 的地址是针对最显著匹配位的客户端主机的IP。 x509:
      使用客户端X500 Principal作为ACL ID标识。ACL表达式是客户端的确切X500主体名称。使用安全端口时,会自动对客户端进行身份验证,并设置x509方案的身份验证信息。

    二.zookeeperACL实战篇

    1>.查看zookeeper的默认权限

    [root@node102.yinzhengjie.org.cn ~]# zkCli.sh -server node103.yinzhengjie.org.cn:2181
    Connecting to node103.yinzhengjie.org.cn:2181
    2019-04-28 10:17:02,821 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
    2019-04-28 10:17:02,824 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=node102.yinzhengjie.org.cn
    2019-04-28 10:17:02,824 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_201
    2019-04-28 10:17:02,826 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
    2019-04-28 10:17:02,826 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/yinzhengjie/softwares/jdk1.8.0_201/jre
    2019-04-28 10:17:02,826 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/target/classes:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../build/classes:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/target/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../build/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-log4j12-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-api-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/netty-3.10.6.Final.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/log4j-1.2.17.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/jline-0.9.94.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/audience-annotations-0.5.0.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-3.4.14.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/src/main/resources/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf:
    2019-04-28 10:17:02,826 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2019-04-28 10:17:02,826 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
    2019-04-28 10:17:02,827 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
    2019-04-28 10:17:02,827 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
    2019-04-28 10:17:02,827 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
    2019-04-28 10:17:02,827 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-957.el7.x86_64
    2019-04-28 10:17:02,827 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
    2019-04-28 10:17:02,827 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
    2019-04-28 10:17:02,827 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/root
    2019-04-28 10:17:02,828 [myid:] - INFO  [main:ZooKeeper@442] - Initiating client connection, connectString=node103.yinzhengjie.org.cn:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5ce65a89
    Welcome to ZooKeeper!
    2019-04-28 10:17:02,845 [myid:] - INFO  [main-SendThread(node103.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server node103.yinzhengjie.org.cn/172.30.1.103:2181. Will not attempt to authenticate using SASL (unknown error)
    JLine support is enabled
    2019-04-28 10:17:02,904 [myid:] - INFO  [main-SendThread(node103.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@879] - Socket connection established to node103.yinzhengjie.org.cn/172.30.1.103:2181, initiating session
    2019-04-28 10:17:02,912 [myid:] - INFO  [main-SendThread(node103.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server node103.yinzhengjie.org.cn/172.30.1.103:2181, sessionid = 0x67001077cdb70001, negotiated timeout = 30000
    
    WATCHER::
    
    WatchedEvent state:SyncConnected type:None path:null
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 0] 
    [root@node102.yinzhengjie.org.cn ~]# zkCli.sh -server node103.yinzhengjie.org.cn:2181          #登陆zookeeper集群
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 0] ls /    
    [zookeeper]
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 1] 
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 1] create /hdfs-ha 'yinzhengjie-hdfs'        #创建“/hdfs-ha”节点,并添加数据为“yinzhengjie-hdfs”
    Created /hdfs-ha
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 2] 
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 2] ls /                           #查看创建的节点
    [hdfs-ha, zookeeper]
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 3] 
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 3] getAcl /hdfs-ha                     #获取刚刚创建的节点的默认权限
    'world,'anyone      #我们可以看到默认的认证方式是‘world’            
    : cdrwa         #其权限为CREATE,DELETE,READ,WRITE,ADMIN权限的缩写名称
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 4] 
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 4] get /hdfs-ha                      #获取hdfs-ha节点的信息
    yinzhengjie-hdfs
    cZxid = 0x1600000011
    ctime = Sun Apr 28 10:17:53 CST 2019
    mZxid = 0x1600000011
    mtime = Sun Apr 28 10:17:53 CST 2019
    pZxid = 0x1600000011
    cversion = 0
    dataVersion = 0
    aclVersion = 0
    ephemeralOwner = 0x0
    dataLength = 16
    numChildren = 0
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 5]

    2>.设置只读权限案例(使用明文密码)

    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 3] getAcl /hdfs-ha                #查看“/hdfs-ha”权限
    'world,'anyone
    : cdrwa
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 4] 
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 4] addauth digest jason:123          #我们给zookeeper创建一个认证用户,同时他会自动给当前终端默认配置为当前用户。
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 5] 
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 5] setAcl /hdfs-ha auth:jason:123:r      #我们为"/hdfs-ha"节点配置只读(r)权限
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 6]
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 6] getAcl /hdfs-ha                #查看"/hdfs-ha"权限
    'digest,'jason:x
    : r
    [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 7]  
    [root@node103.yinzhengjie.org.cn ~]# zkCli.sh -server node101.yinzhengjie.org.cn
    Connecting to node101.yinzhengjie.org.cn
    2019-04-28 11:43:05,009 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
    2019-04-28 11:43:05,012 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=node103.yinzhengjie.org.cn
    2019-04-28 11:43:05,012 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_201
    2019-04-28 11:43:05,013 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
    2019-04-28 11:43:05,013 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/yinzhengjie/softwares/jdk1.8.0_201/jre
    2019-04-28 11:43:05,014 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/target/classes:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../build/classes:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/target/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../build/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-log4j12-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-api-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/netty-3.10.6.Final.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/log4j-1.2.17.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/jline-0.9.94.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/audience-annotations-0.5.0.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-3.4.14.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/src/main/resources/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf:
    2019-04-28 11:43:05,014 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2019-04-28 11:43:05,014 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
    2019-04-28 11:43:05,014 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
    2019-04-28 11:43:05,014 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
    2019-04-28 11:43:05,014 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
    2019-04-28 11:43:05,014 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-957.el7.x86_64
    2019-04-28 11:43:05,014 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
    2019-04-28 11:43:05,014 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
    2019-04-28 11:43:05,014 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/root
    2019-04-28 11:43:05,015 [myid:] - INFO  [main:ZooKeeper@442] - Initiating client connection, connectString=node101.yinzhengjie.org.cn sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5ce65a89
    Welcome to ZooKeeper!
    2019-04-28 11:43:05,035 [myid:] - INFO  [main-SendThread(node101.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server node101.yinzhengjie.org.cn/172.30.1.101:2181. Will not attempt to authenticate using SASL (unknown error)
    JLine support is enabled
    2019-04-28 11:43:05,092 [myid:] - INFO  [main-SendThread(node101.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@879] - Socket connection established to node101.yinzhengjie.org.cn/172.30.1.101:2181, initiating session
    2019-04-28 11:43:05,101 [myid:] - INFO  [main-SendThread(node101.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server node101.yinzhengjie.org.cn/172.30.1.101:2181, sessionid = 0x6500107843430003, negotiated timeout = 30000
    
    WATCHER::
    
    WatchedEvent state:SyncConnected type:None path:null
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 0] 
    [root@node103.yinzhengjie.org.cn ~]# zkCli.sh -server node101.yinzhengjie.org.cn        #上面我们设置了只有jason用户才可以对"/hdfs-ha"节点有读取权限,于是我们开启我们开启另外一个终端操作
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 0] ls /
    [hdfs-ha, zookeeper]
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 1] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 1] getAcl /hdfs-ha                    #我们使用新的终端登陆后,发现我们无法查看"/hdfs-ha"节点的信息了!
    Authentication is not valid : /hdfs-ha
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 2] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 2] get /hdfs-ha
    Authentication is not valid : /hdfs-ha
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 3] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 3] addauth digest jason:123               #我们为当前终端配置为“jason:123”用户时,发现又可以正常访问"/hdfs-ha"节点啦!
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 4] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 4] getAcl /hdfs-ha         
    'digest,'jason:x
    : r
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 5] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 5] get /hdfs-ha            
    yinzhengjie-hdfs
    cZxid = 0x1600000011
    ctime = Sun Apr 28 10:17:53 CST 2019
    mZxid = 0x1600000011
    mtime = Sun Apr 28 10:17:53 CST 2019
    pZxid = 0x1600000011
    cversion = 0
    dataVersion = 0
    aclVersion = 1
    ephemeralOwner = 0x0
    dataLength = 16
    numChildren = 0
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 6] 

    3>.设置只读权限案例(使用密文密码)

      好了,提前揭开加密规则:(就是SHA1加密,然后base64编码)

    static public String generateDigest(String idPassword) throws NoSuchAlgorithmException {
        String parts[] = idPassword.split(":", 2);
        byte digest[] = MessageDigest.getInstance("SHA1").digest(
                idPassword.getBytes());
        return parts[0] + ":" + base64Encode(digest);
    }

      上述操作上面是可以通过Java代码实现,我们可以用命令行轻松实现上述的操作,如下所示:

    [root@node103.yinzhengjie.org.cn ~]# echo -n jason:123 | openssl dgst -binary -sha1 | openssl base64               #我们生称“jason:123”用户密码的密文。    
    r0YEZ4KtjAHo/1PYI0MosMKjrK8=
    [root@node103.yinzhengjie.org.cn ~]# 
    [root@node103.yinzhengjie.org.cn ~]# zkCli.sh -server node101.yinzhengjie.org.cn
    Connecting to node101.yinzhengjie.org.cn
    2019-04-28 14:44:02,191 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
    2019-04-28 14:44:02,194 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=node103.yinzhengjie.org.cn
    2019-04-28 14:44:02,194 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_201
    2019-04-28 14:44:02,195 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
    2019-04-28 14:44:02,195 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/yinzhengjie/softwares/jdk1.8.0_201/jre
    2019-04-28 14:44:02,195 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/target/classes:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../build/classes:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/target/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../build/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-log4j12-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-api-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/netty-3.10.6.Final.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/log4j-1.2.17.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/jline-0.9.94.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/audience-annotations-0.5.0.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-3.4.14.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/src/main/resources/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf:
    2019-04-28 14:44:02,195 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2019-04-28 14:44:02,195 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
    2019-04-28 14:44:02,196 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
    2019-04-28 14:44:02,196 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
    2019-04-28 14:44:02,196 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
    2019-04-28 14:44:02,196 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-957.el7.x86_64
    2019-04-28 14:44:02,196 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
    2019-04-28 14:44:02,196 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
    2019-04-28 14:44:02,196 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/root
    2019-04-28 14:44:02,197 [myid:] - INFO  [main:ZooKeeper@442] - Initiating client connection, connectString=node101.yinzhengjie.org.cn sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5ce65a89
    Welcome to ZooKeeper!
    2019-04-28 14:44:02,215 [myid:] - INFO  [main-SendThread(node101.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server node101.yinzhengjie.org.cn/172.30.1.101:2181. Will not attempt to authenticate using SASL (unknown error)
    JLine support is enabled
    2019-04-28 14:44:02,269 [myid:] - INFO  [main-SendThread(node101.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@879] - Socket connection established to node101.yinzhengjie.org.cn/172.30.1.101:2181, initiating session
    2019-04-28 14:44:02,283 [myid:] - INFO  [main-SendThread(node101.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server node101.yinzhengjie.org.cn/172.30.1.101:2181, sessionid = 0x6500107843430006, negotiated timeout = 30000
    
    WATCHER::
    
    WatchedEvent state:SyncConnected type:None path:null
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 0] 
    [root@node103.yinzhengjie.org.cn ~]# zkCli.sh -server node101.yinzhengjie.org.cn                      #登陆zookeeper集群
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 0] ls /
    [hdfs-ha, zookeeper]
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 1] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 1] create /clickhouse 'yinzhengjie-db'
    Created /clickhouse
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 2] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 2] setAcl /clickhouse digest:jason:r0YEZ4KtjAHo/1PYI0MosMKjrK8=:r    #注意这个格式为,digest:用户名:密码:权限
    cZxid = 0x160000001e
    ctime = Sun Apr 28 14:45:08 CST 2019
    mZxid = 0x160000001e
    mtime = Sun Apr 28 14:45:08 CST 2019
    pZxid = 0x160000001e
    cversion = 0
    dataVersion = 0
    aclVersion = 1
    ephemeralOwner = 0x0
    dataLength = 14
    numChildren = 0
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 3] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 3] getAcl /clickhouse                              #配置授权后,我们发现不需要开启终端,在当前终端就立即生效啦!
    Authentication is not valid : /clickhouse
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 4] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 4] addauth digest jason:123                          #我们使用jason用户进行认证,发现的确生效了,这个密码是123大家知道是怎么来的吗?提示,我们可以用命令行生成
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 5] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 5] getAcl /clickhouse      
    'digest,'jason:x
    : r
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 6]    
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 6] ls /
    [hdfs-ha, zookeeper, clickhouse]
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 7] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 7] get /clickhouse
    yinzhengjie-db
    cZxid = 0x160000001e
    ctime = Sun Apr 28 14:45:08 CST 2019
    mZxid = 0x160000001e
    mtime = Sun Apr 28 14:45:08 CST 2019
    pZxid = 0x160000001e
    cversion = 0
    dataVersion = 0
    aclVersion = 1
    ephemeralOwner = 0x0
    dataLength = 14
    numChildren = 0
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 8] 

      当然,密码处了可以使用命令行的方式生成,还可以使用Java代码去生产,对开发感兴趣的小伙伴可以参考以下文章,我这里就不搬砖了:

        参考链接1:https://www.jianshu.com/p/d3b5db5e03eb

        参考链接2:http://www.cnblogs.com/yjmyzz/p/zookeeper-acl-demo.html

        参考链接3:https://ihong5.wordpress.com/2014/07/24/apache-zookeeper-setting-acl-in-zookeeper-client/

        参考链接4:http://zookeeper.apache.org/doc/r3.4.14/zookeeperProgrammers.html#sc_ZooKeeperAccessControl

     

    三.ACL的注意事项

    1>.删除znode空节点(只要某节点有读权限且无子节点(即空节点)且父节点有删除权限就可以直接删除。

    [zk: node101.yinzhengjie.org.cn(CONNECTED) 0] ls /
    [hdfs-ha, zookeeper]
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 1] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 1] addauth digest jason:123
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 2] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 2] getAcl /hdfs-ha                     #我们可以查看到jason用户对"/hdfs-ha"只有读取权限
    'digest,'jason:x
    : r
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 3] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 3] ls /hdfs-ha                     #我们也知道"/hdfs-ha"节点是一个空节点
    []
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 4] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 4] get /hdfs-ha                    #我们可以使用jason用户查看“/hdfs-ha”节点存储对数据
    yinzhengjie-hdfs
    cZxid = 0x1600000011
    ctime = Sun Apr 28 10:17:53 CST 2019
    mZxid = 0x1600000011
    mtime = Sun Apr 28 10:17:53 CST 2019
    pZxid = 0x1600000011
    cversion = 0
    dataVersion = 0
    aclVersion = 1
    ephemeralOwner = 0x0
    dataLength = 16
    numChildren = 0
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 5] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 5] delete /hdfs-ha                #我们尝试删除该节点,发现是可以成功删除的!神奇吗?因此我们可以总结为,只要某节点有读取权限且无子节点(即空节点)就可以直接删除。
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 6] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 6] ls /
    [zookeeper]
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 7] 

    [zk: node101.yinzhengjie.org.cn(CONNECTED) 8] getAcl /                      #发现没有?根节点的默认认证方式为world,默认权限为“cdrwa”
    'world,'anyone
    : cdrwa
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 9]

    2>.要删除某个节点下的子节点,必须具有对父节点的read权限,以及父节点的delete权限

    [root@node103.yinzhengjie.org.cn ~]# zkCli.sh -server node101.yinzhengjie.org.cn
    Connecting to node101.yinzhengjie.org.cn
    2019-04-28 16:55:38,291 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
    2019-04-28 16:55:38,294 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=node103.yinzhengjie.org.cn
    2019-04-28 16:55:38,294 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_201
    2019-04-28 16:55:38,295 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
    2019-04-28 16:55:38,296 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/yinzhengjie/softwares/jdk1.8.0_201/jre
    2019-04-28 16:55:38,296 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/target/classes:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../build/classes:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/target/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../build/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-log4j12-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-api-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/netty-3.10.6.Final.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/log4j-1.2.17.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/jline-0.9.94.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/audience-annotations-0.5.0.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-3.4.14.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/src/main/resources/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf:
    2019-04-28 16:55:38,296 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2019-04-28 16:55:38,296 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
    2019-04-28 16:55:38,296 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
    2019-04-28 16:55:38,296 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
    2019-04-28 16:55:38,296 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
    2019-04-28 16:55:38,296 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-957.el7.x86_64
    2019-04-28 16:55:38,296 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
    2019-04-28 16:55:38,296 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
    2019-04-28 16:55:38,296 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/root
    2019-04-28 16:55:38,297 [myid:] - INFO  [main:ZooKeeper@442] - Initiating client connection, connectString=node101.yinzhengjie.org.cn sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5ce65a89
    Welcome to ZooKeeper!
    2019-04-28 16:55:38,322 [myid:] - INFO  [main-SendThread(node101.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server node101.yinzhengjie.org.cn/172.30.1.101:2181. Will not attempt to authenticate using SASL (unknown error)
    JLine support is enabled
    2019-04-28 16:55:38,378 [myid:] - INFO  [main-SendThread(node101.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@879] - Socket connection established to node101.yinzhengjie.org.cn/172.30.1.101:2181, initiating session
    2019-04-28 16:55:38,386 [myid:] - INFO  [main-SendThread(node101.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server node101.yinzhengjie.org.cn/172.30.1.101:2181, sessionid = 0x650010784343000c, negotiated timeout = 30000
    
    WATCHER::
    
    WatchedEvent state:SyncConnected type:None path:null
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 0] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 0] create /hadoop 'HADOOP'
    Created /hadoop
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 1] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 1] create /hadoop/hdfs 'hdfs'  
    Created /hadoop/hdfs
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 2] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 2] getAcl /hadoop
    'world,'anyone
    : cdrwa
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 3] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 3] getAcl /hadoop/hdfs
    'world,'anyone
    : cdrwa
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 4] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 4] addauth digest jason:123456
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 5] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 5] setAcl /hadoop auth:jason:123456:ra
    cZxid = 0x1600000035
    ctime = Sun Apr 28 16:59:48 CST 2019
    mZxid = 0x1600000035
    mtime = Sun Apr 28 16:59:48 CST 2019
    pZxid = 0x1600000036
    cversion = 1
    dataVersion = 0
    aclVersion = 1
    ephemeralOwner = 0x0
    dataLength = 6
    numChildren = 1
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 6] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 6] getAcl /hadoop                     
    'digest,'jason:y3tbCkSaneiAqhCGFxNZIKXL7fo=
    : ra
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 7] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 7] getAcl /hadoop/hdfs
    'world,'anyone
    : cdrwa
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 8] 
    [root@node103.yinzhengjie.org.cn ~]# zkCli.sh -server node101.yinzhengjie.org.cn                #准备工作,本次实验的准备工作
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 6] getAcl /hadoop                                     #我们查看"/hadoop"节点的权限为ra
    'digest,'jason:y3tbCkSaneiAqhCGFxNZIKXL7fo=
    : ra
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 7] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 7] getAcl /hadoop/hdfs                           #我们查看"/hadoop"的子节点“/hadoop/hdfs”的权限为cdrwa
    'world,'anyone
    : cdrwa
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 8] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 9] create /hadoop/yarn 'YARN'                     #由于“/hadoop”节点只有读取(ra)权限,因此我们无法在该节点下写入新的子节点。
    Authentication is not valid : /hadoop/yarn
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 10] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 10] create /hadoop/hdfs/namenode 'NAMENODE'            #我们知道上面"/hadoop/hdfs"权限是比“/hadoop”权限范围要大,我们在该目录节点下创建文件时是的
    Created /hadoop/hdfs/namenode
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 11] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 11] addauth digest jason-nn:123                      #我们手动创建一个名称为“jason-nn”的认证用户
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 12]
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 12] setAcl /hadoop/hdfs/namenode auth:jaso-nn:123:da        #我们把刚刚创建的“/hadoop/hdfs/namenode”授权给"jason-nn"用户
    cZxid = 0x160000005a
    ctime = Sun Apr 28 18:04:51 CST 2019
    mZxid = 0x160000005a
    mtime = Sun Apr 28 18:04:51 CST 2019
    pZxid = 0x160000005a
    cversion = 0
    dataVersion = 0
    aclVersion = 1
    ephemeralOwner = 0x0
    dataLength = 8
    numChildren = 0
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 13] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 13] getAcl /hadoop/hdfs/namenode                  #我们可以查看"/hadoop/hdfs/namenode"的路径
    'digest,'jason:y3tbCkSaneiAqhCGFxNZIKXL7fo=
    : da
    'digest,'jason-nn:Bg2TJUg+e7lQNHByXObwMNUWiHY=
    : da
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 14] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 14] get /hadoop/hdfs/namenode                     #我们查看"/hadoop/hdfs/namenode"内容,发现无法读取,这是为什么呢?如果为非要读取该如何是好呢?小伙伴们该如何操作?
    Authentication is not valid : /hadoop/hdfs/namenode
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 15] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 15] ls /hadoop/hdfs                          #查看"/hadoop/hdfs"目录下的节点          
    [namenode]
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 16] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 16] delete /hadoop/hdfs/namenode                            #删除"/hadoop/hdfs/namenode"节点
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 17] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 17] ls /hadoop/hdfs                                #再次查看“/hadoop/hdfs”节点,发现数据被我们成功删除啦!现在我们可以总结出原因了吧?
    []
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 18] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 16] delete /hadoop/hdfs/namenode                     
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 17] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 17] ls /hadoop/hdfs              
    []
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 18] create /hadoop/hdfs/namenode 'NAMENODE'
    Created /hadoop/hdfs/namenode
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 19] setAcl /hadoop/hdfs/namenode auth:jaso-nn:123:rda
    cZxid = 0x160000005d
    ctime = Sun Apr 28 18:08:43 CST 2019
    mZxid = 0x160000005d
    mtime = Sun Apr 28 18:08:43 CST 2019
    pZxid = 0x160000005d
    cversion = 0
    dataVersion = 0
    aclVersion = 1
    ephemeralOwner = 0x0
    dataLength = 8
    numChildren = 0
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 20] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 20] get /hadoop/hdfs/namenode                        
    NAMENODE
    cZxid = 0x160000005d
    ctime = Sun Apr 28 18:08:43 CST 2019
    mZxid = 0x160000005d
    mtime = Sun Apr 28 18:08:43 CST 2019
    pZxid = 0x160000005d
    cversion = 0
    dataVersion = 0
    aclVersion = 1
    ephemeralOwner = 0x0
    dataLength = 8
    numChildren = 0
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 21] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 21] getAcl /hadoop/hdfs/namenode 
    'digest,'jason:y3tbCkSaneiAqhCGFxNZIKXL7fo=
    : dra
    'digest,'jason-nn:Bg2TJUg+e7lQNHByXObwMNUWiHY=
    : dra
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 22] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 22] delete /hadoop/hdfs/namenode                     
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 23] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 23] ls /hadoop/hdfs                                  
    []
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 24] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 19] setAcl /hadoop/hdfs/namenode auth:jaso-nn:123:rda      #如果想要“/hadoop/hdfs/namenode”获取读取权限,我们直接手动给一个即可! 

    3>.想要删除的节点其父节点没有删除权限如何处理?

    [zk: node101.yinzhengjie.org.cn(CONNECTED) 34] getAcl /hadoop              #我们查看“/hadoop”节点的权限为ra
    'digest,'jason:y3tbCkSaneiAqhCGFxNZIKXL7fo=
    : ra
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 35] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 35] getAcl /hadoop/hdfs            #我们在查看“/hadoop”的子节点“/hadoop/hdfs”发现其权限最大化,拥有了cdrwa
    'world,'anyone
    : cdrwa
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 36] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 36] delete /hadoop/hdfs            #但是,当我们删除“/hadoop/hdfs”节点时,发现怎么删除也删不了!这是为什么呢?其实是“/hadoop”节点没删除("d")权限!
    Authentication is not valid : /hadoop/hdfs
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 37]
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 37] setAcl /hadoop auth:jason:123456:rda  #既然上面说“/hadoop”没有“d”权限,我们就给他加一个试试看!
    cZxid = 0x1600000035
    ctime = Sun Apr 28 16:59:48 CST 2019
    mZxid = 0x1600000035
    mtime = Sun Apr 28 16:59:48 CST 2019
    pZxid = 0x1600000036
    cversion = 1
    dataVersion = 0
    aclVersion = 2
    ephemeralOwner = 0x0
    dataLength = 6
    numChildren = 1
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 38] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 38] getAcl /hadoop                #发现添加“d”权限成功啦!
    'digest,'jason:y3tbCkSaneiAqhCGFxNZIKXL7fo=
    : dra
    'digest,'jason-nn:Bg2TJUg+e7lQNHByXObwMNUWiHY=
    : dra
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 39]
    
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 39] getAcl /hadoop/hdfs
    'world,'anyone
    : cdrwa
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 40] 
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 40] delete /hadoop/hdfs            #再次删除"/hadoop/hdfs"节点,我们发现并没有报错信息。
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 41]
    
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 41] ls /hadoop                   #恭喜你,删除成功啦!
    []
    [zk: node101.yinzhengjie.org.cn(CONNECTED) 42]

    温馨提示:

      上面的操作方式是因为我们有用户去操作修改父节点的权限。当我们无法对父节点进行修改时,解决办法就只能到zookeeper配置文件中的dataDir指定目录里清空所有数据,再重启zk,但是这样就相当于所有数据全扔了,所以在设计ACL时,对于delete权限,要谨慎规划,在测试zk集群上做好测试,再转到生产环境操作。

  • 相关阅读:
    Kali视频学习21-25
    20159315《网络攻防实践》第六周学习总结
    Kali视频学习16-20
    20159315《网络攻防实践》第五周学习总结
    一个PE文件的逆向分析
    一个好玩的CTF题
    对于安卓锁屏中知识点小结
    安卓系统安全措施
    安卓防逆向、防动态分析、渗透测试及加固
    安卓组件漏洞防护注意事项
  • 原文地址:https://www.cnblogs.com/yinzhengjie/p/10759845.html
Copyright © 2011-2022 走看看