Hadoop的服务级授权篇

　　　　　　　　　　　　Hadoop的服务级授权篇

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

一.启用服务级授权

　　当客户端连接到Hadoop服务时，根据用户是否具有必须的权限，授予其该服务的权限。例如，用户可能具有将YARN作业提交到集群的必须权限，那么第一个访问控制检查是服务级授权，这回在检查文件权限之前进行。

　　如下图所示，可以通过在"${HADOOP_HOME}/etc/hadoop/core-site.xml"配置文件中设置"hadoop.security.authorization"属性来启用服务级授权。
　　　　<property>
　　　　　　<name>hadoop.security.authorization</name>
　　　　　　<value>true</value>
　　　　　　<description>指定是否启用了服务级别授权,默认值为"false",即不启用服务级授权。</description>
　　　　</property>

　　在集群中启用配置服务级授权后，需要通过每个服务的ACL配置各种Hadoop服务。

二.使用ACL配置服务级授权

　　为了配置服务级授权，可以在"${HADOOP_HOME}/etc/hadoop/hadoop-policy.xml"配置文件中配置每个Hadoop服务的ACL。在每个服务的ACL中，可以列出如下所示的用户和组，以逗号分割的列表授权访问权限：
　　　　user1,user2,user3,group1,group2,group3
　
　　请注意，在服务的ACL定义中的用户和组列表以空格分隔。还可以知仅指定一组用户或一些组，如下所示：
　　　　user1,user2,user3
　　　　group1,group2,group3

　　允许用户和组根据服务列出的ACL访问Hadoop服务的方式如下:
　　　　(1)值为"*"表示所有用户都可以访问Hadoop服务，值为""表示没有用户可以访问;
　　　　(2)如果不为服务指定ACL，则使用为可选属性"security.service.authorization.default.acl"指定的默认值;
　　　　(3)如果不指定"security.service.authorization.default.acl"属性，则允许所有用户访问该服务(这与为ACL指定"*"值相同);
　　　　(4)如果指定user1,user2 group1，则仅允许用户user1,user2和group1的组中的所有用户;
　　　　(5)如果指定user1,user2,最后使用空格，则不允许任何组访问;
　　　　(6)如果使用前导空格指定group1，则只允许来自此组的用户访问

　　以上说明仅供参考，具体使用方法我们应该参考官方文档中"<description>....</description>"标签的解释说明。

[root@hadoop101.yinzhengjie.com ~]# cat ${HADOOP_HOME}/etc/hadoop/hadoop-policy.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
 
 Licensed to the Apache Software Foundation (ASF) under one
 or more contributor license agreements.  See the NOTICE file
 distributed with this work for additional information
 regarding copyright ownership.  The ASF licenses this file
 to you under the Apache License, Version 2.0 (the
 "License"); you may not use this file except in compliance
 with the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>security.client.protocol.acl</name>
    <value>*</value>
    <description>ACL for ClientProtocol, which is used by user code
    via the DistributedFileSystem.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.client.datanode.protocol.acl</name>
    <value>*</value>
    <description>ACL for ClientDatanodeProtocol, the client-to-datanode protocol
    for block recovery.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.datanode.protocol.acl</name>
    <value>*</value>
    <description>ACL for DatanodeProtocol, which is used by datanodes to
    communicate with the namenode.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.inter.datanode.protocol.acl</name>
    <value>*</value>
    <description>ACL for InterDatanodeProtocol, the inter-datanode protocol
    for updating generation timestamp.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.namenode.protocol.acl</name>
    <value>*</value>
    <description>ACL for NamenodeProtocol, the protocol used by the secondary
    namenode to communicate with the namenode.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

 <property>
    <name>security.admin.operations.protocol.acl</name>
    <value>*</value>
    <description>ACL for AdminOperationsProtocol. Used for admin commands.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.refresh.user.mappings.protocol.acl</name>
    <value>*</value>
    <description>ACL for RefreshUserMappingsProtocol. Used to refresh
    users mappings. The ACL is a comma-separated list of user and
    group names. The user and group list is separated by a blank. For
    e.g. "alice,bob users,wheel".  A special value of "*" means all
    users are allowed.</description>
  </property>

  <property>
    <name>security.refresh.policy.protocol.acl</name>
    <value>*</value>
    <description>ACL for RefreshAuthorizationPolicyProtocol, used by the
    dfsadmin and mradmin commands to refresh the security policy in-effect.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.ha.service.protocol.acl</name>
    <value>*</value>
    <description>ACL for HAService protocol used by HAAdmin to manage the
      active and stand-by states of namenode.</description>
  </property>

  <property>
    <name>security.zkfc.protocol.acl</name>
    <value>*</value>
    <description>ACL for access to the ZK Failover Controller
    </description>
  </property>

  <property>
    <name>security.qjournal.service.protocol.acl</name>
    <value>*</value>
    <description>ACL for QJournalProtocol, used by the NN to communicate with
    JNs when using the QuorumJournalManager for edit logs.</description>
  </property>

  <property>
    <name>security.mrhs.client.protocol.acl</name>
    <value>*</value>
    <description>ACL for HSClientProtocol, used by job clients to
    communciate with the MR History Server job status etc. 
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <!-- YARN Protocols -->

  <property>
    <name>security.resourcetracker.protocol.acl</name>
    <value>*</value>
    <description>ACL for ResourceTrackerProtocol, used by the
    ResourceManager and NodeManager to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.resourcemanager-administration.protocol.acl</name>
    <value>*</value>
    <description>ACL for ResourceManagerAdministrationProtocol, for admin commands. 
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.applicationclient.protocol.acl</name>
    <value>*</value>
    <description>ACL for ApplicationClientProtocol, used by the ResourceManager 
    and applications submission clients to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.applicationmaster.protocol.acl</name>
    <value>*</value>
    <description>ACL for ApplicationMasterProtocol, used by the ResourceManager 
    and ApplicationMasters to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.containermanagement.protocol.acl</name>
    <value>*</value>
    <description>ACL for ContainerManagementProtocol protocol, used by the NodeManager 
    and ApplicationMasters to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.resourcelocalizer.protocol.acl</name>
    <value>*</value>
    <description>ACL for ResourceLocalizer protocol, used by the NodeManager 
    and ResourceLocalizer to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.job.task.protocol.acl</name>
    <value>*</value>
    <description>ACL for TaskUmbilicalProtocol, used by the map and reduce
    tasks to communicate with the parent tasktracker.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.job.client.protocol.acl</name>
    <value>*</value>
    <description>ACL for MRClientProtocol, used by job clients to
    communciate with the MR ApplicationMaster to query job status etc. 
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.applicationhistory.protocol.acl</name>
    <value>*</value>
    <description>ACL for ApplicationHistoryProtocol, used by the timeline
    server and the generic history service client to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>

  <property>
    <name>security.collector-nodemanager.protocol.acl</name>
    <value>*</value>
    <description>ACL for CollectorNodemanagerProtocol, used by nodemanager
    if timeline service v2 is enabled, for the timeline collector and nodemanager
    to communicate with each other.
    The ACL is a comma-separated list of user and group names. The user and
    group list is separated by a blank. For e.g. "alice,bob users,wheel".
    A special value of "*" means all users are allowed.</description>
  </property>
</configuration>
[root@hadoop101.yinzhengjie.com ~]# 

三.配置服务级授权

　　可以通过在"${HADOOP_HOME}/etc/hadoop/hadoop-policy.xml"配置文件中指定ACL来配置服务级授权。可将协议分为两种类型：管理和客户端协议。

　　例如，可以配置DataNode和NameNode协议的ACL，以及指定用户是否可以与Hadoop服务进行通信的客户端协议。

　　以下实例显示如何设置客户端协议ACL，以允许用户将YARN作业提交到集群。
　　　　<property>
　　　　　　<name>security.job.client.protocol.acl</name>
　　　　　　<value>Jason,Tom,Tina MapReduce</value>
　　　　　　<description>ACL for MRClientProtocol, used by job clients to
　　　　　　communciate with the MR ApplicationMaster to query job status etc.
　　　　　　The ACL is a comma-separated list of user and group names. The user and
　　　　　　group list is separated by a blank. For e.g. "alice,bob users,wheel".
　　　　　　A special value of "*" means all users are allowed.</description>
　　　　</property>

　　如上所示，使用"security.job.client.protocol.acl"属性可以创建在集群中运行作业的用户好组的白名单。本示例配置仅允许Jason,Tom，Tina用户和MapReduce组中的所有用户将YARN作业提交给Hadoop集群。

　　注意如何使用空格将用户和组列表分开。客户端使用此值与作业的ApplicationMaster服务通信，推荐值为"yarn","mapred"和"hadoop-users"，例如，hadoop-users是我们为运行Hadoop作业的用户创建的组。

　　使用"security.job.task.protocol.acl"属性指定用于YARN作业报告任务进度的协议(map和reduce任务可以使用此协议与NodeManager通信)。由于使用此协议的用户的ID被YARN作业的ID，因此必须将此值属性值设置为"*"。这样做可将用户和作业ID链接在一起。如果为此参数设置了不同的值，作业将失败。
　　　　<property>
　　　　　　<name>security.job.task.protocol.acl</name>
　　　　　　<value>*</value>
　　　　　　<description>ACL for TaskUmbilicalProtocol, used by the map and reduce
　　　　　　tasks to communicate with the parent tasktracker.
　　　　　　The ACL is a comma-separated list of user and group names. The user and
　　　　　　group list is separated by a blank. For e.g. "alice,bob users,wheel".
　　　　　　A special value of "*" means all users are allowed.</description>
　　　　</property>

四.阻止特定用户和组访问服务

　　在某些情况下，需要为服务指定阻止的访问控制列表。这指定了未授权访问服务的用户和组的列表。被阻止的访问控制列表的格式与访问控制列表的格式相同。可以通过$ HADOOP_CONF_DIR / hadoop-policy.xml来指定被阻止的访问控制列表。该属性名称的后缀为“ .blocked”。

　　示例：security.client.protocol.acl的阻止访问控制列表的属性名称将为security.client.protocol.acl.blocked

　　对于服务，可以同时指定访问控制列表和阻止的控制列表。如果用户在访问控制中而不在阻止的访问控制列表中，则该用户有权访问该服务。

　　如果未为服务定义阻止访问控制列表，则将应用security.service.authorization.default.acl.blocked的值。如果未定义security.service.authorization.default.acl.blocked，则将应用空的阻止访问控制列表。

五.控制HDFS管理访问

　　如前所述，在"${HADOOP_HOME}/etc/hadoop/hadoop-policy.xml"配置文件中指定ACL来配置服务级授权。

　　Hadoop可以限制一些管理操作，例如在HA安装程序中将NameNode的手动故障转移功能限制为仅HDFS集群管理员可以使用。可以设置为HDFS超级用户，但不必要。

　　可以在通过在"${HADOOP_HOME}/etc/hadoop/hdfs-site.xml"配置文件中设置"dfs.cluster.administrators"属性来配置HDFS集群管理员，如下所示：
　　　　<property>
　　　　　　<name>dfs.cluster.administrators</name>
　　　　　　<value>hdfs admin_group</value>
　　　　　　<description>指定管理员的ACL，此配置用于控制谁可以访问namenode中的默认servlet等。该值应为逗号分隔的用户和组列表。用户列表排在最前面，并用空格分隔，后跟组列表。此参数默认值为空，这意味着没有用户具有管理访问权限</description>
　　　　</property> 

六.刷新服务级授权配置

　　可以动态修改NameNode和ResourceManager的服务级配置，而无需重启守护程序。

　　可以使用"hdfs dfsadmin -refreshServiceAcl"和"yarn rmadmin -refreshServiceAcl"命令来刷新NameNode和ResourceManager的授权配置，如下所示。
　　　　[root@hadoop101.yinzhengjie.com ~]# hdfs dfsadmin -refreshServiceAcl　　　　　　
　　　　Refresh service acl successful
　　　　[root@hadoop101.yinzhengjie.com ~]# 
　　　　[root@hadoop101.yinzhengjie.com ~]# yarn rmadmin -refreshServiceAcl　　　　　　#服务必须启动才能刷新权限哈，否则会一直尝试，如下图所示。

七.博主推荐阅读

　　关于服务级别认证，官方也给出了很多案例，如果您在集群上打算使用它的话，建议先看看官方文档。下面是我使用Hadoop版本的官方文档。

　　博主推荐阅读:
　　　　http://hadoop.apache.org/docs/r2.10.0/hadoop-project-dist/hadoop-common/ServiceLevelAuth.html