zoukankan html css js c++ java

Spark的Yarn运行模式部署实战案例

　　　　　　　　　　　Spark的Yarn运行模式部署实战案例

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

一.启动Hadoop集群

1>.修改yarn的配置文件

[root@hadoop101.yinzhengjie.org.cn ~]# vim /yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml 
[root@hadoop101.yinzhengjie.org.cn ~]# 
[root@hadoop101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml 
<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<configuration>

<!-- Site specific YARN configuration properties -->

    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
        <description>Reducer获取数据的方式</description>
    </property>

    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
        <description>启用resourcemanager的HA功能</description>
    </property>


    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>yinzhengjie-yarn-ha</value>
        <description>标识集群,以确保RM不会接替另一个群集的活动状态</description>
    </property>


    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm101,rm105</value>
        <description>ResourceManager的逻辑ID列表</description>
    </property>


    <property>
        <name>yarn.resourcemanager.hostname.rm101</name>
        <value>hadoop101.yinzhengjie.org.cn</value>
        <description>指定rm101逻辑别名对应的真实服务器地址,即您可以理解添加映射关系.</description>
    </property>

    <property>
        <name>yarn.resourcemanager.hostname.rm105</name>
        <value>hadoop105.yinzhengjie.org.cn</value>
        <description>指定rm105逻辑别名对应的真实服务器地址,即您可以理解添加映射关系.</description>
    </property>

    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>hadoop101.yinzhengjie.org.cn:2181,hadoop102.yinzhengjie.org.cn:2181,hadoop103.yinzhengjie.org.cn:2181</value>
        <description>指定zookeeper集群的地址</description>
    </property>

    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
        <description>启用自动故障转移(即自动恢复功能)</description>
    </property>

    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
        <description>指定resourcemanager的状态信息存储在zookeeper集群</description>
    </property>

    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
        <description>启用或禁用日志聚合的配置,默认为false,即禁用,将该值设置为true,表示开启日志聚集功能使能</description>
    </property>

    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
        <description>删除聚合日志前要保留多长时间(默认单位是秒),默认值是"-1"表示禁用,请注意,将此值设置得太小,您将向Namenode发送垃圾邮件.</description>
    </property>


    <property>
        <name>yarn.log-aggregation.retain-check-interval-seconds</name>
        <value>3600</value>
        <description>单位为秒,检查聚合日志保留之间的时间.如果设置为0或负值,那么该值将被计算为聚合日志保留时间的十分之一;请注意,将此值设置得太小,您将向名称节点发送垃圾邮件.</description>
    </property>

    <!-- 配置Spark ON YARN-->
     <property>
         <name>yarn.nodemanager.pmem-check-enabled</name>
         <value>false</value>
        <description>是否启动一个线程检查每个任务正使用的物理内存量，如果任务超出分配值，则直接将其杀掉，默认是true</description>
     </property>

     <property>
         <name>yarn.nodemanager.vmem-check-enabled</name>
         <value>false</value>
         <description>是否启动一个线程检查每个任务正使用的虚拟内存量，如果任务超出分配值，则直接将其杀掉，默认是true</description>
     </property>

</configuration>
[root@hadoop101.yinzhengjie.org.cn ~]#

[root@hadoop101.yinzhengjie.org.cn ~]# vim /yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml

[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml
******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml] *******
命令执行成功
******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml] *******
命令执行成功
******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml] *******
命令执行成功
******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml] *******
命令执行成功
******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml] *******
命令执行成功
[root@hadoop101.yinzhengjie.org.cn ~]# 
[root@hadoop101.yinzhengjie.org.cn ~]#

[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml　　　　　　　　#分发配置文件

2>.启动zookeeper集群

[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8239 Jps

hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8745 Jps

hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
9677 Jps

hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6486 Jps

hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8208 Jps

hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6730 Jps

[root@hadoop101.yinzhengjie.org.cn ~]#