zoukankan      html  css  js  c++  java
  • hadoop完全分布式安装部署-笔记

    规划:

    [hadoop@db01 ~]$ cat /etc/hosts
    127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

    192.168.100.231         db01        #(namenode,datanode,JobHistoryServer,nodemanager)
    192.168.100.232         db02        #(resourcemanager,datanode,nodemanager)
    192.168.100.233         db03        #(SecondaryNameNode,datanode,nodemanager)
    192.168.100.234         db04        #(nodemanager,datanode)
    192.168.100.235         db05        #(nodemanager,datanode)


    配置文件:

    [hadoop@db01 hadoop-2.5.0]$ cat etc/hadoop/core-site.xml
    <?xml version="1.0" encoding="UTF-8"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <!--
      Licensed under the Apache License, Version 2.0 (the "License");
      you may not use this file except in compliance with the License.
      You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      See the License for the specific language governing permissions and
      limitations under the License. See accompanying LICENSE file.
    -->

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://db01:9000</value>
        </property>

            <property>
                    <name>hadoop.tmp.dir</name>
                    <value>/usr/local/hadoop-2.5.0/data/tmp</value>
            </property>

            <property>
                    <name>fs.trash.interval</name>
                    <value>7000</value>
            </property>
    </configuration>

    [hadoop@db01 hadoop-2.5.0]$ cat etc/hadoop/hdfs-site.xml
    <?xml version="1.0" encoding="UTF-8"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <!--
      Licensed under the Apache License, Version 2.0 (the "License");
      you may not use this file except in compliance with the License.
      You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      See the License for the specific language governing permissions and
      limitations under the License. See accompanying LICENSE file.
    -->

    <!-- Put site-specific property overrides in this file. -->

    <configuration>

            <property>
                    <name>dfs.namenode.secondary.http-address</name>
                    <value>db03:50090</value>
            </property>

    </configuration>

    [hadoop@db01 hadoop-2.5.0]$ cat etc/hadoop/yarn-site.xml
    <?xml version="1.0"?>
    <!--
      Licensed under the Apache License, Version 2.0 (the "License");
      you may not use this file except in compliance with the License.
      You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      See the License for the specific language governing permissions and
      limitations under the License. See accompanying LICENSE file.
    -->
    <configuration>

        <property>
                <name>yarn.nodemanager.aux-services</name>
                    <value>mapreduce_shuffle</value>
        </property>

            <property>
                    <name>yarn.resourcemanager.hostname</name>
                    <value>db02</value>
            </property>

        <property>
                    <name>yarn.log-aggregation-enable</name>
                    <value>true</value>
            </property>

            <property>
                    <name>yarn.log-aggregation.retain-seconds</name>
                    <value>600000</value>
            </property>
    </configuration>

    [hadoop@db01 hadoop-2.5.0]$ cat etc/hadoop/mapred-site.xml
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <!--
      Licensed under the Apache License, Version 2.0 (the "License");
      you may not use this file except in compliance with the License.
      You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      See the License for the specific language governing permissions and
      limitations under the License. See accompanying LICENSE file.
    -->

    <!-- Put site-specific property overrides in this file. -->

    <configuration>

        <property>
                <name>mapreduce.framework.name</name>
                    <value>yarn</value>
        </property>

            <property>
                    <name>mapreduce.jobhistory.address</name>
                    <value>db01:10020</value>
            </property>

            <property>
                    <name>mapreduce.jobhistory.webapp.address</name>
                    <value>db01:19888</value>
            </property>

    </configuration>

    [hadoop@db01 hadoop-2.5.0]$ cat etc/hadoop/slaves
    db01
    db02
    db03
    db04
    db05

    并且在(etc/hadoop/hadoop-env.sh  etc/hadoop/mapred-env.sh  etc/hadoop/yarn-env.sh)文件中配置相应的JAVA_HOME信息。

  • 相关阅读:
    Kafka系列之-Kafka入门
    2018个人年终总结
    javascript 模块化学习:Class打包出来无法实例化问题
    [SqlServer] Error: 15023
    [Auto Testing] 工具准备:Selenium 与 ChromeDriver
    [Python] 模拟登录网站(。。为了之后操作数据。。)
    [Angular 6] 初学angular,环境全部最新,[ ng serve ] 不能启动,卡在 95% 不动 => 解决方案
    [nuget]VS中包管理器打开后找不到其它工程的问题
    当需要给<span>标签设宽度
    关于近期打算
  • 原文地址:https://www.cnblogs.com/wcwen1990/p/6652073.html
Copyright © 2011-2022 走看看