zoukankan      html  css  js  c++  java
  • hadoop集群搭建

    安装

    masters:

    1台 NameNode

    1台 ResourceManager

    workers:

    剩下的机器既作为 DataNode 也作为 NodeManager

    以 Non-Secure Mode 模式安装

    需要为 hadoop daemon 配置运维环境和运行参数。

    HDFS daemons are NameNode, SecondaryNameNode, and DataNode. YARN daemons are ResourceManager, NodeManager, and WebAppProxy. If MapReduce is to be used, then the MapReduce Job History Server will also be running. For large installations, these are generally running on separate hosts.

    配置运行环境

    需要配置etc/hadoop/hadoop-env.sh、etc/hadoop/mapred-env.sh、etc/hadoop/yarn-env.sh,还要配置JAVA_HOME

    monEnvironment Variable
    NameNode HDFS_NAMENODE_OPTS
    DataNode HDFS_DATANODE_OPTS
    Secondary NameNode HDFS_SECONDARYNAMENODE_OPTS
    ResourceManager YARN_RESOURCEMANAGER_OPTS
    NodeManager YARN_NODEMANAGER_OPTS
    WebAppProxy YARN_PROXYSERVER_OPTS
    Map Reduce Job History Server MAPRED_HISTORYSERVER_OPTS

    配置运行参数

  • 相关阅读:
    etcd的原理分析
    (转)Linux sort命令
    随机森林
    python 类的定义和继承
    python random
    Spark源码阅读(1): Stage划分
    Mac 上安装MySQL
    Python 删除 数组
    在循环中将多列数组组合成大数组
    准确率 召回率
  • 原文地址:https://www.cnblogs.com/zhouj-happy/p/6478102.html
Copyright © 2011-2022 走看看