zoukankan      html  css  js  c++  java
  • hadoop 结构的变化

    http://hadoop.apache.org/docs/r2.4.1/

    Apache Hadoop 2.4.1

    Apache Hadoop 2.4.1 consists of significant improvements over the previous stable release (hadoop-1.x).

    Here is a short overview of the improvments to both HDFS and MapReduce.

    • HDFS Federation

      In order to scale the name service horizontally, federation uses multiple independent Namenodes/Namespaces. The Namenodes are federated, that is, the Namenodes are independent and don't require coordination with each other. The datanodes are used as common storage for blocks by all the Namenodes. Each datanode registers with all the Namenodes in the cluster. Datanodes send periodic heartbeats and block reports and handles commands from the Namenodes.

      More details are available in the HDFS Federation document.

    • MapReduce NextGen aka YARN aka MRv2

      The new architecture introduced in hadoop-0.23, divides the two major functions of the JobTracker: resource management and job life-cycle management into separate components.

      The new ResourceManager manages the global assignment of compute resources to applications and the per-application ApplicationMaster manages the application‚ scheduling and coordination.

      An application is either a single job in the sense of classic MapReduce jobs or a DAG of such jobs.

      The ResourceManager and per-machine NodeManager daemon, which manages the user processes on that machine, form the computation fabric.

      The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks.

      More details are available in the YARN document.

    Getting Started

    The Hadoop documentation includes the information you need to get started using Hadoop. Begin with the Single Node Setup which shows you how to set up a single-node Hadoop installation. Then move on to the Cluster Setup to learn how to set up a multi-node Hadoop installation.

  • 相关阅读:
    es集群的调优2
    es集群中kibana和es集群的高可用设置
    es集群中参数参数discovery.zen.minimum_master_nodes深度解析
    哔哩哔哩适合后端编程人员的elasticsearch快速实战教程学习总结
    Vue2-基本语句
    SpringBoot-ElasticSearch初使用
    Java-SSO单点登录的3种方式【待完善】
    Java-学习日记(函数式编程与@ControllerAdvice)
    SpringBoot-内置Tomcat启动原理
    Java-学习日记(Atomic,Volatile)
  • 原文地址:https://www.cnblogs.com/QAZLIU/p/7850094.html
Copyright © 2011-2022 走看看