zoukankan      html  css  js  c++  java
  • Hadoop(2): Blocks存储管理及读写

    1. Replication: 因为每个HDFS被部署在是低成本的商业硬件上(low cost commodity hardware),所以为了有更佳的Fault Tolerance,HDFS将每个Block备份存储。默认的Replication Factor=3.

    Note: The NameNode collects block report from DataNode periodically to maintain the replication factor. Therefore, whenever a block is over-replicated or under-replicated the NameNode deletes or add replicas as needed.

     

    2. Rack Awareness:Namenode会根据Rack Awareness算法来确保一个Block的3个Replica不在一个Rack上。

    3. HDFS Write:

    a. 建立pipline

     2. Streaming Data

    3.Shutdown and Acknoledgement:

    多Blocks的并行操作:

    For Block A: 1A -> 2A -> 3A -> 4A
    For Block B: 1B -> 2B -> 3B -> 4B -> 5B -> 6B

    HDFS Read:

    https://www.edureka.co/blog/apache-hadoop-hdfs-architecture/#datanode

    https://www.coursera.org/learn/big-data-essentials/lecture/JmzZr/block-and-replica-states-recovery-process-1

  • 相关阅读:
    安卓笔记20170117
    android笔记20170116
    meta 标签的作用
    SASS 初学者入门
    JQuery selector
    浅谈js回调函数
    自己写的jquery 弹框插件
    魔兽种子
    html页面的CSS、DIV命名规则
    各种弹框素材的链接
  • 原文地址:https://www.cnblogs.com/rhyswang/p/10442567.html
Copyright © 2011-2022 走看看