zoukankan      html  css  js  c++  java
  • Hadoop(2): Blocks存储管理及读写

    1. Replication: 因为每个HDFS被部署在是低成本的商业硬件上(low cost commodity hardware),所以为了有更佳的Fault Tolerance,HDFS将每个Block备份存储。默认的Replication Factor=3.

    Note: The NameNode collects block report from DataNode periodically to maintain the replication factor. Therefore, whenever a block is over-replicated or under-replicated the NameNode deletes or add replicas as needed.

     

    2. Rack Awareness:Namenode会根据Rack Awareness算法来确保一个Block的3个Replica不在一个Rack上。

    3. HDFS Write:

    a. 建立pipline

     2. Streaming Data

    3.Shutdown and Acknoledgement:

    多Blocks的并行操作:

    For Block A: 1A -> 2A -> 3A -> 4A
    For Block B: 1B -> 2B -> 3B -> 4B -> 5B -> 6B

    HDFS Read:

    https://www.edureka.co/blog/apache-hadoop-hdfs-architecture/#datanode

    https://www.coursera.org/learn/big-data-essentials/lecture/JmzZr/block-and-replica-states-recovery-process-1

  • 相关阅读:
    javase 超市库存系统
    Javase 简单代码练习
    Javase 简单练习
    SQL表连接查询
    SQL多表查询
    SQL表查询
    SQL数据查询2
    SQL数据查询
    SQL增删改
    AutoCompleteTextView的使用
  • 原文地址:https://www.cnblogs.com/rhyswang/p/10442567.html
Copyright © 2011-2022 走看看