zoukankan      html  css  js  c++  java
  • dfs.replication 参数 动态修改

    首先 dfs.replication这个参数是个client参数,即node level参数。需要在每台datanode上设置。
    其实默认为3个副本已经够用了,设置太多也没什么用。

    一个文件,上传到hdfs上时指定的是几个副本就是几个。以后你修改了副本数,对已经上传了的文件也不会起作用。可以再上传文件的同时指定创建的副本数
    hadoop dfs -D dfs.replication=1 -put 70M logs/2

    可以通过命令来更改已经上传的文件的副本数:
    hadoop fs -setrep -R 3 /

    查看当前hdfs的副本数
    hadoop fsck -locations
    FSCK started by hadoop from /172.18.6.112 for path / at Thu Oct 27 13:24:25 CST 2011
    ....................Status: HEALTHY
    Total size: 4834251860 B
    Total dirs: 21
    Total files: 20
    Total blocks (validated): 82 (avg. block size 58954290 B)
    Minimally replicated blocks: 82 (100.0 %)
    Over-replicated blocks: 0 (0.0 %)
    Under-replicated blocks: 0 (0.0 %)
    Mis-replicated blocks: 0 (0.0 %)
    Default replication factor: 3
    Average block replication: 3.0
    Corrupt blocks: 0
    Missing replicas: 0 (0.0 %)
    Number of data-nodes: 3
    Number of racks: 1
    FSCK ended at Thu Oct 27 13:24:25 CST 2011 in 10 milliseconds
    The filesystem under path '/' is HEALTHY

    某个文件的副本数,可以通过ls中的文件描述符看到
    hadoop dfs -ls
    -rw-r--r-- 3 hadoop supergroup 153748148 2011-10-27 16:11 /user/hadoop/logs/201108/impression_witspixel2011080100.thin.log.gz


    如果你只有3个datanode,但是你却指定副本数为4,是不会生效的,因为每个datanode上只能存放一个副本。
    hadoop fsck -locations 可以看到相应的提示信息,可以看到副本丢失率为33.33%:
    /user/hadoop/logs/test.log: Under replicated blk_-45151128047308146_1147. Target Replicas is 4 but found 3 replica(s).
    Status: HEALTHY
    Total size: 4834251860 B
    Total dirs: 21
    Total files: 20
    Total blocks (validated): 82 (avg. block size 58954290 B)
    Minimally replicated blocks: 82 (100.0 %)
    Over-replicated blocks: 0 (0.0 %)
    Under-replicated blocks: 82 (100.0 %)
    Mis-replicated blocks: 0 (0.0 %)
    Default replication factor: 3
    Average block replication: 3.0
    Corrupt blocks: 0
    Missing replicas: 82 (33.333332 %)
    Number of data-nodes: 3
    Number of racks: 1
    FSCK ended at Thu Oct 27 13:22:14 CST 2011 in 12 milliseconds

  • 相关阅读:
    GridControl 绑定非绑定列
    AutoMapper
    ABP-TaskEver 路线图
    Angular--页面间切换及传值的四种方法
    知行合一,止于至善
    AutoFac 依赖注入--问题集
    【入门】依赖注入(DI)-开始
    利用JAVA生成二维码
    sqlserver如何关闭死锁进程.
    Hadoop/HBase 配置snappy压缩
  • 原文地址:https://www.cnblogs.com/zhangeamon/p/5752636.html
Copyright © 2011-2022 走看看