zoukankan      html  css  js  c++  java
  • HDFS副本设置——默认3

    首先 dfs.replication这个参数是个client参数,即node level参数。需要在每台datanode上设置。
    其实默认为3个副本已经够用了,设置太多也没什么用。

    一个文件,上传到hdfs上时指定的是几个副本就是几个。以后你修改了副本数,对已经上传了的文件也不会起作用。可以再上传文件的同时指定创建的副本数
    Hadoop dfs -D dfs.replication=1 -put 70M logs/2

    可以通过命令来更改已经上传的文件的副本数:
    hadoop fs -setrep -R 3 /

    查看当前hdfs的副本数
    hadoop fsck -locations
    FSCK started by hadoop from /172.18.6.112 for path / at Thu Oct 27 13:24:25 CST 2011
    ....................Status: HEALTHY
    Total size: 4834251860 B
    Total dirs: 21
    Total files: 20
    Total blocks (validated): 82 (avg. block size 58954290 B)
    Minimally replicated blocks: 82 (100.0 %)
    Over-replicated blocks: 0 (0.0 %)
    Under-replicated blocks: 0 (0.0 %)
    Mis-replicated blocks: 0 (0.0 %)
    Default replication factor: 3
    Average block replication: 3.0
    Corrupt blocks: 0
    Missing replicas: 0 (0.0 %)
    Number of data-nodes: 3
    Number of racks: 1
    FSCK ended at Thu Oct 27 13:24:25 CST 2011 in 10 milliseconds
    The filesystem under path '/' is HEALTHY

    某个文件的副本数,可以通过ls中的文件描述符看到
    hadoop dfs -ls
    -rw-r--r-- 3 hadoop supergroup 153748148 2011-10-27 16:11 /user/hadoop/logs/201108/impression_witspixel2011080100.thin.log.gz


    如果你只有3个datanode,但是你却指定副本数为4,是不会生效的,因为每个datanode上只能存放一个副本。

    参考:http://blog.csdn.net/lskyne/article/details/8898666

  • 相关阅读:
    PAT甲题题解-1030. Travel Plan (30)-最短路+输出路径
    PAT甲题题解-1029. Median (25)-求两序列的中位数,题目更新了之后不水了
    PAT甲题题解-1028. List Sorting (25)-水排序
    BZOJ 1492 货币兑换Cash
    Codeforces 276D Little Girl and Maximum XOR
    Codeforces 526E Transmitting Levels
    Codeforces 335B Palindrome
    BZOJ 2527 Meteors
    Codeforces 449D Jzzhu and Numbers
    FJ省队集训DAY4 T3
  • 原文地址:https://www.cnblogs.com/bonelee/p/6485434.html
Copyright © 2011-2022 走看看