zoukankan      html  css  js  c++  java
  • hadoop挂载多硬盘,ZZ-- multiple disks per node

    hadoop挂载多硬盘 ...multiple disks per node

     multiple disks per node Read more at: http://www.queryhome.com/24784/how-to-set-hadoop-tmp-dir-if-i-have-multiple-disks-per-node
     multiple disks per node Read more at: http://www.queryhome.com/24784/how-to-set-hadoop-tmp-dir-if-i-have-multiple-disks-per-node

    http://blog.sina.com.cn/s/blog_b88e09dd01013rd4.html

    Ubuntu - 硬盘分区、格式化、自动挂载配置 | Hard disk add new partition, format, auto mount in ubuntu 

    http://aofengblog.blog.163.com/blog/static/6317021201101502540117/

    http://my.oschina.net/leejun2005/blog/290073

    proper-care-and-feeding-of-drives-in-a-hadoop-cluster-a-conversation-with-stackiqs-dr-bruno

    http://hortonworks.com/blog/proper-care-and-feeding-of-drives-in-a-hadoop-cluster-a-conversation-with-stackiqs-dr-bruno/

    Utilizing-multiple-hard-disks-for-hadoop-HDFS

    http://lucene.472066.n3.nabble.com/Utilizing-multiple-hard-disks-for-hadoop-HDFS-td3553851.html

    =================

    First, Hadoop requires at least two locations for storing it’s files: mapred.local.dir, where MapReduce stores intermediary files,

    and dfs.data.dir, where HDFS stores the HDFS data (there are other locations as well, like hadoop.tmp.dir, where Hadoop and components stores its temporary data).  

    Both of them can cover multiple partitions.  

    While the two locations can be placed on physically different partitions, Cloudera recommends to configure them across the same set of partitions to maximize disk-level parallelism (this might not be an issue if the number of disk is much larger than the number of cores).

    ==

    Hadoop多磁碟設定(ubuntu)
    Hadoop預設只會使用hadoop資料夾的那個磁碟,然而要使用到所有的磁碟必須要做額外的設定,才可以讓HDFS使用到。設定步驟如下:
    1. 設定新磁碟的權限為777(也許不用這麼高),讓Hadoop擁有讀寫的權限
       sudo chmod -c 777 /media/diskName
    2. 修改conf資料夾內的hdfs-site.xml文件,在<configuration></configuration>之間加入property
    <property>
    <name>dfs.data.dir</name>  
    <value> ~/dfs/data,media/diskName/dfs/data</value>
    </property>

    多磁碟間的路徑使用半形逗點分隔,並請確定路徑是否正確

    最後重新啟動HDFS
    ============================
    http://girishkathalagiri.blogspot.com/2012/09/adding-disk-to-hadoop-data-nodesrepeat.html
    ==================
    You need to apply comma-separated lists only to
    dfs.data.dir (HDFS) and mapred.local.dir (MR) directly.
     
    Make sure the subdirectories are different for each, else you may accidentally wipe away your data when you restart MR services.

    The hadoop.tmp.dir property does not accept multiple paths and you should avoid using it in production - its more of a utility property that acts as a default base path for other properties.
    stop-dfs.sh
    start-dfs.sh
  • 相关阅读:
    JDK源码学习之 java.util.concurrent.automic包
    本地工程引入maven工程的配置方式
    JAVA配置&注解方式搭建简单的SpringMVC前后台交互系统
    Spring常用注解简析
    HTTP/HTTPS GET&POST两种方式的实现方法
    【JS语法】作用域与绑定图示
    【编程范式】函数式基础图示
    【技术思维】写好你的博客
    在Nginx服务器上用Jenkins发布Vue/React代码的步骤
    React事件绑定总结
  • 原文地址:https://www.cnblogs.com/GrantYu/p/4106910.html
Copyright © 2011-2022 走看看