zoukankan      html  css  js  c++  java
  • 根据硬件配置后mapred-site.xml和yarn-site.xml

    机器总共16G内存,sqoop导入数据时大约需要2G左右

    mapred-site.xml

    <configuration>
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
    
        <property>
            <name>mapreduce.jobhistory.address</name>
            <value>master:10020</value>
            <description>MapReduce JobHistory Server IPC host:port</description>
        </property>
    
        <property>
            <name>mapreduce.jobhistory.webapp.address</name>
            <value>master:19888</value>
            <description>MapReduce JobHistory Server Web UI host:port</description>
    
        </property>
        <property>
            <name>mapred.child.java.opts</name>
            <value>-Xmx3000m -Xmx3000m -XX:-UseGCOverheadLimit</value>
        </property>
    
        <property>
            <name>mapreduce.reduce.memory.mb</name>
            <value>4096</value>
        </property>
    
        <property>
            <name>mapreduce.map.memory.mb</name>
            <value>4096</value>
    
        </property>
        <property>
            <name>mapreduce.task.timeout</name>
            <value>600000</value>
        </property>
    
    </configuration>

    yarn-site.xml

    
    <configuration>
        <property>
            <name>yarn.resourcemanager.hostname</name>
            <value>master</value>
        </property>
    
    
        <property>
            <description>The address of the applications manager interface in the RM.</description>
            <name>yarn.resourcemanager.address</name>
            <value>${yarn.resourcemanager.hostname}:8032</value>
        </property>
    
    
        <property>
            <description>The address of the scheduler interface.</description>
            <name>yarn.resourcemanager.scheduler.address</name>
            <value>${yarn.resourcemanager.hostname}:8030</value>
        </property>
    
    
        <property>
            <description>The http address of the RM web application.</description>
            <name>yarn.resourcemanager.webapp.address</name>
            <value>${yarn.resourcemanager.hostname}:8088</value>
        </property>
    
    
        <property>
            <description>The https adddress of the RM web application.</description>
            <name>yarn.resourcemanager.webapp.https.address</name>
            <value>${yarn.resourcemanager.hostname}:8090</value>
        </property>
    
    
        <property>
            <name>yarn.resourcemanager.resource-tracker.address</name>
            <value>${yarn.resourcemanager.hostname}:8031</value>
        </property>
    
    
        <property>
            <description>The address of the RM admin interface.</description>
            <name>yarn.resourcemanager.admin.address</name>
            <value>${yarn.resourcemanager.hostname}:8033</value>
        </property>
    
    
        <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
    
        <property>
            <name>yarn.scheduler.maximum-allocation-mb</name>
            <value>9216</value>
            <discription>每个任务最多可用内存,单位MB,默认8182MB</discription>
        </property>
    
        <property>
            <name>yarn.nodemanager.vmem-pmem-ratio</name>
            <value>2.1</value>
        </property>
    
        <property>
            <name>yarn.nodemanager.resource.memory-mb</name>
            <value>10240</value>
            <discription>该节点总共可用内存</discription>
        </property>
      <property>
        <name>yarn.app.mapreduce.am.resource.mb</name>
        <value>3000</value>
        <discription>am管理任务需要的内存</discription>
      </property>
    </configuration>
  • 相关阅读:
    Java method Exception throw with return instance.
    SQL Profiler Practice.
    FTP 文件 循环 Copy
    【SQL遗补】之 SET NOCOUNT (TransactSQL)
    【WinForm窗体控件开发】之三 窗体控件设计时属性Attribute
    【WinForm窗体控件开发】之三续 窗体控件设计时的事件属性
    【WinForm窗体控件开发】之二 简单的窗体控件
    解决删除DataGridView中数据引发的“DataGridView Default Error Dialog 错误”
    .NET开发WinCE程序之使用软键盘(System.WindowsCE.Forms命名空间)兼容WinCE和桌面操作系统之解决方案
    【C#遗补】之Char.IsDigit和Char.IsNumber的区别
  • 原文地址:https://www.cnblogs.com/ggzone/p/10121168.html
Copyright © 2011-2022 走看看