zoukankan      html  css  js  c++  java
  • hadoop 动态调整mapred参数

    bin/hadoop job -set-reduce-capacity job_20151126032920_1142443 1000        调成map数
    bin/hadoop job -set-priority job_20151126032920_1160549 VERY_HIGH          调整优先级
    bin/hadoop job -set-map-capacity job_20151126032920_1142443 1000



    --------------------------------------:$ bin/hadoop job                                                  
    Usage: JobClient <command> <args>
            [-submit <job-file>]
            [-status <job-id>]
            [-info <job-id>]
            [-counter <job-id> <group-name> <counter-name>]
            [-kill <job-id>]
            [-abort <job-id>]
            [-suspend <job-id> [hours]]
            [-recover <job-id> [-force] [-jobconf name=value] [-file local-path] [-cacheArchive] [-cacheFile]]
            [-set-priority <job-id> <priority>]. Valid values for priorities are: VERY_HIGH HIGH NORMAL LOW VERY_LOW
            [-set-map-capacity <job-id> <map-capacity> [ <vertex-id> ] ]
            [-set-reduce-capacity <job-id> <reduce-capacity> [ <vertex-id> ] ]
            [-set-map-over-capacity <job-id> <true/false> [ <vertex-id> ] ]
            [-set-reduce-over-capacity <job-id> <true/false> [ <vertex-id> ] ]
            [-events <job-id> <from-event-#> <#-of-events>]
            [-history <jobOutputDir>]
            [-list [all]]
            [-kill-task <task-id>]
            [-fail-task <task-id>]
            [-schedule-task <task-id>]
            [-input-add <job-id> <input>  [ <vertex-id> ] ]
            [-input-done <job-id>  [ <vertex-id> ] ]
            [-reload-limit-conf]
            [-idle-failover <job-id>]
    Generic options supported are
    -conf <configuration file>     specify an application configuration file
    -D <property=value>            use value for given property
    -fs <local|namenode:port>      specify a namenode
    -jt <local|jobtracker:port>    specify a job tracker
    -files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
    -libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
    -archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

    The general command line syntax is
    bin/hadoop command [genericOptions] [commandOptions]


  • 相关阅读:
    mysql常用基本命令
    mysql8.0.13下载与安装图文教程
    k8s ingress 增加跨域配置
    Jenkins 备份恢复插件 thinBackup 使用
    k8s HA master 节点宕机修复
    nginx 跨域问题解决
    mongodb 3.4.24 主从复制
    k8s 线上安装 jenkins并结合 jenkinsfile 实现 helm 自动化部署
    k8s helm 运用与自建helm仓库chartmuseum
    centos6 源码安装 unzip
  • 原文地址:https://www.cnblogs.com/i80386/p/5085031.html
Copyright © 2011-2022 走看看