zoukankan      html  css  js  c++  java
  • Hadoop优先级调度

    当同时在集群中运行多个作业时,默认情况下,Hadoop将提交的作业放入一个FIFO,一个作业结束后,Hadoop就启动下一个作业。


    当一个运行时间长但是优先级较低的作业先于运行时间短而优先级较高的作业提交时,优先级高的作业会长时间排队等待。


    为了解决这个问题,Hadoop定义了5种不同的作业优先级,分别是:VERY_HIGH,HIGH,NORMAL,LOW,VERY_LOW,作业的默认优先级是NORMAL,可以通过$hadoop job -set-priority进行修改。

    例子:
    1.在集群中启动1个运行时间较长的作业
    caiyong@caiyong:/opt/hadoop$ bin/hadoop jar hadoop-examples-1.2.1.jar pi 2000  2000

    2.查看作业列表
    caiyong@caiyong:/opt/hadoop$ bin/hadoop job -list

    1 jobs currently running
    JobId                                     State    StartTime       UserName    Priority    SchedulingInfo
    job_201503171201_0003   1   1426565671593   caiyong        NORMAL            NA

    3.查看作业的运行状态
    caiyong@caiyong:/opt/hadoop$ bin/hadoop job -status job_201503171201_0003

    Job: job_201503171201_0003
    file: hdfs://127.0.0.1:8020/home/caiyong/tmp/mapred/staging/caiyong/.staging/job_201503171201_0003/job.xml
    tracking URL:http://localhost:50030/jobdetails.jsp?jobid=job_201503171201_0003
    map() completion: 0.012500001
    reduce() completion: 0.0

    Counters: 19
        Job Counters 
            SLOTS_MILLIS_MAPS=117080
            Launched map tasks=26
            Data-local map tasks=26
        File Input Format Counters 
            Bytes Read=2832
        FileSystemCounters
            HDFS_BYTES_READ=5870
            FILE_BYTES_WRITTEN=1316654
        Map-Reduce Framework
            Map output materializedbytes=672
            Map input records=24
            Spilled Records=48
            Map output bytes=432
            Total committed heap usage(bytes)=3815768064
            CPU time spent (ms)=9530
            Map input bytes=576
            SPLIT_RAW_BYTES=3038
            Combine input records=0
            Combine output records=0
            Physical memory (bytes)snapshot=4156928000
            Virtual memory (bytes) snapshot=9500446720
            Map output records=48


    4.把作业的优先级提高为VERY_HIGH
    caiyong@caiyong:/opt/hadoop$ bin/hadoop job -set-priority job_201503171201_0003    VERY_HIGH

    Changed job priority.

    5.查看更改后的作业优先级
    caiyong@caiyong:/opt/hadoop$ bin/hadoop job -list

    1 jobs currently running
    JobId                                     State    StartTime          UserName    Priority    SchedulingInfo
    job_201503171201_0003   1   1426565671593   caiyong      VERY_HIGH   NA


    6.强制结束正在运行的作业
    caiyong@caiyong:/opt/hadoop$ bin/hadoop job -kill job_201503171201_0003

    Killed job job_201503171201_0003

  • 相关阅读:
    今日头条Go建千亿级微服务的实践
    Apache JServ Protocol
    tomcat servlet JSP common gateway interface 公共网关接口
    pollable event-driven Flume source
    kernel.panic
    Linux查找含有某字符串的所有文件
    深入理解Linux修改hostname
    px em rem
    这跟他们的前辈在原生应用程序或印刷出版物中做的设计或多或少有点类似。
    字符串等长切分 视频字幕换行
  • 原文地址:https://www.cnblogs.com/peizhe123/p/5886338.html
Copyright © 2011-2022 走看看