zoukankan      html  css  js  c++  java
  • MapReduce 计数器

    原生的计数器有(格式:<显示名> | <内部名>):

    FileSystemCounters | FileSystemCounters
    ---FILE_BYTES_READ | FILE_BYTES_READ
    ---FILE_BYTES_WRITTEN | FILE_BYTES_WRITTEN
    ---HDFS_BYTES_READ | HDFS_BYTES_READ
    ---HDFS_BYTES_WRITTEN | HDFS_BYTES_WRITTEN

    Job Counters  | org.apache.hadoop.mapred.JobInProgress$Counter
    ---Total time spent by all maps waiting after reserving slots (ms) | FALLOW_SLOTS_MILLIS_MAPS
    ---Total time spent by all reduces waiting after reserving slots (ms) | FALLOW_SLOTS_MILLIS_REDUCES
    ---Rack-local map tasks | RACK_LOCAL_MAPS
    ---SLOTS_MILLIS_MAPS | SLOTS_MILLIS_MAPS
    ---SLOTS_MILLIS_REDUCES | SLOTS_MILLIS_REDUCES
    ---Launched map tasks | TOTAL_LAUNCHED_MAPS
    ---Launched reduce tasks | TOTAL_LAUNCHED_REDUCES

    Map-Reduce Framework | org.apache.hadoop.mapred.Task$Counter
    ---Combine input records | COMBINE_INPUT_RECORDS
    ---Combine output records | COMBINE_OUTPUT_RECORDS
    ---Total committed heap usage (bytes) | COMMITTED_HEAP_BYTES
    ---CPU time spent (ms) | CPU_MILLISECONDS
    ---Map input records | MAP_INPUT_RECORDS
    ---Map output bytes | MAP_OUTPUT_BYTES
    ---Map output materialized bytes | MAP_OUTPUT_MATERIALIZED_BYTES
    ---Map output records | MAP_OUTPUT_RECORDS
    ---Physical memory (bytes) snapshot | PHYSICAL_MEMORY_BYTES
    ---Reduce input groups | REDUCE_INPUT_GROUPS
    ---Reduce input records | REDUCE_INPUT_RECORDS
    ---Reduce output records | REDUCE_OUTPUT_RECORDS
    ---Reduce shuffle bytes | REDUCE_SHUFFLE_BYTES
    ---Spilled Records | SPILLED_RECORDS
    ---SPLIT_RAW_BYTES | SPLIT_RAW_BYTES
    ---Virtual memory (bytes) snapshot | VIRTUAL_MEMORY_BYTES

    File Input Format Counters  | org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter
    ---Bytes Read | BYTES_READ

    File Output Format Counters  | org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter
    ---Bytes Written | BYTES_WRITTEN

    1. 在Map或Reduce中使用:

    增加某个计数器(一般用于自定义的计数器):context.getCounter(<groupName>, <counterName>).increment(<incr>);

    获取某个计数器的值:context.getCounter(<groupName>, <counterName>).getValue();

    2. 在Map或Reduce之外使用:

    job.getCounters().findCounter(groupName, counterName)

    3. Map output bytes | MAP_OUTPUT_BYTES 和 Reduce output records | REDUCE_OUTPUT_RECORDS 记录的是调用Context.write(<Name>, <K>, <V>)的次数(这是新版API, 至于旧版API使用OutputCollector.collect(<K>, <V>)不清楚),对于调用MultipleOutputs.write(<K>, <V>)写出的记录并不记录

  • 相关阅读:
    vue生命周期过程做了什么
    css_css3_实用属性_随时补充更新
    echarts的symbol引用本地图片写法
    无废话设计模式(1)--简单工厂、工厂方法、抽象工厂
    JavaWeb--Maven学习
    SpringCloud Alibaba实战 -引入服务网关Gateway
    从ReentrantLock看AQS (AbstractQueuedSynchronizer) 运行流程 抽象的队列式同步器
    架构的搭建(一)SpringCloud Alibaba
    配置中心之Nacos简介,使用及Go简单集成
    RabbitMQ
  • 原文地址:https://www.cnblogs.com/aprilrain/p/2747787.html
Copyright © 2011-2022 走看看