zoukankan      html  css  js  c++  java
  • Local模式下Spark程序只输出关键信息

    使用spark-submit提交local任务时,会输出很多Info信息:

    -------------------------------------------
    Time: 1495788633000 ms
    -------------------------------------------
    
    17/05/26 08:50:33 INFO scheduler.JobScheduler: Finished job streaming job 1495788633000 ms.2 from job set of time 1495788633000 ms
    17/05/26 08:50:33 INFO scheduler.JobScheduler: Total delay: 0.054 s for time 1495788633000 ms (execution: 0.046 s)
    17/05/26 08:50:33 INFO rdd.MapPartitionsRDD: Removing RDD 177 from persistence list
    17/05/26 08:50:33 INFO storage.BlockManager: Removing RDD 177
    17/05/26 08:50:33 INFO rdd.MapPartitionsRDD: Removing RDD 176 from persistence list
    17/05/26 08:50:33 INFO storage.BlockManager: Removing RDD 176
    17/05/26 08:50:33 INFO rdd.BlockRDD: Removing RDD 175 from persistence list
    17/05/26 08:50:33 INFO dstream.SocketInputDStream: Removing blocks of RDD BlockRDD[175] at socketTextStream at CoGroupTest.scala:21 of time 1495788633000 ms
    17/05/26 08:50:33 INFO storage.BlockManager: Removing RDD 175
    17/05/26 08:50:33 INFO rdd.MapPartitionsRDD: Removing RDD 178 from persistence list
    17/05/26 08:50:33 INFO storage.BlockManager: Removing RDD 178
    17/05/26 08:50:33 INFO rdd.MapPartitionsRDD: Removing RDD 180 from persistence list
    17/05/26 08:50:33 INFO storage.BlockManager: Removing RDD 180
    17/05/26 08:50:33 INFO scheduler.ReceivedBlockTracker: Deleting batches: 1495788631000 ms
    17/05/26 08:50:33 INFO scheduler.InputInfoTracker: remove old batch metadata: 1495788631000 ms
    17/05/26 08:50:34 INFO scheduler.JobScheduler: Added jobs for time 1495788634000 ms
    17/05/26 08:50:34 INFO scheduler.JobScheduler: Starting job streaming job 1495788634000 ms.0 from job set of time 1495788634000 ms

      可以修改log4j的日志级别,只输出关键信息:

    1.修改$SPARK_HOME/conf/log4j.properties

      如果有log4j.properties.template,则复制一份为log4j.properties

      cp log4j.properties.template log4j.properties

    2.修改第一行

      log4j.rootCategory=INFO, console

      为

      log4j.rootCategory=ERROR, console

    3.再次提交任务,只会输出关键信息:

    -------------------------------------------
    Time: 1495788788000 ms
    -------------------------------------------
    (helloworld,helloworld_one)
    (hello,hello_one)
    (join,join_one)
  • 相关阅读:
    左右下划线,中间文字
    sql语言动词
    SQL语言的四个组成部分
    MySQL执行一条查询语句的内部执行过程
    07 | 行锁功过:怎么减少行锁对性能的影响?
    06 | 全局锁和表锁 :给表加个字段怎么有这么多阻碍?
    05 | 深入浅出索引(下)
    04 | 深入浅出索引(上)
    03 | 事务隔离:为什么你改了我还看不见?
    02 | 日志系统:一条SQL更新语句是如何执行的?
  • 原文地址:https://www.cnblogs.com/liugh/p/6909531.html
Copyright © 2011-2022 走看看