zoukankan      html  css  js  c++  java
  • mr的logs的查看

    在map或者reduce函数中使用System.out.println打印的信息沾满查看呢?

    步骤1:启动history server

    /usr/local/hadoop-2.6.0/sbin/mr-jobhistory-daemon.sh start historyserver

    步骤2:使用http://hadoop26:19888/jobhistory来查看作业log

    意外情况:

    如果在查看日志的时候,发生了下面的情况

    在yarn-default.xml中有这样的一个配置

    <property>
    <description>Whether to enable log aggregation. Log aggregation collects
    each container's logs and moves these logs onto a file-system, for e.g.
    HDFS, after the application completes. Users can configure the
    "yarn.nodemanager.remote-app-log-dir" and
    "yarn.nodemanager.remote-app-log-dir-suffix" properties to determine
    where these logs are moved to. Users can access the logs via the
    Application Timeline Server.
    </description>
    <name>yarn.log-aggregation-enable</name>
    <value>false</value>
    </property>

     我们需要把这个值改为true,也就是在yarn-site.xml中再配置一遍

    配置完成后重启historyserver即可。

    现在来看看logs的输出吧:

  • 相关阅读:
    hiveserver2 with kerberos authentication
    python Basic usage
    python Quicksort demo
    Python HeapSort
    mrunit for wordcount demo
    CCDH证书
    Hadoop question list
    Hadoop Yarn core concepts
    Hadoop Resource
    Hadoop could not find or load main class
  • 原文地址:https://www.cnblogs.com/dongdone/p/5687626.html
Copyright © 2011-2022 走看看