zoukankan      html  css  js  c++  java
  • troubleshooting-执行Oozie调度Hive导数脚本抛java.io.IOException: output.properties data exceeds its limit [2048]

    执行Oozie调度Hive导数脚本抛java.io.IOException: output.properties data exceeds its limit [2048]

    原因分析

    shell脚本中一次提交的hql-mr作业量太大,其中包含的信息超过oozie launcher一次容许的最大值2K(2K是默认值)

    解决办法

    1)修改oozie-site.xml:
    <property>
    <name>oozie.action.max.output.data</name>
    <value>204800</value>
    </property>
    2)CDH集群中修改 oozie-site.xml 的 Oozie Server 高级配置代码段(安全阀),如图:

    参考:https://blog.csdn.net/abysscarry/article/details/82120855

    异常日志

    18/09/15 16:21:47 INFO CuratorFrameworkSingleton: Closing ZooKeeper client.
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
    18/09/15 16:21:58 INFO client.RMProxy: Connecting to ResourceManager at master.prodcdh.com/172.18.205.119:8032
    18/09/15 16:21:58 INFO client.RMProxy: Connecting to ResourceManager at master.prodcdh.com/172.18.205.119:8032
    Launcher AM execution failed
    java.io.IOException: output.properties data exceeds its limit [2048]
    at org.apache.oozie.action.hadoop.LocalFsOperations.getLocalFileContentAsString(LocalFsOperations.java:86)
    at org.apache.oozie.action.hadoop.LauncherAM.processActionData(LauncherAM.java:521)
    at org.apache.oozie.action.hadoop.LauncherAM.handleActionData(LauncherAM.java:501)
    at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:229)
    at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
    at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)
    Exception in thread "main" java.io.IOException: output.properties data exceeds its limit [2048]
    at org.apache.oozie.action.hadoop.LocalFsOperations.getLocalFileContentAsString(LocalFsOperations.java:86)
    at org.apache.oozie.action.hadoop.LauncherAM.processActionData(LauncherAM.java:521)
    at org.apache.oozie.action.hadoop.LauncherAM.handleActionData(LauncherAM.java:501)
    at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:229)
    at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
    at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)

  • 相关阅读:
    Atitit. 查找linux 项目源码位置
    Atitit.用户权限服务 登录退出功能
    Atitit.js javascript的rpc框架选型
    Atitit.php  nginx页面空白 并返回500的解决
    Atitit .linux 取回root 密码q99
    Atitit.报名模块的管理
    Atitit.基于时间戳的农历日历历法日期计算
    Atitit.excel导出 功能解决方案 php java C#.net版总集合.doc
    我的博客开通了
    (转)列举ASP.NET 页面之间传递值的几种方式
  • 原文地址:https://www.cnblogs.com/chwilliam85/p/9693201.html
Copyright © 2011-2022 走看看