zoukankan      html  css  js  c++  java
  • 排查sqoop报错:Error running child : java.lang.OutOfMemoryError: Java heap space

    报错栈:

    2017-06-16 19:50:51,002 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: 1=1 AND 1=1
    2017-06-16 19:50:51,043 INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Working on split: 1=1 AND 1=1
    2017-06-16 19:50:51,095 INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select "EXTEND3","EXTEND2","EXTEND1","MEMO","OPER_DATE","OPER_CODE","FILE_CONTENT","FILE_NAME","INPATIENT_NO","ID" from HIS_SDZL."MDT_FILE" tbl where ( 1=1 ) AND ( 1=1 )
    2017-06-16 20:00:22,170 INFO [Thread-13] org.apache.sqoop.mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
    2017-06-16 20:00:22,185 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:514)
        at java.lang.StringBuffer.append(StringBuffer.java:352)
        at java.util.regex.Matcher.appendReplacement(Matcher.java:888)
        at java.util.regex.Matcher.replaceAll(Matcher.java:955)
        at java.lang.String.replaceAll(String.java:2223)
        at QueryResult.readFields(QueryResult.java:205)
        at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
        at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
        at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
        at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

    调小fetchsize参数也不能解决,那问题很可能是某行数据占用空间很大。根据Sqoop生成的导入表对应的实例化类QueryResult.java的244行可定位到报错列是FILE_CONTENT,是个二进制列, 然后查询原库,果然最大的列长达到180M:

    ps: 怎么用标准的sql语句查询 blob字段的大小?
    blob字段有好多种。如果是9i的简单的blob字段则应该是 length,或者lengthb也可。实在不行可以用 dbms_lob.getlength()

  • 相关阅读:
    一个最简单的例子学习SAP Cloud for Customer HTML mashup
    SAP C4C Embedded Component最常见的故障原因分析
    使用SAP CRM Mock framework进行单元测试
    ABAP Debugging Script(调试器脚本)使用的一些实际例子
    SAP ABAP字符和字符串变量隐式转换的一些规则
    两种使用代码获得SAP CRM product sales status的办法
    扎根CNCF社区贡献五年是怎样的体验?听听华为云原生开源团队的负责人怎么说
    LiteOS调测利器:backtrace函数原理知多少
    微服务容错时,这些技术你要立刻想到
    Volcano 监控设计解读,一看就懂
  • 原文地址:https://www.cnblogs.com/aprilrain/p/7118935.html
Copyright © 2011-2022 走看看