zoukankan      html  css  js  c++  java
  • Sqoop export hive to oracle number format issue resolution

    Problem:

    java.lang.NumberFormatException
            at java.math.BigDecimal.<init>(BigDecimal.java:459)
            at java.math.BigDecimal.<init>(BigDecimal.java:728)
            at CDR_D_DETAIL_LUC.__loadFromFields(CDR_D_DETAIL_LUC.java:9803)
            at CDR_D_DETAIL_LUC.parse(CDR_D_DETAIL_LUC.java:9630)
            at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:77)
            at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:36)
            at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
            at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:183)
            at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:811)
            at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
            at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:396)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1113)
            at org.apache.hadoop.mapred.Child.main(Child.java:260)

    Analyisis:
    根据at CDR_D_DETAIL_LUC.parse(CDR_D_DETAIL_LUC.java:9630)这一行,知道是那个字段出了问题,然后检查hive数据源,发现这个字段是\N,是hive中默认的空值。问题的根源在于sqoop将\N当作了一个数值,在解析的时候出现格式异常。

    Solution:
    在sqoop export中指定空串
    sqoop export --connect jdbc:oracle:thin:@192.168.40.119:1521:SPTDI --username CDMA --password sptdi --table CDR_D_DETAIL_LUC --export-dir /user/hive/warehouse/pcmd_1x_hive_etl --input-fields-terminated-by ';' --input-null-string '\\N' --input-null-non-string '\\N' -m 1
    注意,空串变成了\\N 而不是\N,原因在于sqoop会将''中的字符串按字面量传给java,而java中字符串'\N'是非法的,要将\进行转义,所以,应该再加上一个\

  • 相关阅读:
    2-括号配对问题
    14-会场安排问题
    106-背包问题
    12-喷水装置
    HDU-5170
    HDU-1002
    贪吃蛇
    frame与bounds的区别
    MAC下Android的Eclipse开发环境的搭建
    有些人脸上太多的笑是因为他们心中有太多的泪
  • 原文地址:https://www.cnblogs.com/littlesuccess/p/2670014.html
Copyright © 2011-2022 走看看