zoukankan      html  css  js  c++  java
  • HUE中Oozie执行Sqoop

    Oozie执行Sqoop,传入参数(注意,在使用--query时,参数中一定不要带有空格,否则出错)
    1. 新建一个workflow

    2. 拖入一个sqoop

    3. sqoop抽取命令如下(建议先在命令行试下,因为oozie的错误提示很委婉,基本看不出错误原因)

    sqoop import --connect jdbc:mysql://localhost:3306/spider_new --username root --password 1234qwer --target-dir /user/fengz/brand/spider_data/amac/amac_fund_${date} --delete-target-dir --fields-terminated-by '`' --query select * from amac_fund where update_time between '${start_time}' and '${end_time}' and $CONDITIONS -m 1


    这是在命令行可执行的命令,其中有三个动态参数,${date}、${start_time}、${end_time},此处做了一个import的导入操作,把指定数据抽取到HDFS路径下,用了--query写SQL语句抽取想要的数据,此处需要注意,如果使用--query,必须要带$CONDITIONS,否则执行失败。
    4. 填入oozie-sqoop中,需要填入的内容如下

    import --connect jdbc:mysql://localhost:3306/spider_new --username root --password 1234qwer --target-dir /user/fengz/brand/spider_data/amac/amac_fund_${date} --delete-target-dir --fields-terminated-by ` --query select * from amac_fund where update_time between '${start_time}' and '${end_time}' and $CONDITIONS -m 1

    注意:指定分隔符时,要把单引号去掉,下边还有一个大坑
    5. 添加参数

    6. 启动,填入参数

    7. 那么问题来了,报错是必须的
    oozie在解析sqoop语句的时候,是根据空格来分割的,那就恶心了,日期字段中间带有空格,就像这样

    Setting up log4j2
    log4j2 configuration file created at /data1/yarn/nm/usercache/fengz/appcache/application_1552354446915_0759/container_1552354446915_0759_01_000001/sqoop-log4j2.xml
    Sqoop command arguments :
    import
    --connect
    jdbc:mysql://localhost:3306/spider_new
    --username
    root
    --password
    ********
    --target-dir
    /user/fengz/brand/spider_data/amac/amac_fund_20190315
    --delete-target-dir
    --fields-terminated-by
    `
    --query
    "select
    *
    from
    amac_fund
    where
    update_time
    between
    '2019-03-14
    10:57:08'
    and
    '2019-03-14
    14:08:45'
    and
    $CONDITIONS"
    -m
    1
    Fetching child yarn jobs
    tag id : oozie-3e829587c1f3a050998935d2c2ac3ed4


    再看下oozie的错误提示,呵呵呵....发现一点鸟用都没有

    No child hadoop job is executed.
    java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410)
    at org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55)
    at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:223)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
    at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217)
    at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
    at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)
    Caused by: java.lang.SecurityException: Intercepted System.exit(1)
    at org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
    at java.lang.Runtime.exit(Runtime.java:107)
    at java.lang.System.exit(System.java:971)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:251)
    at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:214)
    at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:199)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:101)
    at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:51)
    ... 16 more
    Intercepting System.exit(1)
    Failing Oozie Launcher, Main Class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

    以后用的时候注意下就好了。那这个问题怎么解决?用shell,shell脚本里边写sqoop抽取

  • 相关阅读:
    Java实现LeetCode_0041_FirstMissingPositive
    Java实现 Leetcode 169 求众数
    Java实现 LeetCode 137 只出现一次的数字
    Java实现 LeetCode 137 只出现一次的数字
    Java实现 Leetcode 169 求众数
    TimeUnit用法
    第六章:任务执行——Java并发编程实战
    Java实现的并发任务处理实例
    java三大框架实现任务调度——IRemindService
    如何在一个div中使其子div居中
  • 原文地址:https://www.cnblogs.com/EnzoDin/p/10548442.html
Copyright © 2011-2022 走看看