【Hadoop离线基础总结】oozie调度hive

目录

• 1.拷贝hive的案例模板
• 2.编辑hive模板
• 3.上传工作文件到hdfs
• 4.执行oozie的调度
• 5.查看调度结果

---

• 1.拷贝hive的案例模板

  cd /export/servers/oozie-4.1.0-cdh5.14.0
  cp -ra examples/apps/hive2/ oozie_works/
  

• 2.编辑hive模板

  这里使用的是hiveserver2来进行提交任务，需要注意要启动hiveserver2的服务
  修改job.properties

  cd /export/servers/oozie-4.1.0-cdh5.14.0/oozie_works/hive2
  vim job.properties
  

  nameNode=hdfs://node01:8020
  jobTracker=node01:8032
  queueName=default
  jdbcURL=jdbc:hive2://node03:10000/default
  examplesRoot=oozie_works
  
  oozie.use.system.libpath=true
  # 配置我们文件上传到hdfs的保存路径 实际上就是在hdfs 的/user/root/oozie_works/hive2这个路径下
  oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/hive2
  

  修改workflow.xml
  vim workflow.xml

  <?xml version="1.0" encoding="UTF-8"?>
  <workflow-app xmlns="uri:oozie:workflow:0.5" name="hive2-wf">
      <start to="hive2-node"/>
  
      <action name="hive2-node">
          <hive2 xmlns="uri:oozie:hive2-action:0.1">
              <job-tracker>${jobTracker}</job-tracker>
              <name-node>${nameNode}</name-node>
              <prepare>
                  <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/hive2"/>
                  <mkdir path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data"/>
              </prepare>
              <configuration>
                  <property>
                      <name>mapred.job.queue.name</name>
                      <value>${queueName}</value>
                  </property>
              </configuration>
              <jdbc-url>${jdbcURL}</jdbc-url>
              <script>script.q</script>
              <param>INPUT=/user/${wf:user()}/${examplesRoot}/input-data/table</param>
              <param>OUTPUT=/user/${wf:user()}/${examplesRoot}/output-data/hive2</param>
          </hive2>
          <ok to="end"/>
          <error to="fail"/>
      </action>
  
      <kill name="fail">
          <message>Hive2 (Beeline) action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
      </kill>
      <end name="end"/>
  </workflow-app>
  

  编辑hivesql文件
  vim script.q

  DROP TABLE IF EXISTS test;
  CREATE EXTERNAL TABLE test (a INT) STORED AS TEXTFILE LOCATION '${INPUT}';
  insert into test values(10);
  insert into test values(20);
  insert into test values(30);
  

• 3.上传工作文件到hdfs

  cd /export/servers/oozie-4.1.0-cdh5.14.0/oozie_works
  hdfs dfs -put hive2/ /user/root/oozie_works/
  

• 4.执行oozie的调度

  cd /export/servers/oozie-4.1.0-cdh5.14.0
  bin/oozie job -oozie http://node03:11000/oozie -config oozie_works/hive2/job.properties  -run
  

• 5.查看调度结果