Oozie 3.3.1安装

软件安装路径

软件名称	版本	安装路径
jdk	1.6.0_12	/usr/java/jdk1.6.0_12
maven	3.1.0	/usr/local//apache-maven-3.1.0
Oozie	3.3.1	/usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1
hadoop	1.0.3	/usr/local/hadoop-1.0.3
mysql	5.5.7	/usr/local/mysql

环境变量.bash_profile

export HADOOP_HOME="/home/tnuser/hadoop"
export HADOOP_PREFIX=$HADOOP_HOME
export HBASE_HOME="/home/tnuser/hbase"
export HIVE_HOME="/home/tnuser/hive"
export OOZIE_HOME="/home/tnuser/oozie"
export JAVA_HOME="/usr/java/jdk1.6.0_12"
export JRE_HOME=$JAVA_HOME/jre
export MAVEN_HOME="/usr/local/apache-maven-3.1.0"
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$JRE_HOME/lib
export PATH=$JAVA_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$SQOOP_HOME/bin:$PIG_HOME/bin:$HIVE_HOME/bin:$MAVEN_HOME/bin:$OOZIE_HOME/bin:/usr/bin:/sbin:/usr/sbin/:$PATH
export PIG_CLASSPATH=/usr/java/jdk1.6.0_12/lib/tools.jar:$PIG_CLASSPATH
export HIVE_LIB=$HIVE_HOME/lib/hive-hbase-handler-0.10.0.jar,$HIVE_HOME/lib/hbase-0.92.1.jar,$HIVE_HOME/lib/zookeeper-3.4.3.jar,$HIVE_HOME/lib/guava-r09.jar
export "OOZIE_URL=http://localhost:11000/oozie"

家目录设置

$ ls -lh /home/tnuser/
total 0
lrwxrwxrwx 1 tnuser appuser 39 Jun 11 03:29 hadoop -> /usr/local/hadoop-1.0.3
lrwxrwxrwx 1 tnuser appuser 39 Jun 11 00:42 hbase -> /usr/local/hbase-0.92.1
lrwxrwxrwx 1 tnuser appuser 31 Jun 11 00:42 hdfs -> /usr/local/data
drwxr-xr-x 4 tnuser appuser 45 Jun 11 00:42 logs
lrwxrwxrwx 1 tnuser appuser 83 Jun 17 18:45 oozie -> /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1

1. maven安装

$ wget https://archive.apache.org/dist/maven/maven-3/3.1.0/binaries/apache-maven-3.1.0-bin.tar.gz
$ tar -zxf apache-maven-3.1.0-bin.tar.gz

2.hadoop安装

略

3.mysql安装

略

4. 生成Oozie安装包

$ wget https://archive.apache.org/dist/oozie/3.3.1/oozie-3.3.1.tar.gz
$ tar -zxf oozie-3.3.1.tar.gz
$ cd /usr/local/contentplatform/oozie-3.3.1/bin
$ ./mkdistro.sh -DskipTests

报错及解决

报错

ERROR] Failed to execute goal on project oozie-sharelib-hive: Could not resolve dependencies for project org.apache.oozie:oozie-sharelib-hive:jar:3.3.1: Failed to collect dependencies at org.apache.hive:hive-cli:jar:0.9.0 -> org.apache.hive:hive-metastore:jar:0.9.0 -> javax.jdo:jdo2-api:jar:2.3-ec: Failed to read artifact descriptor for javax.jdo:jdo2-api:jar:2.3-ec: Could not transfer artifact javax.jdo:jdo2-api:pom:2.3-ec from/to Codehaus repository (http://repository.codehaus.org/): repository.codehaus.org: Unknown host repository.codehaus.org -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :oozie-sharelib-hive

ERROR, Oozie distro creation failed

解决

将安装根目录的pom.xml如下代码

        <repository>
            <id>Codehaus repository</id>
            <url>http://repository.codehaus.org/</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>

替换为

        <repository>
            <id>Codehaus repository</id>
            <name>codehaus-mule-repo</name>
            <url>https://repository-master.mulesoft.org/nexus/content/groups/public/</url>
            <layout>default</layout>
        </repository>

重新运行

./mkdistro.sh -DskipTests

出现以下结果说明生成安装包成功

Oozie distro created, DATE[2019.06.18-01:40:02GMT] VC-REV[unavailable], available at [/usr/local/oozie-3.3.1/distro/target]

5.安装Oozie

5.1 启用Oozie Web Console

wget http://archive.cloudera.com/gplextras/misc/ext-2.2.zip

解压到$OOZIE_HOME根目录

5.2 复制$HADOOP_HOME以及$HADOOP_HOME/lib所有jar包到$OOZIE_HOME/libext

$ mkdir $OOZIE_HOME/libext
$ scp  $HADOOP_HOME/*.jar $OOZIE_HOME/libext
$ scp $HADOOP_HOME/lib/*.jar $OOZIE_HOME/libext

5.3 添加mysql数据库驱动到Oozie

$ wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz
$ mv mysql-connector-java-5.1.47.tar.gz $SQOOP_HOME/libext

5.4 安装Oozie

$ oozie-setup.sh -hadoop 0.20.2 $HADOOP_HOME -extjs ext-2.2

输出log

  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"

INFO: Adding extension: /home/tnuser/oozie/libext/asm-3.2.jar
INFO: Adding extension: /home/tnuser/oozie/libext/aspectjrt-1.6.5.jar
INFO: Adding extension: /home/tnuser/oozie/libext/aspectjtools-1.6.5.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-beanutils-1.7.0.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-beanutils-core-1.8.0.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-cli-1.2.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-codec-1.4.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-collections-3.2.1.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-configuration-1.6.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-daemon-1.0.1.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-digester-1.8.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-el-1.0.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-httpclient-3.0.1.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-lang-2.4.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-logging-1.1.1.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-logging-api-1.0.4.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-math-2.1.jar
INFO: Adding extension: /home/tnuser/oozie/libext/commons-net-1.4.1.jar
INFO: Adding extension: /home/tnuser/oozie/libext/core-3.1.1.jar
INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-ant-1.0.2.jar
INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-capacity-scheduler-1.0.2.jar
INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-client-1.0.2.jar
INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-core-1.0.2.jar
INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-examples-1.0.2.jar
INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-fairscheduler-1.0.2.jar
INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-minicluster-1.0.2.jar
INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-test-1.0.2.jar
INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-thriftfs-1.0.2.jar
INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-tools-1.0.2.jar
INFO: Adding extension: /home/tnuser/oozie/libext/hsqldb-1.8.0.10.jar
INFO: Adding extension: /home/tnuser/oozie/libext/jackson-core-asl-1.8.8.jar
INFO: Adding extension: /home/tnuser/oozie/libext/jackson-mapper-asl-1.8.8.jar
INFO: Adding extension: /home/tnuser/oozie/libext/jasper-compiler-5.5.12.jar
INFO: Adding extension: /home/tnuser/oozie/libext/jasper-runtime-5.5.12.jar
INFO: Adding extension: /home/tnuser/oozie/libext/jdeb-0.8.jar
INFO: Adding extension: /home/tnuser/oozie/libext/jersey-core-1.8.jar
INFO: Adding extension: /home/tnuser/oozie/libext/jersey-json-1.8.jar
INFO: Adding extension: /home/tnuser/oozie/libext/jersey-server-1.8.jar
INFO: Adding extension: /home/tnuser/oozie/libext/jets3t-0.6.1.jar
INFO: Adding extension: /home/tnuser/oozie/libext/jetty-6.1.26.jar
INFO: Adding extension: /home/tnuser/oozie/libext/jetty-util-6.1.26.jar
INFO: Adding extension: /home/tnuser/oozie/libext/jsch-0.1.42.jar
INFO: Adding extension: /home/tnuser/oozie/libext/junit-4.5.jar
INFO: Adding extension: /home/tnuser/oozie/libext/kfs-0.2.2.jar
INFO: Adding extension: /home/tnuser/oozie/libext/log4j-1.2.16.jar
INFO: Adding extension: /home/tnuser/oozie/libext/mockito-all-1.8.5.jar
INFO: Adding extension: /home/tnuser/oozie/libext/mysql-connector-java-5.1.24-bin.jar
INFO: Adding extension: /home/tnuser/oozie/libext/oro-2.0.8.jar
INFO: Adding extension: /home/tnuser/oozie/libext/servlet-api-2.5-20081211.jar
INFO: Adding extension: /home/tnuser/oozie/libext/slf4j-api-1.4.3.jar
INFO: Adding extension: /home/tnuser/oozie/libext/slf4j-log4j12-1.4.3.jar
INFO: Adding extension: /home/tnuser/oozie/libext/xmlenc-0.52.jar
Injecting following Hadoop JARs

/home/tnuser/hadoop/hadoop-core-1.0.3.jar

New Oozie WAR file with added 'Hadoop JARs, ExtJS library, JARs' at /home/tnuser/oozie/oozie-server/webapps/oozie.war

INFO: Oozie is ready to be started

5.5 生成Oozie数据库初始化文件

$ ooziedb.sh create -sqlfile oozie.sql

5.6 将初始化文件导入mysql数据库

CREATE DATABASE oozie_data DEFAULT CHARACTER SET uft8 ;
CREATE USER 'oozie'@'*' IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'*'
use oozie;
source oozie.sql;

5.7 配置hadoop core-site.xml，添加以下参数，如之前未添加，需要重启hadoop

<property>
  <name>hadoop.proxyuser.tnuser.hosts</name>
  <value>*</value>
</property>

<property>
  <name>hadoop.proxyuser.tnuser.groups</name>
  <value>*</value>
</property>

5.8 启动Oozie

$ oozie-start.sh
$ netstat -antl | grep 11000
tcp        0      0 0.0.0.0:11000           0.0.0.0:*               LISTEN
$ oozie admin -oozie http://localhost:11000/oozie -status
System mode: NORMAL

5.9 将Oozie共享库导入hadoop

$ cd $OOZIE_HOME
$ tar xvfz oozie-sharelib-3.3.1.tar.gz
$ HADOOP_HOME/bin/hadoop fs -put share share
$ hadoop fs -ls /user/tnuser/share/lib

Found 7 items
drwxr-xr-x   - tnuser supergroup          0 2019-06-19 06:07 /user/tnuser/share/lib/distcp
drwxr-xr-x   - tnuser supergroup          0 2019-06-19 06:07 /user/tnuser/share/lib/hive
drwxr-xr-x   - tnuser supergroup          0 2019-06-19 06:07 /user/tnuser/share/lib/mapreduce-streaming
drwxr-xr-x   - tnuser supergroup          0 2019-06-19 06:07 /user/tnuser/share/lib/oozie
drwxr-xr-x   - tnuser supergroup          0 2019-06-19 06:07 /user/tnuser/share/lib/pig
-rw-r--r--   3 tnuser supergroup       1354 2019-06-19 06:07 /user/tnuser/share/lib/sharelib.properties
drwxr-xr-x   - tnuser supergroup          0 2019-06-19 06:07 /user/tnuser/share/lib/sqoop

 6. 测试Oozie

6.1 修改example包文件对应属性值

$ cd $OOZIE_HOME
$ tar -zxf oozie-examples.tar.gz
$ ls -lh examples/apps/map-reduce/
total 8.0K
-rw-r--r-- 1 tnuser appuser 1021 Jun 20 06:00 job.properties
drwxr-xr-x 2 tnuser appuser   38 Jun 19 07:41 lib
-rw-r--r-- 1 tnuser appuser 2.5K Jan 17  2013 workflow.xml

 将job.properties文件namenode和jobtracker地址修改为实际的地址

$ cat examples/apps/map-reduce/job.properties 

nameNode=hdfs://ec2d-newcntprocnn-01:9000
jobTracker=ec2d-newcntprocnn-01:9001
queueName=default
examplesRoot=examples

oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce
outputDir=map-reduce

 6.2 上传到hdfs

$ hadoop fs -put examples examples
$ hadoop fs -ls /user/tnuser

Found 2 items
drwxr-xr-x   - tnuser supergroup          0 2019-06-20 07:15 /user/tnuser/examples
drwxr-xr-x   - tnuser supergroup          0 2019-06-19 06:07 /user/tnuser/share

 6.3 测试Oozie

报错

$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
Error: E0902 : E0902: Exception occured: [org.apache.hadoop.ipc.RemoteException: User: tnuser is not allowed to impersonate tnuser]

$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
Error: E0902 : E0902: Exception occured: [org.apache.hadoop.ipc.RemoteException: Unauthorized connection for super-user: tnuser from IP 10.189.102.176]

经测试以下hdfs参数的value值为“*“会出现以上报错，需要修改hdfs参数为对应的真实值

<property>
  <name>hadoop.proxyuser.tnuser.hosts</name>
  <value>ec2d-newcntprocdn-04</value>
</property>

<property>
  <name>hadoop.proxyuser.tnuser.groups</name>
  <value>appuser</value>
</property>

正常运行结果

$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
job: 0000000-190620070628363-oozie-tnus-W


$ oozie job -oozie http://localhost:11000/oozie -info 0000000-190620070628363-oozie-tnus-W
Job ID : 0000000-190620070628363-oozie-tnus-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : map-reduce-wf
App Path      : hdfs://ec2d-newcntprocnn-01:9000/user/tnuser/examples/apps/map-reduce
Status        : RUNNING
Run           : 0
User          : tnuser
Group         : -
Created       : 2019-06-20 14:14 GMT
Started       : 2019-06-20 14:14 GMT
Last Modified : 2019-06-20 14:14 GMT
Ended         : -
CoordAction ID: -

Actions
------------------------------------------------------------------------------------------------------------------------------------
ID                                                                            Status    Ext ID                 Ext Status Err Code  
------------------------------------------------------------------------------------------------------------------------------------
0000000-190620070628363-oozie-tnus-W@:start:                                  OK        -                      OK         -         
------------------------------------------------------------------------------------------------------------------------------------
0000000-190620070628363-oozie-tnus-W@mr-node                                  START_RETRY-                      -          JA009     
------------------------------------------------------------------------------------------------------------------------------------


$ oozie job -oozie http://localhost:11000/oozie -info 0000000-190620070628363-oozie-tnus-W
Job ID : 0000000-190620070628363-oozie-tnus-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : map-reduce-wf
App Path      : hdfs://ec2d-newcntprocnn-01:9000/user/tnuser/examples/apps/map-reduce
Status        : SUCCEEDED
Run           : 0
User          : tnuser
Group         : -
Created       : 2019-06-20 14:14 GMT
Started       : 2019-06-20 14:14 GMT
Last Modified : 2019-06-20 14:16 GMT
Ended         : 2019-06-20 14:16 GMT
CoordAction ID: -

Actions
------------------------------------------------------------------------------------------------------------------------------------
ID                                                                            Status    Ext ID                 Ext Status Err Code  
------------------------------------------------------------------------------------------------------------------------------------
0000000-190620070628363-oozie-tnus-W@:start:                                  OK        -                      OK         -         
------------------------------------------------------------------------------------------------------------------------------------
0000000-190620070628363-oozie-tnus-W@mr-node                                  OK        job_201906200714_0002  SUCCEEDED  -         
------------------------------------------------------------------------------------------------------------------------------------
0000000-190620070628363-oozie-tnus-W@end                                      OK        -                      OK         -         
------------------------------------------------------------------------------------------------------------------------------------