oozie：hadoop中的工作流引擎

一、maven安装

1) 从网站上下载Linux版本的Maven安装包；

apache-maven-3.0.5-bin.tar.gz下载

具体解压目录在 /home/maven/

========================================

2) 解压apache-maven-3.0.5-bin.tar.gz进行安装；

tar -xzvf apache-maven-3.0.5-bin.tar.gz

3) 设置maven环境变量；

vim /etc/profile或者vim /etc/environment

 在末尾处增加如下内容：

#maven envirment

export M2_HOME=/home/maven

export M2=$M2_HOME/bin

export MAVEN_OPTS="-Xms256m -Xmx512m"

export PATH=$M2:$PATH

4) 使配置文件生效

source /etc/profile

5)验证maven是否安装成功。

root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin# mvn -version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 21:51:28+0800)
Maven home: /home/maven
Java version: 1.6.0_43, vendor: Sun Microsystems Inc.
Java home: /usr/lib/jvm/jdk1.6.0_43/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-24-server", arch: "amd64", family: "unix"

　　

二、oozie

1、Build

bin/mkdistro.sh -DskipTests

成功build后,会在目录里生成一个distribution. 

hadoop安装时创建的hadoop组和hadoop用户对oozie目录是没有权限，所以最好一开始创建oozie组和oozie用户，同时赋予权限，不然会报错.偷懒用了root用户。　

PS：后来发现，可以单独拷贝/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2下的oozie-3.3.2-distro.tar.gz包到用户自定义目录进行解压使用，除此之外的所有文件可以全部删除。

2、安装

根据官方文档在distibution的目录下（/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2）创建libext，因为装的hadoop版本是1.0.3,所以拷贝$HADOOP_HOME/lib下的所有jar包到libext下，同时拷贝ext-2.2.zip到该目录下。 
还要拷贝$HADOOP_HOME下的所有jar包到libext.　

因为oozie默认并不支持1.0.3版，所以对/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin文件夹下的addtowar.sh文件进行修改：

elif [ "${version}" = "0.20.200" ]; then
    #List is separated by ":"
    hadoopJars="hadoop-core*.jar:jackson-core-asl-*.jar:jackson-mapper-asl-*.jar:commons-configuration-*.jar"
后边增加如下内容：
elif [ "${version}" = "1.0.3" ]; then
    #List is separated by ":"
    hadoopJars="hadoop-core*.jar:jackson-core-asl-*.jar:jackson-mapper-asl-*.jar:commons-configuration-*.jar"

 运行oozie-setup.sh创建oozie.war

root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie-setup.sh -hadoop 1.0.3 /home/hadoop -extjs /home/oozie/ext-2.2.zip

 执行成功后会出现如下界面：

拷贝oozie.war到tomcat部署目录下

root@ubuntu:/home/oozie/webapp/src/main/webapp# cp /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie.war ./

修改配置文件：　　

<property> 
        <name>oozie.service.JPAService.create.db.schema</name> 
        <value>true</value> 
        <description> 
            Creates Oozie DB. 
            If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP. 
            If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up. 
        </description> 
    </property> 

创建Oozie DB

cd /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin  
./ooziedb.sh create -sqlfile oozie.sql -run 

结果

DONE
Check DB schema does not exist
DONE
Check OOZIE_SYS table does not exist
DONE
Create SQL schema
DONE
DONE
Create OOZIE_SYS table
DONE
Oozie DB has been created for Oozie version '3.3.1' 

运行oozie

cd /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin
./oozie-start.sh　　

开启服务：

　　

bin/oozie-run.sh　

运行结果　　

root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie-run.sh
WARN: Use of this script is deprecated; use 'oozied.sh run' instead

Setting OOZIE_HOME:          /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2
Setting OOZIE_CONFIG:        /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf
Sourcing:                    /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf/oozie-env.sh
  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Setting OOZIE_CONFIG_FILE:   oozie-site.xml
Setting OOZIE_DATA:          /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/data
Setting OOZIE_LOG:           /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs
Setting OOZIE_LOG4J_FILE:    oozie-log4j.properties
Setting OOZIE_LOG4J_RELOAD:  10
Setting OOZIE_HTTP_HOSTNAME: dm4
Setting OOZIE_HTTP_PORT:     11000
Setting OOZIE_ADMIN_PORT:     11001
Setting OOZIE_HTTPS_PORT:     11443
Setting OOZIE_BASE_URL:      http://dm4:11000/oozie
Setting CATALINA_BASE:       /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
Setting OOZIE_HTTPS_KEYSTORE_FILE:     /root/.keystore
Setting OOZIE_HTTPS_KEYSTORE_PASS:     password
Setting CATALINA_OUT:        /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/catalina.out
Setting CATALINA_PID:        /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp/oozie.pid

Using   CATALINA_OPTS:        -Xmx1024m -Dderby.stream.error.file=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/derby.log
Adding to CATALINA_OPTS:     -Doozie.home.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2 -Doozie.config.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf -Doozie.log.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs -Doozie.data.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/data -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=dm4 -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://dm4:11000/oozie -Doozie.https.keystore.file=/root/.keystore -Doozie.https.keystore.pass=password -Djava.library.path=

Using CATALINA_BASE:   /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
Using CATALINA_HOME:   /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
Using CATALINA_TMPDIR: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp
Using JRE_HOME:        /usr/lib/jvm/jdk1.6.0_43/jre
Using CLASSPATH:       /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/bin/bootstrap.jar
Using CATALINA_PID:    /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp/oozie.pid
Jun 2, 2013 3:06:46 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: 
Jun 2, 2013 3:06:46 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-11000
Jun 2, 2013 3:06:46 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 338 ms
Jun 2, 2013 3:06:46 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jun 2, 2013 3:06:46 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.36
Jun 2, 2013 3:06:46 PM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor oozie.xml
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib/slf4j-simple-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
log4j:WARN No appenders could be found for logger (org.apache.jasper.compiler.JspRuntimeContext).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Jun 2, 2013 3:06:50 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory ROOT
Jun 2, 2013 3:06:50 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-11000
Jun 2, 2013 3:06:50 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 3976 ms

 界面：

和例子界面类似，没有内容。

　　

案例：

 将/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2中的oozie-examples.tar.gz解压，出现examples文件夹

修改

/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/examples/map-reduce/job.properties文件为如下形式：

nameNode=hdfs://211.80.198.148:9000
jobTracker=211.80.198.148:9001
queueName=default
examplesRoot=examples

oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce
outputDir=map-reduce

将文件夹上传到集群

root@dm4:/home/hadoop# bin/hadoop fs -copyFromLocal /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/examples /user/root

运行命令

root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run

运行结果：

job: 0000000-130602150647008-oozie-root-W

不上传会出现如下错误：

root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
Error: E0504 : E0504: App directory [hdfs://211.80.198.148:9000/user/root/examples/apps/map-reduce] does not exist

结果

 

 PS：中间出现的一些错误

ERROR: Oozie could not be started

REASON: java.lang.NoClassDefFoundError: org/apache/hadoop/util/ReflectionUtils

 将hadoop的jar包（主目录下和lib目录下）拷贝到如下位置

/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib

 之后，运行正常。

参考：
1、http://angelosun.iteye.com/blog/1835881

2、http://practicalcloudcomputing.com/post/26337621577/installing-and-running-apache-oozie-3-2-x-and-possibly

3、http://wsysisibeibei.blog.163.com/blog/static/16536806120126233155505/

 转载请保留：http://www.cnblogs.com/Deron/archive/2013/06/02/3111178.html