zoukankan      html  css  js  c++  java
  • Oozie 3.3.1安装

    软件安装路径

    软件名称 版本 安装路径
    jdk 1.6.0_12

    /usr/java/jdk1.6.0_12

    maven 3.1.0

    /usr/local//apache-maven-3.1.0

    Oozie 3.3.1

    /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1

    hadoop 1.0.3

    /usr/local/hadoop-1.0.3

    mysql 5.5.7

    /usr/local/mysql

    环境变量.bash_profile

    export HADOOP_HOME="/home/tnuser/hadoop"
    export HADOOP_PREFIX=$HADOOP_HOME
    export HBASE_HOME="/home/tnuser/hbase"
    export HIVE_HOME="/home/tnuser/hive"
    export OOZIE_HOME="/home/tnuser/oozie"
    export JAVA_HOME="/usr/java/jdk1.6.0_12"
    export JRE_HOME=$JAVA_HOME/jre
    export MAVEN_HOME="/usr/local/apache-maven-3.1.0"
    export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$JRE_HOME/lib
    export PATH=$JAVA_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$SQOOP_HOME/bin:$PIG_HOME/bin:$HIVE_HOME/bin:$MAVEN_HOME/bin:$OOZIE_HOME/bin:/usr/bin:/sbin:/usr/sbin/:$PATH
    export PIG_CLASSPATH=/usr/java/jdk1.6.0_12/lib/tools.jar:$PIG_CLASSPATH
    export HIVE_LIB=$HIVE_HOME/lib/hive-hbase-handler-0.10.0.jar,$HIVE_HOME/lib/hbase-0.92.1.jar,$HIVE_HOME/lib/zookeeper-3.4.3.jar,$HIVE_HOME/lib/guava-r09.jar
    export "OOZIE_URL=http://localhost:11000/oozie"

    家目录设置

    $ ls -lh /home/tnuser/
    total 0
    lrwxrwxrwx 1 tnuser appuser 39 Jun 11 03:29 hadoop -> /usr/local/hadoop-1.0.3
    lrwxrwxrwx 1 tnuser appuser 39 Jun 11 00:42 hbase -> /usr/local/hbase-0.92.1
    lrwxrwxrwx 1 tnuser appuser 31 Jun 11 00:42 hdfs -> /usr/local/data
    drwxr-xr-x 4 tnuser appuser 45 Jun 11 00:42 logs
    lrwxrwxrwx 1 tnuser appuser 83 Jun 17 18:45 oozie -> /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1

    1. maven安装

    $ wget https://archive.apache.org/dist/maven/maven-3/3.1.0/binaries/apache-maven-3.1.0-bin.tar.gz
    $ tar -zxf apache-maven-3.1.0-bin.tar.gz

    2.hadoop安装

    3.mysql安装

    4. 生成Oozie安装包

    $ wget https://archive.apache.org/dist/oozie/3.3.1/oozie-3.3.1.tar.gz
    $ tar -zxf oozie-3.3.1.tar.gz
    $ cd /usr/local/contentplatform/oozie-3.3.1/bin
    $ ./mkdistro.sh -DskipTests

    报错及解决

    报错

    ERROR] Failed to execute goal on project oozie-sharelib-hive: Could not resolve dependencies for project org.apache.oozie:oozie-sharelib-hive:jar:3.3.1: Failed to collect dependencies at org.apache.hive:hive-cli:jar:0.9.0 -> org.apache.hive:hive-metastore:jar:0.9.0 -> javax.jdo:jdo2-api:jar:2.3-ec: Failed to read artifact descriptor for javax.jdo:jdo2-api:jar:2.3-ec: Could not transfer artifact javax.jdo:jdo2-api:pom:2.3-ec from/to Codehaus repository (http://repository.codehaus.org/): repository.codehaus.org: Unknown host repository.codehaus.org -> [Help 1]
    [ERROR] 
    [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
    [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    [ERROR] 
    [ERROR] For more information about the errors and possible solutions, please read the following articles:
    [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
    [ERROR] 
    [ERROR] After correcting the problems, you can resume the build with the command
    [ERROR]   mvn <goals> -rf :oozie-sharelib-hive
    
    ERROR, Oozie distro creation failed
    View Code

    解决

    将安装根目录的pom.xml如下代码

            <repository>
                <id>Codehaus repository</id>
                <url>http://repository.codehaus.org/</url>
                <snapshots>
                    <enabled>false</enabled>
                </snapshots>
            </repository>

    替换为

            <repository>
                <id>Codehaus repository</id>
                <name>codehaus-mule-repo</name>
                <url>https://repository-master.mulesoft.org/nexus/content/groups/public/</url>
                <layout>default</layout>
            </repository>

    重新运行

    ./mkdistro.sh -DskipTests

    出现以下结果说明生成安装包成功

    Oozie distro created, DATE[2019.06.18-01:40:02GMT] VC-REV[unavailable], available at [/usr/local/oozie-3.3.1/distro/target]

    5.安装Oozie

    5.1 启用Oozie Web Console

    wget http://archive.cloudera.com/gplextras/misc/ext-2.2.zip

    解压到$OOZIE_HOME根目录

    5.2 复制$HADOOP_HOME以及$HADOOP_HOME/lib所有jar包到$OOZIE_HOME/libext

    $ mkdir $OOZIE_HOME/libext
    $ scp  $HADOOP_HOME/*.jar $OOZIE_HOME/libext
    $ scp $HADOOP_HOME/lib/*.jar $OOZIE_HOME/libext

    5.3 添加mysql数据库驱动到Oozie

    $ wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz
    $ mv mysql-connector-java-5.1.47.tar.gz $SQOOP_HOME/libext

    5.4 安装Oozie

    $ oozie-setup.sh -hadoop 0.20.2 $HADOOP_HOME -extjs ext-2.2

    输出log

      setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
    
    INFO: Adding extension: /home/tnuser/oozie/libext/asm-3.2.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/aspectjrt-1.6.5.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/aspectjtools-1.6.5.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-beanutils-1.7.0.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-beanutils-core-1.8.0.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-cli-1.2.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-codec-1.4.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-collections-3.2.1.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-configuration-1.6.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-daemon-1.0.1.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-digester-1.8.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-el-1.0.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-httpclient-3.0.1.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-lang-2.4.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-logging-1.1.1.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-logging-api-1.0.4.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-math-2.1.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/commons-net-1.4.1.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/core-3.1.1.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-ant-1.0.2.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-capacity-scheduler-1.0.2.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-client-1.0.2.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-core-1.0.2.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-examples-1.0.2.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-fairscheduler-1.0.2.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-minicluster-1.0.2.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-test-1.0.2.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-thriftfs-1.0.2.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-tools-1.0.2.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/hsqldb-1.8.0.10.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/jackson-core-asl-1.8.8.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/jackson-mapper-asl-1.8.8.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/jasper-compiler-5.5.12.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/jasper-runtime-5.5.12.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/jdeb-0.8.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/jersey-core-1.8.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/jersey-json-1.8.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/jersey-server-1.8.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/jets3t-0.6.1.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/jetty-6.1.26.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/jetty-util-6.1.26.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/jsch-0.1.42.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/junit-4.5.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/kfs-0.2.2.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/log4j-1.2.16.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/mockito-all-1.8.5.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/mysql-connector-java-5.1.24-bin.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/oro-2.0.8.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/servlet-api-2.5-20081211.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/slf4j-api-1.4.3.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/slf4j-log4j12-1.4.3.jar
    INFO: Adding extension: /home/tnuser/oozie/libext/xmlenc-0.52.jar
    Injecting following Hadoop JARs
    
    /home/tnuser/hadoop/hadoop-core-1.0.3.jar
    
    New Oozie WAR file with added 'Hadoop JARs, ExtJS library, JARs' at /home/tnuser/oozie/oozie-server/webapps/oozie.war
    
    INFO: Oozie is ready to be started
    View Code

    5.5 生成Oozie数据库初始化文件

    $ ooziedb.sh create -sqlfile oozie.sql

    5.6 将初始化文件导入mysql数据库

    CREATE DATABASE oozie_data DEFAULT CHARACTER SET uft8 ;
    CREATE USER 'oozie'@'*' IDENTIFIED BY 'oozie';
    GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'*'
    use oozie;
    source oozie.sql;

    5.7 配置hadoop core-site.xml,添加以下参数,如之前未添加,需要重启hadoop

    <property>
      <name>hadoop.proxyuser.tnuser.hosts</name>
      <value>*</value>
    </property>
    
    <property>
      <name>hadoop.proxyuser.tnuser.groups</name>
      <value>*</value>
    </property>

    5.8 启动Oozie

    $ oozie-start.sh
    $ netstat -antl | grep 11000
    tcp        0      0 0.0.0.0:11000           0.0.0.0:*               LISTEN
    $ oozie admin -oozie http://localhost:11000/oozie -status
    System mode: NORMAL

    5.9 将Oozie共享库导入hadoop

    $ cd $OOZIE_HOME
    $ tar xvfz oozie-sharelib-3.3.1.tar.gz
    $ HADOOP_HOME/bin/hadoop fs -put share share
    $ hadoop fs -ls /user/tnuser/share/lib
    
    Found 7 items
    drwxr-xr-x   - tnuser supergroup          0 2019-06-19 06:07 /user/tnuser/share/lib/distcp
    drwxr-xr-x   - tnuser supergroup          0 2019-06-19 06:07 /user/tnuser/share/lib/hive
    drwxr-xr-x   - tnuser supergroup          0 2019-06-19 06:07 /user/tnuser/share/lib/mapreduce-streaming
    drwxr-xr-x   - tnuser supergroup          0 2019-06-19 06:07 /user/tnuser/share/lib/oozie
    drwxr-xr-x   - tnuser supergroup          0 2019-06-19 06:07 /user/tnuser/share/lib/pig
    -rw-r--r--   3 tnuser supergroup       1354 2019-06-19 06:07 /user/tnuser/share/lib/sharelib.properties
    drwxr-xr-x   - tnuser supergroup          0 2019-06-19 06:07 /user/tnuser/share/lib/sqoop

     6. 测试Oozie

    6.1 修改example包文件对应属性值

    $ cd $OOZIE_HOME
    $ tar -zxf oozie-examples.tar.gz
    $ ls -lh examples/apps/map-reduce/
    total 8.0K
    -rw-r--r-- 1 tnuser appuser 1021 Jun 20 06:00 job.properties
    drwxr-xr-x 2 tnuser appuser   38 Jun 19 07:41 lib
    -rw-r--r-- 1 tnuser appuser 2.5K Jan 17  2013 workflow.xml

     将job.properties文件namenode和jobtracker地址修改为实际的地址

    $ cat examples/apps/map-reduce/job.properties 
    
    nameNode=hdfs://ec2d-newcntprocnn-01:9000
    jobTracker=ec2d-newcntprocnn-01:9001
    queueName=default
    examplesRoot=examples
    
    oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce
    outputDir=map-reduce

     6.2 上传到hdfs

    $ hadoop fs -put examples examples
    $ hadoop fs -ls /user/tnuser
    
    Found 2 items
    drwxr-xr-x   - tnuser supergroup          0 2019-06-20 07:15 /user/tnuser/examples
    drwxr-xr-x   - tnuser supergroup          0 2019-06-19 06:07 /user/tnuser/share

     6.3 测试Oozie

    报错

    $ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
    Error: E0902 : E0902: Exception occured: [org.apache.hadoop.ipc.RemoteException: User: tnuser is not allowed to impersonate tnuser]
    
    $ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
    Error: E0902 : E0902: Exception occured: [org.apache.hadoop.ipc.RemoteException: Unauthorized connection for super-user: tnuser from IP 10.189.102.176]

    经测试以下hdfs参数的value值为“*“会出现以上报错,需要修改hdfs参数为对应的真实值

    <property>
      <name>hadoop.proxyuser.tnuser.hosts</name>
      <value>ec2d-newcntprocdn-04</value>
    </property>
    
    <property>
      <name>hadoop.proxyuser.tnuser.groups</name>
      <value>appuser</value>
    </property>

    正常运行结果

    $ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
    job: 0000000-190620070628363-oozie-tnus-W
    
    
    $ oozie job -oozie http://localhost:11000/oozie -info 0000000-190620070628363-oozie-tnus-W
    Job ID : 0000000-190620070628363-oozie-tnus-W
    ------------------------------------------------------------------------------------------------------------------------------------
    Workflow Name : map-reduce-wf
    App Path      : hdfs://ec2d-newcntprocnn-01:9000/user/tnuser/examples/apps/map-reduce
    Status        : RUNNING
    Run           : 0
    User          : tnuser
    Group         : -
    Created       : 2019-06-20 14:14 GMT
    Started       : 2019-06-20 14:14 GMT
    Last Modified : 2019-06-20 14:14 GMT
    Ended         : -
    CoordAction ID: -
    
    Actions
    ------------------------------------------------------------------------------------------------------------------------------------
    ID                                                                            Status    Ext ID                 Ext Status Err Code  
    ------------------------------------------------------------------------------------------------------------------------------------
    0000000-190620070628363-oozie-tnus-W@:start:                                  OK        -                      OK         -         
    ------------------------------------------------------------------------------------------------------------------------------------
    0000000-190620070628363-oozie-tnus-W@mr-node                                  START_RETRY-                      -          JA009     
    ------------------------------------------------------------------------------------------------------------------------------------
    
    
    $ oozie job -oozie http://localhost:11000/oozie -info 0000000-190620070628363-oozie-tnus-W
    Job ID : 0000000-190620070628363-oozie-tnus-W
    ------------------------------------------------------------------------------------------------------------------------------------
    Workflow Name : map-reduce-wf
    App Path      : hdfs://ec2d-newcntprocnn-01:9000/user/tnuser/examples/apps/map-reduce
    Status        : SUCCEEDED
    Run           : 0
    User          : tnuser
    Group         : -
    Created       : 2019-06-20 14:14 GMT
    Started       : 2019-06-20 14:14 GMT
    Last Modified : 2019-06-20 14:16 GMT
    Ended         : 2019-06-20 14:16 GMT
    CoordAction ID: -
    
    Actions
    ------------------------------------------------------------------------------------------------------------------------------------
    ID                                                                            Status    Ext ID                 Ext Status Err Code  
    ------------------------------------------------------------------------------------------------------------------------------------
    0000000-190620070628363-oozie-tnus-W@:start:                                  OK        -                      OK         -         
    ------------------------------------------------------------------------------------------------------------------------------------
    0000000-190620070628363-oozie-tnus-W@mr-node                                  OK        job_201906200714_0002  SUCCEEDED  -         
    ------------------------------------------------------------------------------------------------------------------------------------
    0000000-190620070628363-oozie-tnus-W@end                                      OK        -                      OK         -         
    ------------------------------------------------------------------------------------------------------------------------------------
  • 相关阅读:
    5.4 省选模拟赛 修改 线段树优化dp 线段树上二分
    一本通 高手训练 1782 分层图 状压dp
    luogu P3830 [SHOI2012]随机树 期望 dp
    5.2 省选模拟赛 或许 线型基
    luogu P4562 [JXOI2018]游戏 组合数学
    一本通 高手训练 1781 死亡之树 状态压缩dp
    luogu P4726 【模板】多项式指数函数 多项式 exp 牛顿迭代 泰勒展开
    4.28 省选模拟赛 负环 倍增 矩阵乘法 dp
    HDU 1756 Cupid's Arrow 计算几何 判断一个点是否在多边形内
    一本通 高手训练 1763 简单树 可持久化线段树 树链刨分 标记永久化
  • 原文地址:https://www.cnblogs.com/ilifeilong/p/11055422.html
Copyright © 2011-2022 走看看