zoukankan      html  css  js  c++  java
  • azkaban的安装部署

    一、所需环境

    1,JDK

    2,HADOOP

    4,mysql

    3,HIVE

    二、安装azkaban

    1,安装git命令: yum install git

    2,下载azkaban源码:git clone https://github.com/azkaban/azkaban.git

    3,安装到 azkaban-tar目录下

    4,更改配置文件:

    vi 安装目录/azkaban-tar/azkaban/git clone https://github.com/azkaban/azkaban.git

    将final boolean isExecuteAsUser = this.sysProps.getBoolean(EXECUTE_AS_USER, true);
    true修改false 

    5,在azkaban目录下执行:./gradlew distTar  

    此步骤会生成5个压缩包,分别放在下图目录中:

    azkaban-db-3.37.0-10-gdd67b02.tar.gz
    azkaban-exec-server-3.37.0-10-gdd67b02.tar.gz
    azkaban-hadoop-security-plugin-3.37.0-10-gdd67b02.tar.gz
    azkaban-solo-server-3.37.0-10-gdd67b02.tar.gz
    azkaban-web-server-3.37.0-10-gdd67b02.tar.gz 

    6,在当前azkaban文件夹下,将该五个压缩包copy出来,放到 安装目录/azkaban-tar文件夹中。

    cp  azkaban-*/build/distributions/*.tar.gz  安装目录/azkaban-tar

     

    7,将这5个压缩包解压,然后重命令,将这5个压缩包放在另外的 安装目录/azkaban下

    8,登录mysql mysql –u root –p

    9,数据库中执行:create database azkaban;

    10,数据库中执行:use azkaban;

    11, 将azkaban sql表结构导入mysql,数据库中执行:source 安装目录/azkaban/azkaban-db-3.46.0-2-gbb24a72/create-all-sql-3.46.0-2-gbb24a72.sql  

    如果执行失败,则将sql语句复制出来执行

     

    12,部署azkaban-web-server

    将 安装目录/azkaban/azkaban-solo-server/ 下的conf目录拷贝到  安装目录/azkaban/ azkaban-web-server/conf

    命令:cp -r  /安装目录/azkaban/azkaban-solo-server/conf   /安装目录/azkaban/azkaban-web-server/conf 

    配置:azkaban.properties

     

    如下(标红的需要更改,其他的可以对比一下):

     

    # Azkaban Personalization Settings

    azkaban.name=Test

    azkaban.label=My Local Azkaban

    azkaban.color=#FF3601

    azkaban.default.servlet.path=/index

    web.resource.dir=/usr/local/azkaban/azkaban-web-server/web

    default.timezone.id=Asia/Shanghai

    # Azkaban UserManager class

    user.manager.class=azkaban.user.XmlUserManager

    user.manager.xml.file=/usr/local/azkaban/azkaban-web-server/conf/azkaban-users.xml

    # Loader for projects

    executor.global.properties=/usr/local/azkaban/azkaban-web-server/conf/global.properties

    azkaban.project.dir=projects

    database.type=mysql

    mysql.port=3306

    mysql.host=localhost

    mysql.database=azkaban

    mysql.user=root

    mysql.password=root

    mysql.numconnections=100

     

    # Velocity dev mode

    velocity.dev.mode=false

    # Azkaban Jetty server properties.

    jetty.use.ssl=false

    jetty.maxThreads=25

    jetty.port=8081

    # Azkaban Executor settings

    executor.maxThreads=50

    executor.port=12321

    executor.flow.threads=30

    # mail settings

    mail.sender=

    mail.host=

    # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.

    # enduser -> myazkabanhost:443 -> proxy -> localhost:8081

    # when this parameters set then these parameters are used to generate email links.

    # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.

    # azkaban.webserver.external_hostname=myazkabanhost.com

    # azkaban.webserver.external_ssl_port=443

    # azkaban.webserver.external_port=8081

    job.failure.email=

    job.success.email=

    lockdown.create.projects=false

    cache.directory=cache

    # JMX stats

    jetty.connector.stats=true

    executor.connector.stats=true

     

    azkaban.native.lib=false

    # Azkaban plugin settings

    azkaban.jobtype.plugin.dir=plugins/jobtypes

     

    13,在conf目录下新增配置文件log4j.properties,内容如下

     

    log4j.rootLogger=INFO,C

    log4j.appender.C=org.apache.log4j.ConsoleAppender

    log4j.appender.C.Target=System.err

    log4j.appender.C.layout=org.apache.log4j.PatternLayout

    log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

     

    14,部署 azkaban-exec-server

    将 /安装目录/azkaban/ azkaban-solo-server下的conf文件拷贝到 /安装文件/azkaban/ azkaban-exec-server

    命令:cp –r /安装目录/azkaban/azkaban-solo-server/conf  /安装目录/azkaban/azkaban-exec-server/conf

     

    配置azkaban.properties:(标红的需要更改,其他的可以对比一下)

     

    # Azkaban Personalization Settings

    azkaban.name=Test

    azkaban.label=My Local Azkaban

    azkaban.color=#FF3601

    azkaban.default.servlet.path=/index

    web.resource.dir=web/

    default.timezone.id=Asia/Shanghai

    # Azkaban UserManager class

    user.manager.class=azkaban.user.XmlUserManager

    user.manager.xml.file=conf/azkaban-users.xml

    # Loader for projects

    executor.global.properties=/usr/local/azkaban/azkaban-exec-server/conf/global.properties

    azkaban.project.dir=projects

    database.type=mysql

    mysql.port=3306

    mysql.host=localhost

    mysql.database=azkaban

    mysql.user=root

    mysql.password=root

    mysql.numconnections=100

    # Velocity dev mode

    velocity.dev.mode=false

    # Azkaban Jetty server properties.

    jetty.use.ssl=false

    jetty.maxThreads=25

    jetty.port=8081

    # Azkaban Executor settings

    executor.maxThreads=50

    executor.port=12321

    executor.flow.threads=30

    # mail settings

    mail.sender=

    mail.host=

    # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.

    # enduser -> myazkabanhost:443 -> proxy -> localhost:8081

    # when this parameters set then these parameters are used to generate email links.

    # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.

    # azkaban.webserver.external_hostname=myazkabanhost.com

    # azkaban.webserver.external_ssl_port=443

    # azkaban.webserver.external_port=8081

    job.failure.email=

    job.success.email=

    lockdown.create.projects=false

    cache.directory=cache

    # JMX stats

    jetty.connector.stats=true

    executor.connector.stats=true

    # Azkaban plugin settings

    azkaban.jobtype.plugin.dir=plugins/jobtypes

     

    azkaban.native.lib=false

     

    在conf目录下,新建配置log4j.properties文件,内容如下:

     

    log4j.rootLogger=INFO,C

    log4j.appender.C=org.apache.log4j.ConsoleAppender

    log4j.appender.C.Target=System.err

    log4j.appender.C.layout=org.apache.log4j.PatternLayout

    log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

     

    15,在hadoop中的/share/Hadoop/common 路径下找到hadoop-common-2.6.4.jar

       在jdk中的/jdk1.8.0_91/db/lib/路径下找到derby.jar

    将这俩个jar包拷贝到azkaban-exec-server和azkaban-web-server
    安装目录下的extlib目录,没有则创建一个,找不到就下载

    16,启动

    进入azkaban-exec-server目录,执行start-exec.sh

    进入azkaban-web-server目录,执行start-web.sh

     

    查看进程:jps

     

    成功执行

    通过http://localhost:8081可访问到web-ui

    16,此时的web界面:

     

    需要更改配置文件解决:

     

    将上图中的web目录的路径加到azkaban.properties配置文件中,如下图

     

    再进入web目录可以看到界面变得美观

     

    17, 用户名密码在 azkaban-web-server/conf/azkaban-users.xml 中配置

  • 相关阅读:
    利用 FFmpeg 和 ImageMagick, AVI 转 GIF(不失真)
    TinyMCE textarea 输入框外部程序动态修改方法
    eclipse快速向下复制行
    ${factoryList }后面有空格不影响
    pre标签
    js备忘录_2
    eclipse 中 大小写切换:ctrl+shift+x 转为大写 ctrl+shift+y 转为小写
    js备忘录_1
    缓存
    myeclipse bug
  • 原文地址:https://www.cnblogs.com/huxinga/p/8706155.html
Copyright © 2011-2022 走看看