Hadoop生态圈-Azkaban部署实战
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.Azkaban部署流程
1>.上传azkaban程序并创建解压目录
[yinzhengjie@s101 data]$ pwd /home/yinzhengjie/data [yinzhengjie@s101 data]$ ll total 401716 -rw-r--r-- 1 yinzhengjie yinzhengjie 16664491 Apr 16 15:37 azkaban-exec-server-3.46.0.tar.gz -rw-r--r-- 1 yinzhengjie yinzhengjie 24333 Apr 16 15:18 azkaban-hadoop-security-plugin-3.46.0.tar.gz -rw-r--r-- 1 yinzhengjie yinzhengjie 349929173 Apr 16 15:11 azkaban-plugins-master.zip -rw-r--r-- 1 yinzhengjie yinzhengjie 23819895 Apr 16 15:18 azkaban-solo-server-3.46.0.tar.gz -rw-r--r-- 1 yinzhengjie yinzhengjie 20899667 Apr 16 15:37 azkaban-web-server-3.46.0.tar.gz -rw-r--r-- 1 yinzhengjie yinzhengjie 11194 Nov 30 2017 create-all-sql-3.46.0.sql [yinzhengjie@s101 data]$ [yinzhengjie@s101 data]$ mkdir /soft/azkaban [yinzhengjie@s101 data]$
2>.分别解压文件到/soft/azkaban中
[yinzhengjie@s101 data]$ tar -xzf azkaban-exec-server-3.46.0.tar.gz -C /soft/azkaban/ [yinzhengjie@s101 data]$ tar -xzf azkaban-web-server-3.46.0.tar.gz -C /soft/azkaban/ [yinzhengjie@s101 data]$
3>.进入到mysql命令行并创建数据库
[yinzhengjie@s101 data]$ mysql -uroot -pyinzhengjie Warning: Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or g. Your MySQL connection id is 2 Server version: 5.6.38 MySQL Community Server (GPL) Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. mysql> create database yinzhengjie_azkaban; Query OK, 1 row affected (0.00 sec) mysql> use yinzhengjie_azkaban; Database changed mysql> source /home/yinzhengjie/data/create-all-sql-3.46.0.sql Query OK, 0 rows affected (0.03 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.01 sec) Query OK, 0 rows affected (0.00 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.00 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.06 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.13 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.22 sec) Query OK, 0 rows affected (0.15 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.18 sec) Query OK, 0 rows affected (0.13 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.17 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.03 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.06 sec) Query OK, 0 rows affected (0.33 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.01 sec) Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.22 sec) Query OK, 0 rows affected (0.12 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.09 sec) Query OK, 0 rows affected (0.10 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.17 sec) Query OK, 0 rows affected (0.18 sec) Query OK, 0 rows affected (0.14 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.12 sec) Query OK, 0 rows affected (0.04 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.08 sec) Query OK, 0 rows affected (0.17 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.16 sec) Query OK, 0 rows affected (0.13 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.13 sec) Query OK, 0 rows affected (0.10 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.04 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected (0.19 sec) Query OK, 0 rows affected (0.23 sec) Query OK, 0 rows affected (0.08 sec) Query OK, 0 rows affected (0.13 sec) Query OK, 0 rows affected (0.01 sec) Query OK, 0 rows affected (0.06 sec) Query OK, 0 rows affected (0.19 sec) Query OK, 0 rows affected (0.27 sec) Query OK, 0 rows affected (0.18 sec) Query OK, 0 rows affected (0.16 sec) Query OK, 0 rows affected (0.17 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.19 sec) mysql> exit; Bye [yinzhengjie@s101 data]$
4>.创建SSL配置（完成下述工作后,将在当前目录生成 keystore 证书文件，需要输出初始密码，选择国家，在输入确认字母y其他按回车即可）
[yinzhengjie@s101 data]$ keytool -keystore keystore -alias jetty -genkey -keyalg RSA Enter keystore password: Re-enter new password: What is your first and last name? [Unknown]: What is the name of your organizational unit? [Unknown]: What is the name of your organization? [Unknown]: What is the name of your City or Locality? [Unknown]: What is the name of your State or Province? [Unknown]: What is the two-letter country code for this unit? [Unknown]: CN Is CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=CN correct? [no]: y Enter key password for <jetty> (RETURN if same as keystore password): [yinzhengjie@s101 data]$ [yinzhengjie@s101 data]$ ll | grep keystore -rw-rw-r-- 1 yinzhengjie yinzhengjie 2232 Jun 24 02:48 keystore [yinzhengjie@s101 data]$
5>.将keystore文件拷贝到web文件夹下
[yinzhengjie@s101 data]$ ll | grep keystore -rw-rw-r-- 1 yinzhengjie yinzhengjie 2232 Jun 24 02:48 keystore [yinzhengjie@s101 data]$ cp keystore /soft/azkaban/azkaban-web-server-3.46.0/ [yinzhengjie@s101 data]$
6>.编辑配置文件【azkaban.properties】
[yinzhengjie@s101 data]$ more /soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban.properties #Azkaban Personalization Settings #服务器UI名称,用于服务器上方显示的名字 azkaban.name=yinzhengjie #描述 azkaban.label=My Local Azkaban #UI颜色 azkaban.color=#FF3601 #指定首页路径 azkaban.default.servlet.path=/index #默认根web目录 web.resource.dir=/soft/azkaban/azkaban-web-server-3.46.0/web/ #默认时区,已改为亚洲/上海 默认为美国 default.timezone.id=Asia/Shanghai #Azkaban UserManager class #用户权限管理默认类 user.manager.class=azkaban.user.XmlUserManager #用户配置,具体配置参加下文 user.manager.xml.file=/soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban-users.xml #Loader for projects # global配置文件所在位置 executor.global.properties=/soft/azkaban/azkaban-web-server-3.46.0/conf/global.properties azkaban.project.dir=projects #数据库类型 database.type=mysql #端口号 mysql.port=3306 #数据库连接IP mysql.host=s101 #数据库实例名 mysql.database=azkaban #数据库用户名 mysql.user=root #数据库密码 mysql.password=yinzhengjie #最大连接数 mysql.numconnections=100 # Velocity dev mode #关闭快速开发模式 velocity.dev.mode=false # Jetty服务器属性. #最大线程数 jetty.maxThreads=25 #Jetty SSL端口 jetty.ssl.port=8443 #Jetty端口 jetty.port=8081 #SSL文件名 jetty.keystore=keystore #SSL文件密码 jetty.password=yinzhengjie #Jetty主密码 与 keystore文件相同 jetty.keypassword=yinzhengjie #SSL文件名 jetty.truststore=keystore # SSL文件密码 jetty.trustpassword=yinzhengjie # 执行服务器属性 #执行服务器端口 executor.port=12321 # 邮件设置 #发送邮箱 mail.sender=xxxxxxxx@163.com #发送邮箱smtp地址 mail.host=smtp.163.com #发送邮件时显示的名称 mail.user=xxxxxxxx #邮箱密码 mail.password=********** #任务失败时发送邮件的地址 job.failure.email=xxxxxxxx@163.com #任务成功时发送邮件的地址 job.success.email=xxxxxxxx@163.com lockdown.create.projects=false #缓存目录 cache.directory=cache # JMX stats jetty.connector.stats=true executor.connector.stats=true # Azkaban JobTypes 插件配置 azkaban.jobtype.plugin.dir=plugins/jobtypes [yinzhengjie@s101 data]$
以上代码只是对该配置文件的一个详细说明，不建议在这个配置文件中写中文！在实际配置中，需要将这个文件拷贝两份，分别放在web和exec目录中，实际使用配置如下：
[yinzhengjie@s101 ~]$ more /soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban.properties # Azkaban Personalization Settings azkaban.name=yinzhengjie azkaban.label=My Local Azkaban azkaban.color=#FF3601 azkaban.default.servlet.path=/index web.resource.dir=/soft/azkaban/azkaban-web-server-3.46.0/web/ default.timezone.id=Asia/Shanghai # Azkaban UserManager class user.manager.class=azkaban.user.XmlUserManager user.manager.xml.file=/soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban-users.xml # Loader for projects executor.global.properties=/soft/azkaban/azkaban-web-server-3.46.0/conf/global.properties azkaban.project.dir=projects database.type=mysql mysql.port=3306 mysql.host=s101 mysql.database=yinzhengjie_azkaban mysql.user=root mysql.password=yinzhengjie mysql.numconnections=100 # Velocity dev mode velocity.dev.mode=false # Azkaban Jetty server properties. jetty.use.ssl=false jetty.maxThreads=25 jetty.port=8081 # Azkaban Executor settings executor.port=12321 # mail settings mail.sender= mail.host= # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users. # enduser -> myazkabanhost:443 -> proxy -> localhost:8081 # when this parameters set then these parameters are used to generate email links. # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used. # azkaban.webserver.external_hostname=myazkabanhost.com # azkaban.webserver.external_ssl_port=443 # azkaban.webserver.external_port=8081 job.failure.email= job.success.email= lockdown.create.projects=false cache.directory=cache # JMX stats jetty.connector.stats=true executor.connector.stats=true # Azkaban plugin settings azkaban.jobtype.plugin.dir=plugins/jobtypes [yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ more /soft/azkaban/azkaban-exec-server-3.46.0/conf/azkaban.properties # Azkaban Personalization Settings azkaban.name=yinzhengjie azkaban.label=My Local Azkaban azkaban.color=#FF3601 azkaban.default.servlet.path=/index web.resource.dir=/soft/azkaban/azkaban-web-server-3.46.0/web/ default.timezone.id=Asia/Shanghai # Azkaban UserManager class user.manager.class=azkaban.user.XmlUserManager user.manager.xml.file=/soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban-users.xml # Loader for projects executor.global.properties=/soft/azkaban/azkaban-web-server-3.46.0/conf/global.properties azkaban.project.dir=projects database.type=mysql mysql.port=3306 mysql.host=s101 mysql.database=yinzhengjie_azkaban mysql.user=root mysql.password=yinzhengjie mysql.numconnections=100 # Velocity dev mode velocity.dev.mode=false # Azkaban Jetty server properties. jetty.use.ssl=false jetty.maxThreads=25 jetty.port=8081 # Azkaban Executor settings executor.port=12321 # mail settings mail.sender= mail.host= # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users. # enduser -> myazkabanhost:443 -> proxy -> localhost:8081 # when this parameters set then these parameters are used to generate email links. # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used. # azkaban.webserver.external_hostname=myazkabanhost.com # azkaban.webserver.external_ssl_port=443 # azkaban.webserver.external_port=8081 job.failure.email= job.success.email= lockdown.create.projects=false cache.directory=cache # JMX stats jetty.connector.stats=true executor.connector.stats=true # Azkaban plugin settings azkaban.jobtype.plugin.dir=plugins/jobtypes [yinzhengjie@s101 ~]$
7>.修改用户配置文件【azkaban-users.xml】
[yinzhengjie@s101 data]$ more /soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban-users.xml <azkaban-users> <user username="azkaban" password="yinzhengjie" roles="admin" groups="azkaban" /> <user username="metrics" password="yinzhengjie" roles="metrics"/> <user username="admin" password="yinzhengjie" roles="admin,metrics" /> <role name="admin" permissions="ADMIN" /> <role name="metrics" permissions="METRICS"/> </azkaban-users> [yinzhengjie@s101 data]$
8>.创建global.properties
[yinzhengjie@s101 data]$ touch /soft/azkaban/azkaban-web-server-3.46.0/conf/global.properties [yinzhengjie@s101 data]$
9>.编写启动脚本
[yinzhengjie@s101 ~]$ more /usr/local/bin/azweb.sh #!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:y1053419035@qq.com #判断用户是否传参 if [ $# -ne 1 ];then echo "无效参数，用法为: $0 {start|stop}" exit fi #获取用户输入的命令 cmd=$1 case $cmd in start) cd /soft/azkaban /soft/azkaban/azkaban-web-server-3.46.0/bin/start-web.sh ;; stop) cd /soft/azkaban /soft/azkaban/azkaban-web-server-3.46.0/bin/shutdown-web.sh ;; *) echo "无效参数，用法为: $0 {start|stop}" exit ;; esac echo ========= azweb $cmd ============= [yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ more /usr/local/bin/azexec.sh #!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:y1053419035@qq.com #判断用户是否传参 if [ $# -ne 1 ];then echo "无效参数，用法为: $0 {start|stop}" exit fi #获取用户输入的命令 cmd=$1 case $cmd in start) cd /soft/azkaban /soft/azkaban/azkaban-exec-server-3.46.0/bin/start-exec.sh ;; stop) cd /soft/azkaban /soft/azkaban/azkaban-exec-server-3.46.0/bin/shutdown-exec.sh ;; *) echo "无效参数，用法为: $0 {start|stop}" exit ;; esac echo ========= azexec $cmd ============= [yinzhengjie@s101 ~]$
10>.启动服务（启动web和executor）
[yinzhengjie@s101 ~]$ jps 4970 Jps [yinzhengjie@s101 ~]$ [yinzhengjie@s101 ~]$ azweb.sh start ========= azweb start ============= [yinzhengjie@s101 ~]$ azexec.sh start ========= azexec start ============= [yinzhengjie@s101 ~]$ jps 5024 AzkabanExecutorServer 4993 AzkabanWebServer 5044 Jps [yinzhengjie@s101 ~]$ netstat -untalp | grep 8081 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp6 0 0 :::8081 :::* LISTEN 4993/java [yinzhengjie@s101 ~]$
11>.测试azkaban连接
输入用户名和密码信息：
登录成功会有以下页面：
二. Azkaban防坑小技巧
1>.job文件有空格
解决方案：
修改command.job配置文件，将里面多余的空格删除掉！然后重新上传job任务。
2>.Missing required property 'azkaban.native.lib'
解决方案：
[yinzhengjie@s101 ~]$ grep azkaban.jobtype.plugin.dir /soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban.properties
azkaban.jobtype.plugin.dir=/soft/azkaban/azkaban-exec-server-3.46.0/plugins/jobtypes/
[yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ grep azkaban.jobtype.plugin.dir /soft/azkaban/azkaban-exec-server-3.46.0/conf/azkaban.properties
azkaban.jobtype.plugin.dir=/soft/azkaban/azkaban-exec-server-3.46.0/plugins/jobtypes/
[yinzhengjie@s101 ~]$
3>.
4>.