在docker容器下安装airflow

本人的环境是基于centos7下来安装的

一、安装docker 

下载docker安装包，下载地址：https://download.docker.com/linux/static/stable/x86_64/

下载到本地后解压

tar -zxf docker-18.09.6.tgz 

将解压出来的docker文件内容移动到 /usr/bin/ 目录下

cp docker/* /usr/bin/

将docker注册为service

新建文件

vim /etc/systemd/system/docker.service

并添加以下内容

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
 
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
 
[Install]
WantedBy=multi-user.target

添加文件权限

chmod +x /etc/systemd/system/docker.service
systemctl daemon-reload

启动docker

systemctl start docker

验证

systemctl status docker			#查看Docker状态
docker -v			#查看Docker版本

　　

二、安装airflow容器

 先在宿主机安装mysql

上传mysql安装包

先卸载自带的mysql相关的包

 rpm -qa|grep mariadb
mariadb-libs-5.5.60-1.el7_5.x86_64
$ rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Permission denied)
$ sudo rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
$ rpm -qa|grep mariadb

再依次安装

bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-common-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-common-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-common-5.7.27-1.e################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-libs-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-libs-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-libs-5.7.27-1.el7################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-libs-compat-5.7.2################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-embedded-compat-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-embedded-compat-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-embedded-compat-5################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-devel-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-devel-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-devel-5.7.27-1.el################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-client-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-client-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-client-5.7.27-1.e################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-server-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-server-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-server-5.7.27-1.e################################# [100%]
[bigdata@bigdata mysql]$

skip-grant-tables #添加这句话，这时候登入mysql就不需要密码

 

登录mysql，输入密码处直接回车就可以

配置msyql

mysql> flush privileges; 
Query OK, 0 rows affected (0.00 sec)

mysql> set password for root@localhost = password('Admin123!');
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> create user 'airflow'@'localhost' identified by 'Airflow123!';
Query OK, 0 rows affected (0.05 sec)

mysql> create database airflow;
Query OK, 1 row affected (0.00 sec)

mysql> create user 'airflow'@'%' identified by 'AirFlow123!';
Query OK, 0 rows affected (0.02 sec)

mysql> GRANT all privileges on airflow.* TO 'airflow'@'%'  IDENTIFIED BY 'AirFlow123!';
Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> GRANT all privileges on airflow.* TO 'root'@'172.17.0.2' IDENTIFIED BY 'Admin123!';
Query OK, 0 rows affected, 1 warning (0.04 sec)

//172.17.0.2 是我的docker容器的地址

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

 修改/etc/my.cnf配置文件，在[mysqld]下添加编码配置

 

character_set_server=utf8
init_connect='SET NAMES utf8'

 vim /etc/my.cnf

 添加这一句

explicit_defaults_for_timestamp=1

编辑保存完 重启mysql服务: sudo systemctl restart mysqld.service

 

三、docker上安装airflow

git clone https://github.com/puckel/docker-airflow.git /root/airflow //下载源码到airflow文件夹

docker run --name airflow6  -v /usr/local/airflow/dags:/usr/local/airflow/dags -v /usr/local/airflow/airflowSql:/usr/local/airflow/airflowSql  -v /home/airflow/airflow.cfg:/usr/local/airflow/airflow.cfg -d -p 8086:8080 --privileged puckel/docker-airflow   //运行容器

/* 我这里把本地的目录映射到容器里面去  通过 -v 参数进行映射**/

  进入容器 

docker exec -it -u root airflow6 bash

修改配置文件

自动扫描.py文件

修改scheduler线程数控制并发量（这个通常在性能调优方面）

修改检查新的dag的间隔

airflow initdb  // 初始化数据库

出现错误：

airflow.exceptions.AirflowException: Could not create Fernet object: Incorrect padding

解决办法：

python -c "from cryptography.fernet import Fernet;

print(Fernet.generate_key().decode())"

 

export AIRFLOW__CORE__FERNET_KEY=oNu9XwewQNyx9mAJT2vZvtm3qzPRZIWRqwk9hSVch4A=

airflow initdb // 重新运行初始化数据库

打开浏览器