zoukankan      html  css  js  c++  java
  • airflow分布式部署(五)开机自启动

    airflow是一个常驻任务,我们希望它在任务失败后可以自动重启,systemd可以帮助我们实现这个功能,首先需要配置如下文件

    /etc/sysconfig/airflow

    AIRFLOW_CONFIG=/root/airflow/airflow.cfg
    AIRFLOW_HOME=/root/airflow

    /usr/lib/systemd/system/airflow-webserver.service

    [Unit]
    Description=Airflow webserver daemon
    
    [Service]
    EnvironmentFile=/etc/sysconfig/airflow
    User=root
    Group=root
    Type=simple
    ExecStart=/root/miniconda3/envs/py36/bin/airflow webserver
    Restart=on-failure
    RestartSec=5s
    PrivateTmp=true
    
    [Install]
    WantedBy=multi-user.target

    /usr/lib/systemd/system/airflow-flower.service

    [Unit]
    Description=Airflow flower daemon
    [Service]
    EnvironmentFile=/etc/sysconfig/airflow
    User=root
    Group=root
    Type=simple
    ExecStart=/root/miniconda3/envs/py36/bin/airflow flower
    Restart=on-failure
    RestartSec=5s
    PrivateTmp=true

    /usr/lib/systemd/system/airflow-scheduler.service

    [Unit]
    Description=Airflow scheduler daemon
    After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
    Wants=redis.service
    
    [Service]
    EnvironmentFile=/etc/sysconfig/airflow
    User=root
    Group=root
    Type=simple
    ExecStart=/root/miniconda3/envs/py36/bin/airflow scheduler
    Restart=always
    RestartSec=5s
    
    [Install]
    WantedBy=multi-user.target

    /usr/lib/systemd/system/airflow-worker.service

    [Unit]
    Description=Airflow celery worker daemon
    After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
    Wants=mysql.service redis.service
    
    [Service]
    EnvironmentFile=/etc/sysconfig/airflow
    User=root
    Group=root
    Type=simple
    ExecStart=/root/miniconda3/envs/py36/bin/airflow worker
    Restart=on-failure
    RestartSec=10s
    
    [Install]
    WantedBy=multi-user.target

    配置完,之后执行以下命令

    #重载所有修改过的配置文件
    sudo systemctl daemon-reload
    #查看启动耗时,关键是可以查看启动失败的原因
    sudo systemd-analyze verify airflow-webserver.service
    #查看启动失败任务的启动日志
    sudo journalctl -u airflow-webserver.service
    #查看服务状态
    systemctl status airflow-scheduler.service
    #判断是否自动重启
    sudo systemctl is-enabled airflow-worker.service
    sudo systemctl enable airflow-worker.service
    #重启任务
    systemctl stop airflow-scheduler.service
    systemctl start airflow-scheduler.service

     报错记录

    FileNotFoundError: [Errno 2] No such file or directory: 'gunicorn'

    这是因为没有把python3的path加入到PATH中,解决思路找到python3的路径加入到PATH即可

    备注:

    又有报错FileNotFoundError: [Errno 2] No such file or directory: 'airflow': 'airflow'

    反反复复始终未能解决,最终webserver/scheduler以systemctl方式启动,worker和flower以airflow worker -D的方式启动



  • 相关阅读:
    正则表达式教程
    LINQ查询基本操作
    C# LINQ 基本操作实例
    Sublime Text 3 常用插件以及安装方法
    记一次矩阵列单元格合并和拆分组件的开发
    解析JavaScript函数的多种写法
    jQuery源码逐行分析学习02(第一部分:jQuery的一些变量和函数)
    jQuery源码逐行分析学习01(jQuery的框架结构简化)
    Vue IIS 403-Forbidden
    数组求和:二分递归
  • 原文地址:https://www.cnblogs.com/wangbin2188/p/13926301.html
Copyright © 2011-2022 走看看