zoukankan      html  css  js  c++  java
  • django+celery+redis实现运行定时任务

    0、目的

    在开发项目中,经常有一些操作时间比较长(生产环境中超过了nginx的timeout时间),或者是间隔一段时间就要执行的任务。
    在这种情况下,使用celery就是一个很好的选择。
     
    celery是一个异步任务队列/基于分布式消息传递的作业队列。
    Celery通过消息(message)进行通信,使用代理(broker)在客户端和工作执行者之间进行交互。
    当开始一个任务时,客户端发送消息到队列并由代理将其发往响应的工作执行者处。
     
    准备使用redis作为消息代理(broker),Django数据库作为结果存储(ResultStore)。
     

    1、安装

    redis:
    windows:
    linux:
    yum install redis-server
    PS:需要在cmd中运行,不能再powercmd。很奇怪。
    pip install celery
    pip install celery-with-redis
    pip install django-celery
     

    2、django代码(whthas_home为project,portal为app)

    修改代码,whthas_home/__init__.py
    1 from __future__ import absolute_import
    2 
    3 from .celery import app as celery_app

    修改代码,whthas_home/setting.py

    复制代码
     1 # Celery settings
     2 import djcelery
     3 djcelery.setup_loader()
     4 
     5 BROKER_URL = 'redis://127.0.0.1:6379/0'
     6 CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'##加密方式CELERY_RESULT_BACKEND = 'redis://:密码@127.0.0.1:6379/0' 
     7 CELERY_ACCEPT_CONTENT = ['json'] 
     8 CELERY_TASK_SERIALIZER = 'json' 
     9 CELERY_RESULT_SERIALIZER = 'json'
     10 CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
     11 CELERY_TIMEZONE = 'Asia/Shanghai'
     12 
     13 # Application definition
     14 
     15 INSTALLED_APPS = [
     16     'django.contrib.auth',
     17     'django.contrib.contenttypes',
     18     'django.contrib.sessions',
     19     'django.contrib.messages',
     20     'django.contrib.staticfiles',
     21     'suit',
     22     'django.contrib.admin',
     23     'DjangoUeditor',
     24     'portal',
     25     'djcelery',
     26 ]
    复制代码

    增加文件, whthas_home/celery.py

    复制代码
     1 from __future__ import absolute_import
     2 
     3 import os
     4 from celery import Celery
     5 from django.conf import settings
     6 
     7 os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'whthas_home.settings')
     8 
     9 app = Celery('portal')
    10 
    11 app.config_from_object('django.conf:settings')
    12 app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
    13 
    14 
    15 @app.task(bind=True)
    16 def debug_task(self):
    17     print('Request: {0!r}'.format(self.request))
    复制代码
    增加文件,portal/tasks.py
    复制代码
     1 from celery import task
     2 from time import sleep
     3 
     4 
     5 @task()
     6 def Task_A(message):
     7     Task_A.update_state(state='PROGRESS', meta={'progress': 0})
     8     sleep(10)
     9     Task_A.update_state(state='PROGRESS', meta={'progress': 30})
    10     sleep(10)
    11     return message
    12 
    13 
    14 def get_task_status(task_id):
    15     task = Task_A.AsyncResult(task_id)
    16 
    17     status = task.state
    18     progress = 0
    19 
    20     if status == u'SUCCESS':
    21         progress = 100
    22     elif status == u'FAILURE':
    23         progress = 0
    24     elif status == 'PROGRESS':
    25         progress = task.info['progress']
    26     
    27     return {'status': status, 'progress': progress}
    复制代码

    3、测试

    启动broker:python manage.py celeryd -l info
    进入shell:python manage.py shell,测试程序是否正常。 
    复制代码
    >>> from portal.tasks import *
    >>> t = TaskA.delay("heel2")
    >>> get_task_status(t.id)
    {'status': u'PROGRESS', 'progress': 0}
    >>> get_task_status(t.id)
    {'status': u'PROGRESS', 'progress': 0}
    >>> get_task_status(t.id)
    {'status': u'PROGRESS', 'progress': 30}
    >>> get_task_status(t.id)
    {'status': u'PROGRESS', 'progress': 30}
    >>> get_task_status(t.id)
    {'status': u'SUCCESS', 'progress': 100}
    >>> 
    复制代码

    同时broker侧能看到: 

    [2017-04-21 16:38:47,023: INFO/MainProcess] Received task: portal.tasks.Task_A[da948495-c64b-4ff9-882b-876721cd5017]
    [2017-04-21 16:39:07,035: INFO/MainProcess] Task portal.tasks.Task_A[da948495-c64b-4ff9-882b-876721cd5017] succeeded in 20.0099999905s: heel

    表示代码能正常运行。

    使用redis-client看到任务状态:
    127.0.0.1:6379> get "celery-task-meta-da948495-c64b-4ff9-882b-876721cd5017"
    "{"status": "SUCCESS", "traceback": null, "result": "heel", "children": []}"

    相关redis命令:keys *

     4、django后台定义任务

    进入后台
     设置任务

    5、执行任务

    启动broker:python manage.py celeryd -l info

    因为这里是个定时任务,所以还需要启动心跳 :python manage.py celery beat  
    broker侧能看到: 
    复制代码
    [2017-04-21 16:56:33,216: INFO/MainProcess] Received task: portal.tasks.Task_A[d8a26977-8413-4bf0-b518-b53052af4cee]
    [2017-04-21 16:56:53,211: INFO/MainProcess] Received task: portal.tasks.Task_A[00f1cab7-eb56-4cc3-9979-a8a68aaaf2de]
    [2017-04-21 16:56:53,220: INFO/MainProcess] Task portal.tasks.Task_A[d8a26977-8413-4bf0-b518-b53052af4cee] succeeded in 20.003000021s: heel2
    [2017-04-21 16:57:13,211: INFO/MainProcess] Received task: portal.tasks.Task_A[aa652612-8525-4110-94ea-9010085ec20b]
    [2017-04-21 16:57:13,223: INFO/MainProcess] Task portal.tasks.Task_A[00f1cab7-eb56-4cc3-9979-a8a68aaaf2de] succeeded in 20.0080001354s: heel2
    [2017-04-21 16:57:33,213: INFO/MainProcess] Received task: portal.tasks.Task_A[9876890e-6a71-4501-bdae-775492ebae88]
    [2017-04-21 16:57:33,219: INFO/MainProcess] Task portal.tasks.Task_A[aa652612-8525-4110-94ea-9010085ec20b] succeeded in 20.0050001144s: heel2
    [2017-04-21 16:57:53,211: INFO/MainProcess] Received task: portal.tasks.Task_A[c12fcffc-3910-4a22-93b3-0df740910728]
    [2017-04-21 16:57:53,221: INFO/MainProcess] Task portal.tasks.Task_A[9876890e-6a71-4501-bdae-775492ebae88] succeeded in 20.0069999695s: heel2
    [2017-04-21 16:58:13,211: INFO/MainProcess] Received task: portal.tasks.Task_A[ccfad575-c0b4-48f5-9385-85ff5dac76fc]
    [2017-04-21 16:58:13,217: INFO/MainProcess] Task portal.tasks.Task_A[c12fcffc-3910-4a22-93b3-0df740910728] succeeded in 20.003000021s: heel2
    [2017-04-21 16:58:33,211: INFO/MainProcess] Received task: portal.tasks.Task_A[0d6f77a6-a29c-4ead-9428-e3e758c754e1]
    [2017-04-21 16:58:33,221: INFO/MainProcess] Task portal.tasks.Task_A[ccfad575-c0b4-48f5-9385-85ff5dac76fc] succeeded in 20.007999897s: heel2
    [2017-04-21 16:58:53,211: INFO/MainProcess] Received task: portal.tasks.Task_A[34b2b3a0-771c-4a28-94a1-92e25763fae1]
    [2017-04-21 16:58:53,217: INFO/MainProcess] Task portal.tasks.Task_A[0d6f77a6-a29c-4ead-9428-e3e758c754e1] succeeded in 20.0039999485s: heel2
    [2017-04-21 16:59:13,211: INFO/MainProcess] Received task: portal.tasks.Task_A[d4be920f-376a-46a2-9edd-095234d29ef2]
    [2017-04-21 16:59:13,217: INFO/MainProcess] Task portal.tasks.Task_A[34b2b3a0-771c-4a28-94a1-92e25763fae1] succeeded in 20.0039999485s: heel2
    复制代码
    可见:设置定时任务成功。
     
    参考资料:
    1、https://gist.github.com/tyrchen/1436486,Django-celery + Redis notes
    2、http://www.cnblogs.com/aguncn/p/4947092.html,django celery redis简单测试
    3、https://my.oschina.net/kinegratii/blog/292395?fromerr=2lvw3H0L,djcelery入门:实现运行定时任务
  • 相关阅读:
    【知了堂学习笔记】java 正则表达式
    【知了堂学习笔记】java 接口与抽象类
    【知了堂学习笔记】java web 简单的登录
    【知了堂学习笔记】java IO流归纳总结
    【知了堂学习笔记】java 自定义异常
    【知了堂学习笔记】java 方法重载与重写的归纳
    编译链接符号重定位流程简述
    项目用到了lua的哪些部分
    lua协程实现简析
    杂乱无章
  • 原文地址:https://www.cnblogs.com/lykops/p/8263109.html
Copyright © 2011-2022 走看看