zoukankan      html  css  js  c++  java
  • 一个有界任务队列的python thradpoolexcutor, 直接捕获错误日志

    基于官方的需要改版

    1、改为有界,官方是吧所有任务添加到线程池的queue队列中,这样内存会变大,也不符合分布式的逻辑(会把中间件的所有任务一次性取完,放到本地的queue队列中,导致分布式变差)

    2、直接打印错误。官方的threadpolexcutor执行的函数,如果不设置回调,即使函数中出错了,自己都不会知道。

    # coding=utf-8
    """
    一个有界任务队列的thradpoolexcutor
    直接捕获错误日志
    """
    from functools import wraps
    import queue
    from concurrent.futures import ThreadPoolExecutor, Future
    # noinspection PyProtectedMember
    from concurrent.futures.thread import _WorkItem
    from app.utils_ydf import LoggerMixin, LogManager
    
    logger = LogManager('BoundedThreadPoolExecutor').get_logger_and_add_handlers()
    
    
    def _deco(f):
        @wraps(f)
        def __deco(*args, **kwargs):
            try:
                return f(*args, **kwargs)
            except Exception as e:
                logger.exception(e)
    
        return __deco
    
    
    class BoundedThreadPoolExecutor(ThreadPoolExecutor, ):
        def __init__(self, max_workers=None, thread_name_prefix=''):
            ThreadPoolExecutor.__init__(self, max_workers, thread_name_prefix)
            self._work_queue = queue.Queue(max_workers * 2)
    
        def submit(self, fn, *args, **kwargs):
            with self._shutdown_lock:
                if self._shutdown:
                    raise RuntimeError('cannot schedule new futures after shutdown')
                f = Future()
                fn_deco = _deco(fn)
                w = _WorkItem(f, fn_deco, args, kwargs)
                self._work_queue.put(w)
                self._adjust_thread_count()
                return f
    
    
    if __name__ == '__main__':
        def fun():
            print(1 / 0)
    
        pool = BoundedThreadPoolExecutor(10)
        pool.submit(fun)
  • 相关阅读:
    ajax怎么打开新窗口具体如何实现
    关于springcloud hystrix 执行 hystrix.stream 跳转失败的问题
    Zookeeper 和Eureka比较
    Maven Install报错:Perhaps you are running on a JRE rather than a JDK?
    Oracle11g卸载步骤
    Oracle数据库备份及恢复
    python是如何进行内存管理的?
    python面试题
    json模块与hashlib模块的使用
    随机验证码、打印进度条、文件copy脚本
  • 原文地址:https://www.cnblogs.com/ydf0509/p/10744570.html
Copyright © 2011-2022 走看看