zoukankan      html  css  js  c++  java
  • 一个有界任务队列的python thradpoolexcutor, 直接捕获错误日志

    基于官方的需要改版

    1、改为有界,官方是吧所有任务添加到线程池的queue队列中,这样内存会变大,也不符合分布式的逻辑(会把中间件的所有任务一次性取完,放到本地的queue队列中,导致分布式变差)

    2、直接打印错误。官方的threadpolexcutor执行的函数,如果不设置回调,即使函数中出错了,自己都不会知道。

    # coding=utf-8
    """
    一个有界任务队列的thradpoolexcutor
    直接捕获错误日志
    """
    from functools import wraps
    import queue
    from concurrent.futures import ThreadPoolExecutor, Future
    # noinspection PyProtectedMember
    from concurrent.futures.thread import _WorkItem
    from app.utils_ydf import LoggerMixin, LogManager
    
    logger = LogManager('BoundedThreadPoolExecutor').get_logger_and_add_handlers()
    
    
    def _deco(f):
        @wraps(f)
        def __deco(*args, **kwargs):
            try:
                return f(*args, **kwargs)
            except Exception as e:
                logger.exception(e)
    
        return __deco
    
    
    class BoundedThreadPoolExecutor(ThreadPoolExecutor, ):
        def __init__(self, max_workers=None, thread_name_prefix=''):
            ThreadPoolExecutor.__init__(self, max_workers, thread_name_prefix)
            self._work_queue = queue.Queue(max_workers * 2)
    
        def submit(self, fn, *args, **kwargs):
            with self._shutdown_lock:
                if self._shutdown:
                    raise RuntimeError('cannot schedule new futures after shutdown')
                f = Future()
                fn_deco = _deco(fn)
                w = _WorkItem(f, fn_deco, args, kwargs)
                self._work_queue.put(w)
                self._adjust_thread_count()
                return f
    
    
    if __name__ == '__main__':
        def fun():
            print(1 / 0)
    
        pool = BoundedThreadPoolExecutor(10)
        pool.submit(fun)
  • 相关阅读:
    Netty 超时机制及心跳程序实现
    ZJUTACM
    寻找素数对
    HDU 1021 Fibonacci Again
    HDU 1019 Least Common Multiple
    HDU 1017 A Mathematical Curiosity
    HDU 1014 Uniform Generator
    HDU 1013 Digital Roots
    HDU 1008 Elevator
    Educational Codeforces Round 2 B. Queries about less or equal elements
  • 原文地址:https://www.cnblogs.com/ydf0509/p/10744570.html
Copyright © 2011-2022 走看看