zoukankan      html  css  js  c++  java
  • Python数据库连接池DBUtils详解

    what's the DBUtils

      DBUtils 是一套用于管理数据库连接池的Python包,为高频度高并发的数据库访问提供更好的性能,可以自动管理连接对象的创建和释放。并允许对非线程安全的数据库接口进行线程安全包装。

    DBUtils提供两种外部接口:

    • PersistentDB :提供线程专用的数据库连接,并自动管理连接。
    • PooledDB :提供线程间可共享的数据库连接,并自动管理连接。

    实测证明 PersistentDB 的速度是最高的,但是在某些特殊情况下,数据库的连接过程可能异常缓慢,而此时的PooledDB则可以提供相对来说平均连接时间比较短的管理方式。

    另外,实际使用的数据库驱动也有所依赖,比如SQLite数据库只能使用PersistentDB作连接池。 下载地址:http://www.webwareforpython.org/downloads/DBUtils/

    DBUtils使用方法

      连接池对象只初始化一次,一般可以作为模块级代码来确保。 PersistentDB的连接例子:

    import DBUtils.PersistentDB 
    persist=DBUtils.PersistentDB.PersistentDB(dbpai=MySQLdb,maxusage=1000,**kwargs)

      这里的参数dbpai指使用的底层数据库模块,兼容DB-API的。maxusage则为一个连接最大使用次数,参考了官方例子。后面的**kwargs则为实际传递给MySQLdb的参数。

      获取连接: conn=persist.connection() 实际编程中用过的连接直接关闭 conn.close() 即可将连接交还给连接池。

    PooledDB使用方法同PersistentDB,只是参数有所不同。

    • dbapi :数据库接口
    • mincached :启动时开启的空连接数量
    • maxcached :连接池最大可用连接数量
    • maxshared :连接池最大可共享连接数量
    • maxconnections :最大允许连接数量
    • blocking :达到最大数量时是否阻塞
    • maxusage :单个连接最大复用次数
    • setsession :用于传递到数据库的准备会话,如 [”set name UTF-8″] 。
    db=pooled.connection()
    cur=db.cursor()
    cur.execute(sql)
    res=cur.fetchone()
    cur.close() # or del cur
    db.close() # or del db
    使用栗子

    python不用连接池的MySQL连接方法

    import MySQLdb
    conn= MySQLdb.connect(host='localhost',user='root',passwd='pwd',db='myDB',port=3306)  
    #import pymysql
    #conn = pymysql.connect(host='localhost', port='3306', db='game', user='root', password='123456', charset='utf8')
    cur=conn.cursor()
    SQL="select * from table1"
    r=cur.execute(SQL)
    r=cur.fetchall()
    cur.close()
    conn.close()

    用连接池后的连接方法

    import MySQLdb
    from DBUtils.PooledDB import PooledDB
    pool = PooledDB(MySQLdb,5,host='localhost',user='root',passwd='pwd',db='myDB',port=3306) #5为连接池里的最少连接数
    
    conn = pool.connection()  #以后每次需要数据库连接就是用connection()函数获取连接就好了
    cur=conn.cursor()
    SQL="select * from table1"
    r=cur.execute(SQL)
    r=cur.fetchall()
    cur.close()
    conn.close()

    DBUtils下载地址:https://pypi.python.org/pypi/DBUtils/

    import sys
    import threading
    import MySQLdb
    import DBUtils.PooledDB
    
    connargs = { "host":"localhost", "user":"user1", "passwd":"123456", "db":"test" }
    def test(conn):
        try:
            cursor = conn.cursor()
            count = cursor.execute("select * from users")
            rows = cursor.fetchall()
            for r in rows: pass
        finally:
            conn.close()
            
    def testloop():
        print ("testloop")
        for i in range(1000):
            conn = MySQLdb.connect(**connargs)
            test(conn)
            
    def testpool():
        print ("testpool")
        pooled = DBUtils.PooledDB.PooledDB(MySQLdb, **connargs)
        for i in range(1000):
            conn = pooled.connection()
            test(conn)
            
    def main():
        t = testloop if len(sys.argv) == 1 else testpool
        for i in range(10):
            threading.Thread(target = t).start()
            
    if __name__ == "__main__":
        main()
    测试代码

    看看 10 线程的测试结果。

    $ time ./main.py  
    testloop  
    testloop  
    testloop  
    testloop  
    testloop  
    testloop  
    testloop  
    testloop  
    testloop  
    testloop  
    real    0m4.471s  
    user    0m0.570s  
    sys     0m4.670s  
    $ time ./main.py -l  
    testpool  
    testpool  
    testpool  
    testpool  
    testpool  
    testpool  
    testpool  
    testpool  
    testpool  
    testpool  
    real    0m2.637s  
    user    0m0.320s  
    sys     0m2.750s  
    View Code

      虽然测试方式不是很严谨,但从测试结果还是能感受到 DBUtils 带来的性能提升。当然,我们我们也可以在 testloop() 中一直重复使用一个不关闭的 Connection,但这却不适合实际开发时的情形。

    DBUtils 提供了几个参数,便于我们更好地调整资源利用。

    DBUtils.PooledDB.PooledDB(self, creator,   
        mincached=0, maxcached=0, maxshared=0, maxconnections=0, blocking=False, maxusage=None,   
        setsession=None, failures=None, *args, **kwargs)  
    Docstring:  
        Set up the DB-API 2 connection pool.  
        creator: either an arbitrary function returning new DB-API 2  
            connection objects or a DB-API 2 compliant database module  
        mincached: initial number of idle connections in the pool  
            (0 means no connections are made at startup)  
        maxcached: maximum number of idle connections in the pool  
            (0 or None means unlimited pool size)  
        maxshared: maximum number of shared connections  
            (0 or None means all connections are dedicated)  
            When this maximum number is reached, connections are  
            shared if they have been requested as shareable.  
        maxconnections: maximum number of connections generally allowed  
            (0 or None means an arbitrary number of connections)  
        blocking: determines behavior when exceeding the maximum  
            (if this is set to true, block and wait until the number of  
            connections decreases, otherwise an error will be reported)  
        maxusage: maximum number of reuses of a single connection  
            (0 or None means unlimited reuse)  
            When this maximum usage number of the connection is reached,  
            the connection is automatically reset (closed and reopened).  
        setsession: optional list of SQL commands that may serve to prepare  
            the session, e.g. ["set datestyle to ...", "set time zone ..."]  
        failures: an optional exception class or a tuple of exception classes  
            for which the connection failover mechanism shall be applied,  
            if the default (OperationalError, InternalError) is not adequate  
        args, kwargs: the parameters that shall be passed to the creator  
            function or the connection constructor of the DB-API 2 module  
    View Code

    DBUtils 仅提供给了连接池管理,实际的数据库操作依然是由符合 DB-API 2 标准的目标数据库模块完成的。

    一个面向对象使用DBUtils的栗子

    # coding=utf-8
    """
    使用DBUtils数据库连接池中的连接,操作数据库
    OperationalError: (2006, ‘MySQL server has gone away’)
    """
    import json
    import pymysql
    import datetime
    from DBUtils.PooledDB import PooledDB
    import pymysql
    
    
    class MysqlClient(object):
        __pool = None;
    
        def __init__(self, mincached=10, maxcached=20, maxshared=10, maxconnections=200, blocking=True,
                     maxusage=100, setsession=None, reset=True,
                     host='127.0.0.1', port=3306, db='test',
                     user='root', passwd='123456', charset='utf8mb4'):
            """
    
            :param mincached:连接池中空闲连接的初始数量
            :param maxcached:连接池中空闲连接的最大数量
            :param maxshared:共享连接的最大数量
            :param maxconnections:创建连接池的最大数量
            :param blocking:超过最大连接数量时候的表现,为True等待连接数量下降,为false直接报错处理
            :param maxusage:单个连接的最大重复使用次数
            :param setsession:optional list of SQL commands that may serve to prepare
                the session, e.g. ["set datestyle to ...", "set time zone ..."]
            :param reset:how connections should be reset when returned to the pool
                (False or None to rollback transcations started with begin(),
                True to always issue a rollback for safety's sake)
            :param host:数据库ip地址
            :param port:数据库端口
            :param db:库名
            :param user:用户名
            :param passwd:密码
            :param charset:字符编码
            """
    
            if not self.__pool:
                self.__class__.__pool = PooledDB(pymysql,
                                                 mincached, maxcached,
                                                 maxshared, maxconnections, blocking,
                                                 maxusage, setsession, reset,
                                                 host=host, port=port, db=db,
                                                 user=user, passwd=passwd,
                                                 charset=charset,
                                                 cursorclass=pymysql.cursors.DictCursor
                                                 )
            self._conn = None
            self._cursor = None
            self.__get_conn()
    
        def __get_conn(self):
            self._conn = self.__pool.connection();
            self._cursor = self._conn.cursor();
    
        def close(self):
            try:
                self._cursor.close()
                self._conn.close()
            except Exception as e:
                print e
    
        def __execute(self, sql, param=()):
            count = self._cursor.execute(sql, param)
            print count
            return count
    
        @staticmethod
        def __dict_datetime_obj_to_str(result_dict):
            """把字典里面的datatime对象转成字符串,使json转换不出错"""
            if result_dict:
                result_replace = {k: v.__str__() for k, v in result_dict.items() if isinstance(v, datetime.datetime)}
                result_dict.update(result_replace)
            return result_dict
    
        def select_one(self, sql, param=()):
            """查询单个结果"""
            count = self.__execute(sql, param)
            result = self._cursor.fetchone()
            """:type result:dict"""
            result = self.__dict_datetime_obj_to_str(result)
            return count, result
    
        def select_many(self, sql, param=()):
            """
            查询多个结果
            :param sql: qsl语句
            :param param: sql参数
            :return: 结果数量和查询结果集
            """
            count = self.__execute(sql, param)
            result = self._cursor.fetchall()
            """:type result:list"""
            [self.__dict_datetime_obj_to_str(row_dict) for row_dict in result]
            return count, result
    
        def execute(self, sql, param=()):
            count = self.__execute(sql, param)
            return count
    
        def begin(self):
            """开启事务"""
            self._conn.autocommit(0)
    
        def end(self, option='commit'):
            """结束事务"""
            if option == 'commit':
                self._conn.autocommit()
            else:
                self._conn.rollback()
    
    
    if __name__ == "__main__":
        mc = MysqlClient()
        sql1 = 'SELECT * FROM shiji  WHERE  id = 1'
        result1 = mc.select_one(sql1)
        print json.dumps(result1[1], ensure_ascii=False)
    
        sql2 = 'SELECT * FROM shiji  WHERE  id IN (%s,%s,%s)'
        param = (2, 3, 4)
        print json.dumps(mc.select_many(sql2, param)[1], ensure_ascii=False)
    View Code

                                

  • 相关阅读:
    webgl-hdr
    color-balance-of-photoshop-using-opencv
    ssr
    hello girl
    牛人收集的162个JavaScript学习教程pdf资源
    What is the order of postprocessing effects?
    glslsandbox
    Geeks3D’s GLSL Shader Library
    face swap
    Leetcode 115.不同的子序列
  • 原文地址:https://www.cnblogs.com/zhuminghui/p/10930846.html
Copyright © 2011-2022 走看看