zoukankan      html  css  js  c++  java
  • 解决mysqldb查询大量数据导致内存使用过高的问题

    1.源码
    connection=MySQLdb.connect(
        host="thehost",user="theuser",
        passwd="thepassword",db="thedb")
    cursor=connection.cursor()
    cursor.execute(query)
    for row in cursor.fetchall():
        print(row)
    2.问题
    普通的操作不管是fetchall()还是fetchone()都是先将数据加载到本地再进行计算,大量的数据会导致内存资源消耗光。解决的方法是使用SSCurosr光标来处理。


    3.优化后的代码  
    import MySQLdb.cursors
    connection=MySQLdb.connect(
        host="thehost",user="theuser",
        passwd="thepassword",db="thedb",
        cursorclass = MySQLdb.cursors.SSCursor)
    cursor=connection.cursor()
    cursor.execute(query)
    for row in cursor:
        print(row)

    參考文档:http://mysql-python.sourceforge.net/MySQLdb.html#

    关键段落截取:
    BaseCursor
    The base class for Cursor objects. This does not raise Warnings.
    CursorStoreResultMixIn
    Causes the Cursor to use the mysql_store_result() function to get the query result. The entire result set is stored on the client side.
    CursorUseResultMixIn
    Causes the cursor to use the mysql_use_result() function to get the query result. The result set is stored on the server side and is transferred row by row using fetch operations.
    CursorTupleRowsMixIn
    Causes the cursor to return rows as a tuple of the column values.

    CursorDictRowsMixIn

    Causes the cursor to return rows as a dictionary, where the keys are column names and the values are column values. Note that if the column names are not unique, i.e., you are selecting from two tables that share column names, some of them will be rewritten as table.column. This can be avoided by using the SQL ASkeyword. (This is yet-another reason not to use * in SQL queries, particularly where JOIN is involved.)
    Cursor
    The default cursor class. This class is composed of CursorWarningMixInCursorStoreResultMixInCursorTupleRowsMixIn, and BaseCursor, i.e. it raises Warning, usesmysql_store_result(), and returns rows as tuples.
    DictCursor
    Like Cursor except it returns rows as dictionaries.
    SSCursor
    A "server-side" cursor. Like Cursor but uses CursorUseResultMixIn. Use only if you are dealing with potentially large result sets.
    SSDictCursor
    Like SSCursor except it returns rows as dictionaries.



  • 相关阅读:
    linux下文件搜索命令学习笔记
    【转】C++格式化输出
    UNIX中的文件类型
    Unix内核中打开文件的表示
    网络编程学习笔记:linux下的socket编程
    TCP协议学习笔记(一)首部以及TCP的三次握手连接四次挥手断开
    C/C++源代码从写完到运行发生了什么
    C++ 函数形参发生动态绑定时指针增长步长与静态类型一致
    C++中为什么要将析构函数定义成虚函数
    C++求一个十进制的二进制中1的个数
  • 原文地址:https://www.cnblogs.com/mengfanrong/p/5106320.html
Copyright © 2011-2022 走看看