zoukankan      html  css  js  c++  java
  • Python urllib2写爬虫时候每次request open以后一定要关闭

    最近用python urllib2写一个爬虫工具,碰到运行一会程序后就会出现scoket connection peer reset错误。经过多次试验发现原来是在每次request open以后没有及时的去close,才导致此问题的出现。所以今天记录一下希望对其他有人有用。直接上代码:

                    request = urllib2.Request(self.url)
                    request.add_header('Cookie','PHPSESSID=79lo60cmtl1ck70h4ufruq6n53; mmf_searchhotkeyandroid=%E5%A4%A9%E6%B6%AF%E7%A4%BE%E5%8C%BA%2C%E7%A9%BF%E8%A1%A3%E5%8A%A9%E6%89%8B%2C%E5%A4%A9%E6%B0%94%2C%E9%B3%84%E9%B1%BC%E5%B0%8F%E9%A1%BD%E7%9A%AE%E7%88%B1%E6%B4%97%E6%BE%A12%2C%E6%B0%B4%E6%9E%9C%E5%BF%8D%E8%80%85%2C%E4%B8%96%E7%95%8COL%2C%E6%88%98%E5%A4%A9; mmf_msisdn=08e2b01ad5dd5b3d297ef6558a60ec26; mmf_us=08e2b01ad5dd5b3d297ef6558a60ec26.39; mmf_userVisitPageIndex=79lo60cmtl1ck70h4ufruq6n53.2')
                    request.add_header('Connection','keep-alive')
                    request.add_header('Accept','*/*')
                    request.add_header('Accept-Language','zh-CN,zh;q=0.8')
                    request.add_header('Accept-Encoding','gzip,deflate,sdch')
                    request.add_header('User-Agent','Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36')
                    response = urllib2.urlopen(request)
                    print response.code
                    if response.info().get('Content-Encoding')=='gzip':
                            buf = StringIO(response.read())
                            f = gzip.GzipFile(fileobj = buf)
                            data = f.read()
                            f.close()

    千万记住f.close()一定得加上。。

  • 相关阅读:
    COCOS2D-X中UI动画导致闪退与UI动画浅析
    使用cocos2d-x3.4结合cocos2.1.5制作小游戏《亲亲小熊》
    algorithm 学习之 for_each
    cocos2dx 3.2 Touch Listen和menu回调实现截屏
    Lua 中string.gsub(sourceString, pattern, replacementString) 返回值有两个
    python 装饰器模式 我的理解
    Https SSL Knowledge & how to get a self-signed certificate on ubuntu.
    公钥和私钥的理解
    Openwrt UCI 快捷切换SS 配置
    iptables ipset 实用命令
  • 原文地址:https://www.cnblogs.com/ansonz/p/3477619.html
Copyright © 2011-2022 走看看