zoukankan      html  css  js  c++  java
  • scrapy设置代理

    在爬取网站内容的时候,最常遇到的问题是:网站对IP有限制,会有防抓取功能,最好的办法就是IP轮换抓取(加代理)

    下面来说一下Scrapy如何配置代理,进行抓取

    1.在Scrapy工程下新建“middlewares.py”

    # Importing base64 library because we'll need it ONLY in case if the proxy we are going to use requires authentication
    import base64
    # Start your middleware class
    class ProxyMiddleware(object):
        # overwrite process request
       
    def process_request(self, request, spider):
            # Set the location of the proxy
           
    request.meta['proxy'] = "http://proxyhk.huawei.com:8080"
     
           
    # Use the following lines if your proxy requires authentication
           
    proxy_user_pass = "username:password"
           
    # setup basic authentication for the proxy
           
    encoded_user_pass = base64.encodestring(proxy_user_pass)
            request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass

    2.在项目配置文件里(./pythontab/settings.py)添加

    DOWNLOADER_MIDDLEWARES = {
    ## 'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': None,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None,
    'wikiSpider.middlewares.ProxyMiddleware': 543
    }
  • 相关阅读:
    (spring-第15回【IoC基础篇】)容器事件
    javascript笔记2-引用类型
    php namespace use 命名空间
    mysql性能优化(二)
    mysql常用命令
    关于有效的性能调优的一些建议[转]
    apache htaccess
    php 换行 空格分割处理
    一些比较好的博文
    php & 引用
  • 原文地址:https://www.cnblogs.com/BigFishFly/p/6337137.html
Copyright © 2011-2022 走看看