zoukankan      html  css  js  c++  java
  • scrapy设置代理

    在爬取网站内容的时候,最常遇到的问题是:网站对IP有限制,会有防抓取功能,最好的办法就是IP轮换抓取(加代理)

    下面来说一下Scrapy如何配置代理,进行抓取

    1.在Scrapy工程下新建“middlewares.py”

    # Importing base64 library because we'll need it ONLY in case if the proxy we are going to use requires authentication
    import base64
    # Start your middleware class
    class ProxyMiddleware(object):
        # overwrite process request
       
    def process_request(self, request, spider):
            # Set the location of the proxy
           
    request.meta['proxy'] = "http://proxyhk.huawei.com:8080"
     
           
    # Use the following lines if your proxy requires authentication
           
    proxy_user_pass = "username:password"
           
    # setup basic authentication for the proxy
           
    encoded_user_pass = base64.encodestring(proxy_user_pass)
            request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass

    2.在项目配置文件里(./pythontab/settings.py)添加

    DOWNLOADER_MIDDLEWARES = {
    ## 'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': None,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None,
    'wikiSpider.middlewares.ProxyMiddleware': 543
    }
  • 相关阅读:
    转 sql 时间转换格式 convert(varchar(10),字段名,转换格式)
    C#页面添加提交数据后跳出小弹窗的功能
    解决粘包问题
    粘包问题
    模拟ssh远程执行命令
    基于TCP协议的socket套接字编程
    Linux和git使用
    osi七层协议
    TCP协议的三次握手和四次挥手
    C/S 和 B/S架构
  • 原文地址:https://www.cnblogs.com/BigFishFly/p/6337137.html
Copyright © 2011-2022 走看看