zoukankan      html  css  js  c++  java
  • python之https爬虫出现 SSL: CERTIFICATE_VERIFY_FAILED (同时打开fiddler就会出现)

    1.参考

    Py 坑之 CERTIFICATE_VERIFY_FAILED

    Python 升级到 2.7.9 之后引入了一个新特性,当你urllib.urlopen一个 https 的时候,会验证一次 SSL 证书,当目标网站使用的是自签名的证书时就会爆出一个 urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)> 的错误消息

    Python Requests throwing up SSLError

    The problem you are having is caused by an untrusted SSL certificate.

    Like @dirk mentioned in a previous comment, the quickest fix is setting verify=False.

    Please note that this will cause the certificate not to be verified. This will expose your application to security risks, such as man-in-the-middle attacks.

    Of course, apply judgment. As mentioned in the comments, this may be acceptable for quick/throwaway applications/scripts, but really should not go to production software.

    If just skipping the certificate check is not acceptable in your particular context, consider the following options, your best option is to set the verifyparameter to a string that is the path of the .pem file of the certificate (which you should obtain by some sort of secure means).

    So, as of version 2.0, the verify parameter accepts the following values, with their respective semantics:

    • True: causes the certificate to validated against the library's own trusted certificate authorities (Note: you can see which Root Certificates Requests uses via the Certifi library, a trust database of RCs extracted from Requests: Certifi - Trust Database for Humans).
    • False: bypasses certificate validation completely.
    • Path to a CA_BUNDLE file for Requests to use to validate the certificates.

    Source: Requests - SSL Cert Verification

    Also take a look at the cert parameter on the same link.

     

     

     

    2.解决办法

    (1)全局设置,对requests不生效 

     ssl._create_default_https_context = ssl._create_unverified_context

     

    (2)urllib2.urlopen 传参

     
    context = ssl._create_unverified_context()
    
    urllib2.urlopen(context=context)

     

    (3)参考 urllib2.urlopen 为opener叠加handler

    复制代码

    # C:Program FilesAnaconda2Liburllib2.py
    # def urlopen
    # elif context:
    # https_handler = HTTPSHandler(context=context)
    # opener = build_opener(https_handler)

    # urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
    try:
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
    resp = opener.open(req)
    except urllib2.URLError as err:
    print err
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar), urllib2.HTTPSHandler(context=context)) #叠加多个 handler
    resp = opener.open(req)

    复制代码

     

    (4)request.get 传参

     
    # requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
    try:
        r = requests.get(url, headers=headers, cookies=cookie)
    except requests.exceptions.SSLError as err:
        print err
        r = requests.get(url, headers=headers, cookies=cookie, verify=False)
  • 相关阅读:
    异常:Unknown lifecycle phase "mvn". You must specify a valid lifecycle
    java中数的表示
    windows7 桌面突然卡住了,点击右键点不了,点击桌面软件点不了,怎么办?
    DJango错误日志生成
    drf框架接口文档
    drf框架中分页组件
    UVa10234 Race
    洛谷P2982 [USACO10FEB]慢下来Slowing down
    UVa10820 Send a Table
    POJ1861 Network
  • 原文地址:https://www.cnblogs.com/piwefei/p/11188071.html
Copyright © 2011-2022 走看看