协程淫威之下的：Requests

前言

• 什么是 Grequests 包
  Grequests 是异步非阻塞的异步请求第三方包，就是在原来的 Requests 模块用 gevent 封装了一层，支持 Requests 所有方法能快速上手
• 怎么安装

pip install grequests

快速上手

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Date    : 2020/5/22 9:54
# @File    : test.py
# @Link    : https://www.cnblogs.com/BenLam/

import grequests

url_list = [
    "http://www.baidu.com",
    "http://python-tablib.org",
    "http://httpbin.org",
    "http://python-requests.org",
    "http://fakedomain/",
    "http://kennethreitz.com"
]
tasks = [grequests.get(u) for u in url_list]
res = grequests.map(tasks)      # tasks 就是多个 requests.get(url) 组成的list

print(res[0])
print(res[0].status_code)
print(res[0].headers)
print(res[0].text)
print(res[0].close)

"""
打印:
    <Response [200]>
    200
    (......)
"""

性能对比

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Date    : 2020/5/22 9:54
# @File    : test.py
# @Link    : https://www.cnblogs.com/BenLam/

import time
import requests
import grequests

urls = [
	"http://www.baidu.com",
	(......)
	"http://www.xxxx.com"
	]

def _requests():
    start_time = time.time()
    response = [requests.get(url) for url in urls]
    print(f"requests 总耗时 - {round(time.time() - start_time, 2)}")
    return response

def _grequests():
    start_time = time.time()
    response = grequests.map([grequests.get(u) for u in urls], size=20)
    print(f"grequests 总耗时 - {round(time.time() - start_time, 2)}")
    return response

def main():
    import threading
    r = threading.Thread(target=_requests)
    g = threading.Thread(target=_grequests)
    r.start()
    g.start()

if __name__ == "__main__":
	main()
"""
打印:
	grequests 总耗时 - 1.02
	requests 总耗时 - 7.43
	[Done] exited with code=0 in 8.149 seconds
"""

对比之下，grequests 比 requests 快多了，所以自己看着办

requests 模块详细使用

调用方法

requests.get(url, params=None, **kwargs)
requests.post(url, data=None, json=None, **kwargs)
requests.put(url, data=None, **kwargs)
requests.head(url, **kwargs)
requests.delete(url, **kwargs)
requests.patch(url, data=None, **kwargs)
requests.options(url, **kwargs)
(......)
requests.request(method, url, **kwargs)

参数

request.request:
- method:  提交方式
- url:     提交地址

- params:  在URL中传递的参数,GET 
    requests.request(
        method="GET",
        url= "https://www.xxxxxx.com",
        params = {"k1":"v1","k2":"v2"}
    )
    # https://www.xxxxxx.com?key1=value1&key2=value2

- data:    在请求体里传递的数据
    requests.request(
        method="POST",
        url= "https://www.xxxxxx.com",
        params = {"key1":"value1","k2":"v2"},
        data = {"user":"张三", "password":"123456", "power":[11, 22, 33]}
    )
请求头:
    content-type: application/json.....(还有很多种格式)
请求体:
    user=张三&password=123456

- json   在请求体里传递的数据
    requests.request(
        method="POST",
        url= "https://www.xxxxxx.com",
        params = {"key1":"value1","k2":"v2"},
        json = {"user":"张三", "password":"123456"}
    )

- headers   请求头
    requests.request(
        method="POST",
        url= "https://www.xxxxxx.com",
        params = {"key1":"value1","k2":"v2"},
        json = {"user":"张三", "password":"123456"}
        headers={
            "Referer": "https://www.xxxxxx.com/login",
            "User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"
        }
    )
- cookies  Cookies
- files    上传文件
- auth      基本认知(headers中加入加密的用户名和密码)
- timeout  请求和响应的超市时间
- allow_redirects  是否允许重定向
- proxies  代理
- verify   是否忽略证书
- cert      证书文件
- session: 用于保存客户端历史访问信息