Python爬虫基础之requests

一、随时随地爬取一个网页下来

　　怎么爬取网页？对网站开发了解的都知道，浏览器访问Url向服务器发送请求，服务器响应浏览器请求并返回一堆HTML信息，其中包括html标签，css样式，js脚本等。我们之前用的是Python标准基础库Urllib实现的，

现在我们使用Python的Requests HTTP库写个脚本开始爬取网页。Requests的口号很响亮“让HTTP服务人类“，够霸气。

二、Python Requests库的基本使用

1.GET和POST请求方式

GET请求

import requests

payload = {"t": "b", "w": "Python urllib"}
response = requests.get('http://zzk.cnblogs.com/s', params=payload)
# print(response.url)  # 打印 http://zzk.cnblogs.com/s?w=Python+urllib&t=b&AspxAutoDetectCookieSupport=1
print(response.text)  

Python requests的GET请求，不需要在作为请求参数前，对dict参数进行urlencode()和手动拼接到请求url后面，get()方法会直接对params参数这样做。

POST请求

import requests

payload = {"t": "b", "w": "Python urllib"}
response = requests.post('http://zzk.cnblogs.com/s', data=payload)
print(response.text)  # u'......'

Python requests的POST请求，不需要在作为请求参数前，对dict参数进行urlencode()和encode()将字符串转换成字节码。raw属性返回的是字节码，text属性直接返回unicode格式的字符串，而不需要再进行decode()将返回的bytes字节码转化为unicode。

相对于Python urllib而言，Python requests更加简单易用。

 2.设置请求头headers

import requests

payload = {"t": "b", "w": "Python urllib"}
headers = {'user_agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
response = requests.get('http://zzk.cnblogs.com/s', params=payload, headers=headers)
print(response.request.headers)  

get方法的请求头，可以通过传递字典格式的参数给headers来实现。response.headers返回服务器响应的请求头信息，response.request.headers返回客户端的请求头信息。

3.设置会话cookie

import requests

cookies = {'cookies_are': 'working'}
response = requests.get('http://zzk.cnblogs.com/', cookies=cookies)
print(response.text)

requests.get()方法cookies参数除了支持dict()字典格式，还支持传递一个复杂的RequestsCookieJar对象，可以指定域名和路径属性。

import requests
import requests.cookies

cookieJar = requests.cookies.RequestsCookieJar()
cookieJar.set('cookies_are', 'working', domain='cnblogs', path='/cookies')
response = requests.get('http://zzk.cnblogs.com/', cookies=cookieJar)
print(response.text)

4.设置超时时间timeout

import requests

response = requests.get('http://zzk.cnblogs.com/', timeout=0.001)
print(response.text)

三、Python Requests库的高级使用

1.Session Object

from requests import Request,Session

s = Session()

s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get('http://httpbin.org/cookies')

print(r.text)
# '{"cookies": {"sessioncookie": "123456789"}}'

通过Session，我们可以在多个请求之间传递cookies信息，不过仅限于同一域名下，否则不会附带上cookie。如果碰到需要登录态的页面，我们可以在登陆的时候保存登录态，再访问其他页面时附带上就好。

2.Prepared Requested

from requests import Request,Session

url = 'http://zzk.cnblogs.com/s'
payload = {"t": "b", "w": "Python urllib"}
headers = {
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36',
        'Content-Type':'application/x-www-form-urlencoded'
}
s = Session()
request = Request('GET', url, headers=headers, data=payload)
prepped = request.prepare()

# do something with prepped.headers
del prepped.headers['Content-Type']
response = s.send(prepped, timeout=3)
print(response.request.headers)

Request对象的prepare()方法返回的对象允许在发送请求前做些额外的工作，例如更新请求体body或者请求头headers.

3.Set Proxy

import requests


# set headers
user_agent = 'Mozilla/5.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/600.1.3 (KHTML, like Gecko) Version/8.0 Mobile/12A4345d Safari/600.1.4'
headers = {'User-Agent': user_agent}
url = 'http://passport.xxx.com/auth/valid.json'
proxies = {'http': 'http://10.1.1.1:80'}  # set http proxy
params = {'uin': 'xxxxxx', 'passwd': 'xxxxxx', 'imgcode': 'ijyk', '_1': '12',
          '_2': '10279', '_3': '23935743', 'url': ''}

response = requests.post(url, headers=headers, data=params, proxies=proxies)
response.raise_for_status()
if response.status_code == requests.codes.ok:
    print(response.text)

requests.get()和post()方法均支持http proxy代理，只要传递proxies = {'http': 'http://10.1.1.1:80'}字典对象，可以实现把请求传递给代理服务器，代理服务器从10.1.1.1:80取回响应数据返回来。

四、Python Requests库的实际应用

 1.GET请求封装

def do_get_request(self, url, headers=None, timeout=3, is_return_text=True, num_retries=2):
        if url is None:
            return None
        print('Downloading:', url)
        if headers is None:  # 默认请求头
            headers = {
                'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
        response = None
        try:
            response = requests.get(url,headers=headers,timeout=timeout)

            response.raise_for_status()  # a 4XX client error or 5XX server error response,raise requests.exceptions.HTTPError
            if response.status_code == requests.codes.ok:
                if is_return_text:
                    html = response.text
                else:
                    html = response.json()
            else:
                html = None
        except requests.Timeout as err:
            print('Downloading Timeout:', err.args)
            html = None
        except requests.HTTPError as err:
            print('Downloading HTTP Error,msg:{0}'.format(err.args))
            html = None
            if num_retries > 0:
                if 500 <= response.status_code < 600:
                    return self.do_get_request(url, headers=headers, num_retries=num_retries - 1)  # 服务器错误，导致请求失败，默认重试2次
        except requests.ConnectionError as err:
            print('Downloading Connection Error:', err.args)
            html = None

        return html

2.POST请求封装

 def do_post_request(self, url, data=None, headers=None, timeout=3, is_return_text=True, num_retries=2):
        if url is None:
            return None
        print('Downloading:', url)
        # 如果请求数据未空，直接返回
        if data is None:
            return
        if headers is None:
            headers = {
                'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
        response = None
        try:
            response = requests.post(url,data=data, headers=headers, timeout=timeout)   # 设置headers timeout无效

            response.raise_for_status()  # a 4XX client error or 5XX server error response,raise requests.exceptions.HTTPError
            if response.status_code == requests.codes.ok:
                if is_return_text:
                    html = response.text
                else:
                    html = response.json()
            else:
                print('else')
                html = None
        except requests.Timeout as err:
            print('Downloading Timeout:', err.args)
            html = None
        except requests.HTTPError as err:
            print('Downloading HTTP Error,msg:{0}'.format(err.args))
            html = None,
            if num_retries > 0:
                if 500 <= response.status_code < 600:
                    return self.do_post_request(url, data=data, headers=headers,
                                                num_retries=num_retries - 1)  # 服务器错误，导致请求失败，默认重试2次
        except requests.ConnectionError as err:
            print('Downloading Connection Error:', err.args)
            html = None

        return html

3.登录态cookie

def save_cookies(self, requeste_cookiejar, filename):
    with open(filename, 'wb')as f:
        pickle.dump(requeste_cookiejar, f)

def load_cookies(self, filename):
    with open(filename, 'rb') as f:
        return pickle.load(f)

# save request cookies
r = requests.get(url)
save_cookies(r.cookies,filename)

# load cookies and do a request
requests.get(url,cookies=load_cookies(filename))