一.获取登录后页面信息的两种方法
1.第一种方法:
人为把有效cookies加到请求头中,代码如下
import urllib.request # 确定url url = "https://weibo.com/u/5811151623/home" # 添加请求头 headers = { "User-Agent": " Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "Cookie": "SINAGLOBAL=1056235723769.1433.1526891663625; Ugrow-G0=1ac418838b431e81ff2d99457147068c; login_sid_t=92c3e5d6461a4a8852629513dba4eb99; cross_origin_proto=SSL; TC-V5-G0=8dc78264df14e433a87ecb460ff08bfe; wb_view_log=1536*8641.25; _s_tentry=passport.weibo.com; Apache=5865173766078.335.1563513209889; ULV=1563513209896:3:1:1:5865173766078.335.1563513209889:1538661475261; SCF=AtThUzjOQECs-GvkBXwLmxzpM3TCF5NC7qGqelck97jPb-KYiDq0-erPkD5uDCEcHbOaMMDc8-xeDffKoiS-W7g.; SUHB=0IIF8F3k8nZAPY; un=310955019@qq.com; TC-Page-G0=52dad2141fc02c292fc30606953e43ef|1563513247|1563513247; wb_view_log_5811151623=1536*8641.25; webim_unReadCount=%7B%22time%22%3A1563513248700%2C%22dm_pub_total%22%3A13%2C%22chat_group_client%22%3A0%2C%22allcountNum%22%3A19%2C%22msgbox%22%3A0%7D; SUBP=0033WrSXqPxfM72wWs9jqgMF55529P9D9W5i-1qlF8XW2aLLs530P9DB5JpV2hepeh.fSK5p1-WpMC4odcXt; SUB=_2AkMqbd6cdcPxrAZZmfkUyWrkbo5H-jyZuLdqAn7uJhMyAxh77g4KqSVutBF-XGCSCDwAF4qC498CuN9Q_vjFiwsA; UOR=www.hejizhan.com,vdisk.weibo.com,login.sina.com.cn" } # 创建请求对象 request = urllib.request.Request(url, headers=headers) # 发送请求 response = urllib.request.urlopen(request) # 读取数据并保存数据 data = response.read() with open("01cookies.html", "wb") as f: f.write(data)
2. 第二种方法
(1)代码登录(post请求),登录成功后得到有效cookie
(2)自动带着cookie去请求
注 cookiejar能自动保存这个cookie 若不用此包的话则每次获取cookie都需要登录,获取cookie(request.get_header('Cookie')),添加cookie("Cookie":相应的cookie),很繁琐。
代码如下
import urllib.request from http import cookiejar from urllib import parse # 登录之前的 登录页的网址https://www.yaozh.com/login/ # 找登录 参数 # 1. 代码登录 # 1.1 登录的网址 login_url = 'https://www.yaozh.com/login' # 1.2 登录的参数 login_form_data = { "username": "xiaomaoera12", "pwd": "lina081012", "formhash": "CE3ADF28C5", "backurl": "https%3A%2F%2Fwww.yaozh.com%2F" } # 1.3 发送登录请求POST cook_jar = cookiejar.CookieJar() # 定义有添加 cook 功能的 处理器 cook_hanlder = urllib.request.HTTPCookieProcessor(cook_jar) # 根据处理器 生成 opener opener = urllib.request.build_opener(cook_hanlder) # 带着参数 发送post请求 # 添加请求头 headers = { 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36' } # 1.参数转译(其是字典); 2. post请求的 data要求是bytes login_str = parse.urlencode(login_form_data).encode('utf-8') login_request = urllib.request.Request(login_url, headers=headers, data=login_str) # 如果登录成功, cookjar自动保存cookie opener.open(login_request) # 2. 代码带着cooke去访问 个人中心 center_url = 'https://www.yaozh.com/member/' center_request = urllib.request.Request(center_url, headers=headers) response = opener.open(center_url) # bytes -->str data = response.read().decode() with open('02cook.html', 'w') as f: f.write(data)
二. 异常
HTTPError与URLError
HTTPError继承自URLError
捕获异常代码
try: response = urllib.request.urlopen(url) except urllib.request.HTTPError as e: print(e.code)
三.requests包
安装方法pip install requests
1. get
(1)具体用法如下代码
import requests url = "http://www.baidu.com/" response = requests.get(url) data = response.content print(type(data)) # 得到<class 'bytes'> data = response.text print(type(data)) # 得到<class 'str'>
content得到的数据类型是字节,而text一般会猜一个编码类型(可以直接去看content,text的源码),有可能会错,所以优先使用content
(2)获取一些具体信息
a. 获取请求头
request_headers = response.request.headers
b. 获取响应头
response_headers = self.response.headers
c. 获取响应状态码
code = response.status_code
d. 请求的cookie以及响应的cookie
request_cookie = response.request._cookies # 请求的cookie response_cookie = response.cookies # 响应的cookie
(3)使用requests包时,不需要对包含中文的url转译(requests自动帮我们转了),使用python自带的urllib时需要自己转译,如下
new_url = urllib.parse.quote(url, safe=string.printable)
相应的请求代码
# url = 'https://www.baidu.com/s?wd=美女' url = 'https://www.baidu.com/s' params = { 'wd':"美女" } headers = { 'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36' } response = requests.get(url,headers=headers, params=params) data = response.content with open('baidu.html', 'wb') as f: f.write(data)
(4) jason
当网站返回的不是html而是标准的jason时,想获取里面信息时,转化成jason时,获取数据更方便import requests
import requests
import json url = 'https://api.github.com/user' headers = { 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36' } # 这个 网址 返回的内容不是html 而是标准的json response = requests.get(url, headers=headers) # str data = response.content.decode() print(type(data)) # 得到数据类型为字符串(jason字符串) print(data) 下面是返回的内容
'''
{ "message": "Requires authentication", "documentation_url": "https://developer.github.com/v3/users/#get-the-authenticated-user" }
'''
# str-- dict
data_dict = json.loads(data)
# json() 自动将json字符串 转换成Python dict list
data = response.json()
print(data['message'])