urllib.request.urlopen 怎样处理了服务器返回的 gzip 乱码数据

urlopen 函数不会自动处理服务器返回的 gzip 数据，出现类似 x1fx8bx08x00 的乱码。

Request header 中的 Accept-Encoding 属性如果为 gzip，服务器返回的是 gzip 压缩后的数据，如下所示：

import urllib.request

req = urllib.request.Request(url)
req.add_header('Accept', '*/*')
req.add_header('Accept-Encoding', 'gzip')
req.add_header('Accept-Language', 'zh-CN,zh;q=0.8,en;q=0.6')
req.add_header('Cache-Control', 'max-age=0')
req.add_header('Connection', 'keep-alive')
req.add_header('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36')

f = urllib.request.urlopen(req)
content = f.read()

返回结果：

x1fx8bx08x00x00x00x00x00x00x03xdd=kx8f$Wuxdf-xf9?Tfcyw5xd5xd3xefxeex19x0bx048vx90x12x92...

解压 gzip

使用 gzip 模块解压字符串，如下所示：

import urllib.request
import gzip

req = urllib.request.Request(url)
req.add_header('Accept', '*/*')
req.add_header('Accept-Encoding', 'gzip')
req.add_header('Accept-Language', 'zh-CN,zh;q=0.8,en;q=0.6')
req.add_header('Cache-Control', 'max-age=0')
req.add_header('Connection', 'keep-alive')
req.add_header('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36')


f = urllib.request.urlopen(req)

# 处理 gzip 压缩的字符串
encoding = f.info().get('Content-Encoding')
if encoding == 'gzip':
    content = gzip.decompress(f.read())
else:
    content = f.read()

content 如果为bytes类型，将其转换为字符串：

if isinstance(html, bytes):
    html = str(html, encoding='utf-8')