python爬虫Urllib实战

Urllib基础

urllib.request.urlretrieve(url,filenname)

直接将网页下载到本地

import urllib.request
>>> urllib.request.urlretrieve("http://www.hellobi.com",filename="D:/1.html")
('D:\/1.html', <http.client.HTTPMessage object at 0x00000000039D3320>)

---

清理下载网页时的缓存

urllib.request.urlcleanup()

---

info

>>> file=urllib.request.urlopen("http://www.hellobi.com")
>>> file.info()
<http.client.HTTPMessage object at 0x0000000003AA98D0>
>>> 

---

getcode() 返回状态吗 202 成功   403 禁止访问

geturl() 查询当前网页url

file.getcode()
200
file.geturl()

'https://www.hellobi.com/

---

超时设置

网页长时间未发出响应

timeout=描述

timeout =1

import urllib.request
file=urllib.request.urlopen("http://www.hellobi.com",timeout=1)

---

>>> import urllib.request
>>> for i in range(0,100):
    try:
        file=urllib.request.urlopen("http://yum.iqianyue.com",timeout=1)
        data=file.read()
        print(len(data))
    except Exceptin as e:
        print("yichang "+str(e))

---