简单爬虫实例

代码工具：jupyter

抓包工具：fiddle

1：搜狗页面内容爬取

import requests

url='https://www.sogou.com/'
response=requests.get(
    url=url
)
text=response.text
text

2:豆瓣电影分类爬取

import requests
url='https://movie.douban.com/j/new_search_subjects'
param={
    'sort':'U',
    'range': '0,10',
    'tags': '',
    'start': '0',
    'genres': '爱情'     
}
headers={
   'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36'
}
response=requests.get(
        url=url,
        headers=headers,
        params=param,
       
)
text=response.json()
text

3：搜索磁条爬取并写入文件

import requests

url='https://www.sogou.com/web'
headers={
   'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36'
}
param={
    'query':'校花'
}
response=requests.get(
    url=url,
    headers=headers,
    params=param
)
text=response.content
with open('xh.html','wb')as f:
    f.write(text)

4：国家药监总监内容爬取。爬取动态生成的内容

import requests


headers={
   'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36'
}
url='http://125.35.6.84:81/xk/itownet/portalAction.do?method=getXkzsList'

data={
   'on': 'true',
   'page': '3',
   'pageSize': '15',
   'productName':'',
   'conditionType': '1',
   'applyname':'',
   'applysn':''
}
response=requests.post(url=url,headers=headers,data=data)
conn_list=response.json()['list']
comm_info_list=[]
for i in conn_list:
    url_c="http://125.35.6.84:81/xk/itownet/portalAction.do?method=getXkzsById"
    data_c={"id":""}
    if i["XC_DATE"]:
        data_c["id"]=i["ID"]
        res=requests.post(url=url_c,data=data_c,headers=headers)
        comm_info_list.append(res.json())
comm_info_list