通用爬虫

xpath解析

import requests
from lxml import etree

url='https://bj.58.com/shunyi/ershoufang/?PGTID=0d30000c-0047-6aa6-0218-69d1ed59a77b&ClickID=3'
headers = {
'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36'
}

data = requests.get(url=url,headers=headers).text
tree = etree.HTML(data)
em=tree.xpath('//ul[@class="house-list-wrap"]/li')
for i in em:
title=i.xpath('./div[@class="list-info"]/h2/a/text()')
price=i.xpath('./div[@class="price"]//text()')
price=''.join(price)

中文乱码问题  
     第一种:
     response=requests.get(url=url,headers=headers)  
     response.encoding = 'utf-8'
      第二种(万能):
     image_name=image_name.endcode('iso-8859-1').decode('gbk')
请求头

   headers = {
   'Connection':'close',
   'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36'
}

   代理ip

    requests.get(url=url,headers=headers,proxies={'http':'111.1.111.1:8080'})