scrapy 爬取链家网站房价爬虫 多页爬取

直接上代码，顺便在这里记录，时间2190906.

刚开始爬贝壳网的，发现有反爬虫，我也不会绕，换了链家网，原来中文也可以做变量。

spider.py

# -*- coding: utf-8 -*-
import scrapy

from beike.items import BeikeItem

class BeikewSpider(scrapy.Spider):
    name = 'beikew'
    allowed_domains = ['lianjia.com']
    start_urls = ['https://su.lianjia.com/ershoufang/']
    page = 1


    def parse(self, response):
        li_list = response.xpath('//*[@id="content"]/div[1]/ul/li')
        for li in li_list:
            item = BeikeItem()
            name = li.xpath('./div[1]/div[1]/a/text()').extract_first()
            单价 = li.xpath('./div[1]/div[6]/div[2]/span/text()').extract_first()
            totalprice = li.xpath('./div[1]/div[6]/div[1]/span/text()').extract_first()
            xiaoqu = li.xpath('./div[1]/div[2]/div/a/text()').extract_first()
            local = li.xpath('./div[1]/div[3]/div/a/text()').extract_first()
            item['name'] = name
            item['单价'] = 单价 #在这里试试中文的，才知道原来中文也可以做变量
            item['totalprice'] = totalprice
            item['xiaoqu'] = xiaoqu
            item['local'] = local
            yield item

        if self.page <= 50:#这里爬取了50页数据，可以随意更改
            self.page += 1
            url_new = str(self.page)
            new_page_url = 'https://su.lianjia.com/ershoufang/pg' + url_new
            yield scrapy.Request(url = new_page_url, callback = (self.parse))

item.py

import scrapy

class BeikeItem(scrapy.Item):
    xiaoqu = scrapy.Field()
    name = scrapy.Field()
    单价 = scrapy.Field()
    totalprice = scrapy.Field()
    local = scrapy.Field()

settings.py

BOT_NAME = 'beike' #这些代码在settings里启用或者添加的。
SPIDER_MODULES = ['beike.spiders']
NEWSPIDER_MODULE = 'beike.spiders'
FEED_EXPORT_ENCODING ='utf-8'
FEED_EXPORT_ENCODING = 'gb18030'
ROBOTSTXT_OBEY = True
DOWNLOAD_DELAY = 1

只用到了3个y文件，其他的都是命令生成的，保持默认。

执行结果：