利用scrapy爬取文件后并基于管道化的持久化存储

一、爬取数据
　　在创建爬虫程序之后写入爬取数据的代码

import scrapy

from boss.items import BossItem

class SecondSpider(scrapy.Spider):
    name = 'second'
    # allowed_domains = ['www.xxx.com']
    start_urls = ['https://www.zhipin.com/c101270100/?query=python%爬虫&ka=sel-city-101270100']

    def parse(self, response):

        li_list = response.xpath('//div[@class="job-list"]/ul/li')
        for li in li_list:
            job_name = li.xpath('./div/div[1]/h3/a/div[1]/text()').extract_first()
            salary = li.xpath('.//div[@class="info-primary"]/h3/a/span/text()').extract_first()
            company = li.xpath('./div/div[2]/div/h3/a/text()').extract_first()
            print(job_name, salary, company)
            # 实例化item对象
            item = BossItem()
            # 将获取到的数据保存到当前对象中
            item['job_name'] = job_name
            item['salary'] = salary
            item['company'] = company

            # 将item提交给管道
            yield item

 之后在items.py 文件里为item对象设置属性

我们将爬取到的信息全部设置为item的属性

import scrapy

class BossItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    # 实例化对象属性
    job_name = scrapy.Field()
    salary = scrapy.Field()
    company = scrapy.Field()

二、写入pipelines.py内容　　

import pymysql
from redis import Redis
import json


class BossPipeline(object):
    fp = None

    def open_spider(self, spider):
        print('开始爬虫')
        self.fp = open('./boss.txt', 'w', encoding='utf8')

    def close_spider(self, spider):
        print('关闭爬虫')
        self.fp.close()

    # 爬虫文件每提交一次item，该方法就会被调用一次
    def process_item(self, item, spider):
        self.fp.write(item['job_name']+':'+item['salary']+':'+item['company'])
        # return 是将item返回给下一个管道类
        return item

我们自定义一个类，将item的数据写入到mysql里（在这之前将mysql和redis都启动）

class MysqlPileLine(object):

    conn = pymysql.connect(host='127.0.0.1', port=3306, user='root', password='', db='bms', charset="utf8")
    cursor = conn.cursor()

    def open_spider(self, spider):
        print('开始爬虫')
        # 建立连接数据库

    def process_item(self, item, spider):

        try:
            self.cursor.execute('insert into boss values("%s","%s","%s")' % (item['job_name'], item['salary'], item['company']))
            print('写入数据。。。。。。。。')
            self.conn.commit()
        except Exception as e:
            print(e)
            self.conn.rollback()

    def close_spider(self, spider):
        self.cursor.close()
        self.conn.close()

在相同的文件下创建redis类写入数据

class RedisPileLine(object):
    conn = Redis(host='127.0.0.1', port=6379)

    def open_spider(self,spider):
        print('开始爬虫')

    def process_item(self, item, spider):
        dic = {
            'name': item['job_name'],
            'salary': item['salary'],
            'company': item['company']
        }
        dic_json = json.dumps(dic)
        self.conn.lpush('boss', dic_json)

三、更改配置文件（其他配置这里不作演示）

最后我们在终端去指定爬虫程序

　　scrapy crawl second