Scrapy之持久化pipelines/items

基于上篇博文存在的问题

　　https://www.cnblogs.com/Alexephor/p/11432195.html　　

　　-无法完成爬虫刚开始 打开连接 爬虫关闭时：关闭连接
　　-分工不明确

本篇博文解决以上两问题

　　上篇博文不足之处主要体现在爬虫部分parse中在爬数据的过程操作了实时打开关闭文件处理，而且爬虫逻辑分工不明确

　　处理方式：1.在爬虫开始位置就打开文件操作或者数据库，在爬完之后就关闭掉即可

　　　　　　　2.爬的数据应该交给process_item处理 

Chouti.py

# -*- coding: utf-8 -*-
import scrapy
from wyb.items import WybItem


class ChoutiSpider(scrapy.Spider):
    name = 'chouti'
    # 爬取定向的网页 只允许这个域名的
    allowed_domains = ['chouti.com']
    start_urls = ['http://chouti.com/']

    def parse(self, response):
        from scrapy.http.response.html import HtmlResponse
        # print(response, type(response))
        # print(response.text)
        item_list = response.xpath('//div[@id="content-list"]/div[@class="item"]')
        for item in item_list:
            text = item.xpath('.//a/text()').extract_first()
            href = item.xpath('.//a/@href').extract_first()
            yield WybItem(text=text, href=href)        
        page_list = response.xpath('//div[@id="dig_lcpage"]//a/@href').extract()
        for page in page_list:
            from scrapy.http import Request
            page = "https://dig.chouti.com"+page
            # 继续发请求，回调函数parse
            yield Request(url=page, callback=self.parse)

pipelines.py

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html

from .settings import HREF_FILE_PATH
from scrapy.exceptions import DropItem


class WybPipeline(object):
    def __init__(self, path):
        self.f = None
        self.path = path

    @classmethod
    def from_crawler(cls, crawler):
        """
        初始化时候，用于创建pipline对象
        :param crawler:
        :return:
        """
        path = crawler.settings.get('HREF_FILE_PATH')
        return cls(path)

    def open_spider(self, spider):
        """
        爬虫开始执行被调用
        :param spider:
        :return:
        """
        # if spider.name == "chouti":
        # self.f = open(HREF_FILE_PATH, 'a+')
        self.f = open(self.path, 'a+')

    def process_item(self, item, spider):
        # item就是yield返回的内容
        # spider就是当前ChoutiSpider类的实例
        # f = open('news.log', 'a+')
        # f.write(item['href'])
        # f.close()
        self.f.write(item['href'] + '
')
        # return item   # 不交给下一个pipeline的process_item去处理
        raise DropItem()  # 后续的 pipeline的process_item不再执行了

    def close_spider(self, spider):
        """
        爬虫关闭时调用
        :param spider:
        :return:
        """
        self.f.close()


class DBPipeline(object):
    def __init__(self, path):
        self.f = None
        self.path = path

    @classmethod
    def from_crawler(cls, crawler):
        """
        初始化时候，用于创建pipline对象
        :param crawler:
        :return:
        """
        path = crawler.settings.get('HREF_DB_PATH')
        return cls(path)

    def open_spider(self, spider):
        """
        爬虫开始执行被调用
        :param spider:
        :return:
        """
        # self.f = open(HREF_DB_PATH, 'a+')
        self.f = open(self.path, 'a+')

    def process_item(self, item, spider):
        # item就是yield返回的内容
        # spider就是当前ChoutiSpider类的实例
        # f = open('db.log', 'a+')
        # f.write(item['href'])
        # f.close()
        self.f.write(item['href'] + '
')
        return item

    def close_spider(self, spider):
        """
        爬虫关闭时调用
        :param spider:
        :return:
        """
        self.f.close()

 items.py

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class WybItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field()
    href = scrapy.Field()

settings.py

ITEM_PIPELINES = {
   'wyb.pipelines.WybPipeline': 300,
   'wyb.pipelines.DBPipeline': 301,
    # 优先级 0-1000   越小优先级越快
}


# 保存路径
HREF_FILE_PATH = 'news.log'
HREF_DB_PATH = 'db.log'

总结：　　

    - 持久化 piplines/items
        a.先写piplines类
            class WybPipeline(object):
                def process_item(self, item, spider):
                    print(item)
                    # item就是yield返回的内容
                    # spider就是当前ChoutiSpider类的实例
                    return item
        b.写Item类
            class WybItem(scrapy.Item):
                # define the fields for your item here like:
                # name = scrapy.Field()
                title = scrapy.Field()
                href = scrapy.Field()
        c.配置文件
            ITEM_PIPELINES = {
                       'wyb.pipelines.WybPipeline': 300,
                        # 优先级 0-1000   越小优先级越高
                    }
        d.爬虫  yield执行一次就执行一次process_item
            yield WybItem(text=text, href=href)

对pipeline文件详情部分

　　源码流程内容

 1.判断当前WybPipeline内中是否有from_crawler
              有：
                    obj = WybPipeline.from_crawler()
              无：
                    obj = WybPipeline()
 2.obj.open_spider()
 3.obj.process_item()/obj.process_item()/obj.process_item()/
 4.obj.close_spider()

在这里看出来再定义Pipeline类时：

　　五方法　　

__init__(self, path)

@classmethod
from_crawler(cls, crawler)

open_spider(self, spider)

process_item(self, item, spider)

close_spider(self, spider)