scrapy框架基础篇

一、scrapy工作流程

（出自该博客：https://www.cnblogs.com/hewanli/p/12090042.html）

 　　1）当爬虫（Spider）要爬取某URL地址的页面时，使用该URL初始化Request对象提交给引擎（Scrapy Engine），并设置回调函数。 

　　　　Spider中初始的Request是通过调用start_requests() 来获取的。start_requests() 读取start_urls 中的URL，并以parse为回调函数生成Request 。

　　 2）Request对象进入调度器(Scheduler)按某种算法进行排队，之后的每个时刻调度器将其出列，送往下载器。

　　 3）下载器（Downloader）根据Request对象中的URL地址发送一次HTTP请求到网络服务器把资源下载下来，并封装成应答包(Response)。

　　 4）应答包Response对象最终会被递送给爬虫（Spider）的页面解析函数进行处理。

　　 5）若是解析出实体（Item），则交给实体管道（Item Pipeline）进行进一步的处理。由Spider返回的Item将被存到数据库(由某些Item Pipeline处理)或使用Feed exports存入到文件中。

　　 6）若是解析出的是链接（URL）,则把URL交给调度器(Scheduler)等待抓取。

　　以上就是Scrapy框架的运行流程，也就是它的工作原理。Request和Response对象是血液，Item是代谢产物。

（1）安装：

pip install scrapy

（2）创建项目

2.1 进入需要创建爬虫项目的文件下，cd 命令

2.2 创建项目：scrapy  startproject  scrapystudy(项目名)

（3） 创建爬虫：scrapy  genspider xxx（爬虫名）xxx.com （爬取域,只加域名即可，不要带上前面的http://或https://）

（4）运行爬虫：scrapy crawl XXX

（5）项目目录

介绍：items：用来保存数据的一个数据结构
　　　middlewares：在爬取过程中定义的一些中间件，可以处理request和response
　　　pipelines：项目管道，输出一些items，负责处理spider从网页中抽取的数据，主要是负责清洗，验证和向数据库中存储数据
　　　settings：定义一些配置信息

二、简单应用

爬取例子：http://quotes.toscrape.com/

2.1、items.py文件定义项目字段（需要爬取的字段），如下：

# Define here the models for your scraped items
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy

class QuotesItem(scrapy.Item):
    # define the fields for your item here like:
    # 定义爬取内容的存储字段
    text = scrapy.Field()
    author = scrapy.Field()
    tags = scrapy.Field()

2.2、quotes.py爬虫文件，处理页面信息，并提取结果，如下：

import scrapy
from scrapystudy.items import QuotesItem

class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    allowed_domains = ['quotes.toscrape.com']
    start_urls = ['http://quotes.toscrape.com/']

    def parse(self, response):
        quotes = response.css(".quote")
        for quote in quotes:
            items = QuotesItem()
            text = quote.css(".text::text").extract_first() #extract_first()提取第一个结果，extract()提取全部结果
            author = quote.css(".author::text").extract_first()
            tags = quote.css(".tags .tag::text").extract() #多个tag处理方式
            # 赋值
            items["text"] = text
            items["author"] = author
            items["tags"] = tags
            yield items
        # 处理翻页
        next_page = response.css(".pager .next a::attr(href)").extract_first()
        url = response.urljoin(next_page) #获取一个完整的url
        yield scrapy.Request(url=url,callback=self.parse) #callback回调函数，自己调用自己
        # 保存数据，命令行输入：scrapy crawl quotes -o quotes.json

 保存数据：

# 命令行输入以下代码，可以保存不同格式的文件
scrapy crawl quotes -o quotes.json
scrapy crawl quotes -o quotes.jl
scrapy crawl quotes -o quotes.csv
scrapy crawl quotes -o quotes.xml

# 保存到json文件，处理中文，在使用scrapy crawl -o file.json时，对于中文，保存的是unicode编码字符，因为需要转换为utf-8中文编码

转换命令：scrapy crawl crawlboke -o boke.json -s FEED_EXPORT_ENCODING=UTF-8

2.3、pipelines.py文件处理items内容长度，以及存储到MongoDB的操作，如下：

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


# useful for handling different item types with a single interface
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
import pymongo

class TextPipeline:

    def __init__(self):
        self.limit = 50

    def process_item(self, item, spider):
        "处理items长度"
        if item['text']:
            if len(item['text'])>self.limit:
                item['text'] = item['text'][0:self.limit].rstrip()+'...'
            return item
        else:
            return DropItem('Missing Text')

class MongoPipeline(object):
    "将数据存储在MongoDB中"

    def __init__(self,mongo_url,mongo_db):
        self.mongo_url = mongo_url
        self.mongo_db = mongo_db

    @classmethod
    def from_crawler(cls,crawler):
        return cls(
            mongo_url = crawler.settings.get('MONGO_URL'),
            mongo_db = crawler.settings.get('MONGO_DB')
        )

    def open_spider(self,spider):
        self.client = pymongo.MongoClient(self.mongo_url)
        self.db = self.client[self.mongo_db]

    def process_item(self,item,spider):
        name = item.__class__.__name__
        self.db[name].insert(dict(item))
        return item

    def close_spider(self,spider):
        self.client.close()

2.4、setting.py文件配置生效信息

# Scrapy settings for scrapystudy project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://docs.scrapy.org/en/latest/topics/settings.html
#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://docs.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'scrapystudy'

SPIDER_MODULES = ['scrapystudy.spiders']
NEWSPIDER_MODULE = 'scrapystudy.spiders'

MONGO_URL = "localhost"
MONGO_DB = "mydb"


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'scrapystudy (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'scrapystudy.middlewares.ScrapystudySpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#    'scrapystudy.middlewares.ScrapystudyDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'scrapystudy.pipelines.TextPipeline': 300,
   'scrapystudy.pipelines.MongoPipeline': 400,
}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'