Scrapy代码实战

1.Spider爬虫代码

# -*- coding: utf-8 -*-
import scrapy
from yszd.items import YszdItem


class YszdSpiderSpider(scrapy.Spider):
    # 爬虫名称，启动爬虫时必须的参数
    name = 'yszd_spider'
    # 爬取域范围，运行爬虫在这个域名下爬取数据（可选）
    allowed_domains = ['itcast.cn']
    # 起始url列表，爬虫执行后第一批请求将从这个列表里获取
    start_urls = ['http://www.itcast.cn/channel/teacher.shtml']

    def parse(self, response):
        # //表示跳级定位，即对当前元素的所有子节点进行查找，一般开头都是使用跳级定位
        # div[@class='li_txt'] : 查找div且属性class='li_txt'的
        node_list = response.xpath("//div[@class='li_txt']")
        # 存储所有item字段
        # items = []
        for node in node_list:
            # 创建item字段对象用来存储信息
            item = YszdItem()
            # extract() : 将xpath对象转换为Unicode字符串
            name = node.xpath("./h3/text()").extract()
            title = node.xpath("./h4/text()").extract()
            info = node.xpath("./p/text()").extract()
            
            item['name'] = name[0]
            item['title'] = title[0]
            item['info'] = info[0]

            yield item
            # items.append(item)

2.Item代码（定义爬取的字段）

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class YszdItem(scrapy.Item):
    name = scrapy.Field()
    title = scrapy.Field()
    info = scrapy.Field()

3.Pipelines管道代码

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html

import json


class YszdPipeline(object):
    def __init__(self):
        self.f = open("yszd.json", "w")

    def process_item(self, item, spider):
        # ensure_ascii默认是True，会把内容转换为unicode
        text = json.dumps(dict(item), ensure_ascii=False) + "
"
        self.f.write(text)
        return item

    def close_spider(self, spider):
        self.f.close()

4.setting代码（开启管道，300表示优先级，越小优先级越高）

5.运行爬虫

　　执行命令：scrapy crawl yszd_spider

　　注意：yszd_spider为你定义爬虫的名称，与1中的第8行代码对应！

6.执行结果