Scrapy爬取慕课网(imooc)所有课程数据并存入MySQL数据库

爬取目标：使用scrapy爬取所有课程数据，分别为

1.课程名 2.课程简介 3.课程等级 4.学习人数

并存入MySQL数据库  （目标网址  http://www.imooc.com/course/list）

一.导出数据文件到本地

1.新建imooc项目

 scrapy startproject imooc 

2.修改 items.py，添加项目item

from  scrapy import Item,Field
class ImoocItem(Item):
    Course_name=Field()#课程名称
    Course_content=Field()#课程内容
    Course_level=Field()#课程等级
    Course_attendance=Field()#课程学习人数

3.在 spiders目录下制作爬虫

vi imooc_spider.py

# -*- coding: utf-8 -*-
from scrapy.spiders import CrawlSpider
from scrapy.selector import Selector
from imooc.items import ImoocItem
from scrapy.http import Request


class Imooc(CrawlSpider):
    name='imooc'
    allowed_domains = ['imooc.com']
    start_urls = []
    for pn in range(1,31):
        url = 'http://www.imooc.com/course/list?page=%s' % pn
        start_urls.append(url)

    def parse(self,response):
        item=ImoocItem()
        selector=Selector(response)
        Course = selector.xpath('//a[@class="course-card"]')

        for eachCourse in Course:
            Course_name = eachCourse.xpath('div[@class="course-card-content"]/h3[@class="course-card-name"]/text()').extract()[0]
            Course_content = eachCourse.xpath('div[@class="course-card-content"]/div[@class="clearfix course-card-bottom"]/p[@class="course-card-desc"]/text()').extract()
            Course_level = eachCourse.xpath('div[@class="course-card-content"]/div[@class="clearfix course-card-bottom"]/div[@class="course-card-info"]/span/text()').extract()[0]
            Course_attendance = eachCourse.xpath('div[@class="course-card-content"]/div[@class="clearfix course-card-bottom"]/div[@class="course-card-info"]/span/text()').extract()[1]
            item['Course_name'] = Course_name
            item['Course_content'] = ';'.join(Course_content)
            item['Course_level'] = Course_level
            item['Course_attendance'] = Course_attendance
            yield item

4.现在可以运行爬虫把数据导出来，现在以cvs格式测试

scrapy crawl imooc -o data.csv -t csv

查看文件

二.爬取数据并存入MySQL数据库

1.这里使用MySQL数据库存储数据，需要用到 MySQLdb包，确保已经安装

首先建立数据库和表

--创建数据库
create database imooc DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;

--创建表
create table imooc_info2(
title varchar(255) NOT NULL COMMENT '课程名称',
content varchar(255) NOT NULL COMMENT '课程简介',
level varchar(255) NOT NULL COMMENT '课程等级',
sums int  NOT NULL COMMENT '课程学习人数'
)

2.修改pipelines.py

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html


import json
from twisted.enterprise import adbapi
from scrapy import log
import MySQLdb
import MySQLdb.cursors
import codecs
import copy

class ImoocPipeline(object):
    def __init__(self):
        self.file = codecs.open('imooc.json', 'w', encoding='utf-8')
    def process_item(self, item, spider):
        line = json.dumps(dict(item), ensure_ascii=False) + "
"
        self.file.write(line)
        return item
    def spider_closed(self, spider):
        self.file.close()

class MySQLPipeline(object):

    def __init__(self):
        self.dbpool = adbapi.ConnectionPool("MySQLdb",
                                           db = "imooc",            # 数据库名
                                           user = "root",       # 数据库用户名
                                           passwd = "hwfx1234",     # 密码
                                           cursorclass = MySQLdb.cursors.DictCursor,
                                           charset = "utf8",
                                           use_unicode = True
                                           )
    def process_item(self, item, spider):
        asynItem=copy.deepcopy(item)
        query = self.dbpool.runInteraction(self._conditional_insert,asynItem)
        query.addErrback(self.handle_error)
        return item

    def _conditional_insert(self, tb, item):
        tb.execute(""" insert into imooc_info (title,content,level,sums) values (%s,%s,%s,%s)""",(item['Course_name'],item['Course_content'],item['Course_level'],item['Course_attendance']))
        log.msg("Item data in db: %s" % item, level=log.DEBUG)

    def handle_error(self, e):
        log.err(e)

3.修改setting.py 

加入MySQL配置，添加pipelines.py 内新建类

# start MySQL database configure setting
MYSQL_HOST = 'localhost'
MYSQL_DBNAME = 'imooc'
MYSQL_USER = 'root'
MYSQL_PASSWD = 'hwfx1234'
# end of MySQL database configure setting
ITEM_PIPELINES = {
            'imooc.pipelines.ImoocPipeline': 300,
            'imooc.pipelines.MySQLPipeline': 300,
                } 

4.开始爬虫

 scrapy crawl imooc 

查看数据库表数据，数据已经入库。

总结：scrapy简单的应用，还没考虑反爬虫、分布式等问题，还需要多练习。