33、豆瓣图书短评

练习介绍

要求：

本练习需要运用scrapy的知识，爬取豆瓣图书TOP250（https://book.douban.com/top250  ）前2页的书籍（50本）的短评数据存储成Excel

    书名

    评论ID

    短评内容

 

1、创建爬虫项目 

 

D:USERDATApython>scrapy startproject duanping
New Scrapy project 'duanping', using template directory 'c:userswww1707appdatalocalprogramspythonpython37libsite-packagesscrapy	emplatesproject', created in:
    D:USERDATApythonduanping

You can start your first spider with:
    cd duanping
    scrapy genspider example example.com

D:USERDATApython>

2、创建爬虫文件 D:USERDATApythonduanpingduanpingspidersduanping.py

 

import scrapy
import bs4
import re
import requests
import math
from ..items import DuanpingItem

class DuanpingItemSpider(scrapy.Spider):
    name = 'duanping'
    allowed_domains = ['book.douban.com']
    start_urls = ['https://book.douban.com/top250?start=0','https://book.douban.com/top250?start=0']

    def parse(self,response):
        bs = bs4.BeautifulSoup(response.text,'html.parser')
        datas = bs.find_all('a',class_='nbg')
        for data in datas:
            book_url = data['href']
            yield scrapy.Request(book_url,callback=self.parse_book)

    def parse_book(self,response):
        book_url = str(response).split(' ')[1].replace('>','')
        print(book_url)
        bs = bs4.BeautifulSoup(response.text,'html.parser')
        comments = int(bs.find('a',href=re.compile('^https://book.douban.com/subject/.*/comments/')).text.split(' ')[1])
        pages = math.ceil(comments / 20) + 1
        #for i in range(1,pages):
        for i in range(1,3):
            comments_url = '{}comments/hot?p={}'.format(book_url,i)
            print(comments_url)
            yield scrapy.Request(comments_url,callback=self.parse_comment)

    def parse_comment(self,response):
        bs = bs4.BeautifulSoup(response.text,'html.parser')
        book_name = bs.find('a',href=re.compile('^https://book.douban.com/subject/')).text
        datas = bs.find_all('li',class_='comment-item')
        for data in datas:
            item = DuanpingItem()
            item['book_name'] = book_name
            item['user_id']= data.find_all('a',href=re.compile('^https://www.douban.com/people/'))[1].text
            item['comment'] = data.find('span',class_='short').text
            yield item

3、编辑item文件 D:USERDATApythonduanpingduanpingitems.py

 

import scrapy

class DuanpingItem(scrapy.Item):
    book_name = scrapy.Field()
    user_id = scrapy.Field()
    comment  = scrapy.Field()

4、编辑文件 D:USERDATApythonduanpingduanpingpipelines.py

 

import openpyxl

class DuanpingPipeline(object):
    def __init__(self):
        self.wb = openpyxl.Workbook()
        self.ws = self.wb.active
        self.ws.append(['书名','评论ID','评论内容'])
    
    def process_item(self, item, spider):
        line = [item['book_name'],item['user_id'],item['comment']]
        self.ws.append(line)
        return item
    def close_spider(self,spider):
        self.wb.save('./save.xlsx')
        self.wb.close()

5、编辑D:USERDATApythonduanpingduanpingsettings.py

 

BOT_NAME = 'duanping'

SPIDER_MODULES = ['duanping.spiders']
NEWSPIDER_MODULE = 'duanping.spiders'

USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'

ROBOTSTXT_OBEY = False

DOWNLOAD_DELAY = 1

ITEM_PIPELINES = {
    'duanping.pipelines.DuanpingPipeline': 300,
}

FEED_URI = './save.csv'
FEED_FORMAT='CSV'
FEED_EXPORT_ENCODING='utf-8-sig'

6、执行命令  scrapy crawl duanping

 

---

以下是草稿部分

1、图书列表页

 

https://book.douban.com/top250?start=0

https://book.douban.com/top250?start=1

 

            <a class="nbg" href="https://book.douban.com/subject/1083428/"

 

 

2、图书详情页

 

https://book.douban.com/subject/1770782/

 

 

评论总数          <a href="https://book.douban.com/subject/1770782/comments/">全部 112943 条</a>

 

 

3、图书短评页

 

https://book.douban.com/subject/1770782/comments/hot?p=1      每页20个

 

 

书名                <a href="https://book.douban.com/subject/1770782/">追风筝的人</a>

 

 

 

find_all    <li class="comment-item" data-cid="693413905">

 

    评论ID[1]       <a title="九尾黑猫" href="https://www.douban.com/people/LadyInSatin/">

                          <a href="https://www.douban.com/people/LadyInSatin/">九尾黑猫</a>

 

    短评内容         <span class="short">“为你，千千万万遍。”我想，小说描写了一种最为诚挚的情感，而且它让你相信有些东西依然存在。在这个没有人相信承诺的年代，让人再次看到承诺背后那些美丽复杂的情感。这是一本好看的书，它让你重新思考。</span>

 

    <li class="comment-item" data-cid="693413905">
            <div class="avatar">
                <a title="福根儿" href="https://www.douban.com/people/fugen/">
                    <img src="https://img3.doubanio.com/icon/u3825598-141.jpg">
                </a>
            </div>
        <div class="comment">
            <h3>
                <span class="comment-vote">
                    <span id="c-693413905" class="vote-count">4756</span>
                        <a href="javascript:;" id="btn-693413905" class="j a_show_login" data-cid="693413905">有用</a>
                </span>
                <span class="comment-info">
                    <a href="https://www.douban.com/people/fugen/">福根儿</a>
                        <span class="user-stars allstar50 rating" title="力荐"></span>
                    <span>2013-09-18</span>
                </span>
            </h3>
            <p class="comment-content">
            
                <span class="short">为你，千千万万遍。</span>
            </p>
        </div>
    </li>