zoukankan      html  css  js  c++  java
  • CrawlSpider一直无法进入回调函数

    CrawlSpider一直无法进入回调函数的

    碰到的两个坑:

    1、正则表达式写的不对

    start_urls = ['http://wz.sun0769.com/political/index/politicsNewest?id=1&page=1']
    
    link = LinkExtractor(allow=r'id=1&page=d+')
    


    2、回调函数中的xpath路径写的不对


    附本次代码:
    import scrapy
    from scrapy.linkextractors import LinkExtractor
    from scrapy.spiders import CrawlSpider, Rule
    from sunCrawlSpider.items import SuncrawlspiderItem
    from sunCrawlSpider.items import Detail_item
    
    
    class SunSpider(CrawlSpider):
        name = 'sun'
        # allowed_domains = ['www.xxx.com']
        start_urls = ['http://wz.sun0769.com/political/index/politicsNewest?id=1&page=1']
    
        link = LinkExtractor(allow=r'id=1&page=d+')
    
        link_detail = LinkExtractor(allow=r'political/politics/index?id=d+')
    
        rules = (
            Rule(link, callback='parse_item', follow=False),
            Rule(link_detail, callback='parse_detail', follow=False),
        )
    
        def parse_item(self, response):
            li_list = response.xpath('/html/body/div[2]/div[3]/ul[2]/li')
            for li in li_list:
                num = li.xpath('./span[1]/text()').extract_first()
                title = li.xpath('./span[3]/a/text()').extract_first()
                # url = 'http://wz.sun0769.com/' + li.xpath('./span[3]/a/@href').extract_first()
                item = SuncrawlspiderItem()
                item['title'] = title
                item['num'] = num
    
                yield item
    
        def parse_detail(self, response):
            content = response.xpath('/html/body/div[3]/div[2]/div[2]/div[2]/pre/text()').extract_first()
            num = response.xpath('/html/body/div[3]/div[2]/div[2]/div[1]/span[4]/text()').extract_first()
            num = num.split(':')[-1]
            item = Detail_item()
            item['num'] = num
            item['content'] = content
    
            yield item
    

      




  • 相关阅读:
    每周学算法/读英文/知识点心得分享 1.28
    ARTS 1.21
    ARTS 1.14
    ARTS 1.7
    ARTS 12.31
    ARTS 12.24
    Leetcode : Median of Two Sorted Arrays
    我是怎样改善遗留系统的
    《大话重构》免费送书活动开始啦
    我的新书终于要出来啦
  • 原文地址:https://www.cnblogs.com/wlj-axia/p/13378474.html
Copyright © 2011-2022 走看看