zoukankan      html  css  js  c++  java
  • CrawlSpider一直无法进入回调函数

    CrawlSpider一直无法进入回调函数的

    碰到的两个坑:

    1、正则表达式写的不对

    start_urls = ['http://wz.sun0769.com/political/index/politicsNewest?id=1&page=1']
    
    link = LinkExtractor(allow=r'id=1&page=d+')
    


    2、回调函数中的xpath路径写的不对


    附本次代码:
    import scrapy
    from scrapy.linkextractors import LinkExtractor
    from scrapy.spiders import CrawlSpider, Rule
    from sunCrawlSpider.items import SuncrawlspiderItem
    from sunCrawlSpider.items import Detail_item
    
    
    class SunSpider(CrawlSpider):
        name = 'sun'
        # allowed_domains = ['www.xxx.com']
        start_urls = ['http://wz.sun0769.com/political/index/politicsNewest?id=1&page=1']
    
        link = LinkExtractor(allow=r'id=1&page=d+')
    
        link_detail = LinkExtractor(allow=r'political/politics/index?id=d+')
    
        rules = (
            Rule(link, callback='parse_item', follow=False),
            Rule(link_detail, callback='parse_detail', follow=False),
        )
    
        def parse_item(self, response):
            li_list = response.xpath('/html/body/div[2]/div[3]/ul[2]/li')
            for li in li_list:
                num = li.xpath('./span[1]/text()').extract_first()
                title = li.xpath('./span[3]/a/text()').extract_first()
                # url = 'http://wz.sun0769.com/' + li.xpath('./span[3]/a/@href').extract_first()
                item = SuncrawlspiderItem()
                item['title'] = title
                item['num'] = num
    
                yield item
    
        def parse_detail(self, response):
            content = response.xpath('/html/body/div[3]/div[2]/div[2]/div[2]/pre/text()').extract_first()
            num = response.xpath('/html/body/div[3]/div[2]/div[2]/div[1]/span[4]/text()').extract_first()
            num = num.split(':')[-1]
            item = Detail_item()
            item['num'] = num
            item['content'] = content
    
            yield item
    

      




  • 相关阅读:
    GIT学习实践笔记
    ubuntu 安装pygit2
    强化vim打造python的IDE
    Python Django学习和实践
    浏览器调试工具网页性能分析中的使用
    公司发版shell脚本重构
    Nightwatch+gulp
    Git
    JavaScript Unit Test with Qunit
    Unit Test Mocking Framework
  • 原文地址:https://www.cnblogs.com/wlj-axia/p/13378474.html
Copyright © 2011-2022 走看看