zoukankan      html  css  js  c++  java
  • CrawlSpider一直无法进入回调函数

    CrawlSpider一直无法进入回调函数的

    碰到的两个坑:

    1、正则表达式写的不对

    start_urls = ['http://wz.sun0769.com/political/index/politicsNewest?id=1&page=1']
    
    link = LinkExtractor(allow=r'id=1&page=d+')
    


    2、回调函数中的xpath路径写的不对


    附本次代码:
    import scrapy
    from scrapy.linkextractors import LinkExtractor
    from scrapy.spiders import CrawlSpider, Rule
    from sunCrawlSpider.items import SuncrawlspiderItem
    from sunCrawlSpider.items import Detail_item
    
    
    class SunSpider(CrawlSpider):
        name = 'sun'
        # allowed_domains = ['www.xxx.com']
        start_urls = ['http://wz.sun0769.com/political/index/politicsNewest?id=1&page=1']
    
        link = LinkExtractor(allow=r'id=1&page=d+')
    
        link_detail = LinkExtractor(allow=r'political/politics/index?id=d+')
    
        rules = (
            Rule(link, callback='parse_item', follow=False),
            Rule(link_detail, callback='parse_detail', follow=False),
        )
    
        def parse_item(self, response):
            li_list = response.xpath('/html/body/div[2]/div[3]/ul[2]/li')
            for li in li_list:
                num = li.xpath('./span[1]/text()').extract_first()
                title = li.xpath('./span[3]/a/text()').extract_first()
                # url = 'http://wz.sun0769.com/' + li.xpath('./span[3]/a/@href').extract_first()
                item = SuncrawlspiderItem()
                item['title'] = title
                item['num'] = num
    
                yield item
    
        def parse_detail(self, response):
            content = response.xpath('/html/body/div[3]/div[2]/div[2]/div[2]/pre/text()').extract_first()
            num = response.xpath('/html/body/div[3]/div[2]/div[2]/div[1]/span[4]/text()').extract_first()
            num = num.split(':')[-1]
            item = Detail_item()
            item['num'] = num
            item['content'] = content
    
            yield item
    

      




  • 相关阅读:
    远程培训数据库迁移
    RMAN系列catalog数据库
    RMAN系列表空间恢复
    典型的一周备份脚本
    [转载]对 Oracle 备份与恢复 的补充说明转自tianlesoft的文章
    RMAN系列基于scn恢复
    使用Profile对用户Session会话进行资源限制
    RMAN系列控制文件、数据文件、redolog全部丢失
    全自动 TSPITR基于RMANLOGSEQ
    Oracle 10g EXPDP和IMPDP使用说明
  • 原文地址:https://www.cnblogs.com/wlj-axia/p/13378474.html
Copyright © 2011-2022 走看看