zoukankan      html  css  js  c++  java
  • CrawlSpider一直无法进入回调函数

    CrawlSpider一直无法进入回调函数的

    碰到的两个坑:

    1、正则表达式写的不对

    start_urls = ['http://wz.sun0769.com/political/index/politicsNewest?id=1&page=1']
    
    link = LinkExtractor(allow=r'id=1&page=d+')
    


    2、回调函数中的xpath路径写的不对


    附本次代码:
    import scrapy
    from scrapy.linkextractors import LinkExtractor
    from scrapy.spiders import CrawlSpider, Rule
    from sunCrawlSpider.items import SuncrawlspiderItem
    from sunCrawlSpider.items import Detail_item
    
    
    class SunSpider(CrawlSpider):
        name = 'sun'
        # allowed_domains = ['www.xxx.com']
        start_urls = ['http://wz.sun0769.com/political/index/politicsNewest?id=1&page=1']
    
        link = LinkExtractor(allow=r'id=1&page=d+')
    
        link_detail = LinkExtractor(allow=r'political/politics/index?id=d+')
    
        rules = (
            Rule(link, callback='parse_item', follow=False),
            Rule(link_detail, callback='parse_detail', follow=False),
        )
    
        def parse_item(self, response):
            li_list = response.xpath('/html/body/div[2]/div[3]/ul[2]/li')
            for li in li_list:
                num = li.xpath('./span[1]/text()').extract_first()
                title = li.xpath('./span[3]/a/text()').extract_first()
                # url = 'http://wz.sun0769.com/' + li.xpath('./span[3]/a/@href').extract_first()
                item = SuncrawlspiderItem()
                item['title'] = title
                item['num'] = num
    
                yield item
    
        def parse_detail(self, response):
            content = response.xpath('/html/body/div[3]/div[2]/div[2]/div[2]/pre/text()').extract_first()
            num = response.xpath('/html/body/div[3]/div[2]/div[2]/div[1]/span[4]/text()').extract_first()
            num = num.split(':')[-1]
            item = Detail_item()
            item['num'] = num
            item['content'] = content
    
            yield item
    

      




  • 相关阅读:
    str_split 分隔中文出现乱码 替代函数
    PHP 浮点数 转化 整数方法对比 ceil,floor,round,intval,number_format
    php 判断字符串之间包含关系
    不解之谜
    正则匹配 特殊的 符号
    PHP 判断字符串 是否 包含另一个字符串
    PHP 删除 数组 指定成员
    HTML 权重标签的使用
    【PAT甲级】1094 The Largest Generation (25 分)(DFS)
    【PAT甲级】1093 Count PAT's (25 分)
  • 原文地址:https://www.cnblogs.com/wlj-axia/p/13378474.html
Copyright © 2011-2022 走看看