zoukankan      html  css  js  c++  java
  • scrapy使用

    1、安装:pip3 install scrapy

      结果安装失败,由于Failed building wheel for Twisted;单独安装Twisted,下载了Twisted-16.6.0-cp35-cp35m-win_amd64.whl文件
      pip3 install 目录Twisted-16.6.0-cp35-cp35m-win_amd64.whl 安装成功,再次执行pip3 install scrapy

    2、新建项目

      cd 项目存在路径

      scrapy startproject 项目名称

    Module `scrapy.conf` is deprecated, use `crawler.settings` attribute instead   from scrapy.conf import settings

     from scrapy.conf import crawler.settings 修改为:from scrapy.settings import Settings

    Module `scrapy.log` has been deprecated, Scrapy now relies on the builtin Python library for logging. Read the updated logging entry in the documentation to learn more.

    如果scrapy crawl xxx运行爬虫后出现结果报错:


    Paste_Image.png
    ImportError: cannot import name '_win32stdio'
    ImportError: No module named 'win32api'

    因为twisted需要安装依赖模块pywin32

    pip install pypiwin32

    Scrapy 'module' object has no attribute 'Spider'错误

     

    在“Scrapy入门教程”中,在创建的“dmoz_spider.py”文件中是通过

    import scrapy
    
    class DmozSpider(scrapy.Spider):

    的方式导入。但是用这种方法会出现错误:

        class DmozSpider(scrapy.Spider):
    AttributeError: 'module' object has no attribute 'Spider'

    需要换一种方式导入

    from scrapy.spider import Spider
    
    class DmozSpider(Spider):

    就可以啦!

    pip源:http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted

    国内:https://pypi.doubanio.com/simple/

    参考资料:

    http://python.gotrained.com/scrapy-tutorial-web-scraping-craigslist/

  • 相关阅读:
    upload.go
    heartbeat.go
    delete.go
    get.go
    handler.go
    uuid.go
    kingpin_parser.go
    disk.go
    logrus_hook.go
    反连接NOT EXISTS子查询中有or 谓词连接条件SQL优化一例
  • 原文地址:https://www.cnblogs.com/tianboblog/p/6340938.html
Copyright © 2011-2022 走看看