zoukankan      html  css  js  c++  java
  • 安装scrapy框架

    安装scrapy框架之前,需要安装几个必备库

    ps.分享个python库下载地址:https://www.lfd.uci.edu/~gohlke/pythonlibs/

    0、wheel(有了这个库之后可以本地安装pyhton库)

    1、lxml

    2、pyOpenSSL

    3、pywin32

    4、twisted

    anaconda可以使用自带的conda install xxxx来安装组件

    必备组件安装完成后,pip install scrapy 或者conda install scrapy

    安装完成后,test一下:

    命令行输入scrapy

    (C:ProgramDataAnaconda3) C:Userswangguoqiang>Scrapy
    Scrapy 1.5.1 - no active project
    
    Usage:
    scrapy <command> [options] [args]
    
    Available commands:
    bench Run quick benchmark test
    fetch Fetch a URL using the Scrapy downloader
    genspider Generate new spider using pre-defined templates
    runspider Run a self-contained spider (without creating a project)
    settings Get settings values
    shell Interactive scraping console
    startproject Create new project
    version Print Scrapy version
    view Open URL in browser, as seen by Scrapy
    
    [ more ] More commands available when run from project directory
    
    Use "scrapy <command> -h" to see more info about a command

    显示如下即代表安装成功

    继续输入Scrapy startproject hello

    (C:ProgramDataAnaconda3) C:Userswangguoqiang>Scrapy startproject hello
    New Scrapy project 'hello', using template directory 'c:\programdata\anaconda
    \lib\site-packages\scrapy\templates\project', created in:
    C:Userswangguoqianghello
    
    You can start your first spider with:
    cd hello
    scrapy genspider example example.com

    cd 进入创建的hello项目中

    输入scrapy genspider baidu www.baidu.com

    (C:ProgramDataAnaconda3) C:Userswangguoqiang>cd hello
    
    (C:ProgramDataAnaconda3) C:Userswangguoqianghello>scrapy genspider baidu ww
    w.baidu.com
    Created spider 'baidu' using template 'basic' in module:
      hello.spiders.baidu

    输入

    scrapy crawl baidu

  • 相关阅读:
    Method总结
    使用CSS的五种方式
    debug js code
    Overload
    fiddler模拟弱网测试
    POJ 1753 Flip Game (IDA*)
    UVA 11400 Lighting System Design(照明系统设计)(dp)
    UVA 12563 Jin Ge Jin Qu hao(劲歌金曲)(01背包+滚动数组)
    UVA 116 Unidirectional TSP (单向TSP)(dp多段图的最短路)
    UVA 1151 Buy or Build (买还是建)(并查集+二进制枚举子集)
  • 原文地址:https://www.cnblogs.com/wang666/p/9467829.html
Copyright © 2011-2022 走看看