安装scrapy框架之前,需要安装几个必备库
ps.分享个python库下载地址:https://www.lfd.uci.edu/~gohlke/pythonlibs/
0、wheel(有了这个库之后可以本地安装pyhton库)
1、lxml
2、pyOpenSSL
3、pywin32
4、twisted
anaconda可以使用自带的conda install xxxx来安装组件
必备组件安装完成后,pip install scrapy 或者conda install scrapy
安装完成后,test一下:
命令行输入scrapy
(C:ProgramDataAnaconda3) C:Userswangguoqiang>Scrapy Scrapy 1.5.1 - no active project Usage: scrapy <command> [options] [args] Available commands: bench Run quick benchmark test fetch Fetch a URL using the Scrapy downloader genspider Generate new spider using pre-defined templates runspider Run a self-contained spider (without creating a project) settings Get settings values shell Interactive scraping console startproject Create new project version Print Scrapy version view Open URL in browser, as seen by Scrapy [ more ] More commands available when run from project directory Use "scrapy <command> -h" to see more info about a command
显示如下即代表安装成功
继续输入Scrapy startproject hello
(C:ProgramDataAnaconda3) C:Userswangguoqiang>Scrapy startproject hello New Scrapy project 'hello', using template directory 'c:\programdata\anaconda \lib\site-packages\scrapy\templates\project', created in: C:Userswangguoqianghello You can start your first spider with: cd hello scrapy genspider example example.com
cd 进入创建的hello项目中
输入scrapy genspider baidu www.baidu.com
(C:ProgramDataAnaconda3) C:Userswangguoqiang>cd hello (C:ProgramDataAnaconda3) C:Userswangguoqianghello>scrapy genspider baidu ww w.baidu.com Created spider 'baidu' using template 'basic' in module: hello.spiders.baidu
输入
scrapy crawl baidu