创建项目
scrapy startproject projectName
cd projectName
scrapy genspider [爬虫名字] [爬虫域名] 爬虫名字不能和projectName重名
运行项目
scrapy crawl [爬虫名字]
或者用配置文件的方式
from scrapy import cmdline cmdline.execute(['scrapy','crawl','dalian_spider'])
爬虫名字
setting.py设置
20 ROBOTSTXT_OBEY = False 40 DEFAULT_REQUEST_HEADERS = { 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36' }