scrapy startproject Redbacktest
cd Redbacktest
创建jobbole爬虫
scrapy genspider jobbole2 blog.jobbole.com
从pycharm中导入后创建main文件
from scrapy.cmdline import execute import sys sys.path.append("D:PycharmProjectsRedbacktest") execute(['scrapy','crawl','jobbole2'])
调试前修改“君子协议”
ROBOTSTXT_OBEY = False
断点调试response是否获取到值
