scrapy shell的使用

首先创建项目test1，这里不能不直接用test作文件名，会出现错误，

scrapy startproject test1

然后在目录/～/test1/test1/spiders  中创建文件spider.py

import scrapy                                                                    
class MySpider(scrapy.Spider):
    name = "myspider"
    start_urls = [ #
            "http://example.com",
            "http://example.org",
            "http://example.net"
            ]  
    def parse(self , response):
        if ".org" in response.url:
            from scrapy.shell import inspect_response
            inspect_response(response , self)

在使用scrapy命令及引用时，首字母不大写， 但在书上都是Scrapy且会出现错误：未找到命令

然后在test1项目目录下，运行爬虫

scrapy crawl myspider

同时shell也会被调用

2020-04-09 18:43:24 [scrapy.utils.log] INFO: Scrapy 2.0.1 started (bot: test1)
2020-04-09 18:43:24 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.6.9 (default, Nov  7 2019, 10:44:02) - [GCC 8.3.0], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d  10 Sep 2019), cryptography 2.8, Platform Linux-5.0.0-27-generic-x86_64-with-Ubuntu-18.04-bionic
2020-04-09 18:43:24 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2020-04-09 18:43:24 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'test1',
 'NEWSPIDER_MODULE': 'test1.spiders',
 'ROBOTSTXT_OBEY': True,
 'SPIDER_MODULES': ['test1.spiders']}
2020-04-09 18:43:24 [scrapy.extensions.telnet] INFO: Telnet Password: 9a20daab62ac5a9e
2020-04-09 18:43:24 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
2020-04-09 18:43:24 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-04-09 18:43:24 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-04-09 18:43:24 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-04-09 18:43:24 [scrapy.core.engine] INFO: Spider opened
2020-04-09 18:43:24 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-04-09 18:43:24 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6024
2020-04-09 18:43:24 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://example.com/robots.txt> (referer: None)
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 12 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 13 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 14 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 18 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 19 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 20 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 22 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 23 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 25 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 27 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 29 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 31 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 32 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 43 without any user agent to enforce it on.
2020-04-09 18:43:24 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://example.org/robots.txt> (referer: None)
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 12 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 13 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 14 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 18 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 19 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 20 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 22 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 23 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 25 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 27 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 29 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 31 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 32 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 43 without any user agent to enforce it on.
2020-04-09 18:43:24 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://example.net/robots.txt> (referer: None)
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 12 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 13 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 14 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 18 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 19 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 20 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 22 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 23 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 25 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 27 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 29 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 31 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 32 without any user agent to enforce it on.
2020-04-09 18:43:24 [protego] DEBUG: Rule at line 43 without any user agent to enforce it on.
2020-04-09 18:43:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://example.com> (referer: None)
2020-04-09 18:43:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://example.org> (referer: None)
2020-04-09 18:43:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://example.net> (referer: None)
[s] Available Scrapy objects:
[s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s]   crawler    <scrapy.crawler.Crawler object at 0x7fe120f3ac88>
[s]   item       {}
[s]   request    <GET http://example.org>
[s]   response   <200 http://example.org>
[s]   settings   <scrapy.settings.Settings object at 0x7fe120c83e80>
[s]   spider     <MySpider 'myspider' at 0x7fe12091f5c0>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   view(response)    View response in a browser


In [1]: response.url           查看地址                                                  
Out[1]: 'http://example.org'