zoukankan      html  css  js  c++  java
  • scrapy Data flow

    The data flow in Scrapy is controlled by the execution engine, and goes like this:
    1. The Engine gets the initial Requests to crawl from the Spider.
    2. The Engine schedules the Requests in the Scheduler and asks for the next Requests to crawl.
    3. The Scheduler returns the next Requests to the Engine.
    4. The Engine sends the Requests to the Downloader, passing through the Downloader Middlewares (see
    process_request()).
    5. Once the page finishes downloading the Downloader generates a Response (with that page) and sends it to the
    Engine, passing through the Downloader Middlewares (see process_response()).
    6. The Engine receives the Response from the Downloader and sends it to the Spider for processing, passing
    through the Spider Middleware (see process_spider_input()).
    7. The Spider processes the Response and returns scraped items and new Requests (to follow) to the Engine,
    passing through the Spider Middleware (see process_spider_output()).
    8. The Engine sends processed items to Item Pipelines, then send processed Requests to the Scheduler and asks
    for possible next Requests to crawl.
    9. The process repeats (from step 1) until there are no more requests from the Scheduler.

  • 相关阅读:
    golang访问数据库
    dynamic与泛型
    新的published和$M+对比
    插入窗体到别的程序里
    淺談怎么样运用Delphi 2009地泛型容器類別
    Delphi随记
    查找文件
    Delphi操作xml
    Delphi图像编程学习笔记
    Ext.net中如何上传文件
  • 原文地址:https://www.cnblogs.com/wuhua1/p/8409913.html
Copyright © 2011-2022 走看看