I apologize for the time it took before writing this part, but you know what they say: Better late then never.
In this part, I’ll explain how I extracted (scraped) the data from the Transports Quebec database mentioned in Part 1 using Python, Scrapy and a few other tools.
This post is not a full-fledged tutorial on using Scrapy, but it should give you a place to start if you’d like to do something similar.
In order to jump directly to the good stuff, I’ll skip the Scrapy installation and assume you are already familiar with Python. If you are not familiar with Python, you’re missing a lot. Get started here, then use some Google-Fu and start hacking on your own project to keep learning.
DISCLAIMER:
- This small project was a quick and dirty drive-by experiment. This code is not: a complete solution, the best solution, production-ready, [insert your own similar statement here]. I’m sure you can do better
(Feel free to post your comments and improvements, however, keep in mind that I don’t plan to maintain or improve this code) - This code follows my late-night style guide.
- For non-French speaking readers: This project extracts data from a French (Quebec, Canada) website, however, you can continue to read, the actual content is not really important and you should be able to follow without problem. You may also learn some French words.
- For French-speaking readers: If you are reading this, you probably won’t mind the English comments in my code. Unless required to do otherwise, I always write my code/comments in English. It’s easier to share this way.
- I’m sure you’ll find typos.
- Everything below was done using Python 2.7 and Scrapy v0.13. I have not tested with Scrapy 0.14+ or
Python 3+ (UPDATE 2011/12/12: As Pablo Hoffman (none other than the lead Scrapy developer) noted in the comments, Scrapy does not support Python 3 and there are currently no plans to support it. Sorry about that!)
That said, let’s get started.
Our target: The Transports Quebec infrastructure database (Well, they call it “Ponts et routes: Information aux citoyens”.)
If you visit the website, you’ll notice that the first page you get is an empty form like this one:
So, to get to the actual data, you need to submit the form by clicking on “Afficher les résultats“. Luckily, the default form values will display all entries. After submitting the form, you’ll get results like these:
This is what our main scraper will have to deal with. Unfortunately, not all the fields we want to extract are available in this table. The rest is in what I call the “details” page. You get there by clicking on the “Fiche No” (first column) link (e.g. 00002). The details page look like this: 
So for each entry found, our scraper will have to follow the “Fiche No” link and scrape the details page too.
In addition, each page of the report only contains 15 items. This means we’ll have to navigate the pages too.
Tools You’ll Need
In order to identify HTML elements to extract, you’ll need these:
NOTE: I used these tools. You may have your own tools that you prefer. Feel free to let me know of other good ones in the comments. (No need to point out that Safari has a web inspector, I know, but I did not find anything equivalent to Firefinder.)
Getting Started
Switch to a work directory then create your base Scrapy project (I called mine mtqinfra):
$ scrapy startproject mtqinfra |
./mtqinfra/spiders/__init__.py |
At this point we have a “skeleton” project. Now let’s create a very simple spider just to see if we can get this to work. (NOTE: There’s a “scrapy genspider” command but I won’t use it here.) Create the file spiders/mtqinfra_spider.py:
04 | from scrapy.spider import BaseSpider |
05 | from scrapy.http import Request |
06 | from scrapy.http import FormRequest |
07 | from scrapy.selector import HtmlXPathSelector |
12 | sys.setdefaultencoding('utf-8') |
14 | class MTQInfraSpider(BaseSpider): |
16 | allowed_domains = ["www.mtq.gouv.qc.ca"] |
21 | def parse(self, response): |
Now, let’s see if this appears to work:
2011-11-22 00:52:09-0500 [scrapy] INFO: Scrapy 0.13.0 started (bot: mtqinfra) |
2011-11-22 00:52:10-0500 [mtqinfra] INFO: Closing spider (finished) |
As you can see, the bot did request our page, got redirected (Status 302) because the initial URL did not include a session ID (2747914247598050) and finally downloaded (Status 200) the page.
Posting the Form
OK, the next step if to get our bot to submit the form for us. Let’s tweak the parse method a bit and add another method to handle the real parsing like this:
01 | def parse(self, response): |
02 | return [FormRequest.from_response(response, |
03 | callback=self.parse_main_list)] |
05 | def parse_main_list(self, response): |
06 | self.log("After submitting form.", level=log.INFO) |
07 | with open("results.html", "w") as f: |
08 | f.write(response.body) |
10 | os.system("open results.html") |
Now, the parse method return a “FormRequest” object that will instruct the bot to submit the form then call “self.parse_main_list” with the response.
Oops. The response contains no results. Whats wrong? Well, if you search a bit (I sniffed the network to compare what is sent by Firefox when the form button is pressed versus what is sent by Scrapy), you’ll find that the “p_request” form field is not set to “RECHR” by Scrapy as when sent by the browser. This is due to the fact that it is empty by default and set by a Javascript function. Let’s fix that:
01 | def parse(self, response): |
02 | return [FormRequest.from_response(response, |
03 | formdata={ "p_request": "RECHR" }, |
04 | callback=self.parse_main_list)] |
06 | def parse_main_list(self, response): |
07 | self.log("After submitting form.", level=log.INFO) |
08 | with open("results.html", "w") as f: |
09 | f.write(response.body) |
11 | os.system("open results.html") |
Ahh, much better. Now we can begin our parser.
Identifying HTML Elements and Their Corresponding XPath
Scrapy uses XPath to select and extract elements from a web page. Well, technically speaking you could parse the response body any way you want (e.g. using regular expressions), but XPath is very powerful so I suggest you give it a try.
I won’t write an XPath tutorial here, but simply put, XPath is a query language that allows you to select elements from HTML like you would do with SQL to extract fields from a table. Although XPath queries can appear intimidating at first, the XPath syntax itself is pretty simple.
Here are some tips to understand, learn and use XPath quickly and identify elements you want to extract.
Use Firebug to identify absolute XPath expressions
In Firebug, the absolute XPath expression to select an HTML element is displayed in a tooltip:
You can also right-click an element and select “Copy XPath” to copy it to the clipboard:
Use Firefinder to test XPath expressions
In Firebug, select the Firefinder tab and enter your XPath expression (or query or filter or selector, whatever you want to call it) then click “Filter”. The matching element(s) will be listed below and highlighted on the page. Because we’ll need to loop on each result table row, try it with this expression:
//table[@id="R10432126941777590"]//table[@summary="Report"]/tbody/tr |
This expression will select each row of the result table.
Use “scrapy shell” to test XPath expressions in Scrapy
Scrapy has a very handy “shell” mode to help you test stuff. In order to bypass the form submission process and get directly on the result page, submit the form with your browser and then copy the URL that includes your session ID. If you do a “GET” on this URL, you’ll get the page you were viewing in your browser (as long as the session ID is still valid). Let’s try it:
01 | $ scrapy shell http://www.mtq.gouv.qc.ca/pls/apex/f?p=102:56:482485043431341::NO:RP:: |
03 | >>> hxs.select('//table[@id="R10432126941777590"]//table[@summary="Report"]/tbody/tr[2]') |
05 | >>> hxs.select('//table[@id="R10432126941777590"]//table[@summary="Report"]/tr[2]') |
06 | [<HtmlXPathSelector xpath='//table[@id="R10432126941777590"]//table[@summary="Report"]/tr[2]' data=u'<tr onmouseover="row_mouse_over104321269'>] |
07 | >>> row2 = hxs.select('//table[@id="R10432126941777590"]//table[@summary="Report"]/tr[2]') |
08 | >>> row2_cells = row2.select('td') |
12 | <HtmlXPathSelector xpath='td' data=u'<td class="t3data" align="center"><a hre'> |
13 | >>> row2_cells[0].extract() |
15 | <a href="f?p=102:53:482485043431341::NO:53:P53_IDE_STRCT_0001:211033"><img title="Fiche de la structure: 00002" src="wwv_flow_file_mgr.get_file?p_security_group_id=1848625384920754&p_fname=detail.gif" alt="Fiche de la structure: 00002" /> |
18 | >>> row2_cells[0].select('a/text()').extract()[0] |
20 | >>> row2_cells[0].select('a/@href').extract()[0] |
21 | u'f?p=102:53:482485043431341::NO:53:P53_IDE_STRCT_0001:211033' |
IMPORTANT: Take note of lines 3 and 4. I am unsure why, but while Firefinder takes the “tbody” tag into account in XPath expressions, Scrapy does not want them. Thus, our previously working XPath returns nothing. If you remove the “tbody” tag (line 5), the expression will work and return the second row of the result table.
Line 8 shows the power of XPath and the Scrapy HtmlXPathSelector object. To extract an array of cells for row #2, on the HtmlXPathSelector for the row we simply call “select(‘td’)”.
The rest of the lines shows how to use the extract() method to extract HTML, text and attribute values.
Create XPath expressions that are general and specific at the same time
Although this does not appear to make much sense, here’s what I mean:
Take this XPath (Firefinder format, remove tbody for Scrapy):
/html/body/form/table/tbody/tr[2]/td/table[4]/tbody/tr/td/table[3]/tbody/tr[2]/td/table |
It is a very specific and absolute XPath to the results table. Should the web page change just a bit (e.g., an extra row in the first table of the form or a new table to hold new information), your XPath will become invalid or point to the wrong table. Now, in the page generated by the underlying reporting logic (Oracle Application Express (APEX) in this case), we noticed that the parent table of the results table has the “id” attribute set to “R10432126941777590″ and that the actual results table has the attribute “summary” set to “Report”. We can then use the following XPath to get to the same table:
//table[@id="R10432126941777590"]//table[@summary="Report"] |
It is more “general” as it skips over everything but two tables. It simply says: “Get me the tables that have their “summary” attribute set to “Report” that are also “under” (in) tables that have their “id” attribute set to “R10432126941777590″. However, because the “id” is very specific (only match one table) and the “summary” is also (somewhat) specific because it only match one table inside that other table, we are unlikely to match anything else. Thats what I mean by general and specific at the same time.
Now, I don’t know Oracle APEX enough to be certain the “id” used above won’t change if the report HTML format is modified, so maybe my solution could break later in this case, however, the principle in general is still good.
Use relative XPath expressions and HtmlXPathSelector objects
Don’t use absolute XPath expressions (as mentioned above) or repeat expressions in your code. Instead use the powerful HtmlXPathSelector objects to navigate in the HTML structure using relative XPath expressions. For example this code gets you columns 1-3 of row 2 or the results table but it sucks:
hxs = HtmlXPathSelector(response) |
row2_cell1 = hxs.select('/html/body/form/table/tr[2]/td/table[4]/tr/td/table[3]/tr[2]/td/table/tr[2]/td[1]') |
row2_cell2 = hxs.select('/html/body/form/table/tr[2]/td/table[4]/tr/td/table[3]/tr[2]/td/table/tr[2]/td[2]') |
row2_cell3 = hxs.select('/html/body/form/table/tr[2]/td/table[4]/tr/td/table[3]/tr[2]/td/table/tr[2]/td[3]') |
This code does the same, but does not suck:
hxs = HtmlXPathSelector(response) |
rows = hxs.select('//table[@id="R10432126941777590"]//table[@summary="Report"]/tr') |
Why? Because in the first case, if anything changes in the HTML, you’ll need to modify 3 XPath expressions. In the second case, you’ll probably need to modify only one (if necessary at all). Of course, this example is simplified a bit to show you the concept (you’ll probably want to loop over rows and cells in your code as we’ll do later), but I hope you get the idea. Unfortunately, sometimes there is no (safe) way to get to an element other than by using an (almost) absolute XPath. Just try to minimize their use in your project.
The MTQInfraItem Object
The parsers that will handle responses as the website is crawled return “Item” objects (or Request objects to instruct the crawler to request more pages). “Item” objects are more or less a “model” of the data you will be scraping. It is a container for the structured data you will be extracting from the HTML pages. Edit the items.py file and replace the existing “MtqinfraItem” with this one:
01 | class MTQInfraItem(Item): |
05 | structure_id = Field() |
06 | structure_name = Field() |
07 | structure_type = Field() |
08 | structure_type_img_href = Field() |
09 | territorial_direction = Field() |
11 | municipality = Field() |
18 | location_href = Field() |
19 | planned_intervention = Field() |
24 | construction_year = Field() |
25 | picture_href = Field() |
26 | last_general_inspection_date = Field() |
27 | next_general_inspection_date = Field() |
28 | average_daily_flow_of_vehicles = Field() |
29 | percent_trucks = Field() |
31 | fusion_marker = Field() |
As you can see, to create your own MTQInfraItem type, you simply subclass the Item class and add a bunch of fields that you later plan to populate and save in your output.
Scraping the Main List
Scraping the main page requires us to do the following:
- Process each row of the results table and for each one:
- Extract all the data we want to keep
- Create a new MTQInfraItem object for the data
- Save the item in a buffer because it is still incomplete as we need to scrape the “details” page to extract the rest of the fields
- Return a “Request” object to the crawler to inform it that the “details” page needs to be requested
- Check if there is another page containing results and if so, return a ”Request” object to the crawler to inform it that another page of results needs to be requested.
The final parser for the main list looks like this:
001 | def parse_main_list(self, response): |
004 | hxs = HtmlXPathSelector(response) |
005 | rows = hxs.select('//table[@id="R10432126941777590"]//table[@summary="Report"]/tr') |
007 | self.log("Failed to extract results table from response for URL '{:s}'. Has 'id' changed?".format(response.request.url), level=log.ERROR) |
010 | cells = row.select('td') |
016 | total_num_records = int(hxs.select('//table[@id="R19176911384131822"]/tr[2]/td/table/tr[8]/td[2]/text()').extract()[0]) |
017 | first_record_on_page = int(cells[0].select('//span[@class="fielddata"]/text()').extract()[0].split('-')[0].strip()) |
018 | last_record_on_page = int(cells[0].select('//span[@class="fielddata"]/text()').extract()[0].split('-')[1].strip()) |
019 | self.log("Scraping details for records {:d} to {:d} of {:d} [{:.2f}% done].".format(first_record_on_page, |
020 | last_record_on_page, total_num_records, float(last_record_on_page)/float(total_num_records)*100), level=log.INFO) |
023 | if last_record_on_page < total_num_records: |
024 | page_links = cells[0].select('//a[@class="fielddata"]/@href').extract() |
025 | if len(page_links) == 1: |
027 | next_page_href = page_links[0] |
029 | next_page_href = page_links[1] |
031 | yield Request(url=response.request.url.split('?')[0]+'?'+next_page_href.split('?')[1], callback=self.parse_main_list) |
038 | record_no = cells[0].select('a/text()').extract()[0].strip() |
039 | record_relative_href = cells[0].select('a/@href').extract()[0] |
040 | record_href = response.request.url.split('?')[0]+'?'+record_relative_href.split('?')[1] |
041 | structure_id = re.sub(ur"^.+:([0-9]+)$", ur'\1', record_href) |
043 | structure_name = "".join(cells[1].select('.//text()').extract()).strip() |
045 | structure_type = cells[2].select('img/@alt').extract()[0] |
046 | structure_type_img_relative_href = cells[2].select('img/@src').extract()[0] |
047 | structure_type_img_href = re.sub(r'/[^/]*
, r'/', response.request.url) + structure_type_img_relative_href
|
049 | territorial_direction = "".join(cells[3].select('b//text()').extract()).strip() |
052 | road = "".join(cells[4].select('.//text()').extract()).strip() |
054 | obstacle = "".join(cells[5].select('.//text()').extract()).strip() |
056 | gci = cells[6].select('nobr/text()').extract()[0].strip() |
059 | ai_code = 'no_restriction' |
060 | if cells[7].select('nobr/img/@alt'): |
061 | ai_desc = cells[7].select('nobr/img/@alt').extract()[0] |
062 | ai_img_relative_href = cells[7].select('nobr/img/@src').extract()[0] |
063 | ai_img_href = re.sub(r'/[^/]*
, r'/', response.request.url) + ai_img_relative_href
|
067 | if cells[7].select('nobr/text()'): |
069 | ai_desc = cells[7].select('nobr/text()').extract()[0] |
075 | if re.search(ur'certaines', ai_desc, re.I): |
076 | ai_code = 'restricted' |
077 | elif re.search(ur'fermée', ai_desc, re.I): |
080 | onclick = cells[8].select('a/@onclick').extract()[0] |
081 | location_href = re.sub(ur"^javascript:pop_url\('(.+)'\);$", ur'\1', onclick) |
083 | planned_intervention = "".join(cells[9].select('.//text()').extract()).strip() |
086 | item = MTQInfraItem() |
087 | item['record_no'] = record_no |
088 | item['record_href'] = record_href |
089 | item['structure_id'] = structure_id |
090 | item['structure_name'] = structure_name |
091 | item['structure_type'] = structure_type |
092 | item['structure_type_img_href'] = structure_type_img_href |
093 | item['territorial_direction'] = territorial_direction |
095 | item['obstacle'] = obstacle |
097 | item['ai_desc'] = ai_desc |
098 | item['ai_img_href'] = ai_img_href |
099 | item['ai_code'] = ai_code |
100 | item['location_href'] = location_href |
101 | item['planned_intervention'] = planned_intervention |
102 | self.items_buffer[structure_id] = item |
104 | yield Request(url=record_href, callback=self.parse_details) |
105 | except Exception as e: |
107 | self.log("Parsing failed for URL '{:s}'".format(response.request.url), level=log.ERROR) |
More details for each lines:
Lines 2,105-108: We wrap our code in a try/except block to log any parsing error with our own message.
Lines 11-13: This is where we skip the header. The logic works because the table header cells are “th” tags, not “td”, so cells is None.
Lines 14-35: This is where we check if we’ve reached the last page or not. If not, we create the “Request” object for the next page.
Lines 21-22: Note the commented “if last_record_on_page < 45:” line. We’ll refer to it in the “Testing It” section below.
Lines 37-84: This is where we extract our data.
Lines 86-101: Here, we create our MTQInfraItem and set the fields we just extracted.
Line 102: Here we save our MTQInfraItem to our internal buffer so we can use it later when we parse the corresponding “details” page.
Line 104: Finally we return a “Request” object so the crawler will request the corresponding “details” page and call our “parse_details” method with the response.
Scraping the Details
I won’t post the code to scrape the “details” page here as it is mostly code similar to lines 37-84 of the previous parser. The only thing to note is that in parse_details(), we actually return the final MTQInfraItem object to the crawler so it can be sent down the pipeline.
Testing It
Before you run this puppy for the first time, you should limit the crawling to a small number of records. I used 45 records because each page has 15. This give us a reasonable sample to validate most of our code. This is where line 22 in parse_main_list() comes handy. Simply uncomment it and comment line 23 to stop processing after 45 records.
If you try to run the crawler as-is on the Transports Quebec website, you’ll probably get errors. At least I did. Apparently, the website does not process concurrent requests using the same session ID. You get an error page when you attempt to do so. By default, Scrapy will attempt to crawl websites more quickly by executing requests concurrently. To disable this completely, add the following lines to settings.py:
CONCURRENT_REQUESTS_PER_DOMAIN = 1 |
REMEMBER: Be polite. Try to minimize the impact of your scraping on the web server. Do your testing on a small number of pages until your are satisfied with your output. Don’t scrape thousand of pages, add a new field and then scrape thousand of pages again. This is particularly true if you test this project. Don’t hammer the Transports Quebec website just for fun, they will simply raise my taxes to buy a bigger server
By default if you simply run “scrapy crawl mtqinfra” Scrapy will print each item on stdout. If you want to save the output in a usable format, you can use the “-o output_file” and “–output-format=format” options. e.g.:
scrapy crawl mtqinfra --output-format=csv -o output.csv |
NOTE: If you attempt to save to XML at this point, you’ll only get a bunch of exceptions because the default XML exporter only handles strings fields and our items have floats. Read on for the solution.
Generating Output Files in Different Formats
OK, you tested the crawler and you are satisfied but you want to save the output in different formats, in a format of your own or in a database. This is where pipelines and exporters come into play.
A pipeline is simply a Python object with a “process_item” method. Once added to our settings.py file, the pipeline object will be instantiated by the crawler and its “process_item” method will be called for each MTQInfraItem. You can then save the item, change it or discard it so other pipelines won’t process it.
Exporters are objects with predefined methods that can be used to persist data in a specific format. Scrapy comes with predefined exporters for CSV, JSON, LineJSON, XML, Pickle (Python) and Pretty Print. You can easily subclass these to modify some of their behavior or subclass the BaseItemExporter class to create your own exporter. On our case, we’ll do both.
Here’s what our exporters.py file looks like:
04 | from scrapy.contrib.exporter import CsvItemExporter |
05 | from scrapy.contrib.exporter import JsonItemExporter |
06 | from scrapy.contrib.exporter import JsonLinesItemExporter |
07 | from scrapy.contrib.exporter import XmlItemExporter |
08 | from scrapy.contrib.exporter import BaseItemExporter |
13 | class MTQInfraXmlItemExporter(XmlItemExporter): |
14 | def serialize_field(self, field, name, value): |
17 | return super(MTQInfraXmlItemExporter, self).serialize_field(field, name, value) |
19 | class MTQInfraJsonItemExporter(JsonItemExporter): |
20 | def __init__(self, file, **kwargs): |
22 | self._configure(kwargs, dont_fail=True) |
24 | self.encoder = json.JSONEncoder(**kwargs) |
25 | self.first_item = True |
27 | class MTQInfraKmlItemExporter(BaseItemExporter): |
28 | def __init__(self, filename, **kwargs): |
29 | self._configure(kwargs, dont_fail=True) |
30 | self.filename = filename |
31 | self.kml = simplekml.Kml() |
34 | def _escape(self, str_value): |
36 | return str_value.replace('&', '&') |
38 | def start_exporting(self): |
41 | def export_item(self, item): |
44 | def finish_exporting(self): |
47 | self.kml.save(self.filename) |
The MTQInfraXmlItemExporter and MTQInfraJsonItemExporter are simply customized versions of their equivalent base Scrapy exporters. The MTQInfraKmlItemExporter is a custom exporter to save output in KML format. It uses the simplekml module. Almost all the work is done in export_item(), which is the method called for each MTQInfraItem created by our parsers. start_exporting/finish_exporting are, as their name imply, called at the start and finish and can be used to setup your exporter or finalize the export process respectively.
Our pipelines.py file contains the following:
04 | from scrapy.xlib.pydispatch import dispatcher |
05 | from scrapy import signals |
06 | from scrapy.exceptions import DropItem |
07 | from scrapy.contrib.exporter import CsvItemExporter |
08 | from scrapy.contrib.exporter import JsonLinesItemExporter |
10 | from exporters import MTQInfraJsonItemExporter |
11 | from exporters import MTQInfraXmlItemExporter |
12 | from exporters import MTQInfraKmlItemExporter |
15 | class MTQInfraPipeline(object): |
17 | self.fields_to_export = [ |
24 | 'structure_type_img_href' |
26 | dispatcher.connect(self.spider_opened, signals.spider_opened) |
27 | dispatcher.connect(self.spider_closed, signals.spider_closed) |
29 | def spider_opened(self, spider): |
30 | self.csv_exporter = CsvItemExporter(open(spider.name+".csv", "w"), |
31 | fields_to_export=self.fields_to_export, quoting=csv.QUOTE_ALL) |
32 | self.json_exporter = MTQInfraJsonItemExporter(open(spider.name+".json", "w"), |
33 | fields_to_export=self.fields_to_export, |
34 | sort_keys=True, indent=4) |
35 | self.jsonlines_exporter = JsonLinesItemExporter(open(spider.name+".linejson", "w"), |
36 | fields_to_export=self.fields_to_export) |
38 | self.xml_exporter = MTQInfraXmlItemExporter(open(spider.name+".xml", "w"), |
39 | fields_to_export=self.fields_to_export, |
40 | root_element="structures", item_element="structure") |
42 | kml_fields = self.fields_to_export[:] |
43 | kml_fields.append('fusion_marker') |
44 | self.kml_exporter = MTQInfraKmlItemExporter(spider.name+".kml", fields_to_export=kml_fields) |
45 | self.csv_exporter.start_exporting() |
46 | self.json_exporter.start_exporting() |
47 | self.jsonlines_exporter.start_exporting() |
48 | self.xml_exporter.start_exporting() |
49 | self.kml_exporter.start_exporting() |
51 | def process_item(self, item, spider): |
52 | self.csv_exporter.export_item(item) |
53 | self.json_exporter.export_item(item) |
54 | self.jsonlines_exporter.export_item(item) |
55 | self.xml_exporter.export_item(item) |
57 | if item['ai_code'] == "no_restriction": |
58 | item['fusion_marker'] = "small_green" |
59 | elif item['ai_code'] == "restricted": |
60 | item['fusion_marker'] = "small_yellow" |
61 | elif item['ai_code'] == "closed": |
62 | item['fusion_marker'] = "small_red" |
64 | item['fusion_marker'] = "small_blue" |
65 | self.kml_exporter.export_item(item) |
68 | def spider_closed(self, spider): |
69 | self.csv_exporter.finish_exporting() |
70 | self.json_exporter.finish_exporting() |
71 | self.jsonlines_exporter.finish_exporting() |
72 | self.xml_exporter.finish_exporting() |
73 | self.kml_exporter.finish_exporting() |
Some notes on the code:
Line 20 and other fields_to_export-related lines: This is used to export fields in a certain order and to exclude the fusion_marker field from all but the KML output.
Lines 29-30: These lines connect Scrapy events to our pipeline. In this case, the spider_opened and spider_closed methods will be called on “start/stop” of the spider, allowing us to setup our exporters and call their start_exporting/finish_exporting methods.
Lines 54-69: This method, as mentioned above, is called for each item created by our parsers. In turns, it calls the export_item method of each exporter.
In order for Scrapy to use our pipeline, we need to add the following lines to settings.py:
'mtqinfra.pipelines.MTQInfraPipeline' |
Running It For Real
When ready to run your scraper on thousand of pages, I suggest you add the following to settings.py:
LOG_FILE = 'mtqinfra.log' |
Or use the –logfile option when running “scrapy crawl”. This will save the Scrapy output to the specified log file. If you still want to see things flowing on your terminal, do a “tail -f” on the log on another terminal, this way you get the best of both worlds.
As mentioned in “Testing It”, be polite. Try to make sure your code generate the proper output with a limited number of pages first. You don’t want to run your scraper for hours (this project does not take hours to crawl but this is an example) and then find out you forgot to include a field and need to reprocess each page.
Also, try to scrape the website during the night, when your traffic has probably less impact.
Finally, if you do run it and then realize your output has errors or needs to be changed, consider “reprocessing” your own results instead of scraping the website again (if possible). For this reason, I strongly suggest you always save your data in LineJSON format as it is super easy to reprocess. See next section for an example.
Reprocessing Results if Needed
If after scraping thousand of pages you realized you had a typo in a generated field (e.g. our KML popup), don’t rescrape the whole website again. Instead, consider reprocessing your own data. Of course, this can only be done if everything you need is already in your previously scraped data. If a field is missing completely and cannot be generated/computed, you’re out of luck.
Here’s an example of how you could reprocess previously saved LineJSON data:
08 | class FakeSpider(object): |
12 | name = "mtqinfra-reprocessed" |
17 | input_file = open("mtqinfra.linejson") |
19 | pipeline = MTQInfraPipeline() |
20 | pipeline.spider_opened(FakeSpider) |
22 | for line in input_file: |
23 | item = MTQInfraItem(json.loads(line)) |
24 | pipeline.process_item(item, FakeSpider) |
26 | pipeline.spider_closed(FakeSpider) |
The Source Code
You can download the complete source code for the scraper on Github.
Final Note
Scrapy is a very powerful scraping framework. It does much more than what I use in this project. Have a look at the documentation to learn more.
I’ll stop here as writing this post actually took more time than coding the project itself. Yes, I’m serious. This either shows you how powerful Python+Scrapy are, or how much I suck at writing blog posts
I hope someone will find this useful. Feel free to share in the comments section.
