PyKHTML, a Python interface to KHTML

PyKHTML, a Python interface to KHTML

PyKHTML is...

A Python module for writing website scrapers/spiders. Whereas traditional methods focus on writing the code to parse HTML/forms themselves, PyKHTML uses the excellent KHTML engine to do all the trudge work. It therefore handles webpages very well (even the severely crufty ones) and is pretty darn fast (implemented in C++). As a bonus the module handles JavaScript and cookies transparently. Hurrah!

How?

PyKHTML requires PyKDE 3 (and hence in turn PyQt 3 + KDE libs). If you would like to run PyKHTML on servers without an X display then Xvfb is required. Fortunately these requirements should come bundled with most modern Linux distributions, and support for Windows/Mac should appear in the next few months.

Show me some code

Okay. Here is an example (one of many examples included in the bundle) that scrapes the title and navigation from this page, with excessive commenting to give you a feel of what programming with PyKHTML is like:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

import pykhtml PyKHTMLUrl = "http://paul.giannaros.org/pykhtml" def extractBitsFromPage(browser): # getElementsByTagName returns a generator, so we convert # to a list and access the first element title = list(browser.document.getElementsByTagName("title"))[0] print "Title:", title.text # Get the text of the navigation items navigation = [] # First get the container of the list items... navigationElement = browser.document.getElementById("navigation") # ... and then loop over the li elements we find for listItem in navigationElement.getElementsByTagName("li"): # Inside the list item is an anchor anchor = listItem.children[0] # And the text inside the anchor is what we want navigation.append(anchor.text) print "Navigation:", " | ".join(navigation) # Stop here, we're done pykhtml.stopEventLoop() def main(): browser = pykhtml.Browser() # the browser is passed as a parameter to extractBitsFromPage # when it is called (when the page has loaded) browser.load(PyKHTMLUrl, extractBitsFromPage) # kick things off pykhtml.startEventLoop() main()

Note of Thanks

Gambit Research, a software company in West London, sponsor PyKHTML development.