一,下载安装
http://www.crummy.com/software/BeautifulSoup
(1)python setup.py build
(2)python setup.py install
(3)小例子
import bs4 from bs4 import BeautifulSoup html_doc = """ <html><head><title>The Dormouse's story</title></head> <body> <p class="title"><b>The Dormouse's story</b></p> <p class="story">Once upon a time there were three little sisters; and their names were <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>, <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>; and they lived at the bottom of a well.</p> <p class="story">...</p> """ soup = BeautifulSoup(html_doc) print(soup.prettify())
二,使用例子
import bs4,os,types,urllib,re from bs4 import BeautifulSoup ###This function is used for crawler the url def parserCategary(urlString): html_doc = urllib.urlopen(urlString).read() soup = BeautifulSoup(html_doc) lis = soup.find_all('li', {'class':'category-item'}) result = {} r = re.compile('/category/([a-zA-Z_]+)\?') for item in lis: result[r.findall(item.contents[0]['href'])[0]] = urlString+item.contents[0]['href'] return result if __name__ == '__main__': urlString = 'https://play.google.com/store' result = parserCategary(urlString) for item in result.items(): print item
一个小问题:
当我去解析一段话的时候,有个这样的节点树:
[u'This can place a load on the CPU. You may feel slow indeed in the Android OS.', <br>We also exhibit the versions of other OS.<br>Downloads : <a href="https://www.google.com/url?q=http://wizapply.com/mp2mark/&sa=D&usg=AFQjCNFp2xblQHypa_Z4hvs_VRlXdqgUKw" target="_blank">http://wizapply.com/mp2mark/</a><p>+ Mobile GPU demonstration [GP2Mark]<br>Search "GP2Mark" in the Google Play!<p>We sell the "manual" and "MP2Mark C source code". If you are interested in is not please contact us.<p>If there is a request, we will receive improvement, etc.<p>tag bench,demo,tegra,multi,core,intel,amd,arm,snapdragon,samsung,exynos,cortex,ndk</p></p></p></br></p></br></br>]
这个列表其实只有两项,一个是开头的unicode字符串,第二个就是一个树。但是因为树中间有逗号,导致本来只有两项的列表,错误的有了很多项。。。
郁闷。
利用DFS去爬网站,必须明确终止条件是什么?每一步处理的问题是什么?
今天花了不少时间去写DFS,遇到不少麻烦:
(1)没有归纳好终止条件是什么
(2)没有明确每一步的任务
(3)虽然是深度,但是在一个函数中,要一个接一个的广度遍历所有的直接孩子。
num-pagination-control
num-pagination-content
input : id="reviewUseAjaxUrl"