1、遍历文档树
使用示例：
html_doc = """ <html><head><title>The Dormouse's story</title></head> <body> <p class="title"><b>The Dormouse's story</b></p> <p class="story">Once upon a time there were three little sisters; and their names were <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>, <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>; and they lived at the bottom of a well.</p> <p class="story">...</p> """ from bs4 import BeautifulSoup soup = BeautifulSoup(html_doc, 'html.parser')
1.1、子节点
一个Tag可能包含多个字符串或其它的Tag,这些都是这个Tag的子节点.Beautiful Soup提供了许多操作和遍历子节点的属性.
注意: Beautiful Soup中字符串节点不支持这些属性,因为字符串没有子节点
1.2、tag的名字
操作文档树最简单的方法就是告诉它你想获取的tag的name.如果想获取 <head> 标签,只要用 soup.head :
soup.head # <head><title>The Dormouse's story</title></head> soup.title # <title>The Dormouse's story</title>
这是个获取tag的小窍门,可以在文档树的tag中多次调用这个方法.下面的代码可以获取<body>标签中的第一个<b>标签:
soup.body.b # <b>The Dormouse's story</b>
通过点取属性的方式只能获得当前名字的第一个tag:
soup.a # <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>
如果想要得到所有的<a>标签,或是通过名字得到比一个tag更多的内容的时候,就需要用到 Searching the tree 中描述的方法,比如: find_all()
soup.find_all('a') # [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>, # <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, # <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
1.3、.contents和.children
tag的 .contents 属性可以将tag的子节点以列表的方式输出:
head_tag = soup.head head_tag # <head><title>The Dormouse's story</title></head> head_tag.contents [<title>The Dormouse's story</title>] title_tag = head_tag.contents[0] title_tag # <title>The Dormouse's story</title> title_tag.contents # [u'The Dormouse's story']
BeautifulSoup 对象本身一定会包含子节点,也就是说<html>标签也是 BeautifulSoup 对象的子节点:
len(soup.contents) # 1 soup.contents[0].name # u'html'
字符串没有 .contents 属性,因为字符串没有子节点:
text = title_tag.contents[0] text.contents # AttributeError: 'NavigableString' object has no attribute 'contents'
通过tag的 .children 生成器,可以对tag的子节点进行循环:
for child in title_tag.children: print(child) # The Dormouse's story
1.4、.descendants
.contents 和 .children 属性仅包含tag的直接子节点.例如,<head>标签只有一个直接子节点<title>
但是<title>标签也包含一个子节点:字符串 “The Dormouse’s story”,这种情况下字符串 “The Dormouse’s story”也属于<head>标签的子孙节点. .descendants 属性可以对所有tag的子孙节点进行递归循环 [5] :
for child in head_tag.descendants: print(child) # <title>The Dormouse's story</title> # The Dormouse's story
1.5、.string
如果tag只有一个 NavigableString 类型子节点,那么这个tag可以使用 .string 得到子节点:
title_tag.string # u'The Dormouse's story'
1.6、.strings和stripped_strings
如果tag中包含多个字符串 [2] ,可以使用 .strings 来循环获取:
for string in soup.strings: print(repr(string)) # u"The Dormouse's story" # u' ' # u"The Dormouse's story" # u' ' # u'Once upon a time there were three little sisters; and their names were ' # u'Elsie' # u', ' # u'Lacie' # u' and ' # u'Tillie' # u'; and they lived at the bottom of a well.' # u' ' # u'...' # u' '
2、父节点
2.1、.parent
通过 .parent 属性来获取某个元素的父节点.在例子“爱丽丝”的文档中,<head>标签是<title>标签的父节点:
title_tag = soup.title title_tag # <title>The Dormouse's story</title> title_tag.parent # <head><title>The Dormouse's story</title></head>
2.2、.parents
通过元素的 .parents 属性可以递归得到元素的所有父辈节点,下面的例子使用了 .parents 方法遍历了<a>标签到根节点的所有节点.
link = soup.a link # <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a> for parent in link.parents: if parent is None: print(parent) else: print(parent.name) # p # body # html # [document] # None
3、兄弟节点
3.1、.next_sibling和.previous_sibling
在文档树中,使用 .next_sibling 和 .previous_sibling 属性来查询兄弟节点:
sibling_soup.b.next_sibling # <c>text2</c> sibling_soup.c.previous_sibling # <b>text1</b>
3.2、.next_siblings和.previous_siblings
通过 .next_siblings 和 .previous_siblings 属性可以对当前节点的兄弟节点迭代输出:
for sibling in soup.a.next_siblings: print(repr(sibling)) # u', ' # <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> # u' and ' # <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a> # u'; and they lived at the bottom of a well.' # None for sibling in soup.find(id="link3").previous_siblings: print(repr(sibling)) # ' and ' # <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> # u', ' # <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a> # u'Once upon a time there were three little sisters; and their names were ' # None