zoukankan      html  css  js  c++  java
  • bs4案例

    #!/usr/bin/python
    import requests 
    from bs4 import BeautifulSoup
    #需求:爬取三国演义小说所有的章节标题和章节内容http://www.shicimingju.com/book/sanguoyanyi.html
    if __name__ == "__main__":
        #对首页的页面数据进行爬取
        headers = {
            'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2626.106 Safari/537.36'
        }
        url = 'http://www.shicimingju.com/book/sanguoyanyi.html'
        page_text = requests.get(url=url,headers=headers).text
    
        #在首页中解析出章节的标题和详情页的url
        #1.实例化BeautifulSoup对象,需要将页面源码数据加载到该对象中
        soup = BeautifulSoup(page_text,'lxml')
        #解析章节标题和详情页的url
        li_list = soup.select('.book-mulu > ul > li')
        fp = open('./sanguo.txt','w',encoding='utf-8')
        for li in li_list:
            title = li.a.string
            detail_url = 'http://www.shicimingju.com'+li.a['href']
            #对详情页发起请求,解析出章节内容
            detail_page_text = requests.get(url=detail_url,headers=headers).text
            #解析出详情页中相关的章节内容
            detail_soup = BeautifulSoup(detail_page_text,'lxml')
            div_tag = detail_soup.find('div',class_='chapter_content')
            #解析到了章节的内容
            content = div_tag.text
            fp.write(title+':'+content+'
    ')
            print(title,'爬取成功')
    
    
    
    
  • 相关阅读:
    5.8 Properties
    5.7(java学习笔记)Vector、Enumeration
    5.6(java学习笔记) queue
    5.5(java学习笔记)TreeSet和TreeMap
    5.4 (Java学习笔记)集合的排序(Collections.sort(),及Arrays.sort())
    UBUNTU18.04安装使用ORB-SLAM2
    UBUNTU18.04安装CUDA
    UBUNTU18.04安装Pangolin
    ubuntu18.08安装eigen
    Ubuntu18.4安装g2o
  • 原文地址:https://www.cnblogs.com/gerenboke/p/13389013.html
Copyright © 2011-2022 走看看