zoukankan      html  css  js  c++  java
  • 爬取全部的校园新闻


    1.从新闻url获取新闻详情: 字典,anews

    import requests
    from bs4 import BeautifulSoup
    from datetime import datetime
    import re
    def click(xbk):
        id=re.findall('(d{1,5})',xbk)[-1]
        clickUrl='http://oa.gzcc.cn/api.php?op=count&id={}&modelid=80'.format(id)
        resClick=requests.get(clickUrl)
        newsClick=int(resClick.text.split('.html')[-1].lstrip("('").rstrip("');"))
        return newsClick
    def newsdt(showinfo):
        newsDate=showinfo.split()[0].split(":")[1]
        newsTime=showinfo.split()[1]
        newsDT=newsDate+' '+newsTime
        dt=datetime.strptime(newsDT,'%Y-%m-%d %H:%M:%S')
        return dt
    def anewsDate(xbk):
        newsDetail={}
        res=requests.get(xbk)
        res.encoding='utf-8'
        soup=BeautifulSoup(res.text,'html.parser')
        newsDetail['nenewsTitle']=soup.select('.show-title')[0].text
        showinfo=soup.select('.show-info')[0].text
        newsDetail['newsDT']=newsdt(showinfo)
        newsDetail['newsClick']=click(newsUrl)
        return newsDetail
    newsZD='http://news.gzcc.cn/html/2005/xiaoyuanxinwen_0710/4.html'
    anewsDate(newsXBK)
    res = requests.get('http://news.gzcc.cn/html/xiaoyuanxinwen/')
    res.encoding='utf-8'
    soup = BeautifulSoup(res.text,'html.parser')
    for news in soup.select('li'):
        if len(news.select('.news-list-title'))>0:
            newsUrl=news.select('a')[0]['href']
            print(anews(newsUrl))
    运行结果如下:


    3.生成所页列表页的url并获取全部新闻 :列表extend(列表) allnews

    ZD='http://news.gzcc.cn/html/xiaoyuanxinwen/'
    res=requests.get(ZD)
    res.encoding='utf-8'
    soup = BeautifulSoup(res.text,'html.parser')
    newsList=[]
    for news in soup.select('li'):
        if len(news.select('.news-list-title'))>0:
            newsUrl=news.select('a')[0]['href']
            newsDesc=news.select('.news-list-description')[0].text
            newsDict=anews(newsUrl)
            newsDict['description']=newsDesc
            newsList.append(newsDict)
    newsList
    
    

    4.设置合理的爬取间隔

    import time

    import random

    time.sleep(random.random()*3)

    5.用pandas做简单的数据处理并保存

    保存到csv或excel文件 

    newsdf.to_csv(r'F:duym爬虫gzccnews.csv')

    保存到数据库

    import sqlite3
    with sqlite3.connect('gzccnewsdb.sqlite') as db:
        newsdf.to_sql('gzccnewsdb',db)

  • 相关阅读:
    课后作业03
    课堂测验02
    构建之法阅读笔记02
    进度条 二
    软件工程个人作业02
    Sprint6
    Sprint5
    Alpha版总结会议
    Beta版总结会议
    Alpha阶段项目Postmortem会议总结
  • 原文地址:https://www.cnblogs.com/068zhengda/p/10698705.html
Copyright © 2011-2022 走看看