zoukankan      html  css  js  c++  java
  • [Python爬虫]猫眼电影榜单爬取

    import requests
    import re
    from requests.exceptions import RequestException
    import json
    from multiprocessing import Pool
    def get_one_page(url):
        headers = {
        'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36'
    }
        try:
            response=requests.get(url,headers=headers)
            if response.status_code==200:
                return response.text
            return None
        except RequestException:
            return None
    def parse_one_page(html):
        pattern=re.compile('<dd>.*?board-index.*?>(d*)</i>.*?data-src="(.*?)".*?name"><a'
                           +'.*?>(.*?)</a>.*?star">(.*?)</p>.*?releasetime">(.*?)</p>'
                           +'.*?integer">(.*?)</i>.*?fraction">(.*?)</i>.*?</dd>',re.S)
        items=re.findall(pattern,html)
        for item in items:
            yield {
                'index':item[0],
                'image':item[1],
                'title':item[2],
                'actor':item[3].strip()[3:],
                'time':item[4].strip()[5:],
                'score':item[5]+item[6]
            }
    
    def write_to_file(content):
        with open('result.txt','a',encoding='utf-8') as f:
            f.write(json.dumps(content,ensure_ascii=False)+'
    ')
            f.close()
    def main(offset):
        url='https://maoyan.com/board/4?offset='+str(offset)
        html=get_one_page(url)
        for item in parse_one_page(html):
            print(item)
            write_to_file(item)
    
    if __name__=='__main__':
        pool=Pool()
        pool.map(main,[i*10 for i in range(10)])
  • 相关阅读:
    java 菜单
    QT 让信号自由飞翔(骚操作)
    QT editLine 无法输入的问题
    易经初学体会
    Cgroup
    springboot pom 引用集合
    使用ab测试工具 进行并发测试
    intellij 设置-试验过的
    【iis错误码】IIS 服务 这些年遇到的错误码
    101个创业失败案例背后的20大原因
  • 原文地址:https://www.cnblogs.com/lightmonster/p/11529647.html
Copyright © 2011-2022 走看看