zoukankan      html  css  js  c++  java
  • 用requests库爬取猫眼电影Top100

    这里需要注意一下,在爬取猫眼电影Top100时,网站设置了反爬虫机制,因此需要在requests库的get方法中添加headers,伪装成浏览器进行爬取

    import requests
    from requests.exceptions import RequestException
    from multiprocessing import Pool
    import re
    import json
    
    def get_one_page(url):
        try:
            headers = {"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"}
            response = requests.get(url, headers=headers)
            if response.status_code == 200:
                return response.text
        except RequestException:
            return None
    
    def parse_one_page(html):
        pattern = re.compile('<dd>.*?board-index.*?>(d+)</i>.*?data-src="(.*?)".*?name"><a'
                             +'.*?>(.*?)</a>.*?star">(.*?)</p>.*?releasetime">(.*?)</p>'
                             +'.*?integer">(.*?)</i>.*?fraction">(.*?)</i>.*?</dd>', re.S)
        items = re.findall(pattern, html)
        for item in items:
            yield {
                'index': item[0],
                'image': item[1],
                'title': item[2],
                'actor': item[3].strip()[3:],
                'time': item[4].strip()[5:],
                'score': item[5] + item[6]
            }
    
    def write_to_file(content):
        with open('result.txt', 'a', encoding='utf-8') as f:
            f.write(json.dumps(content, ensure_ascii=False) + '
    ')
            f.close()
    
    def main(offset):
        url = "http://maoyan.com/board/4?offset=" + str(offset)
        html = get_one_page(url)
        for item in parse_one_page(html):
            print(item)
            write_to_file(item)
    
    
    if __name__ == '__main__':
        pool = Pool()
        pool.map(main, [i*10 for i in range(10)])

    运行结果如下:

  • 相关阅读:
    [Leetcode]847. Shortest Path Visiting All Nodes(BFS|DP)
    [Lintcode]Word Squares(DFS|字符串)
    [Lintcode]Inorder Successor in Binary Search Tree(DFS)
    xampp搭建discuz论坛
    Codeforces Round #459 (Div. 2):D. MADMAX(记忆化搜索+博弈论)
    网址备份
    java并发系列
    java创建多线程
    (转)深入理解Java内存模型之系列篇
    (转)Java并发编程:线程池的使用
  • 原文地址:https://www.cnblogs.com/my_captain/p/9508139.html
Copyright © 2011-2022 走看看