python网络数据抓取三（斗图网图片抓取）

　　继续数据抓取练习，这次选取了一个小网站，目标网址：http://www.bbsnet.com/。目标网站很简陋，网页请求分析也很简单直接就能看到。所以我直接把整个网站的所有图片都爬取了下来（它那个网站上图片应该也是爬取其他网站的）。由于数据比较多所以我使用了多进程抓取。我按照每个链接建立了对应目录，而且有一个简单的进度显示，方便查看使用。我本机测试一共抓取到200多个文件夹合计2700多张图片（网站一直有更新所以结果不一定一样），有些文件夹内是空的，可能是格式没按套路来没抓到，我没太纠结毕竟只是练习而已..下面展示下抓取示例结果和源码：

　

代码：

# -*- coding: UTF-8 -*-

from multiprocessing import Process, Queue
from bs4 import BeautifulSoup
import urllib.request
import os

#存储路径
PATH = 'D:\斗图\'

#爬取目标网站url
CRAWL_TARGET_URL = 'http://www.bbsnet.com/'

#伪装头，有可能不需要
HEADERS = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 
(KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'}

#伪装头1，有可能不需要
HEADERS1 = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 
(KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36', 'Referer': 'http://www.bbsnet.com/'}

#进程数
NUMBER_OF_PROCESSES = 4

def download_image(url, filename):
    try:
        res = urllib.request.urlopen(urllib.request.Request(url=url, headers= HEADERS1), timeout=10)
        with open(filename, "wb") as f:
            f.write(res.read())
    except Exception as e:
        print(e)


def crawl_images(url, path, title):
    try:
        res = urllib.request.urlopen(urllib.request.Request(url=url, headers= HEADERS), timeout=10)
    except Exception as e:
        print(e)
        return

    bf = BeautifulSoup(res.read(), 'html.parser')
    links = bf.find_all('img', title = title)
    i = 0;
    for link in links:
        src = link.get('src')
        download_image(src, path+"\"+str(i)+src[src.rfind('.'):])
        i = i+1


def crawl_urls(sUrls, q):
    for url in sUrls:
        try:
            req =  urllib.request.urlopen(urllib.request.Request(url=url, headers= HEADERS), timeout=10)
        except Exception as e:
            print(e)
            continue
        bf = BeautifulSoup(req.read(), 'html.parser')
        links = bf.find_all('a', class_ = 'zoom')
        for link in links:
            #根据每一个页面标题创建一个文件夹
            title = link.get('title').split()[0]
            path = PATH + title
            sequence = 1
            while os.path.exists(path):
                path = path + str(sequence)
                sequence = sequence + 1
            os.makedirs(path)
            crawl_images(link.get('href'), path, title)
        #完成抓取一个翻页内所有数据向主进程汇报一次进度
        q.put(1)


def start_crawl():
    try:
        req = urllib.request.urlopen(urllib.request.Request(url=CRAWL_TARGET_URL, headers= HEADERS))
    except Exception as e:
        raise e

    #获取翻页url信息
    bs = BeautifulSoup(req.read(), 'html.parser')
    lastPageUrl = bs.find('a', class_ = 'extend')['href']
    pos = lastPageUrl.rfind('/') + 1
    oriPageUrl = lastPageUrl[:pos]
    maxPageNum = int(lastPageUrl[pos:])

    #分配待抓取url到各自进程
    mUrls = {}
    for i in range(NUMBER_OF_PROCESSES):
        mUrls[i] = []
    for n in range(maxPageNum):
        mUrls[n%NUMBER_OF_PROCESSES].append(oriPageUrl+str(n+1))

    #创建进程,使用一个队列来统计完成率
    q = Queue()
    for i in range(NUMBER_OF_PROCESSES):
        p = Process(target=crawl_urls, args=(mUrls[i], q))
        p.start()

    #完成率统计
    schedule = 0
    while 1:
        q.get()
        schedule = schedule+1
        print("已完成：", int(schedule*100/maxPageNum), '%')


if __name__ == '__main__':
    start_crawl()