Scrapy模拟登录知乎

建立项目

scrapy startproject zhihu_login
scrapy genspider zhihu www.zhihu.com

编写spider

• 知乎的登录页url是http://www.zhihu.com/#signin, 为了方便重写sart_requests

# -*- coding: utf-8 -*-
import scrapy

class ZhihuSpider(scrapy.Spider):
    name = "zhihu"
    allowed_domains = ["www.zhihu.com"]

    def start_requests(self):
        # 返回值必须是一个序列
        return [scrapy.Request('http://www.zhihu.com/#signin')]

    def parse(self, response):
        print response

测试能不能正确返回, 返回结果是

[scrapy] DEBUG: Retrying <GET http://www.zhihu.com/robots.txt> (failed 1 times): 500 Internal Server Error

在settings中假如USER_AGENT再进行测试, 返回200, 说明是知乎验证浏览器的问题, 到此可以成功请求到

DEBUG: Crawled (200) <GET http://www.zhihu.com/robots.txt> (referer: None)

• 确定post都需要传入哪些参数, 使用开发者工具得到post值如下(没有出现验证码的情况)

_xsrf    (在html中可以找到)
email
password
remember_me

• 定义login函数, 用于post登录
  以上找大了_xsrf的值

# -*- coding: utf-8 -*-
import scrapy

class ZhihuSpider(scrapy.Spider):
    name = "zhihu"
    allowed_domains = ["www.zhihu.com"]

    def start_requests(self):
        # 返回值必须是一个序列
        return [scrapy.Request('http://www.zhihu.com/#signin', callback=self.login)]

    def login(self, response):
        print '-------'     # 便于测试
        _xsrf = response.xpath(".//*[@id='sign-form-1']/input[2]/@value").extract()[0]
        print _xsrf

使用FormRequest登录

def login(self, response):
        print '-------'     # 便于测试
        _xsrf = response.xpath(".//*[@id='sign-form-1']/input[2]/@value").extract()[0]
        print _xsrf
        return [scrapy.FormRequest(
            url = 'http://www.zhihu.com/login/email',    # 这是post的真实地址
            formdata={
                '_xsrf': _xsrf,
                'email': 'xxxxxxxx',    # email
                'password': 'xxxxxxxx',    # password
                'remember_me': 'true',
            },
            headers=self.headers,
            callback=self.check_login,
        )]

• 检测是否登录成功, 知乎的response会返回一个json, 如果里面r为0的话说明成功登录

def check_login(self, response):
        if json.loads(response.body)['r'] == 0:
            yield scrapy.Request(
                                'http://www.zhihu.com', 
                                headers=self.headers, 
                                callback=self.page_content,
                                dont_filter=True,    # 因为是第二次请求, 设置为True, 默认是False, 否则报错
                                )

spider的完整代码

# -*- coding: utf-8 -*-
import scrapy
import json

class ZhihuSpider(scrapy.Spider):
    name = "zhihu"
    allowed_domains = ["www.zhihu.com"]
    headers = {
            'Host': 'www.zhihu.com',
            'Referer': 'http://www.zhihu.com',
            'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36',
        }

    def start_requests(self):
        # 返回值必须是一个序列
        return [scrapy.Request('http://www.zhihu.com/#signin', callback=self.login)]

    def login(self, response):
        print '-------'     # 便于测试
        _xsrf = response.xpath(".//*[@id='sign-form-1']/input[2]/@value").extract()[0]
        print _xsrf
        return [scrapy.FormRequest(
            url = 'http://www.zhihu.com/login/email',    # 这是post的真实地址
            formdata={
                '_xsrf': _xsrf,
                'email': 'xxxxxxxx',    # email
                'password': 'xxxxxxxx',    # password
                'remember_me': 'true',
            },
            headers=self.headers,
            callback=self.check_login,
        )]

    def check_login(self, response):
        if json.loads(response.body)['r'] == 0:
            yield scrapy.Request(
                                'http://www.zhihu.com', 
                                headers=self.headers, 
                                callback=self.page_content,
                                dont_filter=True,    
                                )

    def page_content(self, response):
        with open('first_page.html', 'wb') as f:
            f.write(response.body)
        print 'done'

注: 也是刚学scrapy, 暂时不知道怎么处理验证码的情况, 还望大牛指教