Splash→Scrapy:
首先,lua脚本中要返回Cookies:
return {cookies = splash:get_cookies()}
然后,在spider中调用response.cookiejar
即可得到返回的Cookies
Scrapy→Splash
把Cookies作为参数传递到SplashRequest()中:
def start_requests(self):
local = "D:\...\VertifiedCookies.txt"
with open(local, 'r+') as f:
cookies = eval(f.read())
url = 'https://www.example.com'
script = """
function main(splash)
splash:clear_cookies()
splash:init_cookies(splash.args.cookies)
assert(splash:go(splash.args.url))
assert(splash:wait(splash.args.wait))
return {cookies = splash:get_cookies(),html = splash:html()}
end
"""
yield SplashRequest(url, self.parse, endpoint='execute', args={'lua_source':script,'wait':1}, cookies=cookies)
传入数据的参数类型与scrapy.Request中的cookies相同: