zoukankan      html  css  js  c++  java
  • 爬虫发送post请求(requests和scrapy)

    一、scrapy框架的

    cookies = {'zgb_abtest':'xxxx','wsKey':'xxxxxx'}
    headers = {
    'Referer': 'xxxxx',
    'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8',
    'Host': 'xxxxx',
    'Connection': 'Keep-Alive',
    'Accept-Encoding': 'gzip'
    }
    data = {'keyword': response.meta['cat_name'],
    'pageNum': str(i),
    'pageSize': '20',
    'frontCatalogId': response.meta['cat_id'],
    }
    request = scrapy.FormRequest(url=url_list,cookies=self.cookie,headers=self.headers,formdata=data, callback=self.get_goods_lists)

    二、requests的

    data = {'keyword':'生肖茅台酒',
    'pageNum':'2',
    'pageSize':'20',
    'frontCatalogId':'1172214559477501952',
    }
    headers = {
    'cookie':'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
    'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36'
    }
    response = requests.post(url=url,data=data,headers=headers)

    总结:
    scrapy框架的cookie和User-Agent要单独写出来,cookie和headers写在spider里面,User-Agent写在setting里面;
    requests的cookie和User-Agent写在headers里面















  • 相关阅读:
    抽象工厂模式
    工厂方法模式
    简单工厂模式
    Zuul
    Turbine
    Hystrix
    Feign
    Ribbon
    Eureka
    @MappedSuperclass的作用
  • 原文地址:https://www.cnblogs.com/qiaoer1993/p/11327642.html
Copyright © 2011-2022 走看看