zoukankan      html  css  js  c++  java
  • ubuntu18.04 搭建scrapy环境(连环踩坑+解决办法)

    ---恢复内容开始---

    预期需求:

    打算搭建scrapy环境,基于python3.x的

    环境描述:

    ubuntu18.04自带了python3.6,打算在虚拟环境vlenv中跑scrapy,装好虚拟环境后,直接pip3 install scrapy命令走起,然后,跑错了

    @ubuntu:~$ pip3 install scrapy
    Collecting scrapy
    Downloading https://files.pythonhosted.org/packages/29/4b/585e8e111ffb01466c59281f34febb13ad1a95d7fb3919fd57c33fc732a5/Scrapy-1.7.3-py2.py3-none-any.whl (234kB)
    100% |████████████████████████████████| 235kB 23kB/s
    Collecting pyOpenSSL (from scrapy)
    Downloading https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl (53kB)
    100% |████████████████████████████████| 61kB 18kB/s
    Collecting Twisted>=13.1.0; python_version != "3.4" (from scrapy)

    。。。

    Downloading https://files.pythonhosted.org/packages/be/70/e5ea8afd6d08a4b99ebfc77bd1845248d56cfcf43d11f9dc324b9580a35c/pyasn1_modules-0.2.6-py2.py3-none-any.whl (95kB)
    100% |████████████████████████████████| 102kB 29kB/s
    Collecting cffi!=1.11.3,>=1.8 (from cryptography>=2.3->pyOpenSSL->scrapy)
    Could not find a version that satisfies the requirement cffi!=1.11.3,>=1.8 (from cryptography>=2.3->pyOpenSSL->scrapy) (from versions: )
    No matching distribution found for cffi!=1.11.3,>=1.8 (from cryptography>=2.3->pyOpenSSL->scrapy)

    安装失败!似乎python版本不对?

    @ubuntu:~$ which python
    /usr/bin/python
    @ubuntu:~$ which python3
    /usr/bin/python3
    @ubuntu:~$ which python3.6
    /usr/bin/python3.6

    上面是环境中的python版本,去scrapy官网查了一下,好像也有提到支持版本2.7和3.4的,

    然后开始百度安装python3.4的办法。按照下面的文章的方法安装成功了。感谢这个作者!

    下载源码直接编译确实靠谱!哈哈

    https://blog.csdn.net/zhongxing9006/article/details/80605126

    下面是装完的python3.4的路径

    @ubuntu:~/Python-3.4.10$ which python3.4
    /usr/local/bin/python3.4

    安装完又用update-alternatives来控制python的版本,可以自定义python默认使用版本。这个方法也是上面链接里提供的,确实好用!

    @ubuntu:~/Python-3.4.10$ sudo update-alternatives --config python
    有 3 个候选项可用于替换 python (提供 /usr/bin/python)。

    选择 路径 优先级 状态
    ------------------------------------------------------------
    * 0 /usr/local/bin/python3.4 3 自动模式
    1 /usr/bin/python2.7 1 手动模式
    2 /usr/bin/python3.6 2 手动模式
    3 /usr/local/bin/python3.4 3 手动模式

    要维持当前值[*]请按<回车键>,或者键入选择的编号:
    leiting@ubuntu:~/Python-3.4.10$
    leiting@ubuntu:~/Python-3.4.10$
    leiting@ubuntu:~/Python-3.4.10$ python
    Python 3.4.10 (default, Aug 20 2019, 21:31:05)
    [GCC 7.4.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>>
    @ubuntu:~/Python-3.4.10$ python3
    Python 3.6.8 (default, Jan 14 2019, 11:02:34)
    [GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>>
    @ubuntu:~/Python-3.4.10$ python2
    Python 2.7.15+ (default, Nov 27 2018, 23:36:35)
    [GCC 7.3.0] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>>
    @ubuntu:~/Python-3.4.10$ ipython3
    Python 3.6.8 (default, Jan 14 2019, 11:02:34)
    Type "copyright", "credits" or "license" for more information.

    IPython 5.5.0 -- An enhanced Interactive Python.
    ? -> Introduction and overview of IPython's features.
    %quickref -> Quick reference.
    help -> Python's own help system.
    object? -> Details about 'object', use 'object??' for extra details.

    In [1]:
    Do you really want to exit ([y]/n)? y
    @ubuntu:~/Python-3.4.10$

    创建虚拟环境,并指定python3.4版本,pip安装scrapy,报错...提示pip版本低,虚拟环境升级pip没成功,又把系统环境的pip升级了。从9.0升级到19.1,虚拟环境还是pip安装不了scrapy,有点崩溃了

    @ubuntu:~/Python-3.4.10$ pip3 list|grep pip
    DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
    pip (9.0.1)

    @ubuntu:~/Python-3.4.10$ pip3 install pip -U
    Collecting pip
    Cache entry deserialization failed, entry ignored
    Downloading https://files.pythonhosted.org/packages/8d/07/f7d7ced2f97ca3098c16565efbe6b15fafcba53e8d9bdb431e09140514b0/pip-19.2.2-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 48kB/s
    Installing collected packages: pip
    Successfully installed pip-19.2.2
    leiting@ubuntu:~/Python-3.4.10$

    升级完pip,好像pip3就命令就没了,全是pip,原因不明

     

    升级完pip,创建新虚拟环境,还是pip安装不起scrapy,烦~后面想,还是不在虚拟环境安装scrapy,不然每个环境都要按照一次scrapy,还是麻烦,打算还是在默认环境安装scrapy,每个虚拟环境创建时,都带上默认环境的包,就可以了,命令用下面这个--system-site-packages

    https://virtualenv.pypa.io/en/latest/reference/#virtualenv-command

    继续默认环境装scrapy,还是pip install scrapy,走起,然后报错了

    @ubuntu:~/Python-3.4.10$ pip install scrapy
    Collecting scrapy

    。。。

    ERROR: pyopenssl 19.0.0 has requirement cryptography>=2.3, but you'll have cryptography 2.1.4 which is incompatible.
    Installing collected packages: cssselect, pyasn1, attrs, pyasn1-modules, service-identity, queuelib, pyOpenSSL, incremental, Automat, zope.interface, constantly, PyHamcrest, hyperlink, Twisted, PyDispatcher, w3lib, parsel, scrapy
    ERROR: Could not install packages due to an EnvironmentError: [Errno 13] 权限不够: '/usr/local/lib/python3.6/dist-packages/cssselect'
    Consider using the `--user` option or check the permissions.

    真的快崩溃了!!!

    百度找了一圈办法,用下面的命令,问题解决!

    pip install --user scrapy

    加上--user参数

    @ubuntu:~/Python-3.4.10$ scrapy
    Scrapy 1.7.3 - no active project

    Usage:
    scrapy <command> [options] [args]

    Available commands:
    bench Run quick benchmark test
    fetch Fetch a URL using the Scrapy downloader
    genspider Generate new spider using pre-defined templates
    runspider Run a self-contained spider (without creating a project)
    settings Get settings values
    shell Interactive scraping console
    startproject Create new project
    version Print Scrapy version
    view Open URL in browser, as seen by Scrapy

    [ more ] More commands available when run from project directory

    Use "scrapy <command> -h" to see more info about a command

    真不容易~~~~

    总结:

    1、用源码安装python各个版本,用update-alternatives来控制python的版本,可以自定义python默认使用版本

    2、pip3 install pip -U,这个-U,意思把pip包升级到最新版本

    3、pip install报错提示Consider using the `--user` option or check the permissions.,就加一个--user参数,有奇效!

  • 相关阅读:
    如何:为 Silverlight 客户端生成双工服务
    Microsoft Sync Framework 2.1 软件开发包 (SDK)
    Windows 下的安装phpMoAdmin
    asp.net安全检测工具 Padding Oracle 检测
    HTTP Basic Authentication for RESTFul Service
    Windows系统性能分析
    Windows Server AppFabric Management Pack for Operations Manager 2007
    Mongo Database 性能优化
    服务器未能识别 HTTP 标头 SOAPAction 的值
    TCP WAIT状态及其对繁忙的服务器的影响
  • 原文地址:https://www.cnblogs.com/noobcoder17/p/11387053.html
Copyright © 2011-2022 走看看