1. OS: debian 9
2. cor@debian:~$ sudo pip3 install Scrapy
cor@debian:~$ sudo pip3 install Scrapy
3. open a terminal , and go to the directory you'd like to start you new project
"scrapy startproject <project name>"
scrapy startproject myscpy
4. "scrapy genspider "
$ scrapy genspider film http://www.rrys2019.com/ --template=crawl :0: UserWarning: You do not have a working installation of the service_identity module: 'cannot import name 'opentype''. Please install it from <https://pypi.python.org/pypi/service_identity>
and make sure all of its dependencies are satisfied. Without the service_identity module, Twisted can perform only rudimentary TLS client hostname verification.
Many valid certificate/hostname mappings may be rejected. Created spider 'film' using template 'crawl' in module: myscpy.spiders.film
"sudo pip3 install service_identity"
sudo pip3 install service_identity
[sudo] password for cor:
Requirement already satisfied: service_identity in /usr/local/lib/python3.5/dist-packages
Requirement already satisfied: attrs>=16.0.0 in /usr/local/lib/python3.5/dist-packages (from service_identity)
Requirement already satisfied: pyasn1-modules in /usr/local/lib/python3.5/dist-packages (from service_identity)
Requirement already satisfied: pyasn1 in /usr/lib/python3/dist-packages (from service_identity)
Requirement already satisfied: cryptography in /usr/local/lib/python3.5/dist-packages (from service_identity)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/local/lib/python3.5/dist-packages (from cryptography->service_identity)
Requirement already satisfied: six>=1.4.1 in /usr/lib/python3/dist-packages (from cryptography->service_identity)
Requirement already satisfied: pycparser in /usr/local/lib/python3.5/dist-packages (from cffi!=1.11.3,>=1.8->cryptography->service_identity)
5.
"sudo pip3 install Twisted==18.9.0", ---> doesn't work
sudo pip3 install Twisted==18.9.0
Collecting Twisted==18.9.0
Downloading https://files.pythonhosted.org/packages/5d/0e/a72d85a55761c2c3ff1cb968143a2fd5f360220779ed90e0fadf4106d4f2/Twisted-18.9.0.tar.bz2 (3.1MB)
100% |████████████████████████████████| 3.1MB 238kB/s
Requirement already satisfied: Automat>=0.3.0 in /usr/local/lib/python3.5/dist-packages (from Twisted==18.9.0)
Requirement already satisfied: PyHamcrest>=1.9.0 in /usr/local/lib/python3.5/dist-packages (from Twisted==18.9.0)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.5/dist-packages (from Twisted==18.9.0)
Requirement already satisfied: constantly>=15.1 in /usr/local/lib/python3.5/dist-packages (from Twisted==18.9.0)
Requirement already satisfied: hyperlink>=17.1.1 in /usr/local/lib/python3.5/dist-packages (from Twisted==18.9.0)
Requirement already satisfied: incremental>=16.10.1 in /usr/local/lib/python3.5/dist-packages (from Twisted==18.9.0)
Requirement already satisfied: zope.interface>=4.4.2 in /usr/local/lib/python3.5/dist-packages (from Twisted==18.9.0)
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from Automat>=0.3.0->Twisted==18.9.0)
Requirement already satisfied: idna>=2.5 in /usr/local/lib/python3.5/dist-packages (from hyperlink>=17.1.1->Twisted==18.9.0)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from zope.interface>=4.4.2->Twisted==18.9.0)
Building wheels for collected packages: Twisted
Running setup.py bdist_wheel for Twisted ... done
Stored in directory: /root/.cache/pip/wheels/57/2e/89/11ba83bc08ac30a5e3a6005f0310c78d231b96a270def88ca0
Successfully built Twisted
Installing collected packages: Twisted
Found existing installation: Twisted 20.3.0
Uninstalling Twisted-20.3.0:
Successfully uninstalled Twisted-20.3.0
Successfully installed Twisted-18.9.0
error messages still:
scrapy crawl film :0: UserWarning: You do not have a working installation of the service_identity module: 'cannot import name 'opentype''.
Please install it from <https://pypi.python.org/pypi/service_identity> and make sure all of its dependencies are satisfied.
Without the service_identity module, Twisted can perform only rudimentary TLS client hostname verification. Many valid certificate/hostname mappings may be rejected. 2020-03-25 14:40:25 [scrapy.utils.log] INFO: Scrapy 2.0.1 started (bot: myscpy) 2020-03-25 14:40:25 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 18.9.0,
Python 3.5.3 (default, Sep 27 2018, 17:25:39) - [GCC 6.3.0 20170516], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.8, Platform Linux-4.9.0-12-amd64-x86_64-with-debian-9.12 2020-03-25 14:40:25 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor 2020-03-25 14:40:25 [scrapy.crawler] INFO: Overridden settings:
"pip3 install service_identity --force --upgrade" works for me(debian)
sudo pip3 install service_identity --force --upgrade
Collecting service_identity
Using cached https://files.pythonhosted.org/packages/e9/7c/2195b890023e098f9618d43ebc337d83c8b38d414326685339eb024db2f6/service_identity-18.1.0-py2.py3-none-any.whl
Collecting cryptography (from service_identity)
Using cached https://files.pythonhosted.org/packages/45/73/d18a8884de8bffdcda475728008b5b13be7fbef40a2acc81a0d5d524175d/cryptography-2.8-cp34-abi3-manylinux1_x86_64.whl
Collecting pyasn1 (from service_identity)
Downloading https://files.pythonhosted.org/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl (77kB)
100% |████████████████████████████████| 81kB 375kB/s
Collecting attrs>=16.0.0 (from service_identity)
Using cached https://files.pythonhosted.org/packages/a2/db/4313ab3be961f7a763066401fb77f7748373b6094076ae2bda2806988af6/attrs-19.3.0-py2.py3-none-any.whl
Collecting pyasn1-modules (from service_identity)
Using cached https://files.pythonhosted.org/packages/95/de/214830a981892a3e286c3794f41ae67a4495df1108c3da8a9f62159b9a9d/pyasn1_modules-0.2.8-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography->service_identity)
Using cached https://files.pythonhosted.org/packages/d6/7f/7acc85c478f5056b98c9961a31697b1e53fbec158ee5f723097e1c355660/cffi-1.14.0-cp35-cp35m-manylinux1_x86_64.whl
Collecting six>=1.4.1 (from cryptography->service_identity)
Downloading https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography->service_identity)
Using cached https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl
Installing collected packages: pycparser, cffi, six, cryptography, pyasn1, attrs, pyasn1-modules, service-identity
Found existing installation: pycparser 2.20
Uninstalling pycparser-2.20:
Successfully uninstalled pycparser-2.20
Found existing installation: cffi 1.14.0
Uninstalling cffi-1.14.0:
Successfully uninstalled cffi-1.14.0
Found existing installation: six 1.10.0
Not uninstalling six at /usr/lib/python3/dist-packages, outside environment /usr
Found existing installation: cryptography 2.8
Uninstalling cryptography-2.8:
Successfully uninstalled cryptography-2.8
Found existing installation: pyasn1 0.1.9
Not uninstalling pyasn1 at /usr/lib/python3/dist-packages, outside environment /usr
Found existing installation: attrs 19.3.0
Uninstalling attrs-19.3.0:
Successfully uninstalled attrs-19.3.0
Found existing installation: pyasn1-modules 0.2.8
Uninstalling pyasn1-modules-0.2.8:
Successfully uninstalled pyasn1-modules-0.2.8
Found existing installation: service-identity 18.1.0
Uninstalling service-identity-18.1.0:
Successfully uninstalled service-identity-18.1.0
Successfully installed attrs-19.3.0 cffi-1.14.0 cryptography-2.8 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 service-identity-18.1.0 six-1.14.0