zoukankan      html  css  js  c++  java
  • pyspider 启动错误遇到的一些坑

    https://blog.csdn.net/SiHann/article/details/88239892

    突然接到一个项目是关于pyspider,遇到了一些小坑,百度一下发现并没有很好的解决所以研究了一下。
    我所使用的python版本是3.6.8,因为pyspider是对pip有版本要求的,所以自动升级一下。

    升级pip
    •# pip install –upgrade pip

    一切配置好之后,就在CMD中运行 pyspider all 来看能否跑起来。
    这是成功的样子,说明可以了。

    不过路上谁能不遇坑,人家各种成功,到了我这里就是各种错误,我就把这些错误整合一下。

    首先第一个错误:python3.7不兼容pyspider问题(出现占用关键字的问题)
    Python 3.5中引入了async和await,它们在Python 3.7中成为关键字。
    所以需要替换一下关键字。

    第二个问题也就是我遇到的问题:[root@localhost python]# pyspider all
    [W 180629 07:08:26 run:413] phantomjs not found, continue running without it.
    [I 180629 07:08:29 result_worker:49] result_worker starting…
    [I 180629 07:08:31 processor:211] processor starting…
    [I 180629 07:08:31 tornado_fetcher:638] fetcher starting…
    [I 180629 07:08:31 scheduler:675] scheduler starting…
    [I 180629 07:08:31 scheduler:614] in 5m: new:0,success:0,retry:0,failed:0
    [I 180629 07:08:31 scheduler:810] scheduler.xmlrpc listening on 127.0.0.1:23333
    [I 180629 07:08:32 app:84] webui exiting…
    Traceback (most recent call last):
    File “/root/.pyenv/versions/3.6.5/bin/pyspider”, line 11, in
    load_entry_point(‘pyspider==0.3.10’, ‘console_scripts’, ‘pyspider’)()
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pyspider-0.3.10-py3.6.egg/pyspider/run.py”, line 754, in main
    cli()
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py”, line 722, in call
    return self.main(*args, **kwargs)
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py”, line 697, in main
    rv = self.invoke(ctx)
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py”, line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py”, line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py”, line 535, in invoke
    return callback(*args, **kwargs)
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/decorators.py”, line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pyspider-0.3.10-py3.6.egg/pyspider/run.py”, line 497, in all
    ctx.invoke(webui, **webui_config)
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py”, line 535, in invoke
    return callback(*args, **kwargs)
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/decorators.py”, line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pyspider-0.3.10-py3.6.egg/pyspider/run.py”, line 384, in webui
    app.run(host=host, port=port)
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pyspider-0.3.10-py3.6.egg/pyspider/webui/app.py”, line 59, in run
    from .webdav import dav_app
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pyspider-0.3.10-py3.6.egg/pyspider/webui/webdav.py”, line 216, in
    dav_app = WsgiDAVApp(config)
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/WsgiDAV-3.0.0a2-py3.6.egg/wsgidav/wsgidav_app.py”, line 122, in init
    _check_config(config)
    File “/root/.pyenv/versions/3.6.5/lib/python3.6/site-packages/WsgiDAV-3.0.0a2-py3.6.egg/wsgidav/wsgidav_app.py”, line 104, in _check_config
    raise ValueError("Invalid configuration: - " + " - ".join(errors))
    ValueError: Invalid configuration:

    Deprecated option ‘domaincontroller’: use ‘domain_controller’ instead.

    这是我遇到的,然后网上居然没有特别明显的解决,费劲千辛万苦才查到,这是WsgiDAV发布了版本 pre-release 3.x导致的,所以只要把版本降下来就好了。
    将wsgidav替换为2.4.1
    •# python -m pip install wsgidav==2.4.1

    然后运行 pyspider all 发现居然可以了。上一张成功的图片。

    然后很欢快的打开浏览器输入:localhost:5000

  • 相关阅读:
    < java.util >-- Set接口
    Codeforces 627 A. XOR Equation (数学)
    Codeforces 161 B. Discounts (贪心)
    Codeforces 161 D. Distance in Tree (树dp)
    HDU 5534 Partial Tree (完全背包变形)
    HDU 5927 Auxiliary Set (dfs)
    Codeforces 27E. Number With The Given Amount Of Divisors (暴力)
    lght oj 1257
    Codeforces 219D. Choosing Capital for Treeland (树dp)
    Codeforces 479E. Riding in a Lift (dp + 前缀和优化)
  • 原文地址:https://www.cnblogs.com/yuyu666/p/10503986.html
Copyright © 2011-2022 走看看