OSX 上安装 Scrapy 的那些坑

Scrapy 这个爬网框架真心不错，但在OSX上安装总是会出现各种的问题，在这里就作一个收集汇总。我的系统环境是 OS X El Capitan (10.11.1)

首先要保证 pip , virtualenv 和 setuptools 是最新版本

pip install --upgrade pip virtualenv setuptools

如果你能直接运行以下的安装指令，只能说明你的人品足够的好：

pip install Scrapy

从上帝的角度来看，每个人都是有罪的，所以我估计也很难出现这样人品出众的人吧。至少，每次我一运行这指令就会弹出各种的错，以下来罗列一下：

ffi 安装失败

出现的跟踪信息：

/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/include/ffi -I/usr/include/libffi -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c c/_cffi_backend.c -o build/temp.macosx-10.6-intel-2.7/c/_cffi_backend.o
    c/_cffi_backend.c:13:10: fatal error: 'ffi.h' file not found
    #include <ffi.h>
             ^
    1 error generated.
    error: command '/usr/bin/clang' failed with exit status 1
    
    ----------------------------------------
Command "/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-ARGDvZ/cffi/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('
', '
'), __file__, 'exec'))" install --record /tmp/pip-BnZEqB-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-ARGDvZ/cffi

解决办法:

brew install libffi

openssl 安装失败

如果出现 openssl 失败，而你又确认真的安装了 Openssl 的话，你将会看到这样的错误。

/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c build/temp.macosx-10.6-intel-2.7/_openssl.c -o build/temp.macosx-10.6-intel-2.7/build/temp.macosx-10.6-intel-2.7/_openssl.o
    build/temp.macosx-10.6-intel-2.7/_openssl.c:408:10: fatal error: 'openssl/aes.h' file not found
    #include <openssl/aes.h>
             ^
    1 error generated.
    error: command '/usr/bin/clang' failed with exit status 1
    
    ----------------------------------------
Command "/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-WKTcA7/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('
', '
'), __file__, 'exec'))" install --record /tmp/pip-FyT7Fx-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-WKTcA7/cryptography

解决办法:

env CRYPTOGRAPHY_OSX_NO_LINK_FLAGS=1 LDFLAGS="$(brew --prefix openssl)/lib/libssl.a $(brew --prefix openssl)/lib/libcrypto.a" CFLAGS="-I$(brew --prefix openssl)/include" pip install scrapy

lxml

这是最可怕的一个包了，尝试过各种的方法和各种的失败，有兴趣的你可以 google 一下 lxml 将会知道有多少人掉到安装的坑里面。

building 'lxml.etree' extension
    /usr/bin/clang -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/opt/openssl/include -I/usr/include/libxml2 -I/private/var/folders/pn/9vl_v4ls1_sfy3mr9cb6ln300000gp/T/pip-build-ZjFZ1r/lxml/src/lxml/includes -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.macosx-10.6-intel-2.7/src/lxml/lxml.etree.o -w -flat_namespace
    In file included from src/lxml/lxml.etree.c:239:
    /private/var/folders/pn/9vl_v4ls1_sfy3mr9cb6ln300000gp/T/pip-build-ZjFZ1r/lxml/src/lxml/includes/etree_defs.h:14:10: fatal error: 'libxml/xmlversion.h' file not found
    #include "libxml/xmlversion.h"
             ^
    1 error generated.
    error: command '/usr/bin/clang' failed with exit status 1
    
    ----------------------------------------
Command "/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -c "import setuptools, tokenize;__file__='/private/var/folders/pn/9vl_v4ls1_sfy3mr9cb6ln300000gp/T/pip-build-ZjFZ1r/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('
', '
'), __file__, 'exec'))" install --record /var/folders/pn/9vl_v4ls1_sfy3mr9cb6ln300000gp/T/pip-ayVXMn-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/pn/9vl_v4ls1_sfy3mr9cb6ln300000gp/T/pip-build-ZjFZ1r/lxml

解决方法:

不要尝试各种网上奇葩的方法，因为基本都不会有用的，真正的原因在 xcode command tools 出现问题，所以只要运行以下指令，呼出 xcode command tools 的安装界面，下载10来分钟，重新运行 scrapy 的安装就好了。

xcode-select --install

当以上的所有的问题都解决后 Scrapy 就安装完成了。