爬虫是一个既好玩又好用的东西，接触它有1年多了，了解一些，算不上精通。爬过新闻、微信公众号，主要用pyspider，scrapy也了解过，python2和3都尝试过。公司业务规模不大，基于 python3.6的pyspider是我最终选择的方案。
环境准备：centos7
centos系统会自带python2，我们最好不理它，使用python3运行python脚本，因为有些程序会依赖python2环境，比如yum。
安装python3.6.4
1. 安装依赖环境
yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel
2.下载Python3
https://www.python.org/downloads/
wget https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tgz
3.安装python3
我个人习惯安装在/usr/local/python3
创建目录：
mkdir -p /usr/local/python3
解压下载好的Python-3.x.x.tgz包
tar -zxvf Python-3.6.4.tgz
4.进入解压后的目录，编译安装。
cd Python-3.6.4 ./configure --prefix=/usr/local/python3
make
make install 或者 make && make install
5.建立python3的软链
ln -s /usr/local/python3/bin/python3 /usr/bin/python3
6.将/usr/local/python3/bin加入PATH
# vim ~/.bash_profile # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs PATH=$PATH:$HOME/bin:/usr/local/python3/bin export PATH
让修改生效
source ~/.bash_profile
检查Python3及pip3是否正常可用：
# python3 -V Python 3.6.4 # pip3 -V pip 9.0.2 from /usr/local/python3/lib/python3.6/site-packages (python 3.6)