AAS代码运行-第11章-1

启动PySpark

export IPYTHON=1 # PySpark也可使用IPython shell
pyspark --master yarn --num-executors 3

发生如下错误:

/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/spark/bin/pyspark: line 135: exec: ipython: not found

原因是没有按照ipython，在google所有一下安装的方法，到网站http://continuum.io/downloads#all上下载Linux-64位的Anaconda版本，

wget https://3230d63b5fc54e62148e-c95ac804525aac4b6dba79b00b39d1d3.ssl.cf1.rackcdn.com/Anaconda-2.3.0-Linux-x86_64.sh

下载完成之后，运行如下命令进行安装：

bash Anaconda-2.3.0-Linux-x86_64.sh

上述命令会进入一个类似shell的交互式环境，安装过程中都选择yes即可。安装完成之后，在安装ipython,Anaconda在交换式环境中输入:

pip install ipython

这时候会安装一大堆的包。安装完成之后会提示是否修改环境变量，输入yes，安装脚本自动将如下脚本写入~/.bashrc中的尾部

export PATH=/root/anaconda/bin:$PATH

运行source ~/.bashrc使新环境变量生效。

source ~/.bashrc

再次运行pyspark命令即可，如下：

pyspark --master yarn --num-executors 3

得到如下输出:

Type "copyright", "credits" or "license" for more information.

IPython 3.2.0 -- An enhanced Interactive Python.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
15/07/04 12:21:02 INFO SecurityManager: Changing view acls to: root
15/07/04 12:21:02 INFO SecurityManager: Changing modify acls to: root
15/07/04 12:21:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/07/04 12:21:03 INFO Slf4jLogger: Slf4jLogger started
15/07/04 12:21:03 INFO Remoting: Starting remoting
15/07/04 12:21:03 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@ip-172-31-25-243.us-west-2.compute.internal:50324]
15/07/04 12:21:03 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@ip-172-31-25-243.us-west-2.compute.internal:50324]
15/07/04 12:21:03 INFO Utils: Successfully started service 'sparkDriver' on port 50324.
15/07/04 12:21:03 INFO SparkEnv: Registering MapOutputTracker
15/07/04 12:21:03 INFO SparkEnv: Registering BlockManagerMaster
15/07/04 12:21:03 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20150704122103-7afc
15/07/04 12:21:03 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
15/07/04 12:21:05 INFO HttpFileServer: HTTP File server directory is /tmp/spark-e2f4c4e3-dc9b-4db0-8fd4-1dcbb8819b05
15/07/04 12:21:05 INFO HttpServer: Starting HTTP Server
15/07/04 12:21:05 INFO Utils: Successfully started service 'HTTP file server' on port 48934.
15/07/04 12:21:05 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/07/04 12:21:05 INFO SparkUI: Started SparkUI at http://ip-172-31-25-243.us-west-2.compute.internal:4040
15/07/04 12:21:05 INFO RMProxy: Connecting to ResourceManager at ip-172-31-25-243.us-west-2.compute.internal/172.31.25.243:8032
15/07/04 12:21:06 INFO Client: Requesting a new application from cluster with 3 NodeManagers
15/07/04 12:21:06 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (4720 MB per container)
15/07/04 12:21:06 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
15/07/04 12:21:06 INFO Client: Setting up container launch context for our AM
15/07/04 12:21:06 INFO Client: Preparing resources for our AM container
15/07/04 12:21:07 INFO Client: Setting up the launch environment for our AM container
15/07/04 12:21:07 INFO SecurityManager: Changing view acls to: root
15/07/04 12:21:07 INFO SecurityManager: Changing modify acls to: root
15/07/04 12:21:07 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/07/04 12:21:07 INFO Client: Submitting application 1 to ResourceManager
15/07/04 12:21:07 INFO YarnClientImpl: Submitted application application_1436008024626_0001
15/07/04 12:21:08 INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
15/07/04 12:21:08 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: root.root
     start time: 1436012467313
     final status: UNDEFINED
     tracking URL: http://ip-172-31-25-243.us-west-2.compute.internal:8088/proxy/application_1436008024626_0001/
     user: root
15/07/04 12:21:09 INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
15/07/04 12:21:10 INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
15/07/04 12:21:11 INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
15/07/04 12:21:12 INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
15/07/04 12:21:13 INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
15/07/04 12:21:14 INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
15/07/04 12:21:15 INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
15/07/04 12:21:16 INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
15/07/04 12:21:17 INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
15/07/04 12:21:18 INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
15/07/04 12:21:19 INFO Client: Application report for application_1436008024626_0001 (state: ACCEPTED)
15/07/04 12:21:19 INFO YarnClientSchedulerBackend: ApplicationMaster registered as Actor[akka.tcp://sparkYarnAM@ip-172-31-25-246.us-west-2.compute.internal:45245/user/YarnAM#-1546613765]
15/07/04 12:21:19 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> ip-172-31-25-243.us-west-2.compute.internal, PROXY_URI_BASES -> http://ip-172-31-25-243.us-west-2.compute.internal:8088/proxy/application_1436008024626_0001), /proxy/application_1436008024626_0001
15/07/04 12:21:19 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
15/07/04 12:21:20 INFO Client: Application report for application_1436008024626_0001 (state: RUNNING)
15/07/04 12:21:20 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: ip-172-31-25-246.us-west-2.compute.internal
     ApplicationMaster RPC port: 0
     queue: root.root
     start time: 1436012467313
     final status: UNDEFINED
     tracking URL: http://ip-172-31-25-243.us-west-2.compute.internal:8088/proxy/application_1436008024626_0001/
     user: root
15/07/04 12:21:20 INFO YarnClientSchedulerBackend: Application application_1436008024626_0001 has started running.
15/07/04 12:21:20 INFO NettyBlockTransferService: Server created on 52594
15/07/04 12:21:20 INFO BlockManagerMaster: Trying to register BlockManager
15/07/04 12:21:20 INFO BlockManagerMasterActor: Registering block manager ip-172-31-25-243.us-west-2.compute.internal:52594 with 265.4 MB RAM, BlockManagerId(<driver>, ip-172-31-25-243.us-west-2.compute.internal, 52594)
15/07/04 12:21:20 INFO BlockManagerMaster: Registered BlockManager
15/07/04 12:21:21 INFO EventLoggingListener: Logging events to hdfs://ns-ha/user/spark/applicationHistory/application_1436008024626_0001
15/07/04 12:21:30 INFO YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-31-25-245.us-west-2.compute.internal:43360/user/Executor#1565208330] with ID 1
15/07/04 12:21:30 INFO RackResolver: Resolved ip-172-31-25-245.us-west-2.compute.internal to /default
15/07/04 12:21:30 INFO YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-31-25-244.us-west-2.compute.internal:39613/user/Executor#-786460899] with ID 2
15/07/04 12:21:30 INFO RackResolver: Resolved ip-172-31-25-244.us-west-2.compute.internal to /default
15/07/04 12:21:30 INFO BlockManagerMasterActor: Registering block manager ip-172-31-25-245.us-west-2.compute.internal:44702 with 530.3 MB RAM, BlockManagerId(1, ip-172-31-25-245.us-west-2.compute.internal, 44702)
15/07/04 12:21:31 INFO BlockManagerMasterActor: Registering block manager ip-172-31-25-244.us-west-2.compute.internal:45974 with 530.3 MB RAM, BlockManagerId(2, ip-172-31-25-244.us-west-2.compute.internal, 45974)
15/07/04 12:21:35 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _ / _ / _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_   version 1.2.0-SNAPSHOT
      /_/

Using Python version 2.7.10 (default, May 28 2015 17:02:03)
SparkContext available as sc.

In [1]: