install scala & spark env

安装Scala 
1，到http://www.scala-lang.org/download/ 下载与Spark版本对应的Scala。Spark1.2对应于Scala2.10的版本。这里下载scala-2.10.4.tgz。 
2，解压安装Scala 
1）， 执行#tar -axvf scala-2.10.4.tgz，解压到/root/spark/scala-2.10.4。 
2），在~/.bash_profile中添加如下配置：

export SCALA_HOME=/root/spark/scala-2.10.4
export PATH=$JAVA_HOME/bin$HADOOP_HOME/bin:$HIVE_HOME/bin:$SCALA_HOME/bin:$PATH

3），使环境变量生效，#source ~/.bash_profile 
3，验证安装，在命令行中输入scala命令，可以进入scala命令控制台。

# scala
Welcome to Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.6.0_45).
Type in expressions to have them evaluated.
Type :help for more information.

scala>

三，安装Spark 
1，到http://spark.apache.org/downloads.html下载spark-1.2.0-bin-hadoop2.4.tgz，解压到/root/spark/spark-1.2.0-bin-hadoop2.4。 
2，在.bash_profile中添加如下配置：

export SPARK_HOME=/root/spark/spark-1.2.0-bin-hadoop2.4
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$SCALA_HOME/bin:$SPARK_HOME/bin:$HIVE_HOME/bin:$PATH

3，使环境变量生效，#source ~/.bash_profile

四，配置Spark 
1，进入Spark的配置文件路径，#cd $SPARK_HOME/conf 
2，执行，#cp spark-env.sh.template spark-env.sh 
3，在spark-env.sh文件中添加如下配置：

export JAVA_HOME=/usr/lib/jdk1.6.0_45
export SCALA_HOME=/root/spark/scala-2.10.4
export HADOOP_CONF_DIR=/root/hadoop/hadoop-2.6.0/etc/hadoop

五，启动Spark 
1，进入spark的安装路径，#cd /root/spark/spark-1.2.0-bin-hadoop2.4 
2，执行#./sbin/start-all.sh命令 
3，执行 #jps命令，会有Master和Worker进程

# jps
38907 RunJar
39030 RunJar
54679 NameNode
26587 Jps
54774 DataNode
9850 Worker
9664 Master
55214 NodeManager
55118 ResourceManager
54965 SecondaryNameNode

4，进入Spark的Web界面：http://datanode-4:8080/ 

5，执行，#./bin/spark-shell命令，可以进入Spark的shell环境，可以通过http://datanode-4:4040，看到SparkUI的情况。 

Last login: Sun Oct  8 05:35:42 2017 from 192.168.1.1
[hadoop@blm ~]$ java -version
java version "1.7.0_65"
Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
Java HotSpot(TM) Client VM (build 24.65-b04, mixed mode)
[hadoop@blm ~]$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:3C:BF:E3  
          inet addr:192.168.1.103  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe3c:bfe3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4461 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5051 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:362317 (353.8 KiB)  TX bytes:411434 (401.7 KiB)
          Interrupt:19 Base address:0x2024

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:325 errors:0 dropped:0 overruns:0 frame:0
          TX packets:325 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:27918 (27.2 KiB)  TX bytes:27918 (27.2 KiB)

[hadoop@blm ~]$ uname -a
Linux blm 2.6.32-358.el6.i686 #1 SMP Thu Feb 21 21:50:49 UTC 2013 i686 i686 i386 GNU/Linux
[hadoop@blm ~]$ ll
total 449508
-rw-rw-r--.  1 hadoop hadoop  80288778 Oct  5 22:25 apache-hive-0.14.0-bin.tar.gz
drwxrwxr-x. 10 hadoop hadoop      4096 Oct  8 05:27 app
-rw-rw-r--.  1 hadoop hadoop         0 Oct  4 20:21 a.txt
-rw-rw-r--.  1 hadoop hadoop         0 Oct  4 20:21 b.txt
-rw-rw-r--.  1 hadoop hadoop         0 Oct  4 20:21 c.txt
drwxrwxr-x.  2 hadoop hadoop      4096 Oct  4 22:34 download
-rwxrw-rw-.  1 hadoop hadoop 160860571 Oct  2 14:19 eclipse-java-luna-SR2-linux-gtk.tar.gz
-rw-rw-r--.  1 hadoop hadoop     27315 Oct  4 00:32 flow.jar
-rw-rw-r--.  1 hadoop hadoop     17765 Oct  4 03:50 flowsum.jar
-rw-rw-r--.  1 hadoop hadoop  15417097 Oct  1 03:33 hadoop-2.4.1-src.tar.gz
-rw-rw-r--.  1 hadoop hadoop 138656756 Oct  1 03:33 hadoop-2.4.1.tar.gz
-rwxrw-rw-.  1 hadoop hadoop      2214 Jul 30  2013 HTTP_20130313143750.dat
drwxr-xr-x.  8 hadoop hadoop      4096 Jun 16  2014 jdk1.7.0_65
lrwxrwxrwx.  1 hadoop hadoop        32 Oct  2 02:33 Link to eclipse -> /home/hadoop/app/eclipse/eclipse
-rw-rw-r--.  1 hadoop hadoop  29937534 Oct 13 08:57 scala-2.10.4.tgz.gz
-rw-rw-r--.  1 hadoop hadoop     10808 Oct  3 01:57 wc.jar
-rw-rw-r--.  1 hadoop hadoop        96 Oct  3 01:41 word.log
drwxrwxr-x.  6 hadoop hadoop      4096 Oct  2 08:10 workspace
-r-xr--r--.  1 hadoop hadoop  35042811 Oct  4 22:39 zookeeper-3.4.10.tar.gz
[hadoop@blm ~]$ tar -zxvf scala-2.10.4.tgz.gz  -C app/
scala-2.10.4/
scala-2.10.4/man/
scala-2.10.4/man/man1/
scala-2.10.4/man/man1/scaladoc.1
scala-2.10.4/man/man1/scalap.1
scala-2.10.4/man/man1/scalac.1
scala-2.10.4/man/man1/fsc.1
scala-2.10.4/man/man1/scala.1
scala-2.10.4/src/
scala-2.10.4/src/scala-library-src.jar
scala-2.10.4/src/scala-swing-src.jar
scala-2.10.4/src/fjbg-src.jar
scala-2.10.4/src/scala-reflect-src.jar
scala-2.10.4/src/scalap-src.jar
scala-2.10.4/src/msil-src.jar
scala-2.10.4/src/scala-compiler-src.jar
scala-2.10.4/src/scala-actors-src.jar
scala-2.10.4/src/scala-partest-src.jar
scala-2.10.4/doc/
scala-2.10.4/doc/README
scala-2.10.4/doc/tools/
scala-2.10.4/doc/tools/index.html
scala-2.10.4/doc/tools/scalap.html
scala-2.10.4/doc/tools/scalac.html
scala-2.10.4/doc/tools/fsc.html
scala-2.10.4/doc/tools/css/
scala-2.10.4/doc/tools/css/style.css
scala-2.10.4/doc/tools/images/
scala-2.10.4/doc/tools/images/scala_logo.png
scala-2.10.4/doc/tools/images/external.gif
scala-2.10.4/doc/tools/scaladoc.html
scala-2.10.4/doc/tools/scala.html
scala-2.10.4/doc/LICENSE
scala-2.10.4/doc/licenses/
scala-2.10.4/doc/licenses/mit_jquery-ui.txt
scala-2.10.4/doc/licenses/mit_tools.tooltip.txt
scala-2.10.4/doc/licenses/mit_sizzle.txt
scala-2.10.4/doc/licenses/mit_jquery-layout.txt
scala-2.10.4/doc/licenses/mit_jquery.txt
scala-2.10.4/doc/licenses/bsd_jline.txt
scala-2.10.4/doc/licenses/apache_jansi.txt
scala-2.10.4/doc/licenses/bsd_asm.txt
scala-2.10.4/examples/
scala-2.10.4/examples/sort2.scala
scala-2.10.4/examples/iterators.scala
scala-2.10.4/examples/monads/
scala-2.10.4/examples/monads/callccInterpreter.scala
scala-2.10.4/examples/monads/stateInterpreter.scala
scala-2.10.4/examples/monads/simpleInterpreter.scala
scala-2.10.4/examples/monads/directInterpreter.scala
scala-2.10.4/examples/monads/errorInterpreter.scala
scala-2.10.4/examples/tcpoly/
scala-2.10.4/examples/tcpoly/monads/
scala-2.10.4/examples/tcpoly/monads/Monads.scala
scala-2.10.4/examples/futures.scala
scala-2.10.4/examples/boundedbuffer.scala
scala-2.10.4/examples/sort1.scala
scala-2.10.4/examples/parsing/
scala-2.10.4/examples/parsing/ListParsers.scala
scala-2.10.4/examples/parsing/ListParser.scala
scala-2.10.4/examples/parsing/ArithmeticParser.scala
scala-2.10.4/examples/parsing/lambda/
scala-2.10.4/examples/parsing/lambda/TestParser.scala
scala-2.10.4/examples/parsing/lambda/Main.scala
scala-2.10.4/examples/parsing/lambda/TestSyntax.scala
scala-2.10.4/examples/parsing/lambda/test/
scala-2.10.4/examples/parsing/lambda/test/test-02.kwi
scala-2.10.4/examples/parsing/lambda/test/test-07.kwi
scala-2.10.4/examples/parsing/lambda/test/test-08.kwi
scala-2.10.4/examples/parsing/lambda/test/test-03.kwi
scala-2.10.4/examples/parsing/lambda/test/test-04.kwi
scala-2.10.4/examples/parsing/lambda/test/test-06.kwi
scala-2.10.4/examples/parsing/lambda/test/test-05.kwi
scala-2.10.4/examples/parsing/lambda/test/test-01.kwi
scala-2.10.4/examples/parsing/JSON.scala
scala-2.10.4/examples/parsing/MiniML.scala
scala-2.10.4/examples/parsing/ArithmeticParsers.scala
scala-2.10.4/examples/fors.scala
scala-2.10.4/examples/patterns.scala
scala-2.10.4/examples/computeserver.scala
scala-2.10.4/examples/oneplacebuffer.scala
scala-2.10.4/examples/sort.scala
scala-2.10.4/examples/package.scala
scala-2.10.4/examples/actors/
scala-2.10.4/examples/actors/seq.scala
scala-2.10.4/examples/actors/producers.scala
scala-2.10.4/examples/actors/links.scala
scala-2.10.4/examples/actors/boundedbuffer.scala
scala-2.10.4/examples/actors/message.scala
scala-2.10.4/examples/actors/auction.scala
scala-2.10.4/examples/actors/channels.scala
scala-2.10.4/examples/actors/fringe.scala
scala-2.10.4/examples/actors/pingpong.scala
scala-2.10.4/examples/actors/looping.scala
scala-2.10.4/examples/xml/
scala-2.10.4/examples/xml/phonebook/
scala-2.10.4/examples/xml/phonebook/phonebook1.scala
scala-2.10.4/examples/xml/phonebook/phonebook2.scala
scala-2.10.4/examples/xml/phonebook/embeddedBook.scala
scala-2.10.4/examples/xml/phonebook/verboseBook.scala
scala-2.10.4/examples/xml/phonebook/phonebook.scala
scala-2.10.4/examples/xml/phonebook/phonebook3.scala
scala-2.10.4/examples/gadts.scala
scala-2.10.4/examples/maps.scala
scala-2.10.4/misc/
scala-2.10.4/misc/scala-devel/
scala-2.10.4/misc/scala-devel/plugins/
scala-2.10.4/misc/scala-devel/plugins/continuations.jar
scala-2.10.4/lib/
scala-2.10.4/lib/typesafe-config.jar
scala-2.10.4/lib/akka-actors.jar
scala-2.10.4/lib/scala-actors.jar
scala-2.10.4/lib/scala-compiler.jar
scala-2.10.4/lib/scala-reflect.jar
scala-2.10.4/lib/scala-library.jar
scala-2.10.4/lib/scala-swing.jar
scala-2.10.4/lib/jline.jar
scala-2.10.4/lib/scala-actors-migration.jar
scala-2.10.4/lib/scalap.jar
scala-2.10.4/bin/
scala-2.10.4/bin/scaladoc.bat
scala-2.10.4/bin/scala.bat
scala-2.10.4/bin/scalac.bat
scala-2.10.4/bin/scala
scala-2.10.4/bin/scaladoc
scala-2.10.4/bin/fsc.bat
scala-2.10.4/bin/fsc
scala-2.10.4/bin/scalac
scala-2.10.4/bin/scalap.bat
scala-2.10.4/bin/scalap
[hadoop@blm ~]$ cd app
[hadoop@blm app]$ ll
total 22732
drwxrwxr-x.  8 hadoop hadoop     4096 Oct  5 22:32 apache-hive-0.14.0-bin
drwxrwxr-x.  9 hadoop hadoop     4096 Oct  3 21:36 eclipse
drwxr-xr-x. 11 hadoop hadoop     4096 Oct  1 05:16 hadoop-2.4.1
drwxr-xr-x. 15 hadoop hadoop     4096 Jun 20  2014 hadoop-2.4.1-src
drwxrwxr-x.  2 hadoop hadoop     4096 Oct  6 03:15 hive
drwxrwxr-x.  2 hadoop hadoop     4096 Oct  6 04:02 hivetestdata
-rw-rw-r--.  1 hadoop hadoop  7232487 Oct  5 23:59 MySQL-client-5.1.73-1.glibc23.i386.rpm
-rw-rw-r--.  1 hadoop hadoop 16004449 Oct  5 23:59 MySQL-server-5.1.73-1.glibc23.i386.rpm
drwxrwxr-x.  9 hadoop hadoop     4096 Mar 18  2014 scala-2.10.4
drwxr-xr-x. 11 root   root       4096 Oct  7 07:44 xx
drwxr-xr-x. 11 root   root       4096 Oct  8 05:27 zookeeper-3.4.5
[hadoop@blm app]$ clear
[hadoop@blm app]$ cd /etc/profile
-bash: cd: /etc/profile: Not a directory
[hadoop@blm app]$ su root
Password:
su: incorrect password
[hadoop@blm app]$ su
Password:
[root@blm app]# clear
[root@blm app]# vi /etc/profile
# /etc/profile

# System wide environment and startup programs, for login setup
# Functions and aliases go in /etc/bashrc

# It's NOT a good idea to change this file unless you know what you
# are doing. It's much better to create a custom.sh shell script in
# /etc/profile.d/ to make custom changes to your environment, as this
# will prevent the need for merging in future updates.

pathmunge () {
    case ":${PATH}:" in
        *:"$1":*)
            ;;
        *)
            if [ "$2" = "after" ] ; then
                PATH=$PATH:$1
            else
                PATH=$1:$PATH
            fi
    esac
}


if [ -x /usr/bin/id ]; then
    if [ -z "$EUID" ]; then
        # ksh workaround
        EUID=`id -u`
        UID=`id -ru`
    fi
    USER="`id -un`"
    LOGNAME=$USER
    MAIL="/var/spool/mail/$USER"
fi

# Path manipulation
if [ "$EUID" = "0" ]; then
    pathmunge /sbin
    pathmunge /usr/sbin
    pathmunge /usr/local/sbin
else
    pathmunge /usr/local/sbin after
    pathmunge /usr/sbin after
    pathmunge /sbin after
fi

HOSTNAME=`/bin/hostname 2>/dev/null`
HISTSIZE=1000
if [ "$HISTCONTROL" = "ignorespace" ] ; then
    export HISTCONTROL=ignoreboth
else
    export HISTCONTROL=ignoredups
fi

export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE HISTCONTROL

export JAVA_HOME=/home/hadoop/jdk1.7.0_65
export SCALA_HOME=/home/hadoop/app/scala-2.10.4
export HADOOP_HOME=/home/hadoop/app/hadoop-2.4.1
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SCALA_HOME/bin
export HIVE_HOME=/home/hadoop/app/apache-hive-0.14.0-bin


# By default, we want umask to get set. This sets it for login shell
# Current threshold for system reserved uid/gids is 200
# You could check uidgid reservation validity in
# /usr/share/doc/setup-*/uidgid file
if [ $UID -gt 199 ] && [ "`id -gn`" = "`id -un`" ]; then
    umask 002
else
    umask 022
fi

for i in /etc/profile.d/*.sh ; do
    if [ -r "$i" ]; then
        if [ "${-#*i}" != "$-" ]; then
            . "$i"
        else
            . "$i" >/dev/null 2>&1
        fi
    fi
done

unset i
"/etc/profile" 85L, 2078C written
[root@blm app]# scala
bash: scala: command not found
[root@blm app]# java
Usage: java [-options] class [args...]
           (to execute a class)
   or  java [-options] -jar jarfile [args...]
           (to execute a jar file)
where options include:
    -d32          use a 32-bit data model if available
    -d64          use a 64-bit data model if available
    -client       to select the "client" VM
    -server       to select the "server" VM
    -hotspot      is a synonym for the "client" VM  [deprecated]
                  The default VM is client.

    -cp <class search path of directories and zip/jar files>
    -classpath <class search path of directories and zip/jar files>
                  A : separated list of directories, JAR archives,
                  and ZIP archives to search for class files.
    -D<name>=<value>
                  set a system property
    -verbose:[class|gc|jni]
                  enable verbose output
    -version      print product version and exit
    -version:<value>
                  require the specified version to run
    -showversion  print product version and continue
    -jre-restrict-search | -no-jre-restrict-search
                  include/exclude user private JREs in the version search
    -? -help      print this help message
    -X            print help on non-standard options
    -ea[:<packagename>...|:<classname>]
    -enableassertions[:<packagename>...|:<classname>]
                  enable assertions with specified granularity
    -da[:<packagename>...|:<classname>]
    -disableassertions[:<packagename>...|:<classname>]
                  disable assertions with specified granularity
    -esa | -enablesystemassertions
                  enable system assertions
    -dsa | -disablesystemassertions
                  disable system assertions
    -agentlib:<libname>[=<options>]
                  load native agent library <libname>, e.g. -agentlib:hprof
                  see also, -agentlib:jdwp=help and -agentlib:hprof=help
    -agentpath:<pathname>[=<options>]
                  load native agent library by full pathname
    -javaagent:<jarpath>[=<options>]
                  load Java programming language agent, see java.lang.instrument
    -splash:<imagepath>
                  show splash screen with specified image
See http://www.oracle.com/technetwork/java/javase/documentation/index.html for more details.
[root@blm app]# su
[root@blm app]# vi /etc/profile
# /etc/profile

# System wide environment and startup programs, for login setup
# Functions and aliases go in /etc/bashrc

# It's NOT a good idea to change this file unless you know what you
# are doing. It's much better to create a custom.sh shell script in
# /etc/profile.d/ to make custom changes to your environment, as this
# will prevent the need for merging in future updates.

pathmunge () {
    case ":${PATH}:" in
        *:"$1":*)
            ;;
        *)
            if [ "$2" = "after" ] ; then
                PATH=$PATH:$1
            else
                PATH=$1:$PATH
            fi
    esac
}


if [ -x /usr/bin/id ]; then
    if [ -z "$EUID" ]; then
        # ksh workaround
        EUID=`id -u`
        UID=`id -ru`
    fi
    USER="`id -un`"
    LOGNAME=$USER
    MAIL="/var/spool/mail/$USER"
fi

# Path manipulation
if [ "$EUID" = "0" ]; then
    pathmunge /sbin
    pathmunge /usr/sbin
    pathmunge /usr/local/sbin
else
    pathmunge /usr/local/sbin after
    pathmunge /usr/sbin after
    pathmunge /sbin after
fi

HOSTNAME=`/bin/hostname 2>/dev/null`
HISTSIZE=1000
if [ "$HISTCONTROL" = "ignorespace" ] ; then
    export HISTCONTROL=ignoreboth
else
    export HISTCONTROL=ignoredups
fi

export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE HISTCONTROL

export JAVA_HOME=/home/hadoop/jdk1.7.0_65
export SCALA_HOME=/home/hadoop/app/scala-2.10.4
export HADOOP_HOME=/home/hadoop/app/hadoop-2.4.1
export SPARK_HOME=/home/hadoop/app/spark-1.2.0-bin-hadoop2.4
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SCALA_HOME/bin:$SPARK_HOME/bin
export HIVE_HOME=/home/hadoop/app/apache-hive-0.14.0-bin


# By default, we want umask to get set. This sets it for login shell
# Current threshold for system reserved uid/gids is 200
# You could check uidgid reservation validity in
# /usr/share/doc/setup-*/uidgid file
if [ $UID -gt 199 ] && [ "`id -gn`" = "`id -un`" ]; then
    umask 002
else
    umask 022
fi

for i in /etc/profile.d/*.sh ; do
    if [ -r "$i" ]; then
        if [ "${-#*i}" != "$-" ]; then
            . "$i"
        else
            . "$i" >/dev/null 2>&1
        fi
    fi
done

unset i
"/etc/profile" 86L, 2155C written

==============================================================================

[hadoop@blm spark-1.2.0-bin-hadoop2.4]$ cd logs
[hadoop@blm logs]$ ll
total 8
-rw-rw-r--. 1 hadoop hadoop 2014 Oct 13 09:40 spark-hadoop-org.apache.spark.deploy.master.Master-1-blm.out
-rw-rw-r--. 1 hadoop hadoop 2091 Oct 13 09:40 spark-hadoop-org.apache.spark.deploy.worker.Worker-1-blm.out
[hadoop@blm logs]$ tail -100f spark-hadoop-org.apache.spark.deploy.master.Master-1-blm.out
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Spark Command: /home/hadoop/jdk1.7.0_65/bin/java -cp ::/home/hadoop/app/spark-1.2.0-bin-hadoop2.4/sbin/../conf:/home/hadoop/app/spark-1.2.0-bin-hadoop2.4/lib/spark-assembly-1.2.0-hadoop2.4.0.jar:/home/hadoop/app/spark-1.2.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar:/home/hadoop/app/spark-1.2.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar:/home/hadoop/app/spark-1.2.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar:/home/hadoop/app/hadoop-2.4.1 -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip blm --port 7077 --webui-port 8080
========================================

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/10/13 09:40:11 INFO Master: Registered signal handlers for [TERM, HUP, INT]
17/10/13 09:40:12 INFO SecurityManager: Changing view acls to: hadoop
17/10/13 09:40:12 INFO SecurityManager: Changing modify acls to: hadoop
17/10/13 09:40:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
17/10/13 09:40:15 INFO Slf4jLogger: Slf4jLogger started
17/10/13 09:40:16 INFO Remoting: Starting remoting
17/10/13 09:40:17 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkMaster@blm:7077]
17/10/13 09:40:17 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkMaster@blm:7077]
17/10/13 09:40:17 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
17/10/13 09:40:18 INFO Master: Starting Spark master at spark://blm:7077
17/10/13 09:40:28 INFO Utils: Successfully started service 'MasterUI' on port 8080.
17/10/13 09:40:28 INFO MasterWebUI: Started MasterWebUI at http://blm:8080
17/10/13 09:40:29 INFO Master: I have been elected leader! New state: ALIVE
17/10/13 09:40:32 INFO Master: Registering worker blm:38727 with 1 cores, 512.0 MB RAM
^C
[hadoop@blm logs]$ cat  spark-hadoop-org.apache.spark.deploy.worker.Worker-1-blm.out
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Spark Command: /home/hadoop/jdk1.7.0_65/bin/java -cp ::/home/hadoop/app/spark-1.2.0-bin-hadoop2.4/sbin/../conf:/home/hadoop/app/spark-1.2.0-bin-hadoop2.4/lib/spark-assembly-1.2.0-hadoop2.4.0.jar:/home/hadoop/app/spark-1.2.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar:/home/hadoop/app/spark-1.2.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar:/home/hadoop/app/spark-1.2.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar:/home/hadoop/app/hadoop-2.4.1 -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m org.apache.spark.deploy.worker.Worker spark://blm:7077
========================================

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/10/13 09:40:16 INFO Worker: Registered signal handlers for [TERM, HUP, INT]
17/10/13 09:40:16 INFO SecurityManager: Changing view acls to: hadoop
17/10/13 09:40:16 INFO SecurityManager: Changing modify acls to: hadoop
17/10/13 09:40:16 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
17/10/13 09:40:18 INFO Slf4jLogger: Slf4jLogger started
17/10/13 09:40:19 INFO Remoting: Starting remoting
17/10/13 09:40:19 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkWorker@blm:38727]
17/10/13 09:40:19 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkWorker@blm:38727]
17/10/13 09:40:19 INFO Utils: Successfully started service 'sparkWorker' on port 38727.
17/10/13 09:40:20 INFO Worker: Starting Spark worker blm:38727 with 1 cores, 512.0 MB RAM
17/10/13 09:40:20 INFO Worker: Spark home: /home/hadoop/app/spark-1.2.0-bin-hadoop2.4
17/10/13 09:40:30 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
17/10/13 09:40:30 INFO WorkerWebUI: Started WorkerWebUI at http://blm:8081
17/10/13 09:40:30 INFO Worker: Connecting to master spark://blm:7077...
17/10/13 09:40:32 INFO Worker: Successfully registered with master spark://blm:7077

export JAVA_HOME=/home/hadoop/jdk1.7.0_65
export SCALA_HOME=/home/hadoop/app/scala-2.10.4
export HADOOP_CONF_DIR=/home/hadoop/app/hadoop-2.4.1


"spark-env.sh" 59L, 3361C written                                                            
[hadoop@blm conf]$ ll
total 28
-rw-rw-r--. 1 hadoop hadoop  303 Dec 10  2014 fairscheduler.xml.template
-rw-rw-r--. 1 hadoop hadoop  620 Dec 10  2014 log4j.properties.template
-rw-rw-r--. 1 hadoop hadoop 5308 Dec 10  2014 metrics.properties.template
-rw-rw-r--. 1 hadoop hadoop   80 Dec 10  2014 slaves.template
-rw-rw-r--. 1 hadoop hadoop  507 Dec 10  2014 spark-defaults.conf.template
-rwxrwxr-x. 1 hadoop hadoop 3361 Oct 13 09:36 spark-env.sh
[hadoop@blm conf]$ jps
4382 Jps
4027 Worker
3890 Master
[hadoop@blm conf]$ ll
total 28
-rw-rw-r--. 1 hadoop hadoop  303 Dec 10  2014 fairscheduler.xml.template
-rw-rw-r--. 1 hadoop hadoop  620 Dec 10  2014 log4j.properties.template
-rw-rw-r--. 1 hadoop hadoop 5308 Dec 10  2014 metrics.properties.template
-rw-rw-r--. 1 hadoop hadoop   80 Dec 10  2014 slaves.template
-rw-rw-r--. 1 hadoop hadoop  507 Dec 10  2014 spark-defaults.conf.template
-rwxrwxr-x. 1 hadoop hadoop 3361 Oct 13 09:36 spark-env.sh
[hadoop@blm conf]$ pwd
/home/hadoop/app/spark-1.2.0-bin-hadoop2.4/conf
[hadoop@blm conf]$ ll
total 28
-rw-rw-r--. 1 hadoop hadoop  303 Dec 10  2014 fairscheduler.xml.template
-rw-rw-r--. 1 hadoop hadoop  620 Dec 10  2014 log4j.properties.template
-rw-rw-r--. 1 hadoop hadoop 5308 Dec 10  2014 metrics.properties.template
-rw-rw-r--. 1 hadoop hadoop   80 Dec 10  2014 slaves.template
-rw-rw-r--. 1 hadoop hadoop  507 Dec 10  2014 spark-defaults.conf.template
-rwxrwxr-x. 1 hadoop hadoop 3361 Oct 13 09:36 spark-env.sh
[hadoop@blm conf]$ cd ..
[hadoop@blm spark-1.2.0-bin-hadoop2.4]$ ll
total 120
drwxrwxr-x. 2 hadoop hadoop  4096 Dec 10  2014 bin
drwxrwxr-x. 2 hadoop hadoop  4096 Oct 13 09:36 conf
drwxrwxr-x. 3 hadoop hadoop  4096 Dec 10  2014 data
drwxrwxr-x. 4 hadoop hadoop  4096 Dec 10  2014 ec2
drwxrwxr-x. 3 hadoop hadoop  4096 Dec 10  2014 examples
drwxrwxr-x. 2 hadoop hadoop  4096 Dec 10  2014 lib
-rw-rw-r--. 1 hadoop hadoop 45242 Dec 10  2014 LICENSE
drwxrwxr-x. 2 hadoop hadoop  4096 Oct 13 09:40 logs
-rw-rw-r--. 1 hadoop hadoop 22559 Dec 10  2014 NOTICE
drwxrwxr-x. 7 hadoop hadoop  4096 Dec 10  2014 python
-rw-rw-r--. 1 hadoop hadoop  3645 Dec 10  2014 README.md
-rw-rw-r--. 1 hadoop hadoop    35 Dec 10  2014 RELEASE
drwxrwxr-x. 2 hadoop hadoop  4096 Dec 10  2014 sbin
drwxrwxr-x. 2 hadoop hadoop  4096 Oct 13 09:40 work
[hadoop@blm spark-1.2.0-bin-hadoop2.4]$ cd bin
[hadoop@blm bin]$ ll
total 108
-rwxrwxr-x. 1 hadoop hadoop 1047 Dec 10  2014 beeline
-rw-rw-r--. 1 hadoop hadoop  953 Dec 10  2014 beeline.cmd
-rw-rw-r--. 1 hadoop hadoop 5374 Dec 10  2014 compute-classpath.cmd
-rwxrwxr-x. 1 hadoop hadoop 6377 Dec 10  2014 compute-classpath.sh
-rw-rw-r--. 1 hadoop hadoop 2065 Dec 10  2014 load-spark-env.sh
-rwxrwxr-x. 1 hadoop hadoop 5049 Dec 10  2014 pyspark
-rw-rw-r--. 1 hadoop hadoop 2412 Dec 10  2014 pyspark2.cmd
-rw-rw-r--. 1 hadoop hadoop 1023 Dec 10  2014 pyspark.cmd
-rwxrwxr-x. 1 hadoop hadoop 2131 Dec 10  2014 run-example
-rw-rw-r--. 1 hadoop hadoop 2869 Dec 10  2014 run-example2.cmd
-rw-rw-r--. 1 hadoop hadoop 1035 Dec 10  2014 run-example.cmd
-rwxrwxr-x. 1 hadoop hadoop 6750 Dec 10  2014 spark-class
-rw-rw-r--. 1 hadoop hadoop 6482 Dec 10  2014 spark-class2.cmd
-rw-rw-r--. 1 hadoop hadoop 1033 Dec 10  2014 spark-class.cmd
-rwxrwxr-x. 1 hadoop hadoop 2884 Dec 10  2014 spark-shell
-rw-rw-r--. 1 hadoop hadoop  971 Dec 10  2014 spark-shell2.cmd
-rwxrwxr-x. 1 hadoop hadoop 1031 Dec 10  2014 spark-shell.cmd
-rwxrwxr-x. 1 hadoop hadoop 1744 Dec 10  2014 spark-sql
-rwxrwxr-x. 1 hadoop hadoop 2562 Dec 10  2014 spark-submit
-rw-rw-r--. 1 hadoop hadoop 2603 Dec 10  2014 spark-submit2.cmd
-rw-rw-r--. 1 hadoop hadoop 1033 Dec 10  2014 spark-submit.cmd
-rwxrwxr-x. 1 hadoop hadoop 2058 Dec 10  2014 utils.sh
[hadoop@blm bin]$ spark-shell
-bash: spark-shell: command not found
[hadoop@blm bin]$ ./spark-shell
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/10/13 09:52:03 INFO SecurityManager: Changing view acls to: hadoop
17/10/13 09:52:03 INFO SecurityManager: Changing modify acls to: hadoop
17/10/13 09:52:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
17/10/13 09:52:03 INFO HttpServer: Starting HTTP Server
17/10/13 09:52:03 INFO Utils: Successfully started service 'HTTP class server' on port 40534.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _ / _ / _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_   version 1.2.0
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) Client VM, Java 1.7.0_65)
Type in expressions to have them evaluated.
Type :help for more information.
17/10/13 09:52:29 INFO SecurityManager: Changing view acls to: hadoop
17/10/13 09:52:29 INFO SecurityManager: Changing modify acls to: hadoop
17/10/13 09:52:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
17/10/13 09:52:31 INFO Slf4jLogger: Slf4jLogger started
17/10/13 09:52:31 INFO Remoting: Starting remoting
17/10/13 09:52:33 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@blm:43213]
17/10/13 09:52:33 INFO Utils: Successfully started service 'sparkDriver' on port 43213.
17/10/13 09:52:34 INFO SparkEnv: Registering MapOutputTracker
17/10/13 09:52:34 INFO SparkEnv: Registering BlockManagerMaster
17/10/13 09:52:34 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20171013095234-d91a
17/10/13 09:52:34 INFO MemoryStore: MemoryStore started with capacity 267.3 MB
17/10/13 09:52:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/10/13 09:52:37 INFO HttpFileServer: HTTP File server directory is /tmp/spark-a2325c17-1794-4c66-a240-6fecb4150ea1
17/10/13 09:52:37 INFO HttpServer: Starting HTTP Server
17/10/13 09:52:38 INFO Utils: Successfully started service 'HTTP file server' on port 41906.
17/10/13 09:52:49 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/10/13 09:52:49 INFO SparkUI: Started SparkUI at http://blm:4040
17/10/13 09:52:50 INFO Executor: Using REPL class URI: http://192.168.1.103:40534
17/10/13 09:52:50 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@blm:43213/user/HeartbeatReceiver
17/10/13 09:52:51 INFO NettyBlockTransferService: Server created on 46708
17/10/13 09:52:51 INFO BlockManagerMaster: Trying to register BlockManager
17/10/13 09:52:51 INFO BlockManagerMasterActor: Registering block manager localhost:46708 with 267.3 MB RAM, BlockManagerId(<driver>, localhost, 46708)
17/10/13 09:52:51 INFO BlockManagerMaster: Registered BlockManager
17/10/13 09:52:52 INFO SparkILoop: Created spark context..
Spark context available as sc.