zoukankan      html  css  js  c++  java
  • Hadoop 2.2.0学习笔记20131209

    1.下载java 7并安装

    [root@server-518 ~]# rpm -ivh jdk-7u40-linux-x64.rpm
    Preparing...                ########################################### [100%]
       1:jdk                    ########################################### [100%]
    Unpacking JAR files...
            rt.jar...
            jsse.jar...
            charsets.jar...
            tools.jar...
            localedata.jar...
            jfxrt.jar...
    [root@server-518 ~]# java -version
    java version "1.7.0_40"
    Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
    Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)
    View Code

    2.下载hadoop2.2.0并解压

    [root@server-518 hadoop]# tar -xzvf hadoop-2.2.0.tar.gz

    3.设置环境变量JAVA_HOME和HADOOP_HOME

    [root@server-518 hadoop]# vi /etc/profile

    在文件末尾添加

    export JAVA_HOME=/usr/java/latest
    export HADOOP_HOME=/root/hadoop-2.2.0

    使环境变量生效

    [root@server-518 hadoop]# source /etc/profile
    [root@server-518 hadoop]# echo $JAVA_HOME
    /usr/java/latest
    [root@server-518 hadoop]# echo $HADOOP_HOME
    /root/hadoop-2.2.0
    [root@server-518 hadoop]#

    4.编辑mapred-site.xml

    [root@server-518 hadoop]# mv mapred-site.xml.template mapred-site.xml

    添加如下内容:

      <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
      </property>

    5.编辑hdfs-site.xml

      <property>
        <name>fs.default.name</name>
        <value>hdfs://192.168.32.23:8020</value>
      </property>

    6.修改yarn-site.xml以是nodemanager启动时加载shuttle server

      <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce-shuffle</value>
      </property>

    7.修改hdfs-site.xml时复制份上该为1

      <property>
        <name>dfs.replication</name>
        <value>1</value>
      </property>

    8.生成无密码登录

    [root@server-518 ~]# ssh-keygen -t rsa
    Generating public/private rsa key pair.
    Enter file in which to save the key (/root/.ssh/id_rsa):
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in /root/.ssh/id_rsa.
    Your public key has been saved in /root/.ssh/id_rsa.pub.
    The key fingerprint is:
    6d:77:86:50:14:75:24:2e:fc:32:16:93:ee:29:73:f6 root@server-518
    The key's randomart image is:
    +--[ RSA 2048]----+
    |           .+oo.o|
    |           o o o |
    |          . * .  |
    |         . o *   |
    |        S o B +  |
    |         . + *   |
    |          o =    |
    |           = .   |
    |              E  |
    +-----------------+
    [root@server-518 ~]# cd .ssh
    [root@server-518 .ssh]# ls
    id_rsa  id_rsa.pub  known_hosts
    [root@server-518 .ssh]# cat id_rsa.pub >> authorized_keys
    View Code

    9.格式化namenode

    [root@server-518 bin]# hadoop namenode -format
    -bash: hadoop: command not found
    [root@server-518 bin]# ./hadoop namenode -format
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
    Instead use the hdfs command for it.
    
    13/12/09 19:47:56 INFO namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG:   host = server-518.novalocal/10.10.96.33
    STARTUP_MSG:   args = [-format]
    STARTUP_MSG:   version = 2.2.0
    STARTUP_MSG:   classpath = /root/hadoop-2.2.0/etc/hadoop:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-logging-1.1.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jersey-json-1.9.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/guava-11.0.2.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-el-1.0.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/avro-1.7.4.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/hadoop-auth-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/stax-api-1.0.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/servlet-api-2.5.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jettison-1.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jersey-core-1.9.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/zookeeper-3.4.5.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/asm-3.2.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jetty-6.1.26.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/xmlenc-0.52.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-lang-2.5.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/log4j-1.2.17.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-cli-1.2.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-math-2.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/paranamer-2.3.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jsp-api-2.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jersey-server-1.9.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-net-3.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-codec-1.4.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/hadoop-annotations-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/xz-1.0.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-digester-1.8.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jsch-0.1.42.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/activation-1.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jets3t-0.6.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/junit-4.8.2.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-io-2.1.jar:/root/hadoop-2.2.0/share/hadoop/common/hadoop-nfs-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0-tests.jar:/root/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/hdfs:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/asm-3.2.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-io-2.1.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-2.2.0-tests.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/avro-1.7.4.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/hamcrest-core-1.1.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/asm-3.2.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/junit-4.10.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/javax.inject-1.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/paranamer-2.3.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/guice-3.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/hadoop-annotations-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/xz-1.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/commons-io-2.1.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-site-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-common-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-common-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-api-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-client-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/junit-4.10.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/commons-io-2.1.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.2.0.jar:/root/hadoop-2.2.0/contrib/capacity-scheduler/*.jar:/root/hadoop-2.2.0/contrib/capacity-scheduler/*.jar
    STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common -r 1529768; compiled by 'hortonmu' on 2013-10-07T06:28Z
    STARTUP_MSG:   java = 1.7.0_40
    ************************************************************/
    13/12/09 19:47:56 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
    Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /root/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
    It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
    13/12/09 19:47:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Formatting using clusterid: CID-af24d15b-93d3-471c-bfaf-d8d61e3b5d0f
    13/12/09 19:47:57 INFO namenode.HostFileManager: read includes:
    HostSet(
    )
    13/12/09 19:47:57 INFO namenode.HostFileManager: read excludes:
    HostSet(
    )
    13/12/09 19:47:57 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
    13/12/09 19:47:57 INFO util.GSet: Computing capacity for map BlocksMap
    13/12/09 19:47:57 INFO util.GSet: VM type       = 64-bit
    13/12/09 19:47:57 INFO util.GSet: 2.0% max memory = 889 MB
    13/12/09 19:47:57 INFO util.GSet: capacity      = 2^21 = 2097152 entries
    13/12/09 19:47:57 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
    13/12/09 19:47:57 INFO blockmanagement.BlockManager: defaultReplication         = 1
    13/12/09 19:47:57 INFO blockmanagement.BlockManager: maxReplication             = 512
    13/12/09 19:47:57 INFO blockmanagement.BlockManager: minReplication             = 1
    13/12/09 19:47:57 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
    13/12/09 19:47:57 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false
    13/12/09 19:47:57 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
    13/12/09 19:47:57 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
    13/12/09 19:47:57 INFO namenode.FSNamesystem: fsOwner             = root (auth:SIMPLE)
    13/12/09 19:47:57 INFO namenode.FSNamesystem: supergroup          = supergroup
    13/12/09 19:47:57 INFO namenode.FSNamesystem: isPermissionEnabled = true
    13/12/09 19:47:57 INFO namenode.FSNamesystem: HA Enabled: false
    13/12/09 19:47:57 INFO namenode.FSNamesystem: Append Enabled: true
    13/12/09 19:47:57 INFO util.GSet: Computing capacity for map INodeMap
    13/12/09 19:47:57 INFO util.GSet: VM type       = 64-bit
    13/12/09 19:47:57 INFO util.GSet: 1.0% max memory = 889 MB
    13/12/09 19:47:57 INFO util.GSet: capacity      = 2^20 = 1048576 entries
    13/12/09 19:47:57 INFO namenode.NameNode: Caching file names occuring more than
    13/12/09 19:47:57 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pc
    13/12/09 19:47:57 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanode
    13/12/09 19:47:57 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension
    13/12/09 19:47:57 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
    13/12/09 19:47:57 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total
    13/12/09 19:47:57 INFO util.GSet: Computing capacity for map Namenode Retry Cach
    13/12/09 19:47:57 INFO util.GSet: VM type       = 64-bit
    13/12/09 19:47:57 INFO util.GSet: 0.029999999329447746% max memory = 889 MB
    13/12/09 19:47:57 INFO util.GSet: capacity      = 2^15 = 32768 entries
    13/12/09 19:47:57 INFO common.Storage: Storage directory /root/hadoop/data/dfs/n
    13/12/09 19:47:57 INFO namenode.FSImage: Saving image file /root/hadoop/data/dfs
    13/12/09 19:47:57 INFO namenode.FSImage: Image file /root/hadoop/data/dfs/name/c
    13/12/09 19:47:57 INFO namenode.NNStorageRetentionManager: Going to retain 1 ima
    13/12/09 19:47:57 INFO util.ExitUtil: Exiting with status 0
    13/12/09 19:47:57 INFO namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at server-518.novalocal/10.10.96.33
    ************************************************************/
    View Code

    10.启动namenode

    [root@server-518 sbin]# ./hadoop-daemon.sh start namenode
    starting namenode, logging to /root/hadoop-2.2.0/logs/hadoop-root-namenode-server-518.out
    Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /root/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
    It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

    11.启动datanode

    [root@server-518 sbin]# ./hadoop-daemon.sh start datanode
    starting datanode, logging to /root/hadoop-2.2.0/logs/hadoop-root-datanode-server-518.out
    Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /root/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
    It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

    12.查看namenode,datanode进程状态,确认服务已启动

    [root@server-518 sbin]# ps -ef | grep java
    root      6942     1  2 20:51 pts/2    00:00:03 /usr/java/latest/bin/java -Dproc_namenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/root/hadoop-2.2.0/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/root/hadoop-2.2.0 -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Djava.library.path=/root/hadoop-2.2.0/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/root/hadoop-2.2.0/logs -Dhadoop.log.file=hadoop-root-namenode-server-518.log -Dhadoop.home.dir=/root/hadoop-2.2.0 -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/root/hadoop-2.2.0/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode
    root      7081     1 10 20:53 pts/2    00:00:03 /usr/java/latest/bin/java -Dproc_datanode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/root/hadoop-2.2.0/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/root/hadoop-2.2.0 -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Djava.library.path=/root/hadoop-2.2.0/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/root/hadoop-2.2.0/logs -Dhadoop.log.file=hadoop-root-datanode-server-518.log -Dhadoop.home.dir=/root/hadoop-2.2.0 -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/root/hadoop-2.2.0/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -server -Dhadoop.security.logger=ERROR,RFAS -Dhadoop.security.logger=ERROR,RFAS -Dhadoop.security.logger=ERROR,RFAS -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.datanode.DataNode
    root      7167  5560  0 20:53 pts/2    00:00:00 grep java
    View Code

    13.启动resource manager

    [root@server-518 sbin]# ./yarn-daemon.sh start resourcemanager
    starting resourcemanager, logging to /root/hadoop-2.2.0/logs/yarn-root-resourcemanager-server-518.out

    14.启动node manager 

    [root@server-518 sbin]# ./yarn-daemon.sh start nodemanager
    starting nodemanager, logging to /root/hadoop-2.2.0/logs/yarn-root-nodemanager-server-518.out

    15.检查resource manager和node manager是否启动成功

    [root@server-518 sbin]# ps -ef | grep java
    root      6942     1  1 20:51 pts/2    00:00:04 /usr/java/latest/bin/java -Dproc_namenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/root/hadoop-2.2.0/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/root/hadoop-2.2.0 -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Djava.library.path=/root/hadoop-2.2.0/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/root/hadoop-2.2.0/logs -Dhadoop.log.file=hadoop-root-namenode-server-518.log -Dhadoop.home.dir=/root/hadoop-2.2.0 -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/root/hadoop-2.2.0/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode
    root      7081     1  1 20:53 pts/2    00:00:03 /usr/java/latest/bin/java -Dproc_datanode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/root/hadoop-2.2.0/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/root/hadoop-2.2.0 -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Djava.library.path=/root/hadoop-2.2.0/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/root/hadoop-2.2.0/logs -Dhadoop.log.file=hadoop-root-datanode-server-518.log -Dhadoop.home.dir=/root/hadoop-2.2.0 -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/root/hadoop-2.2.0/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -server -Dhadoop.security.logger=ERROR,RFAS -Dhadoop.security.logger=ERROR,RFAS -Dhadoop.security.logger=ERROR,RFAS -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.datanode.DataNode
    root      7231     1  4 20:56 pts/2    00:00:03 /usr/java/latest/bin/java -Dproc_resourcemanager -Xmx1000m -Dhadoop.log.dir=/root/hadoop-2.2.0/logs -Dyarn.log.dir=/root/hadoop-2.2.0/logs -Dhadoop.log.file=yarn-root-resourcemanager-server-518.log -Dyarn.log.file=yarn-root-resourcemanager-server-518.log -Dyarn.home.dir= -Dyarn.id.str=root -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/root/hadoop-2.2.0/lib/native -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/root/hadoop-2.2.0/logs -Dyarn.log.dir=/root/hadoop-2.2.0/logs -Dhadoop.log.file=yarn-root-resourcemanager-server-518.log -Dyarn.log.file=yarn-root-resourcemanager-server-518.log -Dyarn.home.dir=/root/hadoop-2.2.0 -Dhadoop.home.dir=/root/hadoop-2.2.0 -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/root/hadoop-2.2.0/lib/native -classpath /root/hadoop-2.2.0/etc/hadoop:/root/hadoop-2.2.0/etc/hadoop:/root/hadoop-2.2.0/etc/hadoop:/root/hadoop-2.2.0/share/hadoop/common/lib/*:/root/hadoop-2.2.0/share/hadoop/common/*:/root/hadoop-2.2.0/share/hadoop/hdfs:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/*:/root/hadoop-2.2.0/share/hadoop/hdfs/*:/root/hadoop-2.2.0/share/hadoop/yarn/lib/*:/root/hadoop-2.2.0/share/hadoop/yarn/*:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/*:/root/hadoop-2.2.0/share/hadoop/mapreduce/*:/root/hadoop-2.2.0/contrib/capacity-scheduler/*.jar:/root/hadoop-2.2.0/contrib/capacity-scheduler/*.jar:/root/hadoop-2.2.0/share/hadoop/yarn/*:/root/hadoop-2.2.0/share/hadoop/yarn/lib/*:/root/hadoop-2.2.0/etc/hadoop/rm-config/log4j.properties org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
    root      7547  5560  0 20:58 pts/2    00:00:00 grep java
    View Code

    没有发现nodemanager进程

    查看启动日志:

    [root@server-518 sbin]# cat ../logs/yarn-root-nodemanager-server-518.log
    2013-12-09 20:58:02,172 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NodeManager
    STARTUP_MSG:   host = server-518/10.10.96.33
    STARTUP_MSG:   args = []
    STARTUP_MSG:   version = 2.2.0
    STARTUP_MSG:   classpath = /root/hadoop-2.2.0/etc/hadoop:/root/hadoop-2.2.0/etc/hadoop:/root/hadoop-2.2.0/etc/hadoop:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-logging-1.1.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jersey-json-1.9.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/guava-11.0.2.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-el-1.0.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/avro-1.7.4.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/hadoop-auth-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/stax-api-1.0.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/servlet-api-2.5.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jettison-1.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jersey-core-1.9.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/zookeeper-3.4.5.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/asm-3.2.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jetty-6.1.26.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/xmlenc-0.52.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-lang-2.5.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/log4j-1.2.17.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-cli-1.2.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-math-2.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/paranamer-2.3.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jsp-api-2.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jersey-server-1.9.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-net-3.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-codec-1.4.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/hadoop-annotations-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/xz-1.0.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-digester-1.8.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jsch-0.1.42.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/activation-1.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/jets3t-0.6.1.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/junit-4.8.2.jar:/root/hadoop-2.2.0/share/hadoop/common/lib/commons-io-2.1.jar:/root/hadoop-2.2.0/share/hadoop/common/hadoop-nfs-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0-tests.jar:/root/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/hdfs:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/asm-3.2.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-io-2.1.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-2.2.0-tests.jar:/root/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/avro-1.7.4.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/hamcrest-core-1.1.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/asm-3.2.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/junit-4.10.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/javax.inject-1.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/paranamer-2.3.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/guice-3.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/hadoop-annotations-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/xz-1.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/commons-io-2.1.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-site-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-common-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-common-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-api-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-client-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/junit-4.10.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/lib/commons-io-2.1.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.2.0.jar:/root/hadoop-2.2.0/contrib/capacity-scheduler/*.jar:/root/hadoop-2.2.0/contrib/capacity-scheduler/*.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-site-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-common-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-common-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-api-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-client-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/avro-1.7.4.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/hamcrest-core-1.1.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/asm-3.2.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/junit-4.10.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/javax.inject-1.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/paranamer-2.3.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/guice-3.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/hadoop-annotations-2.2.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/xz-1.0.jar:/root/hadoop-2.2.0/share/hadoop/yarn/lib/commons-io-2.1.jar:/root/hadoop-2.2.0/etc/hadoop/nm-config/log4j.properties
    STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common -r 1529768; compiled by 'hortonmu' on 2013-10-07T06:28Z
    STARTUP_MSG:   java = 1.7.0_40
    ************************************************************/
    2013-12-09 20:58:02,178 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT]
    2013-12-09 20:58:02,676 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    2013-12-09 20:58:02,989 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
    2013-12-09 20:58:02,990 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
    2013-12-09 20:58:02,991 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
    2013-12-09 20:58:02,991 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
    2013-12-09 20:58:02,991 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
    2013-12-09 20:58:02,992 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
    2013-12-09 20:58:03,012 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
    2013-12-09 20:58:03,012 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class org.apache.hadoop.yarn.server.nodemanager.NodeManager
    2013-12-09 20:58:03,053 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
    2013-12-09 20:58:03,104 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
    2013-12-09 20:58:03,104 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system started
    2013-12-09 20:58:03,147 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
    2013-12-09 20:58:03,147 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: per directory file limit = 8192
    2013-12-09 20:58:03,209 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
    2013-12-09 20:58:03,210 FATAL org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Failed to initialize mapreduce-shuffle
    java.lang.IllegalArgumentException: The ServiceName: mapreduce-shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
            at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
            at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:98)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
            at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:218)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:188)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:338)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:386)
    2013-12-09 20:58:03,211 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed in state INITED; cause: java.lang.IllegalArgumentException: The ServiceName: mapreduce-shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
    java.lang.IllegalArgumentException: The ServiceName: mapreduce-shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
            at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
            at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:98)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
            at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:218)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:188)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:338)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:386)
    2013-12-09 20:58:03,212 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl failed in state INITED; cause: java.lang.IllegalArgumentException: The ServiceName: mapreduce-shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
    java.lang.IllegalArgumentException: The ServiceName: mapreduce-shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
            at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
            at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:98)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
            at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:218)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:188)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:338)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:386)
    2013-12-09 20:58:03,213 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED; cause: java.lang.IllegalArgumentException: The ServiceName: mapreduce-shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
    java.lang.IllegalArgumentException: The ServiceName: mapreduce-shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
            at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
            at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:98)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
            at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:218)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:188)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:338)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:386)
    2013-12-09 20:58:03,213 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager metrics system...
    2013-12-09 20:58:03,214 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system stopped.
    2013-12-09 20:58:03,214 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system shutdown complete.
    2013-12-09 20:58:03,214 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
    java.lang.IllegalArgumentException: The ServiceName: mapreduce-shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
            at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
            at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:98)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
            at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:218)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:188)
            at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:338)
            at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:386)
    2013-12-09 20:58:03,215 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NodeManager at server-518/10.10.96.33
    ************************************************************/
    View Code

    检查yarn-site.xml发现mapreduce-shuffle应该为mapreduce_shuffle,修正之后再启动node manager一切正常

    问题:

    1.启动时出现警告信息:

    Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /root/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
    It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
  • 相关阅读:
    NetSuite Batch Process Status
    NetSuite generated bank files and the Bank Reconciliation process
    Global Search file content in Gitlab repositories, search across repositories, search across all files
    FedEx Package Rate Integration with NetSuite direct integrate by WebServices
    git Merge branches
    git tag and NetSuite deployment tracking
    API 读写cookie的方法
    C# 生成缩略图
    解决jquery操作checkbox全选全不选无法勾选问题
    JS读取写入删除Cookie方法
  • 原文地址:https://www.cnblogs.com/littlesuccess/p/3466109.html
Copyright © 2011-2022 走看看