zoukankan      html  css  js  c++  java

  • hadoop-spark-hive-hbase配置相关说明

    1. zookeeper

    • 配置
    • cp app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/zoo_sample.cfg app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/zoo.cfg
    •  
    • vim app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/zoo.cfg
    • dataDir=/home/cdh5/tmp/zookeeper
    • clientPort=2183
    • server.1=ocdata09:2888:3888
    • mkdir -p /home/cdh5/tmp/zookeeper
    • vim /home/cdh5/tmp/zookeeper/myid
    • echo "1" > /home/cdh5/tmp/zookeeper/myid
    • 初始化操作：

    或者 ./runRemoteCmd.sh '~/och200/zookeeper/bin/zkServer-initialize.sh --myid=1' zoo

    • 分发配置
    • ./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/zoo.cfg app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/ zoo
    • ./runRemoteCmd.sh "app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/zkServer.sh start" zoo
    • ./runRemoteCmd.sh 'echo ruok | nc localhost 2183' zoo
    • ./runRemoteCmd.sh "app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/zkServer.sh stop" zoo
    • 启动
    • 验证
    • 停止

    2. HDFS

    • 配置hadoop
    • hdfs-site.xml
    • <property>
    •     <name>dfs.nameservices</name>
    •     <value>cdh5cluster</value>
    •     <description>
    •         Comma-separated list of nameservices.
    •     </description>
    • </property>
    • <property>
    •     <name>dfs.datanode.address</name>
    •     <value>0.0.0.0:50011</value>
    •     <description>
    •       The datanode server address and port for data transfer.
    •       If the port is 0 then the server will start on a free port.
    •     </description>
    • </property>
    • <property>
    •     <name>dfs.datanode.http.address</name>
    •     <value>0.0.0.0:50076</value>
    •     <description>
    •       The datanode http server address and port.
    •       If the port is 0 then the server will start on a free port.
    •     </description>
    • </property>
    • <property>
    •     <name>dfs.datanode.ipc.address</name>
    •     <value>0.0.0.0:50021</value>
    •     <description>
    •       The datanode ipc server address and port.
    •       If the port is 0 then the server will start on a free port.
    •     </description>
    • </property>
    •  
    • <property>
    •    （）
    •     <name>dfs.nameservices</name>
    •     <value>cdh5cluster</value>
    • </property>
    •  
    • <property>
    •   （命名空间中所有NameNode的唯一标示名称。可以配置多个，使用逗号分隔。该名称是可以让DataNode知道每个集群的所有NameNode.当前，每个集群最多只能配置两个NameNode）
    •     <name>dfs.ha.namenodes.cdh5cluster</name>
    •     <value>nn1,nn2</value>
    •     <description></description>
    • </property>
    •  
    • <property>
    •     <name>dfs.namenode.name.dir</name>
    •     <value>file:///data1/cdh5/dfs/name</value>
    •     <description>Determines where on the local filesystem the DFS name node should store the name table.If this is a comma-delimited list of directories,then name table is replicated in all of the directories,for redundancy.</description>
    •     <final>true</final>
    • </property>
    •  
    • <property>
    •       <name>dfs.datanode.data.dir</name>
    • <value>file:///data1/cdh5/dfs/data,file:///data2/cdh5/dfs/data,file:///data3/cdh5/dfs/data</value>
    •       <final>true</final>
    • </property>
    •  
    • <property>
    •       <name>dfs.replication</name>
    •       <value>3</value>
    • </property>
    •  
    • <property>
    •       <name>dfs.permission</name>
    •       <value>true</value>
    • </property>
    •  
    • <property>
    •     <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
    •     <value>true</value>
    •     <description>
    •       Boolean which enables backend datanode-side support for the experimental DistributedFileSystem*getFileVBlockStorageLocations API.
    •     </description>
    • </property>
    •  
    • <property>
    •     <name>dfs.permissions.enabled</name>
    •     <value>false</value>
    •     <description>
    •       If "true", enable permission checking in HDFS.
    •       If "false", permission checking is turned off,
    •       but all other behavior is unchanged.
    •       Switching from one parameter value to the other does not change the mode,
    •       owner or group of files or directories.
    •     </description>
    • </property>
    •  
    • <property>
    • (每个NAMENODE监听的RPC地址)
    •     <name>dfs.namenode.rpc-address.cdh5cluster.nn1</name>
    •     <value>ocdata09:8030</value>
    •     <description>节点NN1的RPC地址</description>
    • </property>
    •  
    • <property>
    •     <name>dfs.namenode.rpc-address.cdh5cluster.nn2</name>
    •     <value>ocdata08:8030</value>
    •     <description>节点NN2的RPC地址</description>
    • </property>
    •  
    • <property>
    •     <name>dfs.namenode.http-address.cdh5cluster.nn1</name>
    •     <value>ocdata09:50082</value>
    •     <description>节点NN1的HTTP地址</description>
    • </property>
    •  
    • <property>
    •     <name>dfs.namenode.http-address.cdh5cluster.nn2</name>
    •     <value>ocdata08:50082</value>
    •     <description>节点NN2的HTTP地址</description>
    • </property>
    •  
    • <property>
    • （这是NameNode读写JNs的uri。通过这个uri，NameNodes可以读写edit log内容。URI的格式”qjournal://host1:port1;host2:port2;host3:port3/journalId”。这里的host1、host2、host3指的是Journal Node的地址，这里必须是奇数个，至少3个;其中journaId是集群的唯一标示符，对于多个联邦命名空间，也使用同一个journaId。配置如下J
    •     <name>dfs.namenode.shared.edits.dir</name>
    • <value>qjournal://ocdata05:8488;ocdata06:8488;ocdata07:8488/cdh5cluster</value>
    •     <description>采用3个journalnode节点存储元数据，这是IP与端口</description>
    • </property>

     

    • <property>
    •     <name>dfs.journalnode.edits.dir</name>
    •     <value>/home/cdh5/journaldata/jn</value>
    •     <description>journaldata的存储路径</description>
    • </property>
    •  
    • <property>
    •     <name>dfs.journalnode.rpc-address</name>
    •     <value>0.0.0.0:8488</value>
    • </property>
    •  
    • <property>
    •     <name>dfs.journalnode.http-address</name>
    •     <value>0.0.0.0:8483</value>
    • </property>
    •  
    • <property>
    •     <name>dfs.client.failover.proxy.provider.cdh5cluster</name>
    •     <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    •     <description>该类用来判断哪个namenode处于生效状态</description>
    • </property>
    •  
    • <property>
    •     <name>dfs.ha.fencing.methods</name>
    •     <value>shell(/bin/true)</value>
    • </property>
    •  
    • <property>
    •     <name>dfs.ha.fencing.ssh.connect-timeout</name>
    •     <value>10000</value>
    • </property>
    •  
    • <property>
    •     <name>dfs.ha.automatic-failover.enabled</name>
    •     <value>true</value>
    •     <description>
    •       Whether automatic failover is enabled. See the HDFS High
    •       Availability documentation for details on automatic HA
    •       configuration.
    •     </description>
    • </property>
    •  
    • <property>
    •     <name>ha.zookeeper.quorum</name>
    •     <value>ocdata09:2183</value>
    •     <description>1个zookeeper节点</description>
    • </property>
    •  
    • <property>
    •     <name>dfs.datanode.max.xcievers</name>
    •     <value>4096</value>
    • </property>
    •  
    • <property>
    •     <name>dfs.datanode.max.transfer.threads</name>
    •     <value>4096</value>
    •     <description>
    •           Specifies the maximum number of threads to use for transferring data
    •           in and out of the DN.
    •     </description>
    • </property>
    •  
    • <property>
    •     <name>dfs.blocksize</name>
    •     <value>64m</value>
    •     <description>
    •         The default block size for new files, in bytes.
    •         You can use the following suffix (case insensitive):
    •         k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.),
    •         Or provide complete size in bytes (such as 134217728 for 128 MB).
    •     </description>
    • </property>
    •  
    • <property>
    •     <name>dfs.namenode.handler.count</name>
    •     <value>20</value>
    •     <description>The number of server threads for the namenode.</description>
    • </property>
    • <property>
    • （这是一个描述集群中NameNode节点的URI(包括协议、主机名称端口号—)集群里面的每一台机器都要知道NAMENODE的地址。DataNode节点会先在NAMENODE上注册，这样它们的数据才可以被使用。独立的客户端程序通过这个URI跟DATANODE交互，以取得文件的块列表。）
    •     <name>fs.defaultFS</name>
    •     <value>hdfs://cdh5cluster</value>
    • </property>
    •  
    • <property>
    • （hadoop文件系统依赖的基础配置，很多路径都依赖它。如果hdfs-site.xml中不配置namenode和datanode的存放位置，默认就放在这个路径中）
    •     <name>hadoop.tmp.dir</name>
    •     <value>/home/cdh5/tmp/hadoop/hadoop-${user.name}</value>
    • </property>
    •  
    • <property>
    • core-site.xml

                (对本地jar包进行加载)

    •     <name>io.native.lib.available</name>
    •     <value>true</value>
    •     <description>Should native hadoop libraries, if present, be used.</description>
    • </property>
    • (压缩和解压编码类列表，用逗号分隔，这些类是使用java ServiceLoader加载，如果不设置就为null)
    • <property>
    •     <name>io.compression.codecs</name>         <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.SnappyCodec</value>
    • </property>
    • ocdata05
    • ocdata06
    • ocdata07
    • ocdata08
    • ocdata09
    • ocdata05
    • ocdata06
    • export JAVA_HOME=/home/cdh5/app/jdk1.7.0_21
    • ./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc all
    • slaves
    • masters
    • hadoop-env.sh
    • 分发
    • 初始化HDFS：

    主节点执行

    app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/hdfs zkfc -formatZK

    ./runRemoteCmd.sh 'app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/hadoop-daemon.sh start journalnode' jn

    app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/hdfs namenode -format -initializeSharedEdits

    app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/hadoop-daemon.sh start namenode

    备节点执行

    app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/hdfs namenode -bootstrapStandby

    完成

    app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/hadoop-daemon.sh stop namenode

    ./runRemoteCmd.sh 'app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/hadoop-daemon.sh stop journalnode' jn

    • 启动HDFS
    • app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/start-dfs.sh
    • http://10.1.253.99:50082/dfshealth.html (active)
    • http://10.1.253.98:50082/dfshealth.html (standby)
    • http://10.1.253.97:8483/journalstatus.jsp
    • http://10.1.253.96:8483/journalstatus.jsp
    • http://10.1.253.95:8483/journalstatus.jsp
    • app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/stop-dfs.sh
    • 验证：
    • 停止HDFS

    3. Yarn

    配置YARN

    • mapred-site.xml
    • cp app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop/mapred-site.xml.template app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop/mapred-site.xml
    •  
    • vim app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop/mapred-site.xml
    •  
    • <property>
    •     <name>mapreduce.framework.name</name>
    •     <value>yarn</value>
    • </property>
    •  
    • <property>
    •     <name>mapreduce.shuffle.port</name>
    •     <value>8350</value>
    • </property>
    •  
    • <property>
    •     <name>mapreduce.jobhistory.address</name>
    •     <value>0.0.0.0:10121</value>
    • </property>
    •  
    • <property>
    •     <name>mapreduce.jobhistory.webapp.address</name>
    •     <value>0.0.0.0:19868</value>
    • </property>
    •  
    • <property>
    •     <name>mapreduce.jobtracker.http.address</name>
    •     <value>0.0.0.0:50330</value>
    • </property>
    •  
    • <property>
    •     <name>mapreduce.tasktracker.http.address</name>
    •     <value>0.0.0.0:50360</value>
    • </property>
    • vim app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop/yarn-site.xml
    •  
    • <!-- Resource Manager Configs -->
    • <property>
    •     <name>yarn.resourcemanager.connect.retry-interval.ms</name>
    •     <value>2000</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.ha.enabled</name>
    •     <value>true</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
    •     <value>true</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
    •     <value>true</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.cluster-id</name>
    •     <value>yarn-rm-cluster</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.ha.rm-ids</name>
    •     <value>rm1,rm2</value>
    • </property>
    • <property>
    •     <description>Id of the current ResourceManager. Must be set explicitly on each ResourceManager to the appropriate value.</description>
    •     <name>yarn.resourcemanager.ha.id</name>
    •     <value>rm1</value>
    •     <!-- rm1上配置为rm1, rm2上配置rm2 -->
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.recovery.enabled</name>
    •     <value>true</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.store.class</name>
    •     <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.zk.state-store.address</name>
    •     <value>ocdata09:2183</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.zk-address</name>
    •     <value>ocdata09:2183</value>
    • </property>
    • <property>
    •     <name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
    •     <value>5000</value>
    • </property>
    • <!-- RM1 configs -->
    • <property>
    •     <name>yarn.resourcemanager.address.rm1</name>
    •     <value>ocdata08:23140</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.scheduler.address.rm1</name>
    •     <value>ocdata08:23130</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.webapp.address.rm1</name>
    •     <value>ocdata08:23188</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
    •     <value>ocdata08:23125</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.admin.address.rm1</name>
    •     <value>ocdata08:23141</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.ha.admin.address.rm1</name>
    •     <value>ocdata08:23142</value>
    • </property>
    • <!-- RM2 configs -->
    • <property>
    •     <name>yarn.resourcemanager.address.rm2</name>
    •     <value>ocdata09:23140</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.scheduler.address.rm2</name>
    •     <value>ocdata09:23130</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.webapp.address.rm2</name>
    •     <value>ocdata09:23188</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
    •     <value>ocdata09:23125</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.admin.address.rm2</name>
    •     <value>ocdata09:23141</value>
    • </property>
    • <property>
    •     <name>yarn.resourcemanager.ha.admin.address.rm2</name>
    •     <value>ocdata09:23142</value>
    • </property>
    • <!-- Node Manager Configs -->
    • <property>
    •     <description>Address where the localizer IPC is.</description>
    •     <name>yarn.nodemanager.localizer.address</name>
    •     <value>0.0.0.0:23344</value>
    • </property>
    • <property>
    •     <description>NM Webapp address.</description>
    •     <name>yarn.nodemanager.webapp.address</name>
    •     <value>0.0.0.0:23999</value>
    • </property>
    • <property>
    •     <name>yarn.nodemanager.aux-services</name>
    •     <value>mapreduce_shuffle</value>
    • </property>
    • <property>
    •     <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    •     <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    • </property>
    • <property>
    •     <name>yarn.nodemanager.local-dirs</name>
    •     <value>/tmp/pseudo-dist/yarn/local</value>
    • </property>
    • <property>
    •     <name>yarn.nodemanager.log-dirs</name>
    •     <value>/tmp/pseudo-dist/yarn/log</value>
    • </property>
    • ./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc all
    • app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/start-yarn.sh
    • yarn-site.xml
    • 分发
    • Yarn的启动停止 YARN不需要初始化，登录主节点执行

    cdh5 yarn的ha需要手动启动备节点

        ./runRemoteCmd.sh "cd app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin; ./yarn-daemon.sh start resourcemanager" rm2

    验证

    http://10.1.253.98:23188/cluster (有节点列表，active)

    http://10.1.253.99:23188/cluster (无节点列表，standby)

     

    app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/hadoop jar app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar TestDFSIO -write -nrFiles 40 -fileSize 20MB

    停止YARN

    app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/stop-yarn.sh

    手动停止备节点

    ./runRemoteCmd.sh "cd app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin; ./yarn-daemon.sh stop resourcemanager" rm2

    4. hive

    • 配置
    • cp hive-env.sh.template hive-env.sh
    • vim hive-env.sh
    •  
    • export HADOOP_HOME=/home/cdh5/app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT
    •  
    • cp hive-default.xml.template hive-site.xml
    • vim hive-site.xml

    删除其他配置项，只保留：

    <property>

    （配置元数据库,一般为mysql）

        <name>javax.jdo.option.ConnectionURL</name>

    <value>jdbc:mysql://10.1.252.69:3306/cdh5?createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=UTF-8</value>

        </property>

     

    <property>

        (配置元数据库的Driver)

        <name>javax.jdo.option.ConnectionDriverName</name>

        <value>com.mysql.jdbc.Driver</value>

        <description>Driver class name for a JDBC metastore</description>

    </property>

     

    <property>

          (配置元数据库的名称)

        <name>javax.jdo.option.ConnectionUserName</name>

        <value>cdh5</value>

        <description>username to use against metastore database</description>

    </property>

     

    <property>

           (配置元数据库的密码)

        <name>javax.jdo.option.ConnectionPassword</name>

        <value>cdh5</value>

        <description>password to use against metastore database</description>

    </property>

    • 元数据库配置
    • CREATE USER cdh5 IDENTIFIED BY 'cdh5';
    • CREATE DATABASE cdh5;
    • alter database cdh5 character set latin1;
    • grant all privileges on *.* to cdh5@"%" identified by "cdh5";
    • flush privileges;
    • scp mysql-connector-java-5.1.26.jar cdh5@10.1.253.99:/home/cdh5/app/ochadoop-och3.0.0-SNAPSHOT/hive-0.12.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/lib/
    • ./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/hive-0.12.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT app/ochadoop-och3.0.0-SNAPSHOT/ hive
    • nohup ./hiveserver2 &
    • jdbc:
    • jdbc:hive2://10.1.253.99:10000/default
    • org.apache.hive.jdbc.HiveDriver
    • lib: Hadoop和hive下所有jar包
    •  
    • !connect jdbc:hive2://10.1.253.99:10000/default
    • Enter username:dmp
    • Enter password:dmp
    •  
    • show tables;
    • +--------------+
    • |   tab_name   |
    • +--------------+
    • | shaoaq_test  |
    • +--------------+
    •  
    • select * from shaoaq_test;
    • +-----+
    • | id  |
    • +-----+
    • +-----+
    • 上传jdbc jar包
    • 分发
    • 启动
    • 验证

    5. hbase

    • 配置
    • vim app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/regionservers
    •  
    • ocdata05
    • ocdata06
    • ocdata07
    •  
    • vim app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/backup-masters
    •  
    • ocdata08
    •  
    • vim app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/hbase-site.xml
    •  
    • <property>
    •     <name>hbase.rootdir</name>
    •     <value>hdfs://cdh5cluster/hbase</value>
    • </property>
    • <property> 
    •     <name>hbase.cluster.distributed</name> 
    •     <value>true</value> 
    • </property>
    • <property>
    •     <name>hbase.zookeeper.quorum</name>
    •     <value>ocdata09</value>
    • </property>
    • <property>
    •     <name>hbase.zookeeper.property.clientPort</name>
    •     <value>2183</value>
    • </property>
    • <property>
    •     <name>hbase.regionserver.port</name>
    •     <value>60328</value>
    • </property>
    • <property>
    •     <name>hbase.regionserver.info.port</name>
    •     <value>62131</value>
    • </property>
    •  
    • vim app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/hbase-env.sh
    •  
    • export JAVA_HOME=/home/cdh5/app/jdk1.7.0_51
    • export HBASE_CLASSPATH=/home/cdh5/app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop
    • export HBASE_HOME=/home/cdh5/app/hbase
    • export HADOOP_HOME=/home/cdh5/app/hadoop
    • export HADOOP_CONF_DIR=${HADOOP_HOME}/conf
    • export HBASE_LIBRARY_PATH=${HBASE_HOME}/lib/native
    • export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HBASE_HOME}/lib/native
    • export HBASE_MANAGES_ZK=false
    • ./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/ all
    • ./start-hbase.sh
    • ./hbase shell
    • create 'hb_test', 'cf'
    • put 'hb_test','row1','cf:a','123'
    • get 'hb_test','row1'
    • COLUMN                            CELL
    • cf:a                             timestamp=1395204538429, value=123
    • 1 row(s) in 0.0490 seconds
    • quit
    • ./stop-hbase.sh
    • 分发配置
    • 启动
    • 验证
    • 停止

    5. spark

    spark当前可解压即用，yarn-client模式无需分发，只需修改客户端若干配置；

    • spark-1.1.0 on yarn的几个配置说明如下：
    • vim spark-env.sh
    •  
    • MASTER：部署模式，yarn-client/yarn-cluster/local
    • HADOOP_CONF_DIR：（必填）hadoop配置文件目录
    • SCALA_HOME：scala安装路径
    • SPARK_EXECUTOR_INSTANCES：spark申请的yarn worker总数
    • SPARK_EXECUTOR_CORES：每个worker申请的vcore数目
    • SPARK_EXECUTOR_MEMORY：每个worker申请的内存大小
    • SPARK_DRIVER_MEMORY：spark申请的appMaster内存大小
    • SPARK_YARN_APP_NAME：yarn中显示的spark任务名称
    • SPARK_YARN_QUEUE：spark任务队列
    • SPARK_SUBMIT_LIBRARY_PATH：spark任务执行时需要的库目录，如hadoop的native目录
    • SPARK_CLASSPATH：spark任务的classpath
    • SPARK_JAVA_OPTS：JVM进程参数，如gc类型、gc日志、dmp输出等
    • SPARK_HISTORY_OPTS：spark history-server配置参数，一般需要指定webUI端口、记录个数以及Event存储目录等
    •  
    • vim spark-defaults.conf
    •  
    • spark.local.dir：spark任务执行时的本地临时目录
    • spark.yarn.executor.memoryOverhead：每个worker的堆外内存大小，单位MB，yarn模式下需配置以防止内存溢出
    • spark.eventLog.enabled：是否记录Spark事件，用 于应用程序在完成后重构webUI
    • spark.eventLog.dir：保存日志相关信息的路径，可以是hdfs://开头的HDFS路径，也可以是file://开头的本地路径，都需要提前创建
    • spark.eventLog.compress：是否压缩记录Spark事件，前提spark.eventLog.enabled为true，默认使用snappy
    • 启停thrift-server：

    使用spark-sql/thrift-server组件前需将hive-site.xml复制到$SPARK_HOME/conf目录下以使用hive的元数据和若干配置如server端口，可能需要去掉其中的一些多余或不支持的配置项，请留意；

        $SPARK_HOME/sbin/start-thriftserver.sh

        $SPARK_HOME/sbin/stop-thriftserver.sh

    • 启停history-server：
    • $SPARK_HOME/sbin/start-history-server.sh
    • $SPARK_HOME/sbin/stop-history-server.sh
    • 注意：
    1. 如hadoop中启用了lzo压缩需将hadoop-lzo-*.jar复制到SPARK_HOME/lib/目录下；
    2. SPARK-1.1.0版本中spark-examples-*.jar关联的thrift版本与spark-assembly-*.jar不一致，需删除；
    • 配置样例：

    spark-env.sh

        MASTER="yarn-client"

        SPARK_HOME=/home/ochadoop/app/spark

        SCALA_HOME=/home/ochadoop/app/scala

        JAVA_HOME=/home/ochadoop/app/jdk

        HADOOP_HOME=/home/ochadoop/app/hadoop

        HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

     

        SPARK_EXECUTOR_INSTANCES=50

        SPARK_EXECUTOR_CORES=2

        SPARK_EXECUTOR_MEMORY=4G

        SPARK_DRIVER_MEMORY=3G

        SPARK_YARN_APP_NAME="Spark-1.1.0"

        #export SPARK_YARN_QUEUE="default"

     

        SPARK_SUBMIT_LIBRARY_PATH=$SPARK_LIBRARY_PATH:$HADOOP_HOME/lib/native

        SPARK_JAVA_OPTS="-verbose:gc -XX:-UseGCOverheadLimit -XX:+UseCompressedOops -XX:-PrintGCDetails -XX:+PrintGCTimeStamps $SPARK_JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ochadoop/app/spark/`date +%m%d%H%M%S`.hprof"

        export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=1000 -Dspark.history.fs.logDirectory=hdfs://testcluster/eventLog"

    spark-defaults.conf

        spark.serializer                    org.apache.spark.serializer.KryoSerializer

        spark.local.dir                     /data2/ochadoop/data/pseudo-dist/spark/local,/data3/ochadoop/data/pseudo-dist/spark/local

        spark.io.compression.codec          snappy

        spark.speculation                   false

        spark.yarn.executor.memoryOverhead  512

        #spark.storage.memoryFraction       0.4

        spark.eventLog.enabled              true

        spark.eventLog.dir                  hdfs://testcluster/eventLog

        spark.eventLog.compress             true

    如下命令都是用root身份安装，或者在命令前加上sudo
    采用yum安装方式安装
    yum install mysql     #安装mysql客户端
    yum install mysql-server  #安装mysql服务端
    判断MySQL是否已经安装好：
    chkconfig --list|grep mysql
    启动mysql服务：
    service mysqld start或者/etc/init.d/mysqld start
    检查是否启动mysql服务：
    /etc/init.d/mysqld status
    设置MySQL开机启动：
    chkconfig mysqld on 
    检查设置MySQL开机启动是否配置成功：
    chkconfig --list|grep mysql
    显示2 3 4 5为on
    创建root管理员
    mysqladmin -uroot password root
    登录
    mysql -uroot -proot
  • 相关阅读:
    UOS、鸿蒙、麒麟全面出击，国产系统能否干掉Windows？
    黑客给疫情添乱
    人工智能与信息安全
    作为一个程序员，告诉你一些编码知识
    Linus Torvalds 宣布新版Linux系统内核发布
    linux系统root密码忘记了怎么办
    DevOps与NoOps现状分析
    Nginx服务详细介绍
    博客园“可运行"代码
    让setTimeout支持链式
  • 原文地址：https://www.cnblogs.com/yangsy0915/p/4867423.html
Copyright © 2011-2022 走看看