kafka_2.10-0.8.1.1.tgz的1或3节点集群的下载、安装和配置（图文详细教程）绝对干货
一、安装前准备
1.1 示例机器
二、 JDK7 安装
1.1 下载地址
下载地址： http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
[hadoop@hadoop1 ~]$ cd [hadoop@hadoop1 ~]$ rz -bash: rz: command not found [hadoop@hadoop1 ~]$ su root Password: [root@hadoop1 hadoop]# yum -y install lrzsz
hadoop2和hadoop3操作一样，不多赘述。
在安装jdk之前，先要卸载自带的openjdk。
Centos 6.5下的OPENJDK卸载和SUN的JDK安装、环境变量配置
[hadoop@hadoop1 ~]$ pwd /home/hadoop [hadoop@hadoop1 ~]$ ll total 0 [hadoop@hadoop1 ~]$ rz [hadoop@hadoop1 ~]$ ll total 149920 -rw-r--r--. 1 hadoop hadoop 153512879 Oct 23 2015 jdk-7u79-linux-x64.tar.gz [hadoop@hadoop1 ~]$
hadoop2和hadoop3操作一样，不多赘述。
1.2 安装
解压缩
cd /home/hadoop tar zxvf jdk-7u79-linux-x64.gz
[hadoop@hadoop1 ~]$ pwd /home/hadoop [hadoop@hadoop1 ~]$ ll total 149920 -rw-r--r--. 1 hadoop hadoop 153512879 Oct 23 2015 jdk-7u79-linux-x64.tar.gz [hadoop@hadoop1 ~]$ tar -zxvf jdk-7u79-linux-x64.tar.gz
hadoop2和hadoop3操作一样，不多赘述。
建立软连接
ln -s jdk1.7.0_79 jdk
[hadoop@hadoop1 ~]$ ll total 4 drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 [hadoop@hadoop1 ~]$ ln -s jdk1.7.0_79/ jdk [hadoop@hadoop1 ~]$ ll total 4 lrwxrwxrwx. 1 hadoop hadoop 12 Apr 22 22:27 jdk -> jdk1.7.0_79/ drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 [hadoop@hadoop1 ~]$
hadoop2和hadoop3操作一样，不多赘述。
设置环境变量
vim /etc/profile
hadoop2和hadoop3操作一样，不多赘述。
添加如下：
#jdk
export JAVA_HOME=/home/hadoop/jdk export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export PATH=$JAVA_HOME/bin:$PATH
hadoop2和hadoop3操作一样，不多赘述。
设置环境变量
source /etc/profile
hadoop2和hadoop3操作一样，不多赘述。
执行 java -version， 如果有版本显示则说明安装成功
三、安装 zookeeper
1、安装 zookeeper
[hadoop@hadoop1 ~]$ pwd /home/hadoop [hadoop@hadoop1 ~]$ ll total 4 lrwxrwxrwx. 1 hadoop hadoop 12 Apr 22 22:27 jdk -> jdk1.7.0_79/ drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 [hadoop@hadoop1 ~]$ rz [hadoop@hadoop1 ~]$ ll total 27408 lrwxrwxrwx. 1 hadoop hadoop 12 Apr 22 22:27 jdk -> jdk1.7.0_79/ drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 -rw-r--r--. 1 hadoop hadoop 28060242 Mar 20 10:24 zookeeper-3.4.5-cdh5.5.4.gz [hadoop@hadoop1 ~]$
hadoop2和hadoop3都去操作，这里不多赘述。
1.1 安装
解压
cd /home/hadoop tar zxvf zookeeper-3.4.5-cdh5.5.4.gz
[hadoop@hadoop1 ~]$ ll total 27408 lrwxrwxrwx. 1 hadoop hadoop 12 Apr 22 22:34 jdk -> jdk1.7.0_79/ drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 -rw-r--r--. 1 hadoop hadoop 28060242 Mar 20 10:24 zookeeper-3.4.5-cdh5.5.4.gz [hadoop@hadoop1 ~]$ tar -zxvf zookeeper-3.4.5-cdh5.5.4.gz
hadoop2和hadoop3都去操作，这里不多赘述。
1.2 建立软连接
[hadoop@hadoop1 ~]$ pwd /home/hadoop [hadoop@hadoop1 ~]$ ll total 8 lrwxrwxrwx. 1 hadoop hadoop 12 Apr 22 22:27 jdk -> jdk1.7.0_79/ drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 drwxr-xr-x. 14 hadoop hadoop 4096 Apr 26 2016 zookeeper-3.4.5-cdh5.5.4 [hadoop@hadoop1 ~]$ ln -s zookeeper-3.4.5-cdh5.5.4/ zookeeper [hadoop@hadoop1 ~]$ ll total 8 lrwxrwxrwx. 1 hadoop hadoop 12 Apr 22 22:27 jdk -> jdk1.7.0_79/ drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 lrwxrwxrwx. 1 hadoop hadoop 25 Apr 22 22:49 zookeeper -> zookeeper-3.4.5-cdh5.5.4/ drwxr-xr-x. 14 hadoop hadoop 4096 Apr 26 2016 zookeeper-3.4.5-cdh5.5.4 [hadoop@hadoop1 ~]$
hadoop2和hadoop3都去操作，这里不多赘述。
#zookeeper export ZOOKEEPER_HOME=/home/hadoop/zookeeper export PATH=$PATH:$ZOOKEEPER_HOME/bin
hadoop2和hadoop3都去操作，这里不多赘述。
[root@hadoop1 hadoop]# vim /etc/profile [root@hadoop1 hadoop]# source /etc/profile
hadoop2和hadoop3都去操作，这里不多赘述。
1.3 修改配置文件
复制配置文件
cd /home/hadoop/zookeeper/conf cp zoo_sample.cfg zoo.cfg
[hadoop@hadoop1 ~]$ ll total 8 lrwxrwxrwx. 1 hadoop hadoop 12 Apr 22 22:27 jdk -> jdk1.7.0_79/ drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 lrwxrwxrwx. 1 hadoop hadoop 25 Apr 22 22:49 zookeeper -> zookeeper-3.4.5-cdh5.5.4/ drwxr-xr-x. 14 hadoop hadoop 4096 Apr 26 2016 zookeeper-3.4.5-cdh5.5.4 [hadoop@hadoop1 ~]$ cd zookeeper [hadoop@hadoop1 zookeeper]$ cd conf/ [hadoop@hadoop1 conf]$ pwd /home/hadoop/zookeeper/conf [hadoop@hadoop1 conf]$ ll total 12 -rw-rw-r--. 1 hadoop hadoop 535 Apr 26 2016 configuration.xsl -rw-rw-r--. 1 hadoop hadoop 2693 Apr 26 2016 log4j.properties -rw-rw-r--. 1 hadoop hadoop 808 Apr 26 2016 zoo_sample.cfg [hadoop@hadoop1 conf]$ cp zoo_sample.cfg zoo.cfg [hadoop@hadoop1 conf]$ ll total 16 -rw-rw-r--. 1 hadoop hadoop 535 Apr 26 2016 configuration.xsl -rw-rw-r--. 1 hadoop hadoop 2693 Apr 26 2016 log4j.properties -rw-rw-r-- 1 hadoop hadoop 808 Apr 23 08:45 zoo.cfg -rw-rw-r--. 1 hadoop hadoop 808 Apr 26 2016 zoo_sample.cfg [hadoop@hadoop1 conf]$
hadoop2和hadoop3都去操作，这里不多赘述。
[hadoop@hadoop1 conf]$ pwd /home/hadoop/zookeeper/conf [hadoop@hadoop1 conf]$ ll total 16 -rw-rw-r--. 1 hadoop hadoop 535 Apr 26 2016 configuration.xsl -rw-rw-r--. 1 hadoop hadoop 2693 Apr 26 2016 log4j.properties -rw-rw-r-- 1 hadoop hadoop 808 Apr 23 08:45 zoo.cfg -rw-rw-r--. 1 hadoop hadoop 808 Apr 26 2016 zoo_sample.cfg [hadoop@hadoop1 conf]$ vim zoo.cfg
hadoop2和hadoop3都去操作，这里不多赘述。
修改参数
tickTime=2000 initLimit=10 syncLimit=5 dataDir=/home/hadoop/zookeeper/data dataLogDir=/home/hadoop/zookeeper/logs clientPort=2181 server.1=hadoop1:2888:3888 server.2=hadoop2:2888:3888 server.3=hadoop3:2888:3888
tickTime 时长单位为毫秒， 为 zk 使用的基本时间度量单位。 例如， 1 * tickTime 是客户端与 zk 服务端的心跳时间， 2 * tickTime 是客户端会话的超时时间。 tickTime 的默认值为2000 毫秒， 更低的 tickTime 值可以更快地发现超时问题， 但也会导致更高的网络流量(心跳消息)和更高的 CPU 使用率(会话的跟踪处理） 。
clientPort zk 服务进程监听的 TCP 端口， 默认情况下， 服务端会监听 2181 端口。dataDir 无默认配置， 必须配置， 用于配置存储快照文件的目录。 如果没有配置dataLogDir， 那么事务日志也会存储在此目录。
hadoop2和hadoop3都去操作，这里不多赘述。
创建目录
mkdir -p /home/hadoop/zookeeper/data mkdir -p /home/hadoop/zookeeper/logs
hadoop2和hadoop3都去操作，这里不多赘述。
1.3 创建 ID 文件
在 dataDir 目录下（即/home/hadoop/zookeeper/data下）添加 myid 文件， 并把 server.x 中的 x 数字写入文件中
[hadoop@hadoop1 data]$ pwd /home/hadoop/zookeeper/data [hadoop@hadoop1 data]$ ll total 0 [hadoop@hadoop1 data]$ vim myid 1
hadoop2和hadoop3都去操作，这里不多赘述。
2 启动 zookeeper
2.1 启动
cd /home/hadoop/zookeeper/bin ./zkServer.sh start
启动 ZK 服务: ./zkServer.sh start
查看 ZK 服务状态: ./zkServer.sh status
停止 ZK 服务: ./zkServer.sh stop
重启 ZK 服务: ./zkServer.sh restart
hadoop2和hadoop3都去操作，这里不多赘述。
2.1 测试
在 hadoop1 机器上的 zookeeper 中创建节点
cd /home/hadoop/zookeeper/bin ./zkCli.sh create /hello hehe
这里不演示
在 hadoop2 机器上查看节点
cd /home/hadoop/zookeeper/bin ./zkCli.sh
这里不演示
get /hello //如果有值则说明 zookeeper 运行正常
2.1 进程的意义
这里不多赘述。
四、kafka的安装
4.1 安装
在 hadoop1、hadoop2和hadoop3 机器上安装
cd /home/hadoop tar zxvf kafka_2.11-0.8.2.2.tgz ln -s kafka_2.11-0.8.2.2 kafka
[hadoop@hadoop1 ~]$ ll total 28 drwxr-xr-x 8 hadoop hadoop 4096 Apr 26 2016 apache-flume-1.6.0-cdh5.5.4-bin drwxrwxr-x 5 hadoop hadoop 4096 Apr 23 18:34 data lrwxrwxrwx 1 hadoop hadoop 32 Apr 23 18:14 flume -> apache-flume-1.6.0-cdh5.5.4-bin/ lrwxrwxrwx 1 hadoop hadoop 22 Apr 23 09:08 hadoop -> hadoop-2.6.0-cdh5.5.4/ drwxr-xr-x 18 hadoop hadoop 4096 Apr 23 09:50 hadoop-2.6.0-cdh5.5.4 lrwxrwxrwx 1 hadoop hadoop 21 Apr 23 16:55 hbase -> hbase-1.0.0-cdh5.5.4/ drwxr-xr-x 27 hadoop hadoop 4096 Apr 23 17:27 hbase-1.0.0-cdh5.5.4 lrwxrwxrwx 1 hadoop hadoop 20 Apr 23 15:27 hive -> hive-1.1.0-cdh5.5.4/ drwxr-xr-x 10 hadoop hadoop 4096 Apr 26 2016 hive-1.1.0-cdh5.5.4 lrwxrwxrwx. 1 hadoop hadoop 12 Apr 22 22:27 jdk -> jdk1.7.0_79/ drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 lrwxrwxrwx. 1 hadoop hadoop 25 Apr 22 22:49 zookeeper -> zookeeper-3.4.5-cdh5.5.4/ drwxr-xr-x. 16 hadoop hadoop 4096 Apr 23 08:57 zookeeper-3.4.5-cdh5.5.4 [hadoop@hadoop1 ~]$ rz [hadoop@hadoop1 ~]$ ll total 15436 drwxr-xr-x 8 hadoop hadoop 4096 Apr 26 2016 apache-flume-1.6.0-cdh5.5.4-bin drwxrwxr-x 5 hadoop hadoop 4096 Apr 23 18:34 data lrwxrwxrwx 1 hadoop hadoop 32 Apr 23 18:14 flume -> apache-flume-1.6.0-cdh5.5.4-bin/ lrwxrwxrwx 1 hadoop hadoop 22 Apr 23 09:08 hadoop -> hadoop-2.6.0-cdh5.5.4/ drwxr-xr-x 18 hadoop hadoop 4096 Apr 23 09:50 hadoop-2.6.0-cdh5.5.4 lrwxrwxrwx 1 hadoop hadoop 21 Apr 23 16:55 hbase -> hbase-1.0.0-cdh5.5.4/ drwxr-xr-x 27 hadoop hadoop 4096 Apr 23 17:27 hbase-1.0.0-cdh5.5.4 lrwxrwxrwx 1 hadoop hadoop 20 Apr 23 15:27 hive -> hive-1.1.0-cdh5.5.4/ drwxr-xr-x 10 hadoop hadoop 4096 Apr 26 2016 hive-1.1.0-cdh5.5.4 lrwxrwxrwx. 1 hadoop hadoop 12 Apr 22 22:27 jdk -> jdk1.7.0_79/ drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 -rw-r--r-- 1 hadoop hadoop 15773865 Mar 20 10:24 kafka_2.11-0.8.2.2.tgz lrwxrwxrwx. 1 hadoop hadoop 25 Apr 22 22:49 zookeeper -> zookeeper-3.4.5-cdh5.5.4/ drwxr-xr-x. 16 hadoop hadoop 4096 Apr 23 08:57 zookeeper-3.4.5-cdh5.5.4 [hadoop@hadoop1 ~]$
hadoop2和hadoop3操作，不多赘述。
[hadoop@hadoop1 ~]$ ll total 15436 drwxr-xr-x 8 hadoop hadoop 4096 Apr 26 2016 apache-flume-1.6.0-cdh5.5.4-bin drwxrwxr-x 5 hadoop hadoop 4096 Apr 23 18:34 data lrwxrwxrwx 1 hadoop hadoop 32 Apr 23 18:14 flume -> apache-flume-1.6.0-cdh5.5.4-bin/ lrwxrwxrwx 1 hadoop hadoop 22 Apr 23 09:08 hadoop -> hadoop-2.6.0-cdh5.5.4/ drwxr-xr-x 18 hadoop hadoop 4096 Apr 23 09:50 hadoop-2.6.0-cdh5.5.4 lrwxrwxrwx 1 hadoop hadoop 21 Apr 23 16:55 hbase -> hbase-1.0.0-cdh5.5.4/ drwxr-xr-x 27 hadoop hadoop 4096 Apr 23 17:27 hbase-1.0.0-cdh5.5.4 lrwxrwxrwx 1 hadoop hadoop 20 Apr 23 15:27 hive -> hive-1.1.0-cdh5.5.4/ drwxr-xr-x 10 hadoop hadoop 4096 Apr 26 2016 hive-1.1.0-cdh5.5.4 lrwxrwxrwx. 1 hadoop hadoop 12 Apr 22 22:27 jdk -> jdk1.7.0_79/ drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 -rw-r--r-- 1 hadoop hadoop 15773865 Mar 20 10:24 kafka_2.11-0.8.2.2.tgz lrwxrwxrwx. 1 hadoop hadoop 25 Apr 22 22:49 zookeeper -> zookeeper-3.4.5-cdh5.5.4/ drwxr-xr-x. 16 hadoop hadoop 4096 Apr 23 08:57 zookeeper-3.4.5-cdh5.5.4 [hadoop@hadoop1 ~]$ tar -zxvf kafka_2.11-0.8.2.2.tgz
hadoop2和hadoop3操作，不多赘述。
[hadoop@hadoop1 ~]$ ll total 32 drwxr-xr-x 8 hadoop hadoop 4096 Apr 26 2016 apache-flume-1.6.0-cdh5.5.4-bin drwxrwxr-x 5 hadoop hadoop 4096 Apr 23 18:34 data lrwxrwxrwx 1 hadoop hadoop 32 Apr 23 18:14 flume -> apache-flume-1.6.0-cdh5.5.4-bin/ lrwxrwxrwx 1 hadoop hadoop 22 Apr 23 09:08 hadoop -> hadoop-2.6.0-cdh5.5.4/ drwxr-xr-x 18 hadoop hadoop 4096 Apr 23 09:50 hadoop-2.6.0-cdh5.5.4 lrwxrwxrwx 1 hadoop hadoop 21 Apr 23 16:55 hbase -> hbase-1.0.0-cdh5.5.4/ drwxr-xr-x 27 hadoop hadoop 4096 Apr 23 17:27 hbase-1.0.0-cdh5.5.4 lrwxrwxrwx 1 hadoop hadoop 20 Apr 23 15:27 hive -> hive-1.1.0-cdh5.5.4/ drwxr-xr-x 10 hadoop hadoop 4096 Apr 26 2016 hive-1.1.0-cdh5.5.4 lrwxrwxrwx. 1 hadoop hadoop 12 Apr 22 22:27 jdk -> jdk1.7.0_79/ drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 drwxr-xr-x 5 hadoop hadoop 4096 Sep 3 2015 kafka_2.11-0.8.2.2 lrwxrwxrwx. 1 hadoop hadoop 25 Apr 22 22:49 zookeeper -> zookeeper-3.4.5-cdh5.5.4/ drwxr-xr-x. 16 hadoop hadoop 4096 Apr 23 08:57 zookeeper-3.4.5-cdh5.5.4 [hadoop@hadoop1 ~]$ ln -s kafka_2.11-0.8.2.2/ kafka [hadoop@hadoop1 ~]$ ll total 32 drwxr-xr-x 8 hadoop hadoop 4096 Apr 26 2016 apache-flume-1.6.0-cdh5.5.4-bin drwxrwxr-x 5 hadoop hadoop 4096 Apr 23 18:34 data lrwxrwxrwx 1 hadoop hadoop 32 Apr 23 18:14 flume -> apache-flume-1.6.0-cdh5.5.4-bin/ lrwxrwxrwx 1 hadoop hadoop 22 Apr 23 09:08 hadoop -> hadoop-2.6.0-cdh5.5.4/ drwxr-xr-x 18 hadoop hadoop 4096 Apr 23 09:50 hadoop-2.6.0-cdh5.5.4 lrwxrwxrwx 1 hadoop hadoop 21 Apr 23 16:55 hbase -> hbase-1.0.0-cdh5.5.4/ drwxr-xr-x 27 hadoop hadoop 4096 Apr 23 17:27 hbase-1.0.0-cdh5.5.4 lrwxrwxrwx 1 hadoop hadoop 20 Apr 23 15:27 hive -> hive-1.1.0-cdh5.5.4/ drwxr-xr-x 10 hadoop hadoop 4096 Apr 26 2016 hive-1.1.0-cdh5.5.4 lrwxrwxrwx. 1 hadoop hadoop 12 Apr 22 22:27 jdk -> jdk1.7.0_79/ drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 lrwxrwxrwx 1 hadoop hadoop 19 Apr 23 18:41 kafka -> kafka_2.11-0.8.2.2/ drwxr-xr-x 5 hadoop hadoop 4096 Sep 3 2015 kafka_2.11-0.8.2.2 lrwxrwxrwx. 1 hadoop hadoop 25 Apr 22 22:49 zookeeper -> zookeeper-3.4.5-cdh5.5.4/ drwxr-xr-x. 16 hadoop hadoop 4096 Apr 23 08:57 zookeeper-3.4.5-cdh5.5.4 [hadoop@hadoop1 ~]$
hadoop2和hadoop3操作，不多赘述。
环境变量
[hadoop@hadoop1 ~]$ su root Password: [root@hadoop1 hadoop]# vim /etc/profile
hadoop2和hadoop3操作，不多赘述。
#kafka export KAFKA_HOME=/home/hadoop/kafka export PATH=$PATH:$KAFKA_HOME/bin
hadoop2和hadoop3操作，不多赘述。
[hadoop@hadoop1 ~]$ su root Password: [root@hadoop1 hadoop]# vim /etc/profile [root@hadoop1 hadoop]# source /etc/profile [root@hadoop1 hadoop]#
hadoop2和hadoop3操作，不多赘述。
4.1 修改配置文件
1. hadoop1 上修改 config/server.properties
export HBASE_MANAGES_ZK=false broker.id=1 port=9092 host.name=hadoop1 log.dirs=/home/kafka-logs num.partitions=5 log.cleaner.enable=false offsets.storage=kafka dual.commit.enabled=true
delete.topic.enable=true
zookeeper.connect=hadoop1:2181,hadoop2:2181,hadoop3:2181
2. hadoop2 上修改 config/server.properties
export HBASE_MANAGES_ZK=false broker.id=2 port=9092 host.name=hadoop2 log.dirs=/home/kafka-logs num.partitions=5 log.cleaner.enable=false offsets.storage=kafka dual.commit.enabled=true delete.topic.enable=true zookeeper.connect=hadoop1:2181,hadoop2:2181,hadoop3:2181
3. hadoop3 上修改 config/server.properties
export HBASE_MANAGES_ZK=false broker.id=3 port=9092 host.name=hadoop3 log.dirs=/home/kafka-logs num.partitions=5 log.cleaner.enable=false offsets.storage=kafka dual.commit.enabled=true delete.topic.enable=true zookeeper.connect=hadoop1:2181,hadoop2:2181,hadoop3:2181
创建目录
mkdir -p /home/kafka-logs
hadoop1、hadoop2和hadoop3都执行。
[hadoop@hadoop1 kafka]$ mkdir -p /home/kafka-logs mkdir: cannot create directory `/home/kafka-logs': Permission denied [hadoop@hadoop1 kafka]$ su root Password: [root@hadoop1 kafka]# mkdir -p /home/kafka-logs [root@hadoop1 kafka]# chown -R hadoop:hadoop /home/kafka-logs [root@hadoop1 kafka]# cd /home/ [root@hadoop1 home]# ll total 8 drwx------. 26 hadoop hadoop 4096 Apr 23 21:18 hadoop drwxr-xr-x 2 hadoop hadoop 4096 Apr 23 21:18 kafka-logs [root@hadoop1 home]#
注意：在CDH版本的，发行者已经帮我们解决了这个问题。Apache版本，需要解决
首先解决kafka Unrecognized VM option ‘UseCompressedOops’问题
vi /home/hadoop/kafka/bin/kafka-run-class.sh
if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true" fi
去掉-XX:+UseCompressedOops即可
[hadoop@hadoop1 bin]$ pwd /home/hadoop/kafka/bin [hadoop@hadoop1 bin]$ ll total 80 -rwxr-xr-x 1 hadoop hadoop 943 Sep 3 2015 kafka-console-consumer.sh -rwxr-xr-x 1 hadoop hadoop 942 Sep 3 2015 kafka-console-producer.sh -rwxr-xr-x 1 hadoop hadoop 870 Sep 3 2015 kafka-consumer-offset-checker.sh -rwxr-xr-x 1 hadoop hadoop 946 Sep 3 2015 kafka-consumer-perf-test.sh -rwxr-xr-x 1 hadoop hadoop 860 Sep 3 2015 kafka-mirror-maker.sh -rwxr-xr-x 1 hadoop hadoop 884 Sep 3 2015 kafka-preferred-replica-election.sh -rwxr-xr-x 1 hadoop hadoop 946 Sep 3 2015 kafka-producer-perf-test.sh -rwxr-xr-x 1 hadoop hadoop 872 Sep 3 2015 kafka-reassign-partitions.sh -rwxr-xr-x 1 hadoop hadoop 866 Sep 3 2015 kafka-replay-log-producer.sh -rwxr-xr-x 1 hadoop hadoop 872 Sep 3 2015 kafka-replica-verification.sh -rwxr-xr-x 1 hadoop hadoop 4185 Sep 3 2015 kafka-run-class.sh -rwxr-xr-x 1 hadoop hadoop 1333 Sep 3 2015 kafka-server-start.sh -rwxr-xr-x 1 hadoop hadoop 891 Sep 3 2015 kafka-server-stop.sh -rwxr-xr-x 1 hadoop hadoop 868 Sep 3 2015 kafka-simple-consumer-shell.sh -rwxr-xr-x 1 hadoop hadoop 861 Sep 3 2015 kafka-topics.sh drwxr-xr-x 2 hadoop hadoop 4096 Sep 3 2015 windows -rwxr-xr-x 1 hadoop hadoop 1370 Sep 3 2015 zookeeper-server-start.sh -rwxr-xr-x 1 hadoop hadoop 875 Sep 3 2015 zookeeper-server-stop.sh -rwxr-xr-x 1 hadoop hadoop 968 Sep 3 2015 zookeeper-shell.sh [hadoop@hadoop1 bin]$ vim kafka-run-class.sh
这里，不需自己去解决。
在三台机器上的kafka目录下，分别执行以下命令
nohup bin/kafka-server-start.sh config/server.properties &
[hadoop@hadoop1 kafka]$ pwd /home/hadoop/kafka [hadoop@hadoop1 kafka]$ nohup bin/kafka-server-start.sh config/server.properties & [1] 15091 [hadoop@hadoop1 kafka]$ nohup: ignoring input and appending output to `nohup.out' [1]+ Exit 1 nohup bin/kafka-server-start.sh config/server.properties [hadoop@hadoop1 kafka]$
出现以下提示后回车即可。
查看状态
cat nohup.out
[2017-04-23 21:23:35,304] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:23:35,304] INFO Client environment:java.home=/home/hadoop/jdk1.7.0_79/jre (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:23:35,304] INFO Client environment:java.class.path=.:/home/hadoop/jdk/lib/dt.jar:/home/hadoop/jdk/lib/tools.jar:/home/hadoop/kafka/bin/../core/build/dependant-libs-2.10.4*/*.jar:/home/hadoop/kafka/bin/../examples/build/libs//kafka-examples*.jar:/home/hadoop/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/home/hadoop/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/home/hadoop/kafka/bin/../clients/build/libs/kafka-clients*.jar:/home/hadoop/kafka/bin/../libs/jopt-simple-3.2.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-javadoc.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-scaladoc.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-sources.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-test.jar:/home/hadoop/kafka/bin/../libs/kafka-clients-0.8.2.2.jar:/home/hadoop/kafka/bin/../libs/log4j-1.2.16.jar:/home/hadoop/kafka/bin/../libs/lz4-1.2.0.jar:/home/hadoop/kafka/bin/../libs/metrics-core-2.2.0.jar:/home/hadoop/kafka/bin/../libs/scala-library-2.11.5.jar:/home/hadoop/kafka/bin/../libs/scala-parser-combinators_2.11-1.0.2.jar:/home/hadoop/kafka/bin/../libs/scala-xml_2.11-1.0.2.jar:/home/hadoop/kafka/bin/../libs/slf4j-api-1.7.6.jar:/home/hadoop/kafka/bin/../libs/slf4j-log4j12-1.6.1.jar:/home/hadoop/kafka/bin/../libs/snappy-java-1.1.1.7.jar:/home/hadoop/kafka/bin/../libs/zkclient-0.3.jar:/home/hadoop/kafka/bin/../libs/zookeeper-3.4.6.jar:/home/hadoop/kafka/bin/../core/build/libs/kafka_2.10*.jar (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:23:35,304] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:23:35,304] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:23:35,304] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:23:35,305] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:23:35,305] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:23:35,305] INFO Client environment:os.version=2.6.32-431.el6.x86_64 (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:23:35,305] INFO Client environment:user.name=hadoop (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:23:35,305] INFO Client environment:user.home=/home/hadoop (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:23:35,305] INFO Client environment:user.dir=/home/hadoop/kafka_2.11-0.8.2.2 (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:23:35,308] INFO Initiating client connection, connectString=hadoop1:2181,hadoop2:2181,hadoop3:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@31dbfb7a (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:23:35,391] INFO Opening socket connection to server hadoop1/192.168.80.121:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2017-04-23 21:23:35,410] INFO Socket connection established to hadoop1/192.168.80.121:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2017-04-23 21:23:35,530] INFO Session establishment complete on server hadoop1/192.168.80.121:2181, sessionid = 0x15b99fea1e10012, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn) [2017-04-23 21:23:35,542] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient) [2017-04-23 21:23:36,009] INFO Loading logs. (kafka.log.LogManager) [2017-04-23 21:23:36,069] INFO Logs loading complete. (kafka.log.LogManager) [2017-04-23 21:23:36,072] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager) [2017-04-23 21:23:36,110] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager) [2017-04-23 21:23:36,333] INFO Awaiting socket connections on hadoop1:9092. (kafka.network.Acceptor) [2017-04-23 21:23:36,340] INFO [Socket Server on Broker 1], Started (kafka.network.SocketServer) [hadoop@hadoop1 kafka]$
r:/home/hadoop/kafka/bin/../examples/build/libs//kafka-examples*.jar:/home/hadoop/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/home/hadoop/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/home/hadoop/kafka/bin/../clients/build/libs/kafka-clients*.jar:/home/hadoop/kafka/bin/../libs/jopt-simple-3.2.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-javadoc.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-scaladoc.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-sources.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-test.jar:/home/hadoop/kafka/bin/../libs/kafka-clients-0.8.2.2.jar:/home/hadoop/kafka/bin/../libs/log4j-1.2.16.jar:/home/hadoop/kafka/bin/../libs/lz4-1.2.0.jar:/home/hadoop/kafka/bin/../libs/metrics-core-2.2.0.jar:/home/hadoop/kafka/bin/../libs/scala-library-2.11.5.jar:/home/hadoop/kafka/bin/../libs/scala-parser-combinators_2.11-1.0.2.jar:/home/hadoop/kafka/bin/../libs/scala-xml_2.11-1.0.2.jar:/home/hadoop/kafka/bin/../libs/slf4j-api-1.7.6.jar:/home/hadoop/kafka/bin/../libs/slf4j-log4j12-1.6.1.jar:/home/hadoop/kafka/bin/../libs/snappy-java-1.1.1.7.jar:/home/hadoop/kafka/bin/../libs/zkclient-0.3.jar:/home/hadoop/kafka/bin/../libs/zookeeper-3.4.6.jar:/home/hadoop/kafka/bin/../core/build/libs/kafka_2.10*.jar (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:25:34,857] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:25:34,857] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:25:34,857] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:25:34,858] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:25:34,858] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:25:34,858] INFO Client environment:os.version=2.6.32-431.el6.x86_64 (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:25:34,858] INFO Client environment:user.name=hadoop (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:25:34,858] INFO Client environment:user.home=/home/hadoop (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:25:34,858] INFO Client environment:user.dir=/home/hadoop/kafka_2.11-0.8.2.2 (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:25:34,862] INFO Initiating client connection, connectString=hadoop1:2181,hadoop2:2181,hadoop3:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@3764253e (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:25:35,713] INFO Opening socket connection to server hadoop3/192.168.80.123:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2017-04-23 21:25:35,811] INFO Socket connection established to hadoop3/192.168.80.123:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2017-04-23 21:25:35,884] INFO Session establishment complete on server hadoop3/192.168.80.123:2181, sessionid = 0x35b99ff49bf0012, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn) [2017-04-23 21:25:35,890] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient) [2017-04-23 21:25:37,146] INFO Loading logs. (kafka.log.LogManager) [2017-04-23 21:25:37,243] INFO Logs loading complete. (kafka.log.LogManager) [2017-04-23 21:25:37,245] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager) [2017-04-23 21:25:37,300] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager) [2017-04-23 21:25:37,798] INFO Awaiting socket connections on hadoop2:9092. (kafka.network.Acceptor) [2017-04-23 21:25:37,803] INFO [Socket Server on Broker 2], Started (kafka.network.SocketServer) [2017-04-23 21:25:39,744] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$) [2017-04-23 21:25:42,084] INFO Registered broker 2 at path /brokers/ids/2 with address hadoop2:9092. (kafka.utils.ZkUtils$) [2017-04-23 21:25:42,520] INFO [Kafka Server 2], started (kafka.server.KafkaServer) [hadoop@hadoop2 kafka]$
r:/home/hadoop/kafka/bin/../examples/build/libs//kafka-examples*.jar:/home/hadoop/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/home/hadoop/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/home/hadoop/kafka/bin/../clients/build/libs/kafka-clients*.jar:/home/hadoop/kafka/bin/../libs/jopt-simple-3.2.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-javadoc.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-scaladoc.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-sources.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-test.jar:/home/hadoop/kafka/bin/../libs/kafka-clients-0.8.2.2.jar:/home/hadoop/kafka/bin/../libs/log4j-1.2.16.jar:/home/hadoop/kafka/bin/../libs/lz4-1.2.0.jar:/home/hadoop/kafka/bin/../libs/metrics-core-2.2.0.jar:/home/hadoop/kafka/bin/../libs/scala-library-2.11.5.jar:/home/hadoop/kafka/bin/../libs/scala-parser-combinators_2.11-1.0.2.jar:/home/hadoop/kafka/bin/../libs/scala-xml_2.11-1.0.2.jar:/home/hadoop/kafka/bin/../libs/slf4j-api-1.7.6.jar:/home/hadoop/kafka/bin/../libs/slf4j-log4j12-1.6.1.jar:/home/hadoop/kafka/bin/../libs/snappy-java-1.1.1.7.jar:/home/hadoop/kafka/bin/../libs/zkclient-0.3.jar:/home/hadoop/kafka/bin/../libs/zookeeper-3.4.6.jar:/home/hadoop/kafka/bin/../core/build/libs/kafka_2.10*.jar (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:26:10,239] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:26:10,240] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:26:10,240] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:26:10,241] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:26:10,241] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:26:10,241] INFO Client environment:os.version=2.6.32-431.el6.x86_64 (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:26:10,241] INFO Client environment:user.name=hadoop (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:26:10,241] INFO Client environment:user.home=/home/hadoop (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:26:10,241] INFO Client environment:user.dir=/home/hadoop/kafka_2.11-0.8.2.2 (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:26:10,244] INFO Initiating client connection, connectString=hadoop1:2181,hadoop2:2181,hadoop3:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@3764253e (org.apache.zookeeper.ZooKeeper) [2017-04-23 21:26:11,052] INFO Opening socket connection to server hadoop3/192.168.80.123:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2017-04-23 21:26:11,080] INFO Socket connection established to hadoop3/192.168.80.123:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2017-04-23 21:26:11,370] INFO Session establishment complete on server hadoop3/192.168.80.123:2181, sessionid = 0x35b99ff49bf0013, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn) [2017-04-23 21:26:11,505] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient) [2017-04-23 21:26:12,561] INFO Loading logs. (kafka.log.LogManager) [2017-04-23 21:26:12,698] INFO Logs loading complete. (kafka.log.LogManager) [2017-04-23 21:26:12,702] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager) [2017-04-23 21:26:12,918] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager) [2017-04-23 21:26:14,277] INFO Awaiting socket connections on hadoop3:9092. (kafka.network.Acceptor) [2017-04-23 21:26:14,623] INFO [Socket Server on Broker 3], Started (kafka.network.SocketServer) [2017-04-23 21:26:17,519] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$) [2017-04-23 21:26:19,961] INFO Registered broker 3 at path /brokers/ids/3 with address hadoop3:9092. (kafka.utils.ZkUtils$) [2017-04-23 21:26:20,172] INFO [Kafka Server 3], started (kafka.server.KafkaServer) [hadoop@hadoop3 kafka]$
至此，kafka_2.11-0.8.2.2.tgz的3节点集群的搭建完成！
扩展补充