部署impala
impala安装选择rpm包方式进行，这是本次部署唯一一个主要主件采用rpm方式进行安装部署，这里主要原因是cloudera没有提供现成的tar包文件，而源码编译过程会出现各种未知原因，为了方便采用以下方式进行部署。
安装介质如下：
$ ls
bigtop-utils-0.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.noarch.rpm
impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm
impala-shell-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm
impala-catalog-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm
impala-state-store-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm
impala-debuginfo-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm
impala-udf-devel-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm
impala-server-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm
sentry-1.5.1+cdh5.10.0+272-1.cdh5.10.0.p0.70.el7.noarch.rpm
[hadoop@db01 impala270]$ rpm -ivh impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm
warning: impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
error: Failed dependencies:
bigtop-utils >= 0.7 is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
hadoop is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
hadoop-hdfs is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
hadoop-yarn is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
hadoop-mapreduce is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
hbase is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
hive >= 0.12.0+cdh5.1.0 is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
zookeeper is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
hadoop-libhdfs is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
avro-libs is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
parquet is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
sentry >= 1.3.0+cdh5.1.0 is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
sentry is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
/lib/lsb/init-functions is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
libhdfs.so.0.0.0()(64bit) is needed by impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64
[hadoop@db01 impala270]$
1、安装bigtop和sentry
$sudo yum -y install redhat-lsb
sudo rpm -ivh bigtop-utils-0.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.noarch.rpm
warning: bigtop-utils-0.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:bigtop-utils-0.7.0+cdh5.10.0+0-1.################################# [100%]
$sudo rpm -ivh sentry-1.5.1+cdh5.10.0+272-1.cdh5.10.0.p0.70.el7.noarch.rpm --nodeps
warning: sentry-1.5.1+cdh5.10.0+272-1.cdh5.10.0.p0.70.el7.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:sentry-1.5.1+cdh5.10.0+272-1.cdh5################################# [100%]
2、impalad安装
注：impalad需安装在所有datanode服务器上。
sudo rpm -ivh impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm --nodeps
warning: impala-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:impala-2.7.0+cdh5.10.0+0-1.cdh5.1################################# [100%]
3、安装impala server
sudo rpm -ivh impala-server-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm
warning: impala-server-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:impala-server-2.7.0+cdh5.10.0+0-1################################# [100%]
--以上步骤在所有datanode节点安装
4、修改配置文件
sudo vim /etc/default/impala
IMPALA_CATALOG_SERVICE_HOST=db01
IMPALA_STATE_STORE_HOST=db01
这两个ip是要装CATALOG和STATE_STORE的节点，
CATALOG和STATE_STORE这两个服务必须和hive装在同一个节点
在一个节点编辑后，复制到其他节点
sudo scp /etc/default/impala db02:/etc/default/impala
sudo scp /etc/default/impala db03:/etc/default/impala
sudo scp /etc/default/impala db04:/etc/default/impala
vim /etc/default/bigtop-utils
export JAVA_HOME=/opt/service/jdk1.7.0_67
scp bigtop-utils db02:/etc/default/bigtop-utils
scp bigtop-utils db03:/etc/default/bigtop-utils
scp bigtop-utils db04:/etc/default/bigtop-utils
vim hdfs-site.xml
<property>
<name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.file-block-storage-locations.timeout.millis</name>
<value>10000</value>
</property>
5、复制hive和hadoop的配置文件到impala
sudo scp /opt/cdh5/hive-1.1.0-cdh5.10.0/conf/hive-site.xml db01:/etc/impala/conf/
sudo scp /opt/cdh5/hive-1.1.0-cdh5.10.0/conf/hive-site.xml db02:/etc/impala/conf/
sudo scp /opt/cdh5/hive-1.1.0-cdh5.10.0/conf/hive-site.xml db03:/etc/impala/conf/
sudo scp /opt/cdh5/hive-1.1.0-cdh5.10.0/conf/hive-site.xml db04:/etc/impala/conf/
sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/core-site.xml db01:/etc/impala/conf/
sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/core-site.xml db02:/etc/impala/conf/
sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/core-site.xml db03:/etc/impala/conf/
sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/core-site.xml db04:/etc/impala/conf/
sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/hdfs-site.xml db01:/etc/impala/conf/
sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/hdfs-site.xml db02:/etc/impala/conf/
sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/hdfs-site.xml db03:/etc/impala/conf/
sudo scp /opt/cdh5/hadoop-2.6.0-cdh5.10.0/etc/hadoop/hdfs-site.xml db04:/etc/impala/conf/
6、在hive节点安装impala-state-store 和impala-catalog
$ sudo rpm -ivh impala-state-store-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm
warning: impala-state-store-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:impala-state-store-2.7.0+cdh5.10.################################# [100%]
$ sudo rpm -ivh impala-catalog-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm
warning: impala-catalog-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:impala-catalog-2.7.0+cdh5.10.0+0-################################# [100%]
$ sudo cp /mnt/mysql-connector-java-5.1.22-bin.jar /var/lib/impala/
7、所有节点安装impala shell
sudo yum -y install python-setuptools
sudo rpm -ivh impala-shell-2.7.0+cdh5.10.0+0-1.cdh5.10.0.p0.71.el7.x86_64.rpm
8、修改impala依赖jar包，通过软连接方式
sudo rm -rf /usr/lib/impala/lib/hadoop-annotations.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-auth.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-aws.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-common.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-hdfs.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-mapreduce-client-common.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-mapreduce-client-core.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-mapreduce-client-jobclient.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-mapreduce-client-shuffle.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-yarn-api.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-yarn-client.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-yarn-common.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-yarn-server-applicationhistoryservice.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-yarn-server-common.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-yarn-server-nodemanager.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-yarn-server-resourcemanager.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-yarn-server-web-proxy.jar
sudo rm -rf /usr/lib/impala/lib/hbase-annotations.jar
sudo rm -rf /usr/lib/impala/lib/hbase-client.jar
sudo rm -rf /usr/lib/impala/lib/hbase-common.jar
sudo rm -rf /usr/lib/impala/lib/hbase-protocol.jar
sudo rm -rf /usr/lib/impala/lib/hive-ant.jar
sudo rm -rf /usr/lib/impala/lib/hive-beeline.jar
sudo rm -rf /usr/lib/impala/lib/hive-common.jar
sudo rm -rf /usr/lib/impala/lib/hive-exec.jar
sudo rm -rf /usr/lib/impala/lib/hive-hbase-handler.jar
sudo rm -rf /usr/lib/impala/lib/hive-metastore.jar
sudo rm -rf /usr/lib/impala/lib/hive-serde.jar
sudo rm -rf /usr/lib/impala/lib/hive-service.jar
sudo rm -rf /usr/lib/impala/lib/hive-shims-common.jar
sudo rm -rf /usr/lib/impala/lib/hive-shims.jar
sudo rm -rf /usr/lib/impala/lib/hive-shims-scheduler.jar
sudo rm -rf /usr/lib/impala/lib/zookeeper.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/common/lib/hadoop-annotations-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-annotations.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/common/lib/hadoop-auth-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-auth.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/mapreduce1/lib/hadoop-aws-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-aws.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/common/hadoop-common-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-common.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/hdfs/hadoop-hdfs-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-hdfs.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-common.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-core.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-jobclient.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-shuffle.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-api-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-api.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-client-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-client.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-common-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-common.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-server-applicationhistoryservice.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-server-common.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-server-nodemanager.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-server-resourcemanager.jar
sudo ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0-cdh5.10.0.jar /usr/lib/impala/lib/hadoop-yarn-server-web-proxy.jar
sudo ln -s /opt/cdh5/hbase-1.2.0-cdh5.10.0/lib/hbase-annotations-1.2.0-cdh5.10.0.jar /usr/lib/impala/lib/hbase-annotations.jar
sudo ln -s /opt/cdh5/hbase-1.2.0-cdh5.10.0/lib/hbase-client-1.2.0-cdh5.10.0.jar /usr/lib/impala/lib/hbase-client.jar
sudo ln -s /opt/cdh5/hbase-1.2.0-cdh5.10.0/lib/hbase-common-1.2.0-cdh5.10.0.jar /usr/lib/impala/lib/hbase-common.jar
sudo ln -s /opt/cdh5/hbase-1.2.0-cdh5.10.0/lib/hbase-protocol-1.2.0-cdh5.10.0.jar /usr/lib/impala/lib/hbase-protocol.jar
sudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-ant-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-ant.jar
sudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-beeline-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-beeline.jar
sudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-common-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-common.jar
sudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-exec-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-exec.jar
sudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-hbase-handler-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-hbase-handler.jar
sudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-metastore-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-metastore.jar
sudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-serde-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-serde.jar
sudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-service-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-service.jar
sudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-shims-common-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-shims-common.jar
sudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-shims-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-shims.jar
sudo ln -s /opt/cdh5/hive-1.1.0-cdh5.10.0/lib/hive-shims-scheduler-1.1.0-cdh5.10.0.jar /usr/lib/impala/lib/hive-shims-scheduler.jar
sudo ln -s /opt/cdh5/zookeeper-3.4.5-cdh5.10.0/zookeeper-3.4.5-cdh5.10.0.jar /usr/lib/impala/lib/zookeeper.jar
rm -rf /usr/lib/impala/lib/libhadoop.so
rm -rf /usr/lib/impala/lib/libhadoop.so.1.0.0
rm -rf /usr/lib/impala/lib/libhdfs.so
rm -rf /usr/lib/impala/lib/libhdfs.so.0.0.0
ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/lib/native/libhadoop.so /usr/lib/impala/lib/libhadoop.so
ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/lib/native/libhadoop.so.1.0.0 /usr/lib/impala/lib/libhadoop.so.1.0.0
ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/lib/native/libhdfs.so /usr/lib/impala/lib/libhdfs.so
ln -s /opt/cdh5/hadoop-2.6.0-cdh5.10.0/lib/native/libhdfs.so.0.0.0 /usr/lib/impala/lib/libhdfs.so.0.0.0
9、启动impala
#安装了hive的节点启动state-store和catalog服务
service impala-state-store start
service impala-catalog start
#所有datanode节点启动impalad
service impala-server start