Cluster OS&Platform
redhat/centos7, docker, mesos, cloudera manager(cdh)
Checklist
1 check user & password & network reachability, make sure everything is fine to login all remote servers by ssh client
2 check linux release, upgrade or reinstall if necessary
3 check yum repo
4 check hardware: memory, cpu, disk & partition, network device
5 check network: lan & wlan environment
5 check data disk, make sure all data disks are mounted by fdisk/mke2fs/mount, /etc/fstab
6 check user & group, running processes, free memory, free disk, directory structure
6 check iptables, make sure only the necessary protocols and ports are allowed
7 check systemd, service
Preparation
1 login the first server by ssh client, it will be acted as deploy center
2 upload 'deploy' directory(more than 15G: iso8G, cdh3G,registry2G) and 'deploy.sh' to $HOME by sftp or other tools
3 use deploy.sh to generate deploy commands
4 deploy the cluster offline
Offline Deployment Details
1 make sure root user login without password by ssh client between all servers
2 create yum repo file using iso on local server
3 install ansible on local server
4 create & push /etc/hosts to all servers by ansible
5 install & start docker on local server
6 docker load registry image from local file
7 docker run registry on local server
8 docker run nginx on local server as 'yum repo server', 'docker registry server' and 'cloudera manager proxy server'
9 docker run mysql on local server temporarily
10 create mysql databases & users and flush privileges
11 create & push yum repo file to all servers by ansible
12 push deploy directory to all servers by ansible
13 install ansible on other servers
14 install and start docker on other servers
15 install kerberos on all servers
16 start kdc on local server
17 install jdk on all servers
18 install cloudera manager & cdh on all servers
19 start cloudera manager anget on all servers
20 start cloudera manager master on local server
21 deploy zookeeper by cloudera manager
22 install mesos and marathon on all servers
23 start mesos agent on all servers
24 start mesos master on other two servers
25 start marathon master on other two servers
26 deploy marathon-lb on marathon by api
27 docker stop mysql and redeploy mysql master&slave on marathon by api
28 deploy redis, elasticsearch, logstash, kibana, kafka, airflow on marathon by api
29 deploy hdfs, yarn, hive, hbase, oozie, kudu, impala, sentry, hue by cloudera manager
30 enable kerberos on cloudera manager
31 deploy all kinds of client
32 confirm all components working
33 initialize data structure in mysql, hive, es, impala
34 deploy business components like tomcat, logstash on marathon
35 trigger all crontab stuff in airflow
36 confirm all business working
Optional
use zabbix to monitor all servers