Flink ON YARN模式 一、安装Flink 1、下载 1.1、下载 flink 包 官网地址：https://archive.apache.org/dist/flink/ 采用flink-1.8.0-bin-scala_2.11.tgz安装，因目前Flink尚未集成hadoop2.9版本，因此选择2.6稳定版进行安装（兼容），flink-shaded-hadoop-2-uber-2.6.5-7.0.jar flink-1.8.0-bin-scala_2.11.tgz下载路径： https://archive.apache.org/dist/flink/flink-1.8.0/flink-1.8.0-bin-scala_2.11.tgz 1.2、下载 hadoop 依赖包 官网下载flink-shaded-hadoop-2-uber-2.6.5-7.0.jar 拷贝到${flink_home}/lib/目录下 下载地址 https://flink.apache.org/downloads.html 2、解压 解压安装在 /opt/module 目录下 $ cd /opt/module $ tar -zxvf flink-1.8.0-bin-scala_2.11.tgz 3、配置环境变量 $ vim /etc/profile export FLINK_HOME=/opt/module/flink-1.8.0 export PATH=$FLINK_HOME/bin:$PATH $ source /etc/profile $ flink #执行成功，表示Flink安装成功 注意：非root用户，配在.bash_profile文件里 4、添加 hadoop 依赖包 把 flink-shaded-hadoop-2-uber-2.6.5-7.0.jar 拷贝到 /opt/module/flink-1.8.0/lib 目录下 4、修改yarn-site.xml文件 $ sudo vim /opt/module/hadoop-2.7.6/etc/hadoop/yarn-site.xml 4.1、配置AM在尝试重启的最大次数 <property> <name>yarn.resourcemanager.am.max-attempts</name> <value>4</value> <description>The maximum number of application master execution attempts</description> </property> 5、修改 flink-conf.yaml 文件 $ sudo vim /opt/module/flink-1.8.0/conf/flink-conf.yaml #jobmanager taskmanager配置 jobmanager.rpc.address: localhost jobmanager.rpc.port: 6123 jobmanager.heap.mb: 256 taskmanager.heap.mb: 512 taskmanager.numberOfTaskSlots: 1 #是否应在 TaskManager 启动时预先分配 TaskManager 管理的内存 #默认不进行预分配，这样在我们不使用flink集群时候不会占用集群资源 taskmanager.memory.preallocate: false parallelism.default: 1 jobmanager.web.port: 33069 #默认端口8081 # yarn ApplicationMaster 能接受最多的失败 container 数，直到 YARN 会话失败。 # 需配置，注意":"之后有个空格 yarn.maximum-failed-containers: 99999 #akka config # 需配置，注意":"之后有个空格 akka.watch.heartbeat.interval: 5 s akka.watch.heartbeat.pause: 20 s akka.ask.timeout: 60 s akka.framesize: 20971520b #high-avaliability(高可用配置) # 注释打开 high-availability: zookeeper ## 根据安装的zookeeper信息填写 #high-availability.zookeeper.quorum: 10.141.61.226:2181,10.141.53.244:2181,10.141.18.219:2181 high-availability.zookeeper.quorum: localhost:2181 high-availability.zookeeper.path.root: /data/flink/flink-on-yarn ## HA 信息存储到HDFS的目录，根据各自的Hdfs情况修改，hdfs://mycluster/取自配置fs.defaultFS的值 #high-availability.zookeeper.storageDir: hdfs://mycluster/flink/recovery high-availability.zookeeper.storageDir: hdfs://mycluster/flink/recovery ##ApplicationMaster尝试次数 yarn.application-attempts=10 #checkpoint config(容错配置) ##支持Backend有memory、fs、rocksdb state.backend: rocksdb ## checkpoint到HDFS的目录 根据各自安装的HDFS情况修改 state.backend.fs.checkpointdir: hdfs://mycluster/flink/checkpoint ## 对外checkpoint到HDFS的目录 state.checkpoints.dir: hdfs://mycluster/flink/savepoint #memory config env.java.opts: -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 - XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -XX:+HeapDumpOnOutOfMemoryError yarn.heap-cutoff-ratio: 0.2 taskmanager.memory.off-heap: true 6、Flink ON YARN相关服务启动 启动Zookeeper集群 $ /opt/module/zookeeper-3.4.6/bin/zkServer.sh start ##zookeeper重启可以改用如下命令 $ /opt/module/zookeeper-3.4.6/bin/zkServer.sh restart 启动HDFS集群 $ start-dfs.sh 验证hadoop是否启动成功 http://182.61.*.60:50070/ http://106.12.*.89:50070/ 启动YARN集群 $ start-yarn.sh 查看服务状态 [admin@145 sbin]$ yarn rmadmin -getServiceState rm1 standby [admin@145 sbin]$ yarn rmadmin -getServiceState rm2 active http://106.12.241.145:33069 sudo vim /opt/module/hadoop-2.7.6/etc/hadoop/yarn-site.xml <!-- ResourceManager对外web ui地址。用户可通过该地址在浏览器中查看集群各类信息。--> <property> <name>yarn.resourcemanager.webapp.address</name> <value>node145:33069</value> </property> 3、以yarnSession方式提交job 3.1、创建session 创建yarnsession $ yarn-session.sh -nm test -n 2 -tm 1024 -s 2 #-nm Yarn的应用名字 #-n (container) taskmanager的数量 #-tm 每个taskmanager的内存大小 #-s (slot) 每个taskmanager的slot 数量 yarn-session的信息会写到：/tmp/.yarn-properties-hadoop 这种机制不太好（手动获取appid） 问题一： The configuration directory ('/opt/module/flink-1.8.0/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them. https://www.cnblogs.com/frankdeng/p/9047698.html https://blog.csdn.net/magic_kid_2010/article/details/97004746?depth_1-utm_source=distribute.pc_relevant.none-task&utm_source=distribute.pc_relevant.none-task zookeper 安装教程 https://www.cnblogs.com/linjiqin/p/8407084.html