我这里搭建了3节点的集群
操作系统：centos7
配置静态ip
配置三节点的免密通讯
配置时钟同步
安装jdk1.8
以上的具体步骤我就不多说了。
下载flink安装包：https://archive.apache.org/dist/flink/flink-1.8.0/
上传安装包到集群
解压安装包
直接启动flink
./start-cluster.sh
这两个进程在的话证明启动成功
打开浏览器访问：http://192.168.215.161:8081/ 大家根据自己的实际ip地址来
可以看到我们通过8081端口访问Local模式的flink集群
通过idea的maven把之前的项目打包
D:MyCodeFlinkStudyflink>mvn clean package D:MyCodeFlinkStudyflink>set JAVA_HOME=D:SoftWaresJDK [INFO] Scanning for projects... [INFO] [INFO] ---------------------------< com.gong:flink >--------------------------- [INFO] Building Flink Quickstart Job 1.0 [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ flink --- [INFO] Deleting D:MyCodeFlinkStudyflink arget [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ flink --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ flink --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 4 source files to D:MyCodeFlinkStudyflink argetclasses [INFO] [INFO] --- scala-maven-plugin:3.2.2:compile (default) @ flink --- [WARNING] Expected all dependencies to require Scala version: 2.11.12 [WARNING] org.apache.flink:flink-runtime_2.11:1.6.2 requires scala version: 2.11.12 [WARNING] org.scala-lang:scala-reflect:2.11.12 requires scala version: 2.11.12 [WARNING] org.apache.flink:flink-scala_2.11:1.6.2 requires scala version: 2.11.12 [WARNING] org.apache.flink:flink-scala_2.11:1.6.2 requires scala version: 2.11.12 [WARNING] org.scala-lang:scala-compiler:2.11.12 requires scala version: 2.11.12 [WARNING] org.scala-lang.modules:scala-xml_2.11:1.0.5 requires scala version: 2.11.7 [WARNING] Multiple versions of scala libraries detected! [INFO] Using incremental compilation [INFO] Compiling 2 Scala sources and 4 Java sources to D:MyCodeFlinkStudyflink argetclasses... [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ flink --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:MyCodeFlinkStudyflinksrc est esources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ flink --- [INFO] No sources to compile [INFO] [INFO] --- scala-maven-plugin:3.2.2:testCompile (default) @ flink --- [WARNING] Expected all dependencies to require Scala version: 2.11.12 [WARNING] org.apache.flink:flink-runtime_2.11:1.6.2 requires scala version: 2.11.12 [WARNING] org.scala-lang:scala-reflect:2.11.12 requires scala version: 2.11.12 [WARNING] org.apache.flink:flink-scala_2.11:1.6.2 requires scala version: 2.11.12 [WARNING] org.apache.flink:flink-scala_2.11:1.6.2 requires scala version: 2.11.12 [WARNING] org.scala-lang:scala-compiler:2.11.12 requires scala version: 2.11.12 [WARNING] org.scala-lang.modules:scala-xml_2.11:1.0.5 requires scala version: 2.11.7 [WARNING] Multiple versions of scala libraries detected! [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ flink --- [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ flink --- [INFO] Building jar: D:MyCodeFlinkStudyflink argetflink-1.0.jar [INFO] [INFO] --- maven-shade-plugin:3.0.0:shade (default) @ flink --- [INFO] Including org.apache.flink:flink-java:jar:1.6.2 in the shaded jar. [INFO] Including org.apache.flink:flink-core:jar:1.6.2 in the shaded jar. [INFO] Including org.apache.flink:flink-annotations:jar:1.6.2 in the shaded jar. [INFO] Including org.apache.flink:flink-metrics-core:jar:1.6.2 in the shaded jar. [INFO] Including com.esotericsoftware.kryo:kryo:jar:2.24.0 in the shaded jar. [INFO] Including com.esotericsoftware.minlog:minlog:jar:1.2 in the shaded jar. [INFO] Including org.objenesis:objenesis:jar:2.1 in the shaded jar. [INFO] Including commons-collections:commons-collections:jar:3.2.2 in the shaded jar. [INFO] Including org.apache.commons:commons-compress:jar:1.4.1 in the shaded jar. [INFO] Including org.tukaani:xz:jar:1.0 in the shaded jar. [INFO] Including org.apache.flink:flink-shaded-asm:jar:5.0.4-4.0 in the shaded jar. [INFO] Including org.apache.commons:commons-lang3:jar:3.3.2 in the shaded jar. [INFO] Including org.apache.commons:commons-math3:jar:3.5 in the shaded jar. [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.7 from the shaded jar. [INFO] Excluding com.google.code.findbugs:jsr305:jar:1.3.9 from the shaded jar. [INFO] Excluding org.apache.flink:force-shading:jar:1.6.2 from the shaded jar. [INFO] Including org.apache.flink:flink-streaming-java_2.11:jar:1.6.2 in the shaded jar. [INFO] Including org.apache.flink:flink-runtime_2.11:jar:1.6.2 in the shaded jar. [INFO] Including org.apache.flink:flink-queryable-state-client-java_2.11:jar:1.6.2 in the shaded jar. [INFO] Including org.apache.flink:flink-hadoop-fs:jar:1.6.2 in the shaded jar. [INFO] Including commons-io:commons-io:jar:2.4 in the shaded jar. [INFO] Including org.apache.flink:flink-shaded-netty:jar:4.1.24.Final-4.0 in the shaded jar. [INFO] Including org.apache.flink:flink-shaded-jackson:jar:2.7.9-4.0 in the shaded jar. [INFO] Including commons-cli:commons-cli:jar:1.3.1 in the shaded jar. [INFO] Including org.javassist:javassist:jar:3.19.0-GA in the shaded jar. [INFO] Including com.typesafe.akka:akka-actor_2.11:jar:2.4.20 in the shaded jar. [INFO] Including com.typesafe:config:jar:1.3.0 in the shaded jar. [INFO] Including org.scala-lang.modules:scala-java8-compat_2.11:jar:0.7.0 in the shaded jar. [INFO] Including com.typesafe.akka:akka-stream_2.11:jar:2.4.20 in the shaded jar. [INFO] Including org.reactivestreams:reactive-streams:jar:1.0.0 in the shaded jar. [INFO] Including com.typesafe:ssl-config-core_2.11:jar:0.2.1 in the shaded jar. [INFO] Including com.typesafe.akka:akka-protobuf_2.11:jar:2.4.20 in the shaded jar. [INFO] Including com.typesafe.akka:akka-slf4j_2.11:jar:2.4.20 in the shaded jar. [INFO] Including org.clapper:grizzled-slf4j_2.11:jar:1.0.2 in the shaded jar. [INFO] Including com.github.scopt:scopt_2.11:jar:3.5.0 in the shaded jar. [INFO] Including org.xerial.snappy:snappy-java:jar:1.1.4 in the shaded jar. [INFO] Including com.twitter:chill_2.11:jar:0.7.4 in the shaded jar. [INFO] Including com.twitter:chill-java:jar:0.7.4 in the shaded jar. [INFO] Including org.apache.flink:flink-clients_2.11:jar:1.6.2 in the shaded jar. [INFO] Including org.apache.flink:flink-optimizer_2.11:jar:1.6.2 in the shaded jar. [INFO] Including org.apache.flink:flink-shaded-guava:jar:18.0-4.0 in the shaded jar. [INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.7 from the shaded jar. [INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar. [INFO] Including org.apache.flink:flink-scala_2.11:jar:1.6.2 in the shaded jar. [INFO] Including org.scala-lang:scala-reflect:jar:2.11.12 in the shaded jar. [INFO] Including org.scala-lang:scala-library:jar:2.11.12 in the shaded jar. [INFO] Including org.scala-lang:scala-compiler:jar:2.11.12 in the shaded jar. [INFO] Including org.scala-lang.modules:scala-xml_2.11:jar:1.0.5 in the shaded jar. [INFO] Including org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4 in the shaded jar. [INFO] Including org.apache.flink:flink-streaming-scala_2.11:jar:1.6.2 in the shaded jar. [INFO] Including org.apache.flink:flink-table_2.10:jar:1.3.2 in the shaded jar. [INFO] Replacing original artifact with shaded artifact. [INFO] Replacing D:MyCodeFlinkStudyflink argetflink-1.0.jar with D:MyCodeFlinkStudyflink argetflink-1.0-shaded.jar [INFO] Dependency-reduced POM written at: D:MyCodeFlinkStudyflinkdependency-reduced-pom.xml [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 14.482 s [INFO] Finished at: 2020-09-22T20:03:08+08:00 [INFO] ------------------------------------------------------------------------ D:MyCodeFlinkStudyflink>
将这个包上传到集群
在集群运行
bin/flink run -c com.gong.flink.batch.WordCount flink-1.0.jar --input /opt/modules/flink-1.8.0/test.txt --output /opt/modules/flink-1.8.0/out.txt
这里我们知道输入路径和输出路径
下面的是一个官方的例子
先启动nc
重新打开一个终端启动官方的例子
bin/flink run examples/streaming/SocketWindowWordCount.jar --hostname master --port 9999
在nc端输入一些字符
通过浏览器的界面，我们可以看到流处理的结果
通过界面把作业停止掉
可以看到已经停止了
