zoukankan      html  css  js  c++  java
  • Flink安装排坑指南

    Flink安装排坑指南

    Flink依赖与java环境,请先安装jdk,最好1.8以上,配置好JAVA_HOME环境变量

    首先我是在window上安装flink的,下载flink

    下载之后解压到任意目录,进入bin目录运行start-cluster.bat脚本启动flink服务,但是Task Slots却为0,查看log发现是TaskManager没有初始化成功

    image-20200314182933049

    2020-03-14 18:28:37,888 ERROR org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - TaskManager initialization failed.
    org.apache.flink.configuration.IllegalConfigurationException: Failed to create TaskExecutorResourceSpec
    	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.resourceSpecFromConfig(TaskExecutorResourceUtils.java:72)
    	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.startTaskManager(TaskManagerRunner.java:356)
    	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<init>(TaskManagerRunner.java:152)
    	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:308)
    	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$runTaskManagerSecurely$2(TaskManagerRunner.java:322)
    	at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
    	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerSecurely(TaskManagerRunner.java:321)
    	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.main(TaskManagerRunner.java:287)
    Caused by: org.apache.flink.configuration.IllegalConfigurationException: The required configuration option Key: 'taskmanager.cpu.cores' , default: null (fallback keys: []) is not set
    	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.checkConfigOptionIsSet(TaskExecutorResourceUtils.java:90)
    	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.lambda$checkTaskExecutorResourceConfigSet$0(TaskExecutorResourceUtils.java:84)
    	at java.util.Arrays$ArrayList.forEach(Unknown Source)
    	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.checkTaskExecutorResourceConfigSet(TaskExecutorResourceUtils.java:84)
    	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.resourceSpecFromConfig(TaskExecutorResourceUtils.java:70)
    	... 7 more
    2020-03-14 18:28:37,892 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Shutting down BLOB cache
    2020-03-14 18:28:37,892 INFO  org.apache.flink.runtime.blob.PermanentBlobCache              - Shutting down BLOB cache
    

    以为是没有权限创建文件夹,用管理员权限试了也报一样的错

    搞了半天也没个结果,最后果断放弃了window环境。

    在vmware上弄了个centos7,在centos7上搭建了jdk和flink环境,推荐centos7最少给2G内存,不然后面启动flink时会被内存不够的问题折腾死。

    flink最少需要192M的内存才能启动,jdk1.8之后堆内存采用的是元空间,元空间是根据内存大小自动分配的。如果元空间给的太小,flink将会启动不起来。

    The derived from fraction jvm overhead memory (19.200mb (20132659 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
    

    在启动flink时可以通过jvm参数指定元空间大小

    • -XX:MetaspaceSize,初始空间大小,达到该值就会触发垃圾收集进行类型卸载,同时GC会对该值进行调整:如果释放了大量的空间,就适当降低该值;如果释放了很少的空间,那么在不超过MaxMetaspaceSize时,适当提高该值。

    flink的cofig/flink-conf.yaml配置的jobmanager.heap.sizetaskmanager.memory.process.size也不能超过元空间大小。它们默认大小分别是:1024m和1568m

    配置方法:env.java.opts: -XX:MetaspaceSize=512m,指定元空间为512m

    如果电脑内存够大的话,推荐还是直接给centos弄个2G吧,能省很多事。

    运行./bin/start-cluster.sh,在linux上顺利启动,在win10上通过浏览器访问

    image-20200314203247067

    win10上启动不了不知道是不是因为权限太严格导致的,希望有路过的大佬能在评论区指点一二

  • 相关阅读:
    解决content-type为"application/json"的post过来的数据在php端接受不到的问题
    webshell导致项目崩溃
    mysql启动报错 mysql InnoDB: Error: could not open single-table tablespace file
    php性能优化
    post表单翻页保存搜索条件
    PHP7 MongDB 安装与使用
    Mac下编译Thrift的时候Python2.7会报错 site-packages': Operation not permitted
    苹果系统通过brew安装sshpass
    volatile关键字深入理解
    java语言中application异常退出和线程异常崩溃的捕获方法,并且在捕获的钩子方法中进行异常处理
  • 原文地址:https://www.cnblogs.com/dagger9527/p/12494052.html
Copyright © 2011-2022 走看看