zoukankan      html  css  js  c++  java
  • Flink开发部署踩坑记录

    ~~~~踩坑一:springboot+flink项目的日志问题~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    现象:开发环境IDE里调试运行都ok,部署到flink服务器上运行报如下错误: 

    The main method caused an error: LoggerFactory is not a Logback LoggerContext but Logback is on the classpath. Either remove Logback or the competing implementation (class org.apache.logging.slf4j.Log4jLoggerFactory loaded from file:/usr/flink/lib/log4j-slf4j-impl-2.12.1.jar). If you are using WebLogic you will need to add 'org.slf4j' to prefer-application-packages in WEB-INF/weblogic.xml: org.apache.logging.slf4j.Log4jLoggerFactory

    原因:java常用的日志框架(或者说api)有Commons Logging(对应的具体实现是log4j)与slf4j(对应的具体实现是logback)两种。springboot与flink默认使用的应该都是slf4j(即logback,详见官网:Apache Flink 1.7 Documentation: Best Practices),因此,除开我们的项目pom.xml文件里要排除掉log4j的引用之外,还需要在flink服务器上部署logback的jar包,否则就会出现上述错误。在我部署的flink服务器目录下(位置:/usr/flink/lib/),没有logback的包,只有log4j的包,所以就出错了。可以参考这篇帖子:(1条消息) Flink使用logback须知_cloudera首席用户的博客-CSDN博客_flink logback

     相关jar包下载位置:

    ——logback:Central Repository: ch/qos/logback (maven.org)

    ——slf4j:Central Repository: org/slf4j (maven.org)

    ~~~~踩坑二:springboot+flink项目本地IDE调试OK,发布到服务器执行失败~~~~~~~~~~~~~~~~~

    现象1:开发环境IDE里调试运行都ok,采用maven打包,部署到flink服务器上运行报如下错误: 

    org.apache.flink.streaming.runtime.tasks.StreamTaskException: Could not instantiate outputs in order.
        at org.apache.flink.streaming.api.graph.StreamConfig.getOutEdgesInOrder(StreamConfig.java:485)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.createRecordWriters(StreamTask.java:1341)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.createRecordWriterDelegate(StreamTask.java:1325)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.<init>(StreamTask.java:327)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.<init>(StreamTask.java:308)
        at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.<init>(SourceStreamTask.java:76)
        at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.<init>(SourceStreamTask.java:72)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.flink.runtime.taskmanager.Task.loadAndInstantiateInvokable(Task.java:1525)
        at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:730)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
        at java.lang.Thread.run(Thread.java:748)
    Caused by: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner.keySelector of type org.apache.flink.api.java.functions.KeySelector in instance of org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner
        at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
        at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2410)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2404)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
        at java.util.ArrayList.readObject(ArrayList.java:799)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2295)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
        at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:615)
        at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:600)
        at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:587)
        at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:541)
        at org.apache.flink.streaming.api.graph.StreamConfig.getOutEdgesInOrder(StreamConfig.java:482)
     
    现象2:开发环境IDE里调试运行都ok,采用idea打包,部署到flink服务器上运行报如下错误:
    org.apache.flink.util.FlinkRuntimeException: Could not execute application. at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:88) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70) at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:102) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ... 7 more Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No auto configuration classes found in META-INF/spring.factories. If you are using a custom packaging, make sure that file is correct. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84) ... 10 more Caused by: java.lang.IllegalArgumentException: No auto configuration classes found in META-INF/spring.factories. If you are using a custom packaging, make sure that file is correct. at org.springframework.util.Assert.notEmpty(Assert.java:470) at org.springframework.boot.autoconfigure.AutoConfigurationImportSelector.getCandidateConfigurations(AutoConfigurationImportSelector.java:180) at org.springframework.boot.autoconfigure.AutoConfigurationImportSelector.getAutoConfigurationEntry(AutoConfigurationImportSelector.java:123) at org.springframework.boot.autoconfigure.AutoConfigurationImportSelector$AutoConfigurationGroup.process(AutoConfigurationImportSelector.java:434) at org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorGrouping.getImports(ConfigurationClassParser.java:879) at org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorGroupingHandler.processGroupImports(ConfigurationClassParser.java:809) at org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorHandler.process(ConfigurationClassParser.java:780) at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:193) at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:331) at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:247) at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:311) at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:112) at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:746) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:564) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434) at org.springframework.boot.SpringApplication.run(SpringApplication.java:338) at catsti.innovation.flink.common.SpringBootApplicationUtil.run(SpringBootApplicationUtil.java:38) at catsti.innovation.flink.CarCountApplication.main(CarCountApplication.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)  
     
    原因:还未找到。这坑如果爬不出来估计就得放弃springboot了。
  • 相关阅读:
    【原】从/dev/null重新打开标准输出
    Go 接口转换的一个例子
    关于软件编译安装的出错处理
    【原】GO 语言常见错误
    HP平台由于变量声明冲突导致程序退出时的core
    动态链接库加载出错:cannot restore segment prot after reloc: Permission denied
    Windows VC++常见问题汇总
    .net:System.Web.Mail vs System.Net.Mail应该用哪个
    网络管理的功能
    Hello World! — 用 Groovy 编写的 Java 程序
  • 原文地址:https://www.cnblogs.com/coldlight/p/15502645.html
Copyright © 2011-2022 走看看