zoukankan      html  css  js  c++  java
  • problems_flume

    problems_flume

    1 kafka-console-consumer接收不到flume推送过来的消息

    原因和解决方法:需要先启动kafka,再启动flume,两者启动有先后顺序。

    2 flume启动报错

    desc:
    执行flume-ng agent -c conf -f conf/load_balancer_server.conf -n a1 -Dflume.root.logger=DEBUG,console ,报错

    errorlog:

    org.apache.flume.FlumeException: Failed to set up server socket
    at org.apache.flume.source.AvroSource.start(AvroSource.java:248)
    at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
    at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: node03/192.168.8.120:52020
    at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
    at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:106)
    at org.apache.flume.source.AvroSource.start(AvroSource.java:243)
    ... 9 more
    Caused by: java.net.BindException: Address already in use
    

    RCA: 上一次的flume进程还未关闭。
    执行jps可以看到4191 Application ,这个就是flume的进程。原因是关闭flume时,使用了Ctrl+Z,所以导致前台关闭了flume,但是后台进程未关闭。

    solution: kill -9 4191. 另外,下次关闭flume,尽量使用Ctrl+C。

    3 (未解决)flume监控目录,抓取文件内容推送给kafka,报错

    desc: flume监控目录,抓取文件内容推送给kafka,报错:
    errorlog:

    /datas/destFile/220104_YT1013_8c5f13f33c299316c6720cc51f94f7a0_2016101912_318.txt
    2019-08-06 23:04:31,434 (pool-3-thread-1) [ERROR - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:280)] FATAL: Spool Directory source r1: { spoolDir: /datas/destFile }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
    java.nio.charset.MalformedInputException: Input length = 1
    at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
    at org.apache.flume.serialization.ResettableFileInputStream.readChar(ResettableFileInputStream.java:283)
    at org.apache.flume.serialization.LineDeserializer.readLine(LineDeserializer.java:132)
    at org.apache.flume.serialization.LineDeserializer.readEvent(LineDeserializer.java:70)
    at org.apache.flume.serialization.LineDeserializer.readEvents(LineDeserializer.java:89)
    at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readDeserializerEvents(ReliableSpoolingFileEventReader.java:343)
    at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:331)
    at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:250)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    

    正常处理完文件后,应该删除该文件,应该提示这个信息:

    2019-08-06 23:04:31,412 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.deleteCurrentFile(ReliableSpoolingFileEventReader.java:492)] Preparing to delete file 
    

    问题原因:文件的字符集编码是UTF-8,要手动设置一下字符集编码。

    解决方案:a1.sources.r1.inputCharset = UTF-8 //将server.properties配置文件中的 inputCharset 设置为 UTF-8
    还是没用,未解决。

    貌似修复了程序的bug,这个问题就没再发生了,不过又有新的问题出现了:

    2019-08-07 01:41:45,377 (pool-3-thread-1) [ERROR - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:280)] FATAL: Spool Directory source r1: { spoolDir: /datas/destFile }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
    java.lang.IllegalStateException: File has been modified since being read: /datas/destFile/220605_YT1013_ba701bb6bf74afb2f14a92de320bc023_2016101808_113.txt
    at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.retireCurrentFile(ReliableSpoolingFileEventReader.java:406)
    at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:326)
    at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:250)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    ^C2019-08-07 01:42:11,310 (agent-shutdown-hook) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.stop(LifecycleSupervisor.java:78)] Stopping lifecycle supervisor 10
    2019-08-07 01:42:11,320 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:149)] Component type: CHANNEL, name: c1 stopped
    

    4

    5

  • 相关阅读:
    #背包方案 ——整数划分(Acwing900)
    #分组背包 #背包方案 ——Acwing 1013 机器分配
    #背包 #二进制优化 ——Acwing 5. 多重背包问题 II(二进制优化)
    #背包方案 AcWing 532. 货币系统
    #背包方案 ——AcWing 1021. 货币系统2
    背包问题求方案数
    有依赖的背包问题
    分组背包问题
    二维费用的背包问题
    混合背包问题
  • 原文地址:https://www.cnblogs.com/mediocreWorld/p/15184992.html
Copyright © 2011-2022 走看看