zoukankan      html  css  js  c++  java
  • flume 使用遇到问题及解决

    1. ../flume/fchannel/spool/data/ 目录下发生缓存文件积压

      可能原因:同一时间同一客户端下向两个监控目录mv文件;或同一时间多个客户端向服务端上传文件

    2.清空../flume/fchannel/spool/data/ 目录下文件重启后,监控目录下的文件发生积压,没有上传

      flume.log里面重复一个异常:

    java.lang.IllegalStateException: Channel closed [channel=c1]. Due to java.lang.NullPointerException: null
            at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:352)
            at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
            at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:181)
            at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:188)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
            at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
            at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
            at java.lang.Thread.run(Thread.java:662)
    Caused by: java.lang.NullPointerException
            at org.apache.flume.channel.file.Log.writeCheckpoint(Log.java:1028)
            at org.apache.flume.channel.file.Log.replay(Log.java:472)
            at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
            at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
            ... 9 more

      原因是:清空../flume/fchannel/spool/data/ 目录下文件的同时也需要清空../flume/fchannel/spool/checkpoint/ 目录下的文件

    另,有几个关于flume的问题:

    1. flume支持监控目录有子目录的情况,但是同时向两个或多个子目录mv文件的时候,对文件大小有没有限制?

    2. flume支持同时多少客户端向同一监控目录下mv文件呢?

    3. flume客户端的/opt/soft/flume/fchannel/spool/data目录下的文件的作用是?

    4. 怎么有效验证flume客户端上传文件是否成功呢?

  • 相关阅读:
    bugku细心地大象
    【学术篇】一些水的不行的dp
    【笔记篇】莫队算法(一)
    【学术篇】luogu1351 [NOIP2014提高组] 联合权值
    【学术篇】网络流24题——方格取数加强版
    【学术篇】SDOI2009 SuperGCD
    【学术篇】网络流24题——方格取数问题
    【模板篇】A* 寻路算法
    【模板篇】k短路 SDOI2010 魔法猪学院
    【学术篇】SDOI2009 最优图像
  • 原文地址:https://www.cnblogs.com/-wangjiannan/p/4072108.html
Copyright © 2011-2022 走看看