zoukankan      html  css  js  c++  java
  • 记录~采集机器出现load异常

    症状描述
    1、机器load增加从原来的不到1,疯狂增长到20+
    2、问题期间,磁盘容量正常、cpu正常、内存正常

    解决办法
    根据load问题,查找磁盘读写,使用iotop命令查看哪些进程磁盘读写异常

    iotop命令参数
    -o:只显示有io操作的进程
    -b:批量显示,无交互,主要用作记录到文件。
    -n NUM:显示NUM次,主要用于非交互式模式。
    -d SEC:间隔SEC秒显示一次。
    -p PID:监控的进程pid。
    -u USER:监控的进程用户。

    快捷键
    左右箭头:改变排序方式,默认是按IO排序。
    r:改变排序顺序。
    o:只显示有IO输出的进程。
    p:进程/线程的显示方式的切换。
    a:显示累积使用量。
    q:退出。

    查出flume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink进程读写磁盘异常如下,修改java配置调高内存,并重启后问题解决。
    9572 be/4 flume 479.49 K/s 1302.39 K/s 0.00 % 6.21 % java -Xmx100m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=data-etl-01:6188 -cp /usr/hdp/current/flume-server/conf/adlog:/usr/hdp/current/flume-server/lib/::/usr/lib/flume/li~p/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-
    9569 be/4 flume 664.65 K/s 2.25 M/s 0.00 % 6.10 % java -Xmx100m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=data-etl-01:6188 -cp /usr/hdp/current/flume-server/conf/adlog:/usr/hdp/current/flume-server/lib/
    ::/usr/lib/flume/li~p/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-
    9576 be/4 flume 749.31 K/s 2.08 M/s 0.00 % 3.96 % java -Xmx100m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=data-etl-01:6188 -cp /usr/hdp/current/flume-server/conf/adlog:/usr/hdp/current/flume-server/lib/::/usr/lib/flume/li~p/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-
    9632 be/4 flume 0.00 B/s 407.49 K/s 0.00 % 3.84 % java -Xmx100m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=data-etl-01:6188 -cp /usr/hdp/current/flume-server/conf/adlog:/usr/hdp/current/flume-server/lib/
    ::/usr/lib/flume/li~p/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-
    9574 be/4 flume 357.64 K/s 1482.79 K/s 0.00 % 3.70 % java -Xmx100m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=data-etl-01:6188 -cp /usr/hdp/current/flume-server/conf/adlog:/usr/hdp/current/flume-server/lib/::/usr/lib/flume/li~p/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-
    9566 be/4 flume 380.59 K/s 1899.78 K/s 0.00 % 2.97 % java -Xmx100m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=data-etl-01:6188 -cp /usr/hdp/current/flume-server/conf/adlog:/usr/hdp/current/flume-server/lib/
    ::/usr/lib/flume/li~p/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-
    9564 be/4 flume 531.72 K/s 1708.30 K/s 0.00 % 2.34 % java -Xmx100m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink -Dflume.monitoring.node=data-etl-01:6188 -cp /usr/hdp/current/flume-server/conf/adlog:/usr/hdp/current/flume-server/lib/*::/usr/lib/flume/li~p/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-

  • 相关阅读:
    Atitit.eclise的ide特性-------abt 编译
    Atitit.eclise的ide特性-------abt 编译
    Atitit python3.0 3.3 3.5 3.6 新特性 Python2.7新特性1Python 3_x 新特性1python3.4新特性1python3.5新特性1值得关注的新特性1Python3.6新特性2  Python2.7新特性Python 2.7的新特性
    Atitit python3.0 3.3 3.5 3.6 新特性 Python2.7新特性1Python 3_x 新特性1python3.4新特性1python3.5新特性1值得关注的新特性1Pyth
    Atitit intellij idea的使用总结attilax
    Atitit.eclipse 4.3 4.4  4.5 4.6新特性
    Atitit.eclipse 4.3 4.4  4.5 4.6新特性
    Atitit.eclipse 4.3 4.4  4.5 4.6新特性
    Atitit 软件国际化原理与概论
    Atitit 软件国际化原理与概论
  • 原文地址:https://www.cnblogs.com/isenhome/p/6216270.html
Copyright © 2011-2022 走看看