【原创】大叔经验分享（101）marathon反复重启

marathon反复重启

# systemctl status marathon
● marathon.service - Marathon
   Loaded: loaded (/usr/lib/systemd/system/marathon.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2020-01-10 19:57:20 CST; 284ms ago
  Process: 9919 ExecStart=/usr/share/marathon/bin/marathon (code=exited, status=103)
  Process: 9916 ExecStartPre=/bin/chmod 755 /run/marathon (code=exited, status=0/SUCCESS)
  Process: 9913 ExecStartPre=/bin/chown marathon:marathon /run/marathon (code=exited, status=0/SUCCESS)
  Process: 9910 ExecStartPre=/bin/mkdir -p /run/marathon (code=exited, status=0/SUCCESS)
 Main PID: 9919 (code=exited, status=103)

通过journalctl查看日志如下：

Jan 10 19:20:53 test-002 marathon[26905]: [2020-01-10 19:20:53,773] ERROR abdicateLeadership was called (mesosphere.marathon.core.election.ElectionServiceImpl:scala-execut
Jan 10 19:20:53 test-002 marathon[26905]: mesosphere.marathon.Exception: abdicateLeadership
Jan 10 19:20:53 test-002 marathon[26905]: at mesosphere.marathon.core.election.ElectionServiceImpl.abdicateLeadership(ElectionService.scala:154)
Jan 10 19:20:53 test-002 marathon[26905]: at mesosphere.marathon.MarathonSchedulerService.$anonfun$startLeadership$7(MarathonSchedulerService.scala:254)
Jan 10 19:20:53 test-002 marathon[26905]: at mesosphere.marathon.MarathonSchedulerService.$anonfun$startLeadership$7$adapted(MarathonSchedulerService.scala:253)
Jan 10 19:20:53 test-002 marathon[26905]: at scala.Option.foreach(Option.scala:257)
Jan 10 19:20:53 test-002 marathon[26905]: at mesosphere.marathon.MarathonSchedulerService.$anonfun$startLeadership$6(MarathonSchedulerService.scala:253)
Jan 10 19:20:53 test-002 marathon[26905]: at mesosphere.marathon.MarathonSchedulerService.$anonfun$startLeadership$6$adapted(MarathonSchedulerService.scala:240)
Jan 10 19:20:53 test-002 marathon[26905]: at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
Jan 10 19:20:53 test-002 marathon[26905]: at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
Jan 10 19:20:53 test-002 marathon[26905]: at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
Jan 10 19:20:53 test-002 marathon[26905]: at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
Jan 10 19:20:53 test-002 marathon[26905]: at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
Jan 10 19:20:53 test-002 marathon[26905]: at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

检查mesos发现mesos异常，最终定位原因为zookeeper节点之间数据不同步，mesos在zookeeper上维护master leader信息

[zk: 192.168.0.1:2181(CONNECTED) 1] ls /mesos

[json.info_0000000114, log_replicas, json.info_0000000115]

这个数据不同步导致mesos找不到master，进而导致marathon异常，将有问题的zookeeper节点stop，清空数据目录后重启，数据一致后问题解决；