How to check Logstash's pulse

Have you ever wondered if Logstash was sending data to your outputs? There's a brand new way to check if Logstash has a "pulse." Introducing the heartbeat input plugin! It’s bundled with Logstash 1.5 so you can start using it immediately!

Why?

Logstash currently has a single pipeline. All events generated by inputs travel through the filter block, and then out of Logstash through the output block.

Even if you have multiple outputs and are separating events using conditionals all events pass through this single pipeline. If any one of your outputs backs up, the entire pipeline stops flowing. The heartbeat plugin takes advantage of this to help you know when the flow of events slows, or stops altogether.

How?

The heartbeat plugin sends a message at a definable interval. Here are the options available for the message configuration parameter:

• Any string value: The message field will contain the specified string value.  If unset, the message field will contain the string value ok
• epoch: Rather than a message field, this will result in a clock field which will contain the current epoch timestamp (UTC). If you are unfamiliar with this, it means the number of seconds elapsed since Jan 1, 1970.
• sequence: Rather than a message field, this will result in a clock field which will contain a number. At start time, the sequence starts at zero and will increment each time your specified interval time has elapsed. Note that this means that if you restart Logstash, the counter resets to zero again.

Examples

Be sure to assign a type to your heartbeat events. This will make it possible to conditionally act on these events later on.

"ok" Message

Perhaps you only want to know that Logstash is still sending messages. Your monitoring system can interpret an "ok" received within a time window as an indicator that everything is working. Your monitoring system would be responsible for tracking the time between "ok" messages.

I can send the default "ok" message every 10 seconds like this:

input {
  heartbeat {
    interval => 10
    type => "heartbeat"
  }
  # ... other input blocks go here
}

The events would look like this:

{"message":"ok","host":"example.com","@version":"1","@timestamp":"2015-03-18T17:05:24.696Z","type":"heartbeat"}
{"message":"ok","host":"example.com","@version":"1","@timestamp":"2015-03-18T17:05:34.696Z","type":"heartbeat"}
{"message":"ok","host":"example.com","@version":"1","@timestamp":"2015-03Read More

Epoch timestamp

Perhaps your monitoring system uses unix timestamps to track event timing (like Zabbix, for example). If so, you can use the epoch timestamp in the clock field to calculate the difference between "now" and when Logstash generated the heartbeat event. You can calculate lag in this way. This may be especially useful if you inject the heartbeat before events go into a broker, or buffering system, like Redis, RabbitMQ, or Kafka. If the buffer begins to fill up, the time difference will become immediately apparent. You could use this to track the elapsed time--from event creation, to indexing--for your entire Logstash pipeline.

This example will send the epoch timestamp in the clock field:

input {

  heartbeat {
    message => "epoch"
    interval => 10
    type => "heartbeat"
  }
  # ... other input blocks go here
}

The events would look like this:

{"clock":1426698365,"host":"example.com","@version":"1","@timestamp":"2015-03-18T17:06:05.360Z","type":"heartbeat"}
{"clock":1426698375,"host":"example.com","@version":"1","@timestamp":"2015-03-18T17:06:15.364Z","type":"heartbeat"}
{"clock":1426698385,"host":"example.com","@version":"1","@timestamp":"2015Read More

Sequence of numbers

This example makes it easy to immediately check if new events are occurring because the clock will continuously increase.

input {
  heartbeat {
    message => "sequence"
    interval => 10
    type => "heartbeat"
  }
  # ... other input blocks go here
}

The events would look like this:

{"clock":1,"host":"example.com","@version":"1","@timestamp":"2015-03-18T17:08:13.024Z","type":"heartbeat"}
{"clock":2,"host":"example.com","@version":"1","@timestamp":"2015-03-18T17:08:23.027Z","type":"heartbeat"}
{"clock":3,"host":"example.com","@version":"1","@timestamp":"2015-03-18T17Read More

 

Output

Now let's add a conditional to send this to our monitoring system, and not to our other outputs:

output {
  if [type] == "heartbeat" {
    # Define the output block for your monitoring system here
  } else {
  # ... other output blocks go here
  }
}

Of course, if you do want your heartbeat messages to be indexed alongside your log data, you are free to do so.

Conclusion

The new heartbeat plugin provides a simple, but effective way to monitor the availability of your Logstash instances right now. We have big plans for the future, though.  Take a look at our road map!

In the future we plan to have a full API, complete with visibility into the pipeline, plugin performance, queue status, event throughput and so much more.  We are super excited to bring these improvements to you!

Happy Logstashing!

input {
    heartbeat {
        tags => ["heartbeat"]
        type => "heartbeat"
        message => "epoch"
        interval => 15
    }
}

output {
    if "heartbeat" in [tags] {
        file {
            path => "/var/log/cloudchef/logstash/logstash-hearbeat.log"
        }
    }
}