Logstash… When splunk is too expensive/complex.

What is it?

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs. It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.

It’s really simple… Want to monitor your syslog and send it to ElasticSearch? Check the tutorial here.

You can build your log monitoring setup by using an “INPUT”…

• amqp
• exec
• file
• gelf
• redis
• stdin
• stomp
• syslog
• tcp
• twitter
• xmpp
• zeromq

Then adding a FILTER to the data…

• date
• dns
• gelfify
• grep
• grok
• grokdiscovery
• json
• multiline
• mutate
• split

And transfering it to an OUTPUT

• amqp
• elasticsearch
• elasticsearch_river
• file
• ganglia
• gelf
• graphite
• internal
• loggly
• mongodb
• nagios
• null
• redis
• statsd
• stdout
• stomp
• tcp
• websocket
• xmpp
• zabbix
• zeromq

Practical use-case…? Use logstash to monitor certain logfiles (syslog, Oracle alert file, catalina.out, etc), add a filter to limit only see the messages that are worthwhile, and send them to a message queue (RabbitMQ?). Then use logstash on your Nagios server to read that messaging queue and add it to Nagios & Elastic Search. That way you get the needed alerting in your Nagios and are able to search via Elastic Search.