Graylog

Install

I wanted to run Graylog in a VM and Graylog supplies appliance images in OVA format, among others.
We can extract the disk from the OVA file and convert it to a format I can use. This applies not just to the Graylog appliance, but any OVA image you might have.
Let's get started...

Download the newest .ova Graylog virtual appliance from here. At this time that is graylog-2.0.3-1.ova.

$ wget https://packages.graylog2.org/releases/graylog-omnibus/ova/graylog-2.0.3-1.ova

I'm using KVM/QEMU so this OVA file isn't going to do very well for me. I just need the disk image.
OVA Files are just tarballs. You can change the file extension to .tar if it makes this clearer, but this is not required.

$ mv graylog-2.0.3.ova graylog-2.0.3.tar

Extract it

$ tar xvf graylog-2.0.3.ova

$ qemu-img -h

Will give you a list of supported formats

Convert the disk to your favorite format. I chose qcow2.

$ cd graylog
$ qemu-img convert -p -O qcow2 graylog-disk1.vmdk graylog-disk1.qcow2

UPDATE: you can skip these steps by just grabbing a qcow2 image directly from Graylog here

Now create a new VM that will boot your newly converted disk image.
Use the graylog.ovf file for hints on cpu/ram specs. Now turn it on.

Continue with the official Graylog VM Appliance setup from here.

Inputs

I used to have custom imput, but realized I didn't need it.
Here is the default input that comes configured on the appliance, for reference.

allow_override_date: true
bind_address: 0.0.0.0
expand_structured_data: false
force_rdns: false
override_source: <empty>
port: 514
recv_buffer_size: 262144
store_full_message: false

Remote rsyslog

Edit /etc/rsyslog.conf on most modern linux distributions, or create a new .conf file in /etc/rsyslog.d/ if your system supports it.
Add the following, replacing YOUR_GRAYLOG_IP and GRAYLOG_INPUT_PORT with appropriate info for your setup.

*.* @YOUR_GRAYLOG_IP:GRAYLOG_INPUT_PORT;RSYSLOG_SyslogProtocol23Format

Aaaaand don't forget to restart the rsyslog service

$ sudo service rsyslog restart

Streams

I have a number of hosts on my network pushing logs to graylog. (getting pfsense to send sane logs was a chore. see the Pfsense Syslog? project.

I use 2 streams. One to capture SIP authentication failures from my asterisk system, the other to capture ssh authentication failures from any host.

For asterisk, my rules are:

  • application_name field must match "asterisk"
  • message field must contain SecurityEvent="InvalidPassword"

For SSH, my rules are:

  • application_name field must be sshd
  • message field must contain authentication failure

Alarms

Now that I have streams set up to trigger on authentication failures (i.e. some bot trying to guess my root password via ssh or trying to enumerate extensions on my asterisk system), what's next?

Oi! Push no'ifications!

I want my phone to ding

I'm so hardcore I want to be notified via my phone every time one of these streams gets a message (more exactly they are set up to trigger when there are "more than 0 entries in the last 1 minute" so it only checks every minute)

I use Notify My Android (NMA).
Two things we need to make this work are:

  1. The Execute Script Plugin that lets you run an arbitrary shell command as an alarm callback, and
  2. NMA provides a neat perl script that we can use

Install the Execute Script plugin and copy the nma.pl script to the graylog server. Make sure to make the script executable. I put the script in /opt/graylog/bin. Also create a file in the same directory to hold your NMA API key. Put that in /opt/graylog/bin/api.key.

$ chmod +x nma.pl

Now, in each of our auth failure streams, set up a new callback of type "Execute Script" and enter your nma.pl command. Here's the one I'm currently using for the sshd notifications:

/opt/graylog/bin/nma.pl -apikeyfile=/opt/graylog/bin/api.key -application=graylog '-event=sshd auth attempt failure' '-notification=Someone Is smacking their balls on your sshd' -priority=1

At this point we have a system that sends a push notification every time a minute elapses that has at least 1 failed asterisk or sshd login attempt. Then I go into pfSense and add the offending IP into my static floating blacklist rule. Certainly not very efficient and quickly the amount of dinging is irritating to me and everyone in my family who wants to sleep for more than about 3 minutes at a time.

Good god, make it stop!

I no longer want my phone to ding

Still to come. how to make it not ding quite as much.... in fact the dinging is completely optional when we're done.

Page last modified on September 02, 2016