Home Monitoring: Sending Zeek logs to ELK

Prerequisites

This post marks the second instalment of the “Create enterprise monitoring at home” series, here is part one in case you missed it. In this post, we’ll be looking at how to send Zeek logs to ELK Stack using Filebeat. A few things to note before we get started,

Installing Elastic

Installing Elastic is fairly straightforward, firstly add the PGP key used to sign the Elastic packages.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

If you need to, add the apt-transport-https package.

sudo apt-get install apt-transport-https

Then add the elastic repository to your source list.

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Finally install the ElasticSearch package.

sudo apt-get update && sudo apt-get install elasticsearch

Once installed, we need to make one small change to the ElasticSearch config file, /etc/elasticsearch/elasticsearch.yml. We’re going to set the bind address as 0.0.0.0, this will allow us to connect to ElasticSearch from any host on our network. It’s worth noting, that putting the address 0.0.0.0 here isn’t best practice, and you wouldn’t do this in a production environment, but as we are just running this on our home network it’s fine.

Once that’s done, let’s start the ElasticSearch service, and check that it’s started up properly.

sudo service elasticsearch start
sudo service elasticsearch status 

You should get a green light and an active running status if all has gone well. Next, we want to make sure that we can access Elastic from another host on our network. I’m going to use my other Linux host running Zeek to test this. Run the curl command below from another host, and make sure to include the IP of your Elastic host.

curl -X GET "IP OF YOUR ELASTIC HOST:9200/?pretty" 

If all has gone right, you should get a reponse simialr to the one below.

• 

Installing Kibana

Now it’s time to install and configure Kibana, the process is very similar to installing elastic search. We’ve already added the Elastic APT repository so it should just be a case of installing the Kibana package.

sudo apt-get update && sudo apt-get install kibana

One it’s installed we want to make a change to the config file, similar to what we did with ElasticSearch. Change the server host to 0.0.0.0 in the /etc/kibana/kibana.yml file.

• 

Once it’s installed, start the service and check the status to make sure everything is working properly.

sudo service kibana start
sudo service kibana status 

You should get a green light and an active running status if all has gone well. Now let’s check that everything is working and we can access Kibana on our network. Browse to the IP address hosting kibana and make sure to specify port 5601, or whichever port you defined in the config file. You should see a page similar to the one below.

• 

Configuring Zeek

Now that we’ve got ElasticSearch and Kibana set up, the next step is to get our Zeek data ingested into ElasticSearch. There are a couple of ways to do this. Kibana has a Filebeat module specifically for Zeek, so we’re going to utilise this module.

First, go to the SIEM app in Kibana, do this by clicking on the SIEM symbol on the Kibana toolbar, then click the “add data” button. As shown in the image below, the Kibana SIEM supports a range of log sources, click on the “Zeek logs” button.

• 

You have to install Filebeats on the host where you are shipping the logs from. So in our case, we’re going to install Filebeat onto our Zeek server. Follow the instructions specified on the page to install Filebeats, once installed edit the filebeat.yml configuration file and change the appropriate fields. The username and password for Elastic should be kept as the default unless you’ve changed it. Make sure to change the Kibana output fields as well.

Once that is done, we need to configure Zeek to convert the Zeek logs into JSON format. First, stop Zeek from running.

zeekctl stop

Then edit the line @load policy/tuning/json-logs.zeek to the file  /opt/zeek/share/zeek/site/local.zeek

• 

Restart zeek

zeekctl deploy

And now check that the logs are in JSON format. Even if you are not familiar with JSON, the format of the logs should look noticeably different than before.

tail -f /opt/zeek/logs/current/dns.log

Now we need to configure the Zeek Filebeat module. First, enable the module.

sudo filebeat modules enable zeek

Then edit the config file, /etc/filebeat/modules.d/zeek.yml. We need to specify each individual log file created by Zeek, or at least the ones that we wish for Elastic to ingest. For each log file in the /opt/zeek/logs/ folder, the path of the “current” log, and any previous log have to be defined, as shown below.

dns:
   enabled: true
   var.paths: [ "/opt/zeek/logs/current/dns.log", "/opt/zeek/logs/*.dns.json" ]

If there are some default log files in the opt folder, like capture_loss.log that you do not wish to be ingested by Elastic then simply set the “enabled” field as false. It’s important to set any logs sources which do not have a log file in /opt/zeek/logs as enabled: false, otherwise, you’ll receive an error. Also be sure to be careful with spacing, as YML files are space sensitive.

Once that’s done, you should be pretty much good to go, launch Filebeat, and start the service.

sudo filebeat setup
sudo service filebeat start

If everything has gone right, you should get a successful message after checking the

• 

If you go the network dashboard within the SIEM app you should see the different dashboards populated with data from Zeek! The dashboards here give a nice overview of some of the data collected from our network. We’ll learn how to build some more protocol-specific dashboards in the next post in this series.

Enriching with Suricata

This next step is an additional extra, it’s not required as we have Zeek up and working already. However adding an IDS like Suricata can give some additional information to network connections we see on our network, and can identify malicious activity. Some people may think adding Suricata to our SIEM is a little redundant as we already have an IDS in place with Zeek, but this isn’t really true. While Zeek is often described as an IDS, it’s not really in the traditional sense. Zeek collects metadata for connections we see on our network, while there are scripts and additional packages that can be used with Zeek to detect malicious activity, it does not necessarily do this on its own. Suricata is more of a traditional IDS and relies on signatures to detect malicious activity. Now I often question the reliability of signature-based detections, as they are often very false positive heavy, but they can still add some value, particularly if well-tuned.

Installing Suricata

I’m not going to detail every step of installing and configuring Suricata, as there are already many guides online which you can use. I used this guide as it shows you how to get Suricata set up quickly. I’m going to install Suricata on the same host that is running Zeek, but you can set up and new dedicated VM for Suricata if you wish. Just make sure you assign your mirrored network interface to the VM, as this is the interface in which Suricata will run against.

Once you have Suricata set up its time configure Filebeat to send logs into ElasticSearch, this is pretty simple to do. Navigate to the SIEM app in Kibana, click on the “add data” button, and select Suricata Logs

Follow the instructions, they’re all fairly straightforward and similar to when we imported the Zeek logs earlier. Step 3 is the only step that’s not entirely clear, for this step, edit the /etc/filebeat/modules.d/suricata.yml by specifying the path of your suricata.json file.

var.paths: ["/my/path/suricata.json"]

Once that’s done, complete the setup with the following commands.

./filebeat setup
./filebeat -e

If all has gone right, you should recieve a success message when checking if data has been ingested.

We can also confirm this by checking the networks dashboard in the SIEM app, here we can see a break down of events from Filebeat.

Conclusion

And that brings this post to an end! It’s fairly simple to add other log source to Kibana via the SIEM app now that you know how. I’d recommend adding some endpoint focused logs, Winlogbeat is a good choice.

I’d say the most difficult part of this post was working out how to get the Zeek logs into ElasticSearch in the correct format with Filebeat. It’s not very well documented. In the next post in this series, we’ll look at how to create some Kibana dashboards with the data we’ve ingested.