Tag: bro

Home Monitoring: Sending Zeek logs to ELK

June 25, 2020 /

Prerequisites

This post marks the second instalment of the “Create enterprise monitoring at home” series, here is part one in case you missed it. In this post, we’ll be looking at how to send Zeek logs to ELK Stack using Filebeat. A few things to note before we get started,

  • I’m running ELK in its own VM, separate from my Zeek VM, but you can run it on the same VM if you want.
    • ELK is running on a Ubuntu 18.04 VM.
  • It’s pretty easy to break your ELK stack as it’s quite sensitive to even small changes, I’d recommend taking regular snapshots of your VMs as you progress along.

Installing Elastic

Installing Elastic is fairly straightforward, firstly add the PGP key used to sign the Elastic packages. 

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

If you need to, add the apt-transport-https package. 

sudo apt-get install apt-transport-https

Then add the elastic repository to your source list. 

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Finally install the ElasticSearch package.

sudo apt-get update && sudo apt-get install elasticsearch

Once installed, we need to make one small change to the ElasticSearch config file, /etc/elasticsearch/elasticsearch.yml. We’re going to set the bind address as 0.0.0.0, this will allow us to connect to ElasticSearch from any host on our network. It’s worth noting, that putting the address 0.0.0.0 here isn’t best practice, and you wouldn’t do this in a production environment, but as we are just running this on our home network it’s fine.

Once that’s done, let’s start the ElasticSearch service, and check that it’s started up properly. 

sudo service elasticsearch start
sudo service elasticsearch status

You should get a green light and an active running status if all has gone well. Next, we want to make sure that we can access Elastic from another host on our network. I’m going to use my other Linux host running Zeek to test this. Run the curl command below from another host, and make sure to include the IP of your Elastic host. 

curl -X GET "IP OF YOUR ELASTIC HOST:9200/?pretty"

If all has gone right, you should get a reponse simialr to the one below.

Installing Kibana

Now it’s time to install and configure Kibana, the process is very similar to installing elastic search. We’ve already added the Elastic APT repository so it should just be a case of installing the Kibana package. 

sudo apt-get update && sudo apt-get install kibana

One it’s installed we want to make a change to the config file, similar to what we did with ElasticSearch. Change the server host to 0.0.0.0 in the /etc/kibana/kibana.yml file.

Once it’s installed, start the service and check the status to make sure everything is working properly. 

sudo service kibana start
sudo service kibana status

You should get a green light and an active running status if all has gone well. Now let’s check that everything is working and we can access Kibana on our network. Browse to the IP address hosting kibana and make sure to specify port 5601, or whichever port you defined in the config file. You should see a page similar to the one below.

Configuring Zeek

Now that we’ve got ElasticSearch and Kibana set up, the next step is to get our Zeek data ingested into ElasticSearch. There are a couple of ways to do this. Kibana has a Filebeat module specifically for Zeek, so we’re going to utilise this module.

First, go to the SIEM app in Kibana, do this by clicking on the SIEM symbol on the Kibana toolbar, then click the “add data” button. As shown in the image below, the Kibana SIEM supports a range of log sources, click on the “Zeek logs” button.

You have to install Filebeats on the host where you are shipping the logs from. So in our case, we’re going to install Filebeat onto our Zeek server. Follow the instructions specified on the page to install Filebeats, once installed edit the filebeat.yml configuration file and change the appropriate fields. The username and password for Elastic should be kept as the default unless you’ve changed it. Make sure to change the Kibana output fields as well.

Once that is done, we need to configure Zeek to convert the Zeek logs into JSON format. First, stop Zeek from running. 

zeekctl stop

Then edit the line @load policy/tuning/json-logs.zeek to the file  /opt/zeek/share/zeek/site/local.zeek

Restart zeek

zeekctl deploy

And now check that the logs are in JSON format. Even if you are not familiar with JSON, the format of the logs should look noticeably different than before.

tail -f /opt/zeek/logs/current/dns.log

Now we need to configure the Zeek Filebeat module. First, enable the module. 

sudo filebeat modules enable zeek

Then edit the config file, /etc/filebeat/modules.d/zeek.yml. We need to specify each individual log file created by Zeek, or at least the ones that we wish for Elastic to ingest. For each log file in the /opt/zeek/logs/ folder, the path of the “current” log, and any previous log have to be defined, as shown below. 

dns:
   enabled: true
   var.paths: [ "/opt/zeek/logs/current/dns.log", "/opt/zeek/logs/*.dns.json" ]

If there are some default log files in the opt folder, like capture_loss.log that you do not wish to be ingested by Elastic then simply set the “enabled” field as false. It’s important to set any logs sources which do not have a log file in /opt/zeek/logs as enabled: false, otherwise, you’ll receive an error. Also be sure to be careful with spacing, as YML files are space sensitive.

Once that’s done, you should be pretty much good to go, launch Filebeat, and start the service.

sudo filebeat setup
sudo service filebeat start

If everything has gone right, you should get a successful message after checking the

If you go the network dashboard within the SIEM app you should see the different dashboards populated with data from Zeek! The dashboards here give a nice overview of some of the data collected from our network. We’ll learn how to build some more protocol-specific dashboards in the next post in this series.

Enriching with Suricata

This next step is an additional extra, it’s not required as we have Zeek up and working already. However adding an IDS like Suricata can give some additional information to network connections we see on our network, and can identify malicious activity. Some people may think adding Suricata to our SIEM is a little redundant as we already have an IDS in place with Zeek, but this isn’t really true. While Zeek is often described as an IDS, it’s not really in the traditional sense. Zeek collects metadata for connections we see on our network, while there are scripts and additional packages that can be used with Zeek to detect malicious activity, it does not necessarily do this on its own. Suricata is more of a traditional IDS and relies on signatures to detect malicious activity. Now I often question the reliability of signature-based detections, as they are often very false positive heavy, but they can still add some value, particularly if well-tuned.

Installing Suricata

I’m not going to detail every step of installing and configuring Suricata, as there are already many guides online which you can use. I used this guide as it shows you how to get Suricata set up quickly. I’m going to install Suricata on the same host that is running Zeek, but you can set up and new dedicated VM for Suricata if you wish. Just make sure you assign your mirrored network interface to the VM, as this is the interface in which Suricata will run against.

Once you have Suricata set up its time configure Filebeat to send logs into ElasticSearch, this is pretty simple to do. Navigate to the SIEM app in Kibana, click on the “add data” button, and select Suricata Logs

Follow the instructions, they’re all fairly straightforward and similar to when we imported the Zeek logs earlier. Step 3 is the only step that’s not entirely clear, for this step, edit the /etc/filebeat/modules.d/suricata.yml by specifying the path of your suricata.json file. 

var.paths: ["/my/path/suricata.json"]

Once that’s done, complete the setup with the following commands.

./filebeat setup
./filebeat -e

If all has gone right, you should recieve a success message when checking if data has been ingested.

We can also confirm this by checking the networks dashboard in the SIEM app, here we can see a break down of events from Filebeat.

Conclusion

And that brings this post to an end! It’s fairly simple to add other log source to Kibana via the SIEM app now that you know how. I’d recommend adding some endpoint focused logs, Winlogbeat is a good choice.

I’d say the most difficult part of this post was working out how to get the Zeek logs into ElasticSearch in the correct format with Filebeat. It’s not very well documented. In the next post in this series, we’ll look at how to create some Kibana dashboards with the data we’ve ingested.

Create enterprise monitoring at home with Zeek and Elk (Part 1)

April 16, 2020 /

Intro

In this series, I’m going to show you how you can utilise open source technology to build your own network monitoring solution good enough to be deployed in any enterprise environment! The two core technologies that we’re going to use are Zeek (formerly Bro) and ELK.

For those unaware, Zeek is an open-source network monitoring framework which creates alerts and events based from data collected by a network tap. One way in which I used to describe Zeek to people is that it’s essentially an IDS but on steroids. It’s used throughout the industry, especially in the network anomaly space, in fact, the UK cybersecurity company Darktrace uses Zeek as a key component of their product.

The plan for this solution is to tap our home network with Zeek and feed the logs into Elk, with Elk we can run queries across our data, build out some beautiful dashboards with Kibana, and even create some analytics to automate some detections. We will also look into deploying an endpoint agent on some devices to and feed those logs into ELK too.

Prerequisites

In order to follow along, you’ll need

  • A server/old PC capable of running Zeek and ELK
    • In my case, I have an HP Proliant ML350e running ESXI
    • The server needs to have a minimum of 2 free network ports
    • And for it to run smoothly at least 8GB memory and a decent processor.
  • A managed network switch which is capable of port mirroring.

Network Architecture

The diagram below shows a rough guide to my home network. The key points to take from this diagram are the mirror tap on the network and the mirror connection. To get the most out of Zeek, you need to tap the connection on your switch that goes to your router, essentially the connection from your LAN to the internet. By doing this we will capture all traffic from our home networking going out to the internet, from devices like iPhones connected to the Wi-Fi and the PiHole acting as our DNS server.

VMware Setup

You should first check that your physical NICs on your server are visible in ESXI. Go to networking>Physical NICs in ESXi and you should see them there.

Both of these physical connections should connect into your switch, with one going into your mirror port, and the other a regular switch port. You can set your mirror port in your switch config. I have a Netgear GS110TP where port 2 is the one that goes into my router, and I have port 6 free, so that is the one I’ll configure to be the mirror port.

You’ll need to know which ports on your server correspond to which ESXi physical ports. For example, port 6 on my switch plugs into Eth1 on my server, and Eth1 is called vmnic1 in ESXi. So vmnic1 will be the mirror NIC.

Next, we need to create a virtual switch will be used for the mirror data. Go to Networking > Virtual Switches > Add Standard Virtual Switch, and enter the details shown below, selecting your mirror NIC as “Uplink 1”. Make sure that you set promiscuous mode as “accept”.

Now add a port group by going to Networking> Port Groups > Add port group, and assign the virual switch you just created to it. Again make sure Promiscuous mode is enabled.

Now you’re ready to create your virtual machine, I’m using Ubuntu Server 18.04 for mine. When creating you VM, make sure you add both network adapters, including the one you’re using for the Mirrored traffic.

Preparing for Zeek

Once you’ve got Ubuntu installed, do the usual updates. 

sudo apt-get update
sudo apt-get upgrade

Now check that both of your network interfaces are detected by Ubuntu. It is highly likely that your Mirror port will be down. As you can see from the image below, both of my interfaces are detected but only the management interface (ens160) is up with an IP address assigned.

To fix this, let’s first put the interface into promiscuous mode, and then bring it “up”. 

ip link set "your mirror int" promisc on
ip link set "your mirror int" up

Lets now check that we’re receiving traffic on the mirror port by running tcpdump.

tcpdump -i "your mirror int"

If everything has worked, you should see the mirrored traffic flowing through the interface similar to the image below.

Installing Zeek

We’re now ready to crack on and install Zeek. To start, we need to install all the perquisites. Do so by running the command below. 

sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev

Next we need to create the working directory for Zeek, for some reason Zeek does not do this by default on install. 

sudo mkdir /opt/zeek
sudo chown -R zeek:zeek /opt/zeek
sudo chmod 740 /opt/zeek

Next download Zeek with GIT 

git clone -–recursive https://github.com/zeek/zeek

Unpack the compressed files, and enter the Zeek download directory. Then set the /opt/Zeek directory we created earlier as the install directory. 

cd /home/paul/zeek
./configure --prefix=/opt/zeek

Now we’re ready to install Zeek, run the following make commands, and leave it to install (this can take some time) 

make
make install

Next we need to add the PATH environment variable

export PATH=/usr/local/zeek/bin:$PATH

And now we need to do add some basic config to the node.cfg file which is located in the /opt/etc/ directory. Uncomment the manager, proxy and worker-1 settings, and define your mirror interface in the worker-1 settings.

Now we’re ready to deploy Zeek by running the command below

zeekctl deploy

If all goes well we should not get any errors, and we can check to make sure everything started up properly by checking the status of zeek. 

zeekctl status

Everything looks good! So now let’s go and check our logs where our captured data is being written to. Logs are located in directory /opt/zeek/logs/
The directory “current” holds the logs for the current day, while logs from previous days are archived off into their own directories. There’s also a different log file for different data types, like DNS connections, HTTP connections, etc…

Lets check the DNS logs

tail -f dns.log

Awesome! So everything is working correctly!

That concludes this post, keep an eye out for part 2 of this series, we’re going to deploy ELK and feed our logs into Elastic Search where we can build some beautiful dashboards to display our captured data.

Copyright © 2024 On The Hunt

Theme by Anders Noren — Up ↑