Threat Hunting Fundamentals

  Threat hunting is the process of implementing active controls that aim to detect adversaries inside the enterprise perimeter. This approach provides another level of defense by actively finding any vulnerabilities and looking for signs of malicious indications within the organization’s network. With this type of approach, it can help the hunters uncover hidden threats on the network. The purpose of this activity is to reduce the dwell time of detecting the presence of an attacker on the organization’s perimeter or to intercept potential attacks before the damage is done or to mitigate the damage of an attack in progress.

The increasing sophistication of cyber criminals breaching organizations today is rising at a high rate, causing massive damage to the public and to big or small organizations’ property. Seeing today’s metrics, cyber crime activities seems to show no signs of slowing down. The reason is that breaching from organization to organization pays well for these criminals, and for them, it is a profitable business and allows them to have a budget for research and development that can enhance their weapons that can be used for future attacks.

To patch this problem, many organizations now implement an active approach of defense because automated security tools are not enough to actively detect these sophisticated attackers on the network. Security tools start by protecting the organization’s perimeter from attackers, while active approach or threat hunting starts by drafting a hypothesis that assumes the organization is breached and the only way to gather evidence is to actively investigate through drafted hypothesis and gather artifacts and evidence that can be used as indicators of compromise.

Threat hunting requires skills and expertise since it relies on humans rather than security products. Although it relies on human skills, threat hunting cannot be done without technologies and processes. What is special about humans is, that they can formulate their own methodical approach to help solve the problem at hand, which technology cannot.

An effective threat hunting strategy is based on a mindset and a methodical approach that allows the security analyst to think like a threat actor and use that understanding to determine what clues to look for that could indicate an attack is underway.

APT Attack Common Stages

APTs are a stealthy threat group that is typically a nation state or state-sponsored group. Their aim is to either steal confidential information or gain authorized access to another organization’s network and remain undetected for an extended period of time in order to spy on their victim. What makes them an advanced threat is that they are sponsored and constantly improving their TTP (Technique, Tactiques, and Procedure) through continuous research and development.

In this stage, like a typical burglar forcing their way through the door, cyber criminals usually gain entry through an infected file, a USB stick, an exploited vulnerability, or in some cases, someone giving them the key to the door.

In most cases, attackers gain initial access, or what we call a stage 1 Initial Compromise, through the following:

      • Phishing
      • Spear Phishing
      • Whaling

Then they typically exploit a vulnerability, either a human vulnerability or a software vulnerability.

In some special cases, for example the famous Solarwinds backdoor which, companies that use their service are vulnerable since the threat actors distribute their malware together with a signed binary that will not be flagged as malicious by security tools. 

To mitigate these types of attacks, organizations can invest in user awareness and conduct phishing tests, which can test their organization.

In this stage, a level 1 initial compromise is established, just like a burglar would after they successfully breach the house. Now they contact the other party for further instruction.

What happens on this stage is that actors perform an enumeration of the system, exploring the territory and looking for possible accounts with administrator privileges to compromise. But, at this stage, threat actors aim to be as stealthy as possible. What they do to become stealthy is that they move slowly. Sometimes they let their malware sleep for a few days, weeks, or months to avoid the noise it can make that might blow up their cover.

At this stage, typically, you may want to look for possible enumeration commands like:

      • whoami
      • net user *
      • tasklist 
      • Get-Services 

#note: System Administrators use these commands everyday to complete their tasks; look for anomalies such as executing these commands within a minute and the account name associated with it.  Again, these commands are used for legitimate purposes, just look for anomalies.

At this stage, an account with administrator privilege is identified and compromised. With this account, attackers can now perform to move laterally inside the network, which aims to elevate from level 1 initial compromise to level 2 compromise, jumping from machine to machine to reach the keys to the kingdom.

At this stage, attackers perform their moves as stealthily as possible to avoid detection. Attackers perform “living off the land” techniques, which use legitimate tools that are used by administrators to propagate to the network.

Threat actors enumerate the following in this stage:

      • local routing tables: routeprint
      • ARP cache: arp cache
      • User and Account Privs: net user / net groups
      • System Data: tasklist, wmi, powershell

Those are some of the commands to monitor. Again, those commands are legitimate and used by administrators in their day-to-day tasks. Just look for anomalies and odd usage of the tool and you are set.

At this stage, threat actors have successfully leveraged from level1 compromise to the keys to the kingdom at this point. Once adversaries have found valuable data within the network, their next task is to copy that data outside of the company’s network, to a target location of their choosing, presumably a place outside of the jurisdiction of the authorities of the victim organization and one that masks the identity of the attackers.

This final step is the payoff for the attackers, allowing them to monetize their operation by ransoming the data, selling it on the black market, or otherwise exploiting it.

What threat actors had done on this stage are:

      • Archive: Gzip, zip, tar, winrar
      • Tunneling: FTP Server, Dropbox, Google Drive 
      • Establish Destination: FQDN, DNS, URL

As a hunter, we can monitor this type of data inside our network for anomalies. In such cases, for example monitoring data transfer that exceeds up to 500MB in a single transfer or day.

#note: You must have IDS/IPS and data sources to detect these types of events inside your network. Without log/data sources it is impossible to detect exfiltration.

In this stage, threat actors try to stay on the compromised network as stealthily as possible. At this point, registry keys, kernel configurations, and security tools may have been altered, allowing them to move freely without fear of detection.

Advance threats keep the system monitoring their target for as long as possible, giving them an advantage in seeing what tactics the security team is using to hunt for specific threats and changing their TTPs to avoid detection. Also, by maintaining persistence, threat actors are able to capture new data such as company’s intellectual property.

In this stage, threat actors maintain and persist on the system through the following registry keys:

      • Services 
      • Winlogon
      • Run
      • RunOnce
      • Known_DLLs

These are some of the most common persistent techniques used by attackers.

#note: To detect such changes, hunters can use security tools and actively compare the current system’s baseline to the previous baseline with a clean state status.


Methodology of Effective Threat Hunting

Threat hunters must define and assume that the organization has already been breached rather than generally searching for random types of threats.

Before diving deep, hunters must define a specific, industry-focused threat that could be inside the environment. The scenario can be based on current threat intelligence feeds such as the MITRE ATT&CK framework, which enables hunters to search for specific APT that targets your industry, or can be a result of available threat research teams from your organization or could be from other teams.

In this step, the hunter should conduct research on the current TTPs that could be used by the attackers to target your industry. This includes the following:

      • Initial compromise
      • CNC
      • Lateral Movement
      • Exfiltration
      • Persistence

For some specific reasons hunters, must also know if there are documented known vulnerabilities like unpatched software that the organization has not yet implemented.

In this step, hunters assume that a breach has already occurred and that attackers have gained one initial foothold. The hunter assesses the goals of attacker based on the the cyber attack chain, then formulates a “guess or questions” about what the techniques, tactics, and procedures that the attacker might use and what the possible evidence they might have created that can reveal their activities.

Formulating a hypothesis is guessing and asking questions like:

    • “If adversaries successfully gain an initial foothold, then they must have found a way, which can be through phishing.”
     
    • “If adversaries are already performing lateral movement, they might use the living off the land technique.”
     
    • “If adversaries are already inside, they might cover their tracks by clearing the system event logs.”
     
    • “If adversaries are already inside, they will persist in the system through the registry.”

Those are the samples of guesses and questions that hunters already assume that they are breached. From formulating a hypothesis, hunters can start their investigation and move to step 3, which is to identify and gather evidence.

In this step, hunt teams must identify and assemble the data sources they can analyze within their hunt, as they seek to find proof or disprove their formulated hypothesis.

In this step, hunters must document the steps, including the data sources where their data comes from, to ensure that the hunt can be justified during reporting.

In this step, hunters identify and gather their evidence through the following:

      • Event log management or SIEM   
      • Email logs
      • Network logs
      • Endpoint logs
      • Firewall logs
      • Server logs

Performing a hunt with a ton of data sources can be overwhelming. This can drown the hunt team and can perform a hunt ineffectively. To patch this lurking problem, the hunt team must consider that there are legitimate activities in the system performed by administrators. What they need to look for are anomalies during its usage, such as unidentified account names, time zone, enumeration and, etc. and correlate them with other evidence sources. Effective threat hunting can also rely on endpoint, system, and network baselines, which can effectively detect if there are changes in the system or the network.

Evidence are gathered from the previous step.

In this step, gathered evidence and data are being correlated with the help of human analytical and visualization techniques. This is why threat hunting can be effectively done by humans with the help of technology because the combination of these two can uncover relationships within evidence inside the data sources.

In this step, hunters must conduct comparisons from the past baselines or have a quick chat with other IT teams to answer specific questions and concerns. Effective threat hunting must “not live by bread alone.” This means that they cannot rely on their own understanding, that they need other people like IT leaders from different departments to help them correlate what they’ve found and to know whether it is normal or malicious. Hunters on this stage must have a good understanding of what data patterns are associated with an adversary’s activity for the given stage in the attack chain, and this can also be done by correlating evidence with the MITRE ATT&CK framework.

Documentation.
 
In this step, hunters must present the types of evidence they collected, the techniques, tactics and procedures of how the analysis is performed during the hunt, and the conclusions that are proved during the process.
 
This can be the following:
 
      • Uncovered Vulnerabilities
      • Detected phishing incident
      • Data Exfiltration
      • Data stolen
      • Account usage anomalies
      • Living Off The Land tools detected
      • Metrics of the hunt process
      • Root cause of compromise detected
      • Scope of affected machines, accounts and applications
      • Description of the technique revealed
      • IOC to be used to detect similar attacks
      • Lesson learned and Remediation
      • Recommendation
 
During this stage, reporting the data evidence presented to the management will initiate a discussion that can lead to incident response, patching newly uncovered vulnerabilities, and improving the holes in your environment that were found during the hunt process. This will also lead to improvements in the organization’s infrastructure, such as patching blind spots and monitoring the specific targets that can help prevent future attacks and improve the whole organization’s infrastructure.