Register for our May 18th Webinar with Carahsoft: Optimize Your Network from Core to Edge to Cloud Register Now
Skip to Main Content

Encrypted Traffic Analysis

Encryption protects end-user privacy, and its adoption is increasing rapidly. Unfortunately, one side-effect of the increased use of encryption is the erosion of visibility for network defenders. Encrypted Traffic Analysis is a way to restore network visibility for defenders while maintaining privacy for users.

To understand encrypted traffic analysis, we first need to define cryptanalysis. Cryptanalysis is derived from two Greek root words, CRYPT – hidden, ANALYSIS – loosen, which investigates the hidden aspects of communication systems. Historically, there are two kinds of cryptanalysis in the context of network security: breaking encryption and side-channel analysis of potential information “leaks.” Encrypted traffic analysis is a side-channel analysis that allows network defenders to identify malware communications and threat actors hiding activity in secure encrypted traffic.

There are three levels and categories of Encrypted Traffic Analysis:

Level Category Methodology Technology
Level 1 Simple Traffic Analysis Network Transaction Monitoring
Level 2 Enhanced Certificate Analysis Deep Packet Inspection
Level 3 Advanced Cryptanalysis Deep Packet Dynamics (DPD)

LEVEL 1: Traffic Analysis – Information available in the network transaction (IP address, ports, protocol, and timing)

LEVEL 2: Certificate Analysis – Looking at the particulars of the encryption used (cipher suites and extensions, etc.)

LEVEL 3: Deep Packet Dynamics (DPD) – Looking at network traffic characteristics and traits, such as patterns in the sequence of packet lengths and times. Learn more about DPD.

Techniques for Encrypted Traffic Analysis ThreatEye groups ETA techniques into three main categories:

LiveAction-fingerprint
Fingerprint

Unique identification of network entities such as devices, domains, IPs, users, and connections

LiveAction-map
Map

Identify meaningful relationships between network entities on the globe, in the network, and with similar features

LiveAction-profile
Profile

Observe changing behavior of network entities over time with comparisons to established baselines.

Level Fingerprint Map Profile
Level 1: Traffic Analysis Protocol Fingerprint – each machine has a protocol fingerprint based on the services it utilizes or provides Shared IP or ASN – often multi-tenant servers host multiple malicious sites in the same location Pattern of life/time of day – traffic at odd hours of the day or night can indicate malicious traffic
Level 2: Certificate Analysis TLS Fingerprinting – unique combinations of cipher suites and extensions Malware use of TLS – identify malware propensity with specific fingerprints Novel Fingerprints – the emergence of new fingerprints can indicate the presence of malware or other unwanted software on the network
Level 3: Deep Packet Dynamics OS Fingerprinting – identify host and IoT device types from “instinctive” packet header details Application ID – characterize applications based on similar byte patterns of typical usage Interactive Sessions – detect usage of Remote Access Toolkits (RATs) by identifying the characteristic patterns of transmission of individual keystrokes

Encrypted Traffic Analysis to Uncover Command & Control (C2) Activity

Malicious threat actors and malware system operators communicate with infected target systems using a set of techniques called Command and Control (C2). Threat actors employ C2 techniques to mimic expected, benign traffic using common ports and standard encryption protocols to avoid detection. Despite these precautions, Encrypted Traffic Analysis with machine learning effectively uncovered different types of C2 activity.

Level 1 Defends Against: Beaconing

An infected system uses beaconing to reestablish contact with the control infrastructure. This activity is characterized by sending identical messages at a specified interval. When repeated messages surface, Level 1 ETA recognizes potential beaconing activity by capturing patterns within both the communication intervals and the byte totals in both directions.

Level 2 Defends Against: TLS Fingerprinting

The encryption software libraries used by malware often differ from the encryption libraries used by the browser, apps, and other legitimate software. When beaconing activity identifies a suitable command host, an encrypted C2 protocol initiates a secure connection using these same libraries. These events create a distinctive signature that can be identified on the network. ThreatEye will identify new encrypted sessions, both through libraries used and by identifying encrypted fingerprints, alerted as “new tls sha1 found” and “new tls ja3s found”.

Encrypted Fingerprinting

Level 3 Defends Against: Sequence of Packet Lengths

Once a secure connection is made, communication between the C2 infrastructure and the infected target begins. Due to the specific nature of the C2 commands, the number and size of the packets being exchanged over this connection often have characteristic signatures that distinguish them from typical web traffic. Here, real-time analysis of packet traits like these can yield signature deviations that point to C2 activity.

In summary, ETA combined with machine learning techniques effectively identifies malicious C2 activity on the network. Despite having no visibility into the content of the exchange, ETA tells us a great deal about encrypted traffic and provides valuable insights to aid network defenders.

Defending Against Exfiltration with Encrypted Traffic Analysis

Once a threat actor has identified information of value, they must find a way to transport that data out of the network to resources they control. Bulk transfers of large data sets are easily detectable; therefore, attackers use other, stealthy techniques to exfiltrate data.

Level 1 Defends Against: “Low and Slow”

Rather than exfiltrating the data in a single transfer, threat actors can choose to release small amounts of data over time. Basic traffic analysis recognizes this “low and slow” technique by tracking byte totals over time.

Level 2 Defends Against: Tunneling

Tunneling encapsulates one protocol—or layer—of encryption within another one. This type of traffic has a different packet dynamic profile than standard traffic on that port. ThreatEye NV’s parsing capabilities can even detect nested layers of encryption. Some forms of tunneling, such as DNS tunneling, are also detectable by analyzing the ratio of bytes transferred in each direction during a connection.

Level 3 Defends Against: Cloud Service

Each cloud application has a highly recognizable packet dynamics fingerprint tied to its typical usage. Exfiltration to cloud-based accounts requires extensive data transfer. Profiling behavioral usage can highlight and identify exfiltration events outside regular enterprise activities.

Encrypted Traffic Analysis, coupled with machine learning capabilities, evaluates complex data patterns over time and highlights which activities grade as normal (potentially benign) or abnormal (potentially malicious)—all without access to the content of the data being transferred.

Free Trial

ThreatEye Free Trial