Encryption protects end-user privacy, and its adoption is increasing rapidly. Unfortunately, one side-effect of the increased use of encryption is the erosion of visibility for network defenders. Encrypted Traffic Analysis is a way to restore network visibility for defenders while maintaining privacy for users.
To understand encrypted traffic analysis, we first need to define cryptanalysis. Cryptanalysis is derived from two Greek root words, CRYPT – hidden, ANALYSIS – loosen, which investigates the hidden aspects of communication systems. Historically, there are two kinds of cryptanalysis in the context of network security: breaking encryption and side-channel analysis of potential information “leaks.” Encrypted traffic analysis is a side-channel analysis that allows network defenders to identify malware communications and threat actors hiding activity in secure encrypted traffic.
There are three levels and categories of Encrypted Traffic Analysis:
|Level 1||Simple||Traffic Analysis||Network Transaction Monitoring|
|Level 2||Enhanced||Certificate Analysis||Deep Packet Inspection|
|Level 3||Advanced||Cryptanalysis||Deep Packet Dynamics (DPD)|
LEVEL 1: Traffic Analysis – Information available in the network transaction (IP address, ports, protocol, and timing)
LEVEL 2: Certificate Analysis – Looking at the particulars of the encryption used (cipher suites and extensions, etc.)
LEVEL 3: Deep Packet Dynamics (DPD) – Looking at network traffic characteristics and traits, such as patterns in the sequence of packet lengths and times. Learn more about DPD.
Techniques for Encrypted Traffic Analysis ThreatEye groups ETA techniques into three main categories:
Unique identification of network entities such as devices, domains, IPs, users, and connections
Identify meaningful relationships between network entities on the globe, in the network, and with similar features
Observe changing behavior of network entities over time with comparisons to established baselines.
|Level 1: Traffic Analysis||Protocol Fingerprint – each machine has a protocol fingerprint based on the services it utilizes or provides||Shared IP or ASN – often multi-tenant servers host multiple malicious sites in the same location||Pattern of life/time of day – traffic at odd hours of the day or night can indicate malicious traffic|
|Level 2: Certificate Analysis||TLS Fingerprinting – unique combinations of cipher suites and extensions||Malware use of TLS – identify malware propensity with specific fingerprints||Novel Fingerprints – the emergence of new fingerprints can indicate the presence of malware or other unwanted software on the network|
|Level 3: Deep Packet Dynamics||OS Fingerprinting – identify host and IoT device types from “instinctive” packet header details||Application ID – characterize applications based on similar byte patterns of typical usage||Interactive Sessions – detect usage of Remote Access Toolkits (RATs) by identifying the characteristic patterns of transmission of individual keystrokes|
Encrypted Traffic Analysis to Uncover Command & Control (C2) Activity
Malicious threat actors and malware system operators communicate with infected target systems using a set of techniques called Command and Control (C2). Threat actors employ C2 techniques to mimic expected, benign traffic using common ports and standard encryption protocols to avoid detection. Despite these precautions, Encrypted Traffic Analysis with machine learning effectively uncovered different types of C2 activity.
Level 1 Defends Against: Beaconing
An infected system uses beaconing to reestablish contact with the control infrastructure. This activity is characterized by sending identical messages at a specified interval. When repeated messages surface, Level 1 ETA recognizes potential beaconing activity by capturing patterns within both the communication intervals and the byte totals in both directions.
Level 2 Defends Against: TLS Fingerprinting
The encryption software libraries used by malware often differ from the encryption libraries used by the browser, apps, and other legitimate software. When beaconing activity identifies a suitable command host, an encrypted C2 protocol initiates a secure connection using these same libraries. These events create a distinctive signature that can be identified on the network. ThreatEye will identify new encrypted sessions, both through libraries used and by identifying encrypted fingerprints, alerted as “new tls sha1 found” and “new tls ja3s found”.
Level 3 Defends Against: Sequence of Packet Lengths
Once a secure connection is made, communication between the C2 infrastructure and the infected target begins. Due to the specific nature of the C2 commands, the number and size of the packets being exchanged over this connection often have characteristic signatures that distinguish them from typical web traffic. Here, real-time analysis of packet traits like these can yield signature deviations that point to C2 activity.
In summary, ETA combined with machine learning techniques effectively identifies malicious C2 activity on the network. Despite having no visibility into the content of the exchange, ETA tells us a great deal about encrypted traffic and provides valuable insights to aid network defenders.
Defending Against Exfiltration with Encrypted Traffic Analysis
Once a threat actor has identified information of value, they must find a way to transport that data out of the network to resources they control. Bulk transfers of large data sets are easily detectable; therefore, attackers use other, stealthy techniques to exfiltrate data.
Level 1 Defends Against: “Low and Slow”
Rather than exfiltrating the data in a single transfer, threat actors can choose to release small amounts of data over time. Basic traffic analysis recognizes this “low and slow” technique by tracking byte totals over time.
Level 2 Defends Against: Tunneling
Tunneling encapsulates one protocol—or layer—of encryption within another one. This type of traffic has a different packet dynamic profile than standard traffic on that port. ThreatEye NV’s parsing capabilities can even detect nested layers of encryption. Some forms of tunneling, such as DNS tunneling, are also detectable by analyzing the ratio of bytes transferred in each direction during a connection.
Level 3 Defends Against: Cloud Service
Each cloud application has a highly recognizable packet dynamics fingerprint tied to its typical usage. Exfiltration to cloud-based accounts requires extensive data transfer. Profiling behavioral usage can highlight and identify exfiltration events outside regular enterprise activities.
Encrypted Traffic Analysis, coupled with machine learning capabilities, evaluates complex data patterns over time and highlights which activities grade as normal (potentially benign) or abnormal (potentially malicious)—all without access to the content of the data being transferred.