Inferring Fine-grained Control Flow Inside SGX Enclaves with Branch Shadowing

In this paper, we explore a new, yet critical, side-channel attack against Intel Software Guard Extension (SGX), called a branch shadowing attack, which can reveal fine-grained control flows (i.e., each branch) of an enclave program running on real SGX hardware. The root cause of this attack is that Intel SGX does not clear the branch history when switching from enclave mode to non-enclave mode, leaving the fine-grained traces to the outside world through a branch-prediction side channel. However, exploiting the channel is not so straightforward in practice because 1) measuring branch prediction/misprediction penalties based on timing is too inaccurate to distinguish fine-grained control-flow changes and 2) it requires sophisticated control over the enclave execution to force its execution to the interesting code blocks. To overcome these challenges, we developed two novel exploitation techniques: 1) Intel PT- and LBR-based history-inferring techniques and 2) APIC-based technique to control the execution of enclave programs in a fine-grained manner. As a result, we could demonstrate our attack by breaking recent security constructs, including ORAM schemes, Sanctum, SGX-Shield, and T-SGX. Not limiting our work to the attack itself, we thoroughly studied the feasibility of hardware-based solutions (e.g., branch history clearing) and also proposed a software-based countermeasure, called Zigzagger, to mitigate the branch shadowing attack in practice.

💡 Research Summary

The paper introduces a novel side‑channel attack against Intel Software Guard Extensions (SGX) called the branch shadowing attack. The authors observe that SGX does not clear the processor’s branch‑prediction state (the Branch Target Buffer and associated history) when switching between enclave and non‑enclave modes. Consequently, the fine‑grained execution trace—whether each conditional, indirect, or unconditional branch was taken—remains visible to an external observer.

To exploit this leakage, the authors develop two complementary techniques. First, they leverage Intel Processor Trace (PT) and the Last Branch Record (LBR) feature available on Skylake and Kaby Lake CPUs. LBR records up to 32 recent branches, including the source address, target address, a misprediction flag, and the elapsed core cycles between entries (timed LBR). Because LBR reports prediction outcomes directly and with low variance, the attacker can reconstruct the exact branch outcomes of the enclave without relying on noisy timing measurements.

Second, they gain precise control over enclave execution by manipulating the local Advanced Programmable Interrupt Controller (APIC) timer. By programming the APIC to fire at a high frequency and inserting malicious code into the timer interrupt handler, the attacker can interrupt the enclave at carefully chosen points, read the LBR, and then resume execution. This “interrupt‑and‑probe” loop allows the attacker to capture branch history after any specific branch of interest. The authors also optionally disable the CPU cache during critical measurements to amplify the timing difference between predicted and mispredicted paths.

Together, these techniques enable the attacker—assumed to be a malicious operating system, which is the threat model for SGX—to infer fine‑grained control‑flow information at the basic‑block level. This is a substantial improvement over the previously known controlled‑channel attack, which only reveals page‑level memory‑access patterns. The authors demonstrate the practicality of their attack on real SGX hardware by extracting sensitive data from several representative applications:

• String literals and formatted strings from the Intel SGX SDK,
• RSA private keys from the mbed TLS library,
• Machine‑learning model parameters from LIBSVM, and
• HTTP request contents from an Apache web server running inside an enclave.

Moreover, they evaluate the attack against recent SGX hardening proposals—deterministic multiplexing, Sanctum, SGX‑Shield, and T‑SGX—and show that all of them are bypassed because those defenses focus on mitigating page‑fault side channels, not branch‑prediction leakage.

In response, the paper proposes hardware‑based and software‑based mitigations. On the hardware side, they suggest that the CPU should automatically flush the BTB and branch‑history registers on every enclave entry/exit. Simulations indicate that this would incur negligible performance impact. On the software side, they introduce Zigzagger, a compiler‑level transformation that inserts randomized indirect branches after each conditional or indirect branch and uses multi‑target indirect branches to dilute BTB collisions. Zigzagger effectively reduces the attacker’s ability to create a “shadow” branch that maps to the same BTB entry as the target branch. Empirical evaluation shows an average runtime overhead of less than 1.3× and modest code‑size increase, making it a practical countermeasure.

The authors also discuss limitations of their approach, such as the need for the attacker to maintain control over the virtual address space (which is realistic for a compromised OS) and the potential for future attacks that combine branch‑shadowing with other micro‑architectural side channels (e.g., power analysis or speculative execution attacks).

In conclusion, the work reveals that SGX’s isolation does not extend to the processor’s branch‑prediction state, exposing a powerful, low‑noise side channel that can reconstruct enclave control flow at a granularity far finer than previously thought. The paper’s contributions are threefold: (1) a concrete demonstration of fine‑grained branch‑level leakage, (2) a practical exploitation methodology using PT, LBR, and APIC‑timer manipulation, and (3) viable mitigation strategies with acceptable performance costs. This research underscores the necessity of considering micro‑architectural state when designing trusted execution environments and prompts both hardware vendors and software developers to incorporate branch‑history sanitization into future SGX designs.