MemoryRanger Prevents Hijacking FILE_OBJECT Structures in Windows Kernel

Windows OS kernel memory is one of the main targets of cyber-attacks. By launching such attacks, hackers are succeeding in process privilege escalation and tampering with users data by accessing kernel mode memory. This paper considers a new example of such an attack, which results in access to the files opened in an exclusive mode. Windows built-in security features prevent such legal access, but attackers can circumvent them by patching dynamically allocated objects. The research shows that the Windows 10, version 1809 x64 is vulnerable to this attack. The paper provides an example of using MemoryRanger, a hypervisor-based solution to prevent such attack by running kernel-mode drivers in isolated kernel memory enclaves.

💡 Research Summary

The paper investigates a concrete kernel‑mode attack against Windows 10 version 1809 (x64) that subverts the operating system’s exclusive‑file protection by directly tampering with the dynamically allocated FILE_OBJECT structure. In a normal scenario, when a file is opened with the FILE_SHARE_NONE (or exclusive) flag, the I/O manager checks the FO_EXCLUSIVE bit inside the FILE_OBJECT and denies any subsequent open attempts from other processes. The authors demonstrate that this safeguard can be bypassed when a malicious driver obtains a reference to the target FILE_OBJECT via ObReferenceObjectByHandle, maps the underlying physical page into its address space using an MDL, changes the page‑table protection to RWX, and clears the FO_EXCLUSIVE flag. After the flag is cleared, other processes can open the same file without receiving STATUS_ACCESS_DENIED, effectively achieving unauthorized read/write access to a file that should be locked.

The attack chain relies on two realistic prerequisites: (1) the attacker can load a kernel‑mode driver with sufficient privileges—either by exploiting a signed driver vulnerability, abusing a legitimate driver’s IOCTL, or leveraging a compromised system; (2) the attacker can manipulate page‑table attributes, which is feasible through undocumented kernel APIs or by directly editing the page tables after gaining write access to physical memory. The authors reproduce the attack in a controlled lab environment, confirming that the file’s exclusive lock is removed and that the file’s contents become accessible to any process.

To counter this threat, the paper introduces MemoryRanger, a hypervisor‑based memory‑isolation framework that creates separate “enclaves” for each kernel driver. MemoryRanger runs in VMX root mode and leverages Intel VT‑x (or AMD‑V) Extended Page Tables (EPT) to enforce per‑driver memory policies. When a driver is loaded, MemoryRanger intercepts the load event, identifies the driver’s code and data pages, and constructs an EPT view that grants the driver read/write/execute rights only to its own enclave. All other kernel memory, including the global FILE_OBJECT instances belonging to other drivers or the core OS, is mapped with “no‑access” permissions for that driver. Consequently, a malicious driver that attempts the MDL‑based mapping of a foreign FILE_OBJECT will trigger an EPT violation; the hypervisor blocks the access before the driver can modify the flag.

The implementation details are described thoroughly. MemoryRanger is loaded early in the boot process, registers callbacks for PsSetLoadImageNotifyRoutine and PsSetCreateProcessNotifyRoutine to monitor driver load/unload events, and dynamically updates the EPT entries whenever a driver’s memory layout changes (e.g., due to paging or hot‑patching). The framework also defines a small set of shared pages for inter‑driver communication, which are explicitly whitelisted; any attempt to access non‑whitelisted pages results in a VM‑exit and a denial.

The evaluation consists of two main experiments. First, the authors repeat the FILE_OBJECT flag‑clearing attack on a system protected by MemoryRanger. The malicious driver successfully obtains a reference to the target FILE_OBJECT but fails at the step where it tries to change the page protection; the EPT violation aborts the operation, the exclusive flag remains set, and the subsequent open attempts from other processes are still denied. No kernel panic occurs, demonstrating that MemoryRanger can safely intercept and neutralize the attack. Second, the authors measure performance impact on typical I/O workloads (file open, read, write) and on overall system throughput using the PCMark benchmark. The results show an average latency increase of 2.1 % and a maximum CPU overhead of 3 % under heavy I/O, indicating that the security benefits come at a modest cost.

The discussion acknowledges current limitations. MemoryRanger presently supports only 64‑bit drivers; legacy 32‑bit drivers would require additional translation layers. The hypervisor itself becomes part of the trusted computing base, so its code must be minimal and formally verified to avoid becoming a new attack surface. Future work is suggested in the areas of dynamic enclave resizing, multi‑core EPT synchronization, and integration with policy‑based access control frameworks that could also mediate user‑mode requests.

In conclusion, the paper makes two significant contributions. First, it provides the first publicly documented, reproducible method for bypassing Windows’ exclusive‑file protection by directly manipulating the kernel’s FILE_OBJECT structure, highlighting a gap in the OS’s memory‑integrity guarantees. Second, it demonstrates that a hypervisor‑level memory‑isolation solution such as MemoryRanger can effectively close this gap, preventing malicious drivers from tampering with critical kernel objects while preserving compatibility with existing drivers and imposing only minimal performance penalties. The work underscores the importance of hardware‑assisted isolation for kernel‑mode security and offers a practical blueprint for future OS hardening efforts.