Kernel Guard Strips Execute Bit to Catch Windows Shellcode Injection

Shellcode injection lets attackers run arbitrary code inside a legitimate Windows process without ever writing a file to disk, and it remains a common technique in modern malware and advanced persistent threats. A new open-access study published in the journal Cybersecurity on 3 November 2026 presents XMemGuard, a kernel-mode detection framework that attacks the problem at its root: the moment a process asks the kernel to create executable memory in another process. The work comes from a research team including Chengchun Wang, Xing Tian, Xiaoyu Wang, Yuxuan Zhu, Yaqin Cao, Jie Yin, Yuling Liu and Qixu Liu.

The framework enforces what the authors call the cross-process write-XOR-execute principle. Traditional W^X policy keeps a process from having memory that is both writable and executable at the same time, and Windows has long supported this through Data Execution Prevention, which uses the NX bit so that attempts to execute a data page trigger a page-fault exception. But DEP only stops a process from executing its own data pages. Arbitrary Code Guard goes further by restricting dynamic code operations initiated by a protected process, yet it is an application-level, opt-in mitigation. Neither mechanism prevents a remote process from acquiring executable memory in a target process through cross-process system calls such as NtAllocateVirtualMemory or NtProtectVirtualMemory. When one process makes such a request, the kernel mainly verifies whether the caller holds the required handle and access rights; it does not judge whether the remote acquisition of executable memory is legitimate from a security standpoint.

XMemGuard closes that gap by making the relationship between the requesting process and the target process the basis of the decision. Cross-process system calls must not create attacker-controlled executable memory in the target process, and execute permission requested by one process for a different process is not granted unless the requesting process is trusted for that target. The design exploits an asymmetry the authors identify in the anatomy of a shellcode injection chain, which decomposes into three stages: acquisition, in which a memory region with executable permission is established in the target process; writing, in which the shellcode is copied into that region; and triggering, in which control flow is redirected to the shellcode entry point. The acquisition stage is relatively stable, the writing stage is not by itself an indicator of malicious behavior, and the triggering stage is highly diverse, with new techniques appearing almost every year, including thread-pool work items, fiber switching, loader notifications and waiting-thread hijacking.

Because any trigger mechanism requires attacker-controlled executable memory as a prerequisite, the framework shifts the defensive focus upstream to the acquisition stage. The implementation relies on a pre-dispatch system-call interception mechanism built on Windows Event Tracing. On Windows x64, each system call passes through the ETW and Performance Monitoring Counter collection path after entering KiSystemServiceRepeat but before reaching the System Service Dispatch Table. The driver replaces the HalCollectPmcCounters entry in the kernel HalPrivateDispatchTable with a pointer to a custom detour function, so it gains control before the actual SSDT dispatch without modifying the SSDT or patching kernel code. The authors argue that documented kernel extension interfaces cannot express the required enforcement: object callbacks can strip access rights when a handle is opened but are not invoked on the memory operation itself, and process, thread and image notification routines report lifecycle events rather than memory operations.

By applying two selection criteria to the public Windows x64 system-call tables, the team identified five system calls in three semantic categories: allocating a new private region whose requested protection already permits execution, adding the execute bit to the protection of an existing region, and mapping a view of a section object with a protection that includes the execute bit. When a suspicious cross-process executable-memory acquisition attempt passes the filters, the driver takes a single uniform action: it removes the execute bit from the requested protection, so the target process receives an ordinary data page rather than an executable page. For example, PAGE_EXECUTE_READWRITE is downgraded to PAGE_READWRITE. The downgrade is silent, returning STATUS_SUCCESS rather than STATUS_ACCESS_DENIED, so an attacker cannot infer the presence of the defense from the system-call return value. When a queried range falls within a controlled region, the driver even reports the originally requested protection to NtQueryVirtualMemory, so that probing tools see the protection that was requested.

The affected address range is registered as a controlled-region record containing the base address, size, source process identifier, current effective protection and originally requested protection. If this is the first controlled region observed in the target process, the driver injects a monitoring component into that process through kernel-mode manual mapping, because conventional user-mode approaches such as CreateRemoteThread with LoadLibrary may deadlock on the loader lock and APC-based injection depends on the target thread entering an alertable wait state. When the attacker later attempts to execute code from a controlled region, the CPU NX mechanism triggers a page fault. The monitoring component captures this exception at the user-mode exception-dispatch boundary, before user-registered SEH or VEH handlers run, by instrumenting KiUserExceptionDispatcher. An attacker cannot receive the exception first by pre-registering a handler, yet the full user-mode thread context remains available for recording. The driver matches the faulting address against its controlled-region records and can then terminate the faulting thread or process, or continue monitoring and restore the original execute permission to collect more attack information.

In the evaluation, the team compared XMemGuard with three existing detection methods: FalconEye, a kernel-mode method that hooks injection-related system calls and applies explicit detection rules; Starink’23, which matches fine-grained API traces against behavior models; and Di Pietro’25, which captures injection traces through dynamic binary instrumentation and assigns labels offline. On a set of 33 proof-of-concept samples covering remote thread injection, APC injection, reflective DLL injection, process hollowing, thread hijacking, the eight PoolParty variants and seven emerging techniques reported between 2022 and 2025, XMemGuard detected all 33, a 100 percent detection rate. Di Pietro’25 detected 30, while FalconEye and Starink’23 each detected 15. Neither FalconEye nor Starink’23 detected any PoolParty variant or any emerging injection sample. Di Pietro’25 missed PoolParty Start Routine, Thread Name-Calling and Waiting-thread hijacking because their execution-triggering mechanisms fell outside its predefined detection rules, even though all three acquired executable memory through ordinary NtAllocateVirtualMemory and NtProtectVirtualMemory calls that XMemGuard intercepted.

On in-the-wild samples, the evaluation used 1,146 malware samples and a benign set of 1,144 samples comprising 1,030 Windows system programs and 114 portable applications. After removing samples for which Di Pietro’25 produced no analyzable traces, the final evaluation set contained 1,143 malware and 1,091 benign samples. XMemGuard achieved 99.79 percent precision and 81.54 percent recall, with the highest accuracy, recall and F1 score among the compared methods; its F1 score was 89.74 percent. FalconEye and Di Pietro’25 both achieved 100.00 percent precision, with recall values of 66.84 percent and 60.89 percent respectively. XMemGuard’s precision was only 0.21 percentage points lower, corresponding to two false positives in the full evaluation set: draw.io, whose Electron-style bootstrap loads initialization code into child address spaces, and 7+ Taskbar Tweaker, which injects its own code into explorer.exe. Among the malware samples, 122 were detected only by XMemGuard. For 150 samples that other methods caught but XMemGuard did not count, cross-process executable-memory acquisition was observed and intercepted, but no final outcome could be counted within the fixed 180-second observation window, which the authors interpret as a difference in runtime observability rather than a gap at the acquisition-call level.

Log analysis of the in-the-wild samples showed that NtAllocateVirtualMemory is the dominant acquisition syscall, appearing in 90.03 percent of all samples and providing exclusive coverage for 76.99 percent of them. NtProtectVirtualMemory appeared in 17.59 percent of samples, and NtMapViewOfSection in only 1.40 percent, though one sample relied entirely on that call. Performance evaluations showed the overhead is concentrated rather than uniform. In microbenchmarks, calls requesting execute permission showed a much larger increase than calls released early by the filtering conditions. Under concurrent load, throughput with XMemGuard enabled grew from 144,460 operations per second at one thread to 996,625 at sixteen, while the baseline delivered 3.4 times the throughput at one thread but only 1.8 times at sixteen, showing the gap narrows as threads are added. Among seven application workloads, only Git clone showed a clear slowdown, at +4.81 percent, because cross-process memory operations occur there often enough for the detour cost to accumulate. The PCMark 10 overall score dropped by 3.22 percent, and no category dropped by more than 6.98 percent. The framework held 344.0 KB of kernel pool with no target process under control, with each controlled region adding 128 bytes, and a continuous 24-hour run produced no growth in that footprint, no bugcheck and no loss of detection capability.

The authors also delineate what the framework does not cover. Code-reuse attacks such as Return-Oriented Programming do not create new executable memory and remain the concern of control-flow integrity mechanisms like CET and CFG. Mockingjay exploits pre-existing read-write-execute sections in signed DLLs rather than acquiring new executable memory, so it falls outside the monitoring points, though its applicability is limited because such DLLs are relatively rare. An attacker who writes read-write shellcode cross-process and then uses an in-process ROP chain to invoke VirtualProtect also escapes the constraint, a scenario the authors say should in principle be handled by control-flow defenses. The threat model assumes the attacker has arbitrary user-mode code execution in a source process, holds a process handle with the rights required for cross-memory operations, and has no kernel-mode execution capability, with kernel integrity mechanisms such as PatchGuard remaining effective. Even if an attacker tampers with the user-mode monitor, the authors note, this affects only whether the exception is recorded as a detection event, not whether the memory region remains non-executable. The evaluation was performed on Windows 10 22H2 x64, and the main framework implementation, proof-of-concept source code and malware sample hashes are publicly available on GitHub.

Subject of Research: Real-time kernel-mode detection of Windows shellcode injection by enforcing a cross-process write-XOR-execute principle on executable-memory acquisition system calls

Article Title: Real-Time Detection of Windows Shellcode Injection via the Cross-Process W^X Principle

Article References: Wang, C., Tian, X., Wang, X., Zhu, Y., Cao, Y., Yin, J., Liu, Y., & Liu, Q. (2026). Real-Time Detection of Windows Shellcode Injection via the Cross-Process W^X Principle. Cybersecurity, 9(1), Article 250. https://doi.org/10.1186/s42400-026-00690-4

Image Credits: AI Generated

DOI: 10.1186/s42400-026-00690-4

Keywords: shellcode injection, Windows, XMemGuard, kernel-mode detection, W^X principle, cross-process system calls, Data Execution Prevention, Arbitrary Code Guard, process injection, malware detection, ETW interception, runtime overhead