Service Oriented Architecture in Network Security - a novel Organisation in Security Systems

Current network security systems are a collection of various security components, which are directly installed in the operating system. These check the whole node for suspicious behaviour. Armouring intrusions e.g. have the ability to hide themselves from being checked. We present in this paper an alternative organisation of security systems. The node is completely virtualized with current virtualization systems so that the operating system with applications and the security system is distinguished. The security system then checks the node from outside and the right security components are provided through a service oriented architecture. Due to the running in a virtual machine, the infected nodes can be halted, duplicated, and moved to other nodes for further analysis and legal aspects. This organisation is in this article analysed and a preliminary implementation showing promising results are discussed.

💡 Research Summary

The paper begins by critiquing the conventional design of network security solutions, which typically consist of a collection of anti‑virus, firewall, and intrusion detection components that are installed directly inside the operating system of each host. This “in‑OS” placement creates several problems: (1) malicious code that employs armouring techniques can hide from the OS‑level sensors; (2) the security modules operate largely in isolation, lacking cooperative workflows that could correlate events across components; (3) redundant scanning consumes unnecessary CPU, memory, and I/O resources. To address these shortcomings, the authors propose a fundamentally different architecture that leverages full host virtualization together with a Service‑Oriented Architecture (SOA) for security services.

In the proposed design each physical node runs a hypervisor (e.g., VMware or KVM) that hosts multiple virtual machines (VMs). One of these VMs is dedicated to security and runs a “security environment” that has privileged access to the other guest VMs. The remaining VMs host the user operating system and applications. The security VM monitors the other VMs from the outside, using the hypervisor’s introspection capabilities (memory snapshots, CPU registers, disk I/O) to detect suspicious behaviour without being subject to the same armouring tricks that affect in‑OS agents.

The SOA layer sits on top of this virtualization foundation. A set of security servers maintains a catalogue of security‑service images (e.g., antivirus, firewall, IDS) that can be instantiated on demand. When a new host joins the network, the DHCP process is extended so that the host is only granted network access after the security server has provisioned a suitable security VM and attached the required security services. This dynamic provisioning enables rapid adaptation to heterogeneous devices such as mobile hand‑helds, thin clients, or IoT nodes, each of which may have different resource constraints and threat profiles.

A key advantage of the architecture is its handling of infections. If the security VM flags a guest VM as compromised, it immediately pauses the VM, creates a snapshot, and duplicates the snapshot to the central security server. The duplicated image can be analysed in depth for forensic evidence, satisfying legal and investigative requirements that are often impossible in traditional setups where the infected host is either cleaned (destroying evidence) or left untouched (risking further spread). After analysis, the security server can spin up a clean VM image with the necessary applications pre‑installed and hand it back to the user, minimizing downtime.

Implementation considerations are discussed in detail. While modern hypervisors already provide most of the required primitives (snapshot, pause, migration), the authors note that a mechanism is still needed for the security VM to read the memory and disk of other VMs safely. They suggest extending the hypervisor with a privileged API or a virtual device driver that mediates such access. All communication between security services, the security VM, and the security servers is protected using a public‑key infrastructure (PKI) with digital signatures and integrity checks, preventing an adversary from spoofing or tampering with the security components.

The paper also acknowledges several security and operational challenges. The hypervisor itself becomes a high‑value target; any compromise at this layer could undermine the entire protection scheme. Managing the PKI at scale, ensuring timely revocation of compromised keys, and handling key distribution across distributed sites add operational complexity. Performance overhead is another concern: the extra VM layer and the introspection activities consume CPU cycles and memory, potentially affecting latency‑sensitive applications. The authors propose mitigating these issues through hardware‑rooted trust (TPM), regular integrity verification of the hypervisor and security VM images, and selective activation of intensive scanning only when anomalies are detected.

Preliminary results are modest: the prototype successfully boots a security VM, pauses and snapshots a guest VM, and transfers the snapshot to a remote analysis server. However, quantitative performance measurements, scalability tests with dozens of nodes, and real‑world malware detection experiments remain future work.

In conclusion, the authors argue that combining full host virtualization with a service‑oriented delivery model offers a promising path toward more resilient network security. It enables rapid, automated deployment of up‑to‑date security services, preserves forensic evidence, and reduces the attack surface presented by in‑OS agents. The paper outlines a roadmap that includes building a VMware‑based testbed, integrating artificial immune system concepts, and exploring multi‑agent coordination to further enhance detection capabilities. The work is supported by the Luxembourg Ministry of Education and Research and the University of Luxembourg.