Q-ESP: a QoS-compliant Security Protocol to enrich IPSec Framework

IPSec is a protocol that allows to make secure connections between branch offices and allows secure VPN accesses. However, the efforts to improve IPSec are still under way; one aspect of this improvement is to take Quality of Service (QoS) requirements into account. QoS is the ability of the network to provide a service at an assured service level while optimizing the global usage of network resources. The QoS level that a flow receives depends on a six-bit identifier in the IP header; the so-called Differentiated Services code point (DSCP). Basically, Multi-Field classifiers classify a packet by inspecting IP/TCP headers, to decide how the packet should be processed. The current IPSec standard does hardly offer any guidance to do this, because the existing IPSec ESP security protocol hides much of this information in its encrypted payloads, preventing network control devices such as routers and switches from utilizing this information in performing classification appropriately. To solve this problem, we propose a QoS-friendly Encapsulated Security Payload (Q-ESP) as a new IPSec security protocol that provides both security and QoS supports. We also present our NetBSD kernel-based implementation as well as our evaluation results of Q-ESP.

💡 Research Summary

The paper addresses a fundamental limitation of the IPsec Encapsulating Security Payload (ESP) protocol: by encrypting the transport‑layer header, ESP hides the source and destination ports, the protocol identifier, and consequently the five‑tuple information that network devices need for QoS classification. While the Differentiated Services Code Point (DSCP) field remains in the IP header, the lack of visibility into the transport header prevents routers and switches from applying Multi‑Field Classifier (MFC) based active admission control, which is essential for guaranteeing priority treatment of latency‑sensitive traffic such as VoIP and video conferencing.

To overcome this, the authors propose Q‑ESP, a “QoS‑friendly ESP” that retains the security guarantees of ESP while exposing the necessary QoS fields in clear text. The core idea is to insert a QoS Extension Header between the ESP header and the encrypted payload. This extension carries the DSCP value (6 bits) together with the full five‑tuple: source IP, destination IP, source port, destination port, and the transport‑layer protocol identifier. By keeping these fields unencrypted, network devices can classify packets according to the standard MFC rules without decrypting the traffic.

The paper details the packet format for both IPv4 and IPv6, supporting transport‑mode and tunnel‑mode operation. In IPv4, after the standard ESP header, a 1‑byte DSCP field, a 2‑byte source‑port, a 2‑byte destination‑port, and a 1‑byte protocol field are placed, followed by the encrypted payload and the integrity check value (ICV). IPv6 follows the same layout, with the Next Header field indicating the presence of Q‑ESP. The authors emphasize that the QoS Extension Header is covered by the ESP authentication tag, so any tampering with DSCP or port numbers will cause authentication failure and the packet will be discarded, preserving integrity.

Processing on the sender side proceeds as follows: (1) the original packet is captured and classified; (2) the QoS Extension Header is built and inserted; (3) the payload (including the original transport header) is encrypted using the chosen ESP cipher (e.g., AES‑CBC); (4) an ICV is computed over the entire packet, including the clear‑text QoS fields. The receiver first validates the ICV, then decrypts the payload, and finally extracts the QoS Extension Header to restore the original five‑tuple for further processing.

Security analysis demonstrates that Q‑ESP provides confidentiality, integrity, and authentication equivalent to standard ESP. Although QoS fields are exposed, they are authenticated, preventing an attacker from forging higher‑priority markings. Moreover, because the QoS fields are not encrypted, the protocol eliminates the “passive admission control” problem where a malicious sender could claim unwarranted priority; any unauthorized modification is detectable.

Implementation was carried out as a kernel module in NetBSD 7.0. The authors integrated Q‑ESP into the existing IPsec stack, allowing runtime selection between ESP and Q‑ESP. Performance evaluation involved throughput tests across a range of packet sizes (64 B to 4096 B) and cryptographic suites (AES‑CBC with HMAC‑SHA1, AES‑GCM, etc.). Results show that Q‑ESP incurs a modest overhead of roughly 3 %–7 % relative to plain ESP, with the impact diminishing for larger packets where the fixed header cost is amortized. Priority‑control experiments confirm that DSCP‑based traffic segregation works as intended: when QoS processing is enabled, high‑priority flows achieve higher throughput and lower latency compared to a baseline ESP deployment without QoS awareness.

In conclusion, Q‑ESP successfully bridges the gap between security and QoS in IPsec environments. By exposing the necessary classification fields while preserving cryptographic protection, it enables active admission control without requiring changes to existing network infrastructure. The authors suggest future work on hardware acceleration, integration with Software‑Defined Networking (SDN) controllers for dynamic QoS policy enforcement, and extending the approach to other security protocols such as IPsec Authentication Header (AH) or newer encapsulation mechanisms.