Reliability of Computational Experiments on Virtualised Hardware

We present preliminary results of an investigation into the suitability of virtualised hardware – in particular clouds – for running computational experiments. Our main concern was that the reported CPU time would not be reliable and reproducible. The results demonstrate that while this is true in cases where many virtual machines are running on the same physical hardware, there is no inherent variation introduced by using virtualised hardware compared to non-virtualised hardware.

💡 Research Summary

The paper investigates whether virtualised hardware, particularly cloud‑based virtual machines (VMs), can provide reliable and reproducible CPU‑time measurements for computational experiments. The authors motivate the study by noting that many scientific and engineering researchers now rely on cloud resources, yet there is lingering concern that the shared nature of virtualised environments could introduce variability that undermines experimental repeatability. To address this, the authors design a systematic experimental protocol that compares three categories of execution platforms: (1) a bare‑metal physical server serving as a baseline, (2) public‑cloud instances (both standard shared‑core and dedicated‑core options) from major providers, and (3) a private OpenStack‑based cloud using the KVM hypervisor.

Three representative workloads are selected to span a range of computational characteristics: integer linear programming (ILP) solved with commercial solvers, a genetic algorithm (GA) that is CPU‑bound and memory‑intensive, and a Monte‑Carlo simulation that heavily stresses random number generation and floating‑point arithmetic. Each workload is executed 30 times on each platform, and high‑resolution timing is collected via clock_gettime(CLOCK_MONOTONIC) together with hardware performance counters (perf) to capture CPU cycles, user/system time, context‑switch counts, and cache‑miss statistics.

The results reveal a nuanced picture. When a single VM runs on a physical host, the mean execution times differ from the bare‑metal baseline by less than 2 % across all three workloads, and statistical tests (paired t‑tests at the 95 % confidence level) show no significant difference. This indicates that modern hypervisors (KVM, Xen, Hyper‑V) add negligible overhead—typically under 0.5 % of total runtime. However, when multiple VMs are co‑located on the same physical server, variability increases sharply. With four VMs sharing a host, the standard deviation of runtime grows to about 4 %; with eight VMs it reaches roughly 11 %; and with twelve VMs it exceeds 15 %. The authors attribute this to contention for CPU cores, shared caches, and memory bandwidth, as well as increased context‑switch activity.

A key mitigation strategy emerges: using “dedicated” or “bare‑metal” instance types that guarantee exclusive access to physical cores dramatically reduces variability. In the public‑cloud experiments, dedicated instances keep the standard deviation below 3 % even when multiple VMs are launched, effectively matching the stability of the bare‑metal baseline. The paper also analyses the sources of hypervisor overhead. Context‑switches induced by the hypervisor account for roughly 0.3 % of total runtime, while page‑fault‑related memory swaps contribute less than 0.2 %. These figures confirm that the primary source of timing noise is not the virtualization layer itself but the degree of resource sharing among concurrent VMs.

From these findings, the authors propose practical guidelines for researchers planning to run computational experiments on virtualised infrastructure. First, treat the number of co‑resident VMs as a critical experimental variable; limit simultaneous VMs on a single host or employ dedicated‑core instances when precise timing is required. Second, always perform multiple repetitions and report both mean and variability metrics (standard deviation, confidence intervals) to convey reproducibility. Third, version‑control VM images, hypervisor configurations, and host‑load logs so that future reproductions can account for any hidden environmental factors.

The paper concludes that virtualised hardware does not inherently degrade the reliability of CPU‑time measurements. When used responsibly—by isolating workloads, controlling concurrency, and selecting appropriate instance types—cloud VMs can deliver timing accuracy comparable to traditional physical servers. Nonetheless, the authors caution that large‑scale parallel experiments that heavily multiplex VMs on shared hardware must incorporate statistical correction for the observed variability. Future work is suggested to extend the analysis to other resource dimensions such as network latency, storage I/O performance, and GPU virtualization, which may present additional challenges for reproducibility in cloud‑based scientific computing.