Best practices for HPM-assisted performance engineering on modern multicore processors

Many tools and libraries employ hardware performance monitoring (HPM) on modern processors, and using this data for performance assessment and as a starting point for code optimizations is very popular. However, such data is only useful if it is interpreted with care, and if the right metrics are chosen for the right purpose. We demonstrate the sensible use of hardware performance counters in the context of a structured performance engineering approach for applications in computational science. Typical performance patterns and their respective metric signatures are defined, and some of them are illustrated using case studies. Although these generic concepts do not depend on specific tools or environments, we restrict ourselves to modern x86-based multicore processors and use the likwid-perfctr tool under the Linux OS.

💡 Research Summary

The paper presents a systematic methodology for leveraging hardware performance monitoring (HPM) counters within a structured performance‑engineering workflow, targeting modern x86‑based multicore processors. The authors argue that raw counter values are only useful when interpreted in the context of well‑defined performance patterns and that the choice of metrics must be driven by the specific optimization goal.

Workflow definition
The proposed workflow consists of four stages: (1) goal definition, (2) profiling, (3) pattern identification, and (4) targeted refactoring. In the profiling stage, both high‑level metrics (runtime, memory footprint, CPU utilization) and low‑level HPM counters are collected. The pattern‑identification stage translates the raw counter data into a set of “performance patterns” such as memory‑bandwidth saturation, cache‑reuse deficiency, pipeline stalls, branch‑prediction failures, floating‑point inefficiency, synchronization imbalance, and power‑state variability.

Pattern signatures
For each pattern the authors define a characteristic “metric signature” – a combination of counters that together reveal the underlying bottleneck. Examples include:

• Memory‑bandwidth saturation – the ratio of total load/store instructions (MEM_INST_RETIRED.ALL_LOADS + MEM_INST_RETIRED.ALL_STORES) to L3‑cache‑miss traffic (L3_CACHE_MISS).
• Cache‑reuse deficiency – L1/L2 miss rates (L1D_REPLACEMENT, L2_RQSTS.MISS) together with CACHE_REFERENCES vs. CACHE_MISSES.
• Pipeline stalls – UOPS_ISSUED.ANY and CYCLE_ACTIVITY.STALLS_TOTAL.
• Branch misprediction – BR_MISP_RETIRED.ALL_BRANCHES divided by BR_INST_RETIRED.ALL_BRANCHES.

These signatures are deliberately tool‑agnostic; they can be implemented with any HPM framework (e.g., likwid‑perfctr, perf, Intel VTune, PAPI).

Tool chain and experimental platform
The study focuses on Linux systems equipped with Intel Xeon and AMD EPYC CPUs. The primary measurement tool is likwid-perfctr, chosen for its ability to define event groups, collect per‑thread counters, and present real‑time statistics. The authors also discuss how to translate the signatures to other tools, emphasizing that the methodology does not depend on a specific software stack.

Case studies
Two representative scientific codes are examined.

1. Computational Fluid Dynamics (CFD) application – Initial profiling revealed a high L1 miss rate (≈12 %) and a memory‑bandwidth utilization of 85 %. The combined signature identified both “cache‑reuse deficiency” and “bandwidth saturation”. Optimizations included (a) restructuring data from an array‑of‑structures to a structure‑of‑arrays layout to improve SIMD friendliness, and (b) applying loop blocking to increase temporal locality. After these changes, L1 miss rate dropped to 4 % and overall runtime improved by 28 %.

2. Large‑scale dense matrix multiplication library – The branch‑misprediction signature showed a 18 % misprediction rate, indicating costly conditional branches inside the innermost loops. The authors removed the conditionals from the hot loop, moved them outside, and switched the OpenMP scheduling policy from static,chunk to guided. This reduced misprediction to below 5 % and decreased per‑thread stall cycles by 30 %, yielding a noticeable speed‑up.

Accuracy, overhead, and environment control
The paper stresses that HPM measurements are sensitive to CPU frequency scaling, power‑saving states, hyper‑threading, and OS scheduling. To obtain reproducible data, the authors recommend fixing the governor to “performance”, disabling turbo boost, and pinning threads to physical cores before each measurement. They also discuss how to minimize measurement overhead by limiting the number of concurrent events and by using sampling intervals that balance granularity with perturbation.

Conclusions and future work
The authors conclude that HPM becomes a powerful engineering instrument when embedded in a “quantify → pattern → fix” loop. The defined metric signatures enable developers to pinpoint multiple concurrent bottlenecks without relying on a single counter. The methodology is portable across tools and can be extended to non‑x86 architectures (ARM, RISC‑V) with appropriate event mapping. Future research directions include automated pattern detection using machine‑learning classifiers, integration of HPM data into just‑in‑time (JIT) compilation pipelines, and scaling the approach to cloud‑native workloads where hardware counters are virtualized.

Overall, the paper provides a practical, reproducible framework for turning raw hardware counters into actionable performance insights, demonstrating its effectiveness through real‑world scientific applications and offering clear guidance on measurement hygiene, tool selection, and systematic optimization.