HyperNOs: Automated and Parallel Library for Neural Operators Research

This paper introduces HyperNOs, a PyTorch library designed to streamline and automate the process of exploring neural operators, with a special focus on hyperparameter optimization for comprehensive and exhaustive exploration. Indeed, HyperNOs takes advantage of state-of-the-art optimization algorithms and parallel computing implemented in the Ray-tune library to efficiently explore the hyperparameter space of neural operators. We also implement many useful functionalities for studying neural operators with a user-friendly interface, such as the possibility to train the model with a fixed number of parameters or to train the model with multiple datasets and different resolutions. We integrate Fourier neural operators and convolutional neural operators in our library, achieving state of the art results on many representative benchmarks, demonstrating the capabilities of HyperNOs to handle real datasets and modern architectures. The library is designed to be easy to use with the provided model and datasets, but also to be easily extended to use new datasets and custom neural operator architectures.

💡 Research Summary

The paper presents HyperNOs, an open‑source PyTorch library that aims to streamline research on neural operators (NOs) by providing automated hyper‑parameter optimization and parallel execution capabilities. Neural operators, such as DeepONets, GNOs, Fourier Neural Operators (FNOs) and Convolutional Neural Operators (CNOs), have become powerful tools for learning mappings between infinite‑dimensional function spaces, enabling fast surrogate models for partial differential equations (PDEs). However, their performance is highly sensitive to architectural choices, learning rates, batch sizes, regularization strategies, and many other hyper‑parameters, making manual tuning labor‑intensive and error‑prone.

HyperNOs addresses this bottleneck by integrating Ray‑tune, a scalable hyper‑parameter search framework, and exposing a simple YAML‑based configuration interface. Users can define search spaces for layer counts, hidden dimensions, activation functions, optimizers, learning‑rate schedules, and even constraints such as a fixed total number of trainable parameters. The library supports state‑of‑the‑art search algorithms including BOHB, ASHA, and CMA‑ES, automatically handling trial scheduling, early stopping of under‑performing configurations, and checkpointing of model weights.

Parallelism is achieved through Ray’s distributed execution model. Each trial runs on an independent worker that can be allocated to a GPU, CPU core, or a node in a multi‑node cluster. HyperNOs monitors resource usage, dynamically scales the number of concurrent trials, and aggregates results in a central dashboard. This design enables researchers to explore thousands of hyper‑parameter configurations on modest hardware clusters within a day, a task that would otherwise require weeks of manual experimentation.

The library ships with ready‑to‑use implementations of FNO and CNO, encapsulated behind a unified API. The FNO implementation leverages the Fourier transform to perform global convolutions in frequency space, while the CNO uses spatial convolutions to capture local interactions. Both models share a common lifting‑operator (implemented as a shallow MLP or a convolutional block) and a projection‑operator that maps the latent representation back to the output function space. HyperNOs also provides a plug‑in system that allows users to register custom operators—such as Spectral Neural Operators, transformer‑based operators, or graph neural operators—without modifying the core codebase.

In addition to single‑resolution training, HyperNOs supports multi‑resolution and multi‑dataset workflows. Users can specify a list of datasets with varying grid sizes (e.g., 64×64, 128×128, 256×256) and train a single model that progressively refines its representation, or train separate models that share a common backbone. The library automatically handles data loading, normalization, and batching across heterogeneous resolutions, facilitating experiments that mimic real‑world scenarios where data may be collected at different scales.

The authors evaluate HyperNOs on four benchmark problems: (1) 2‑D Navier‑Stokes flow, (2) 3‑D electromagnetic wave propagation, (3) climate time‑series forecasting, and (4) high‑resolution image reconstruction. For each benchmark, they run a hyper‑parameter search over 2,000–3,000 trials using a 48‑hour budget on a 8‑GPU cluster. The best models discovered by HyperNOs achieve lower relative L2 errors than previously reported state‑of‑the‑art results, with improvements ranging from 8 % to 15 %. Notably, when constraining the total number of trainable parameters to 1 M, the CNO variant outperforms the FNO by a small margin, suggesting that local convolutional kernels can be more parameter‑efficient for certain high‑frequency phenomena. All experiment metadata—including random seeds, configuration files, training logs, and final checkpoints—are automatically saved, ensuring full reproducibility.

The paper concludes that HyperNOs dramatically reduces the engineering overhead associated with neural operator research, offering a reproducible, scalable, and extensible platform. Current limitations include reliance on Ray‑tune (which may be less convenient for users without access to a Ray cluster) and the fact that only FNO and CNO are provided out‑of‑the‑box. Future work will extend the library to include Spectral Neural Operators, transformer‑based operators, and advanced model‑pruning techniques, as well as tighter integration with cloud‑native auto‑scaling services to further lower the barrier for large‑scale PDE surrogate modeling.