Joint Parameter and State-Space Bayesian Optimization: Using Process Expertise to Accelerate Manufacturing Optimization

Bayesian optimization (BO) is a powerful method for optimizing black-box manufacturing processes, but its performance is often limited when dealing with high-dimensional multi-stage systems, where we can observe intermediate outputs. Standard BO models the process as a black box and ignores the intermediate observations and the underlying process structure. Partially Observable Gaussian Process Networks (POGPN) model the process as a Directed Acyclic Graph (DAG). However, using intermediate observations is challenging when the observations are high-dimensional state-space time series. Process-expert knowledge can be used to extract low-dimensional latent features from the high-dimensional state-space data. We propose POGPN-JPSS, a framework that combines POGPN with Joint Parameter and State-Space (JPSS) modeling to use intermediate extracted information. We demonstrate the effectiveness of POGPN-JPSS on a challenging, high-dimensional simulation of a multi-stage bioethanol production process. Our results show that POGPN-JPSS significantly outperforms state-of-the-art methods by achieving the desired performance threshold twice as fast and with greater reliability. The fast optimization directly translates to substantial savings in time and resources. This highlights the importance of combining expert knowledge with structured probabilistic models for rapid process maturation.

💡 Research Summary

The paper addresses a critical limitation of conventional Bayesian optimization (BO) when applied to high‑dimensional, multi‑stage manufacturing processes that generate rich intermediate observations. Standard BO treats the entire process as a black‑box, ignoring both the causal structure of the process and the high‑dimensional state‑space time‑series that can be measured at each sub‑process. While Partially Observable Gaussian Process Networks (POGPN) improve upon earlier Gaussian Process Networks (GPN) by modeling each node with a latent output and using doubly stochastic variational inference (DSVI), they still struggle with the sheer dimensionality of intermediate data.

To overcome this, the authors combine POGPN with Joint Parameter and State‑Space (JPSS) modeling, which leverages domain‑expert knowledge to compress high‑dimensional sensor streams (S(v)) into low‑dimensional latent features (h(v)). These features retain the essential dynamical information while dramatically reducing the computational burden of variational inference. The resulting framework, named POGPN‑JPSS, proceeds as follows:

1. Process Representation – The manufacturing system is expressed as a directed acyclic graph (DAG) (A). Each node (v) corresponds to a sub‑process (P(v)) with controllable inputs (x(v)) and observable outputs (y(v)). The overall input vector has dimensionality (D_x = 26) (6, 10, and 10 variables for the three stages of a bio‑ethanol seed‑train).

2. Expert‑Driven Feature Extraction – High‑dimensional time‑series sensor data (S(v)) (temperature, pH, concentrations, etc.) are passed through an expert‑designed dimensionality‑reduction pipeline (e.g., physics‑guided auto‑encoders or tailored PCA). The pipeline outputs a compact latent vector (h(v)) for each stage.

3. Probabilistic Modeling – Each node is modeled by a Sparse Variational Gaussian Process (SV‑GP). The latent outputs (\tilde f_{P_a(v)}) from parent nodes, together with the extracted features (h(v)), serve as inputs to the child node’s GP. DSVI jointly optimizes the evidence lower bound (ELBO) across the whole network, learning inducing points, variational means, and covariances.

4. Acquisition and Optimization – With the posterior predictive distribution in hand, the framework employs Expected Improvement (EI) or its numerically stable variant LogEI as the acquisition function. Inducing points are placed using a Greedy Improvement Reduction (GIR) strategy, which biases selection toward regions that have already shown improvement, thereby balancing exploration and exploitation.

5. Iterative BO Loop – New candidate input configurations (x) are proposed, the process is simulated (or run on the plant), intermediate observations are collected, compressed into (h(v)), and the model is updated. This loop repeats until a predefined performance threshold—here the space‑time yield (STY) of ethanol—is reached.

The authors validate POGPN‑JPSS on a realistic, high‑dimensional simulation of a three‑stage bio‑ethanol production line. The simulation includes ordinary differential equations governing biomass, ethanol, substrate, and volume dynamics, with nonlinear kinetics (Luedeking‑Piret, Pirt, Andrews inhibition, and Ghose‑Tyagi ethanol inhibition). Fifty random initial runs generate a training set of 50 evaluations. Comparative experiments involve:

• Standard Single‑Task GP BO (treating the whole plant as a black box),
• Original POGPN (without JPSS feature extraction),
• High‑dimensional GP with scaled kernels (LogEI + dimensional priors).

Key results:

• Speed of Convergence – POGPN‑JPSS reaches the target STY after an average of ~12 BO iterations, roughly half the number required by the next‑best method (≈20–24 iterations). This translates into a two‑fold reduction in experimental or simulation time.
• Reliability – Across 30 independent runs, POGPN‑JPSS achieves the threshold in >95 % of cases, whereas the baseline methods show success rates between 70 % and 85 %.
• Uncertainty Management – The combination of DSVI and GIR keeps posterior variance well‑behaved, preventing the acquisition function from becoming overly optimistic or overly conservative, which is a common failure mode in high‑dimensional BO.
• Computational Efficiency – By compressing the intermediate state‑space, the number of inducing points per node stays modest (≈30–40), keeping the overall computational complexity manageable even with three interconnected SV‑GPs.

The paper’s contributions are threefold:

1. Methodological Integration – It demonstrates how expert‑driven dimensionality reduction can be seamlessly embedded within a hierarchical probabilistic model, enabling BO to exploit rich intermediate data that were previously unusable.
2. Scalable Variational Inference – The use of DSVI together with a performance‑oriented inducing‑point allocation (GIR) yields a scalable inference scheme suitable for multi‑stage processes with dozens of controllable variables.
3. Empirical Validation – Through a demanding bio‑ethanol seed‑train case study, the authors provide concrete evidence that the proposed framework not only speeds up convergence but also improves robustness, thereby delivering tangible time and resource savings.

In summary, POGPN‑JPSS represents a significant step forward for data‑efficient optimization of complex manufacturing systems. By marrying process‑expert knowledge with structured Bayesian networks, it unlocks the value of high‑dimensional intermediate observations, accelerates the attainment of performance targets, and offers a practical pathway for rapid process maturation in industrial settings.