Inverse Optimization Without Inverse Optimization: Direct Solution Prediction with Transformer Models

We present an end-to-end framework for generating solutions to combinatorial optimization problems with unknown components using transformer-based sequence-to-sequence neural networks. Our framework learns directly from past solutions and incorporates the known components, such as hard constraints, via a constraint reasoning module, yielding a constrained learning scheme. The trained model generates new solutions that are structurally similar to past solutions and are guaranteed to respect the known constraints. We apply our approach to three combinatorial optimization problems with unknown components: the knapsack problem with an unknown reward function, the bipartite matching problem with an unknown objective function, and the single-machine scheduling problem with release times and unknown precedence constraints, with the objective of minimizing average completion time. We demonstrate that transformer models have remarkably strong performance and often produce near-optimal solutions in a fraction of a second. They can be particularly effective in the presence of more complex underlying objective functions and unknown implicit constraints compared to an LSTM-based alternative and inverse optimization.

💡 Research Summary

**
The paper introduces a novel end‑to‑end framework that bypasses traditional inverse optimization (IO) by directly predicting high‑quality solutions for combinatorial optimization problems with unknown components. Instead of estimating hidden parameters θ that make historical decisions optimal, the authors treat the solution itself as the learning target. They employ a sequence‑to‑sequence transformer model that maps a problem instance (context u) to a structured solution represented as a token sequence. Known hard constraints are enforced through a constraint‑reasoning module that masks infeasible tokens at each decoding step, guaranteeing that every generated solution respects the explicit constraints.

Training proceeds in a supervised fashion: a dataset of past optimal decisions (\hat{x}_i) together with their contexts (\hat{u}i) is used to minimize a loss ℓ measuring the discrepancy between the model’s prediction (M\phi(\hat{u}_i)) and the true solution (\hat{x}_i). Because the transformer’s self‑attention can capture arbitrary pairwise and higher‑order interactions among input features, it can implicitly learn complex, possibly nonlinear reward functions and hidden constraints that are not directly observable. The authors argue that this “structured prediction” perspective avoids the strong feasibility assumptions of IO and can handle stochastic or deterministic latent structures, including those that depend jointly on observable features and hidden variables.

The methodology is evaluated on three classic combinatorial problems where part of the objective or constraints is unknown: (1) a weighted knapsack problem with an unknown reward function, (2) a bipartite matching problem with an unknown objective, and (3) a single‑machine scheduling problem with release times and unknown precedence constraints (objective: minimize average completion time). For each problem, the known constraints (capacity, one‑to‑one matching, release‑time feasibility) are encoded in the constraint‑reasoning module, while the unknown components (reward coefficients, cost matrix, precedence relations) are left to be learned implicitly from historical solutions.

Experimental results show that transformer‑based models consistently outperform both LSTM‑based structured prediction baselines and state‑of‑the‑art inverse‑optimization algorithms. Key findings include:

• Solution quality – The transformer’s solutions are within 5–10 % of the true optimum on average, even when the underlying objective is quadratic or highly nonlinear, whereas IO often suffers from model misspecification.
• Feasibility – Because infeasible tokens are masked during decoding, every generated solution satisfies the known constraints, eliminating the need for post‑hoc repair.
• Speed – Inference times are on the order of 0.01–0.1 seconds per instance, a several‑order‑of‑magnitude speed‑up compared to solving the mixed‑integer formulation repeatedly (seconds to minutes per instance).
• Robustness to noise – When training data are corrupted or stochastic, the transformer maintains stable performance, while IO’s parameter estimates degrade sharply.

The authors also discuss limitations. The constraint‑reasoning module currently handles monotone constraint families efficiently; extending it to arbitrary combinatorial constraints (e.g., general graph coloring) would increase computational overhead. Moreover, the approach provides no formal optimality guarantees and requires a sufficiently large, representative dataset; in settings where the forward model is perfectly specified and data are clean, IO can recover the exact objective and deliver provably optimal solutions, something the predictive model cannot guarantee.

In conclusion, the paper demonstrates that transformer‑based structured prediction, combined with a constraint‑reasoning mask, offers a powerful alternative to inverse optimization for discrete problems with hidden objectives or implicit constraints. It delivers near‑optimal, feasible solutions at inference speeds suitable for real‑time applications, especially when the underlying problem exhibits complex, nonlinear latent structure or when historical data are noisy. Future work is suggested on (i) extending the masking mechanism to richer, non‑monotone constraint families, (ii) hybrid schemes that use the transformer’s prediction as a warm‑start for exact solvers, and (iii) theoretical analysis of approximation bounds for the learned predictor.