LLM-Inspired Pretrain-Then-Finetune for Small-Data, Large-Scale Optimization

We consider small-data, large-scale decision problems in which a firm must make many operational decisions simultaneously (e.g., across a large product portfolio) while observing only a few, potentially noisy, data points per instance. Inspired by the success of large language models (LLMs), we propose a pretrain-then-finetune approach built on a designed Transformer model to address this challenge. The model is first pretrained on large-scale, domain-informed synthetic data that encode managerial knowledge and structural features of the decision environment, and is then fine-tuned on real observations. This new pipeline offers two complementary advantages: pretraining injects domain knowledge into the learning process and enables the training of high-capacity models using abundant synthetic data, while finetuning adapts the pretrained model to the operational environment and improves alignment with the true data-generating regime. While we have leveraged the Transformer’s state-of-the-art representational capacity, particularly its attention mechanism, to efficiently extract cross-task structure, our approach is not an off-the-shelf application. Instead, it relies on problem-specific architectural design and a tailored training procedure to match the decision setting. Theoretically, we develop the first comprehensive error analysis regarding Transformer learning in relevant contexts, establishing nonasymptotic guarantees that validate the method’s effectiveness. Critically, our analysis reveals how pretraining and fine-tuning jointly determine performance, with the dominant contribution governed by whichever is more favorable. In particular, finetuning exhibits an economies-of-scale effect, whereby transfer learning becomes increasingly effective as the number of instances grows.

💡 Research Summary

This paper tackles a class of decision‑making problems that are increasingly common in modern operations management: firms must simultaneously solve thousands or even millions of stochastic optimization tasks (e.g., pricing, inventory, assortment) while each individual task provides only a handful of noisy observations. The authors refer to this setting as “small‑data, large‑scale optimization.” Traditional data‑driven approaches such as Sample Average Approximation (SAA) become infeasible because the per‑instance sample size is far too small to estimate the underlying demand distribution reliably.

Inspired by the success of large language models (LLMs), the authors propose a two‑stage learning pipeline—pretrain‑then‑fine‑tune—built around a purpose‑designed Transformer architecture. In the pretraining stage, they generate massive synthetic datasets that embed domain knowledge (managerial heuristics, stylized theoretical models, or outputs of generative models). Each synthetic instance contains the true parameters of the stochastic demand distribution together with the optimal decision for that instance. This abundant data allows a high‑capacity Transformer to learn rich representations of the underlying decision structure.

The Transformer is not used as a generic function approximator; rather, it serves as a parameter estimator. Input tokens encode variable‑length information about each product (features, historical sales, contextual attributes). Multi‑head attention aggregates information across heterogeneous instances, enabling the model to capture cross‑task regularities that are crucial for transfer learning. The output of the network is an estimate of the latent demand parameters, which are then fed into a downstream optimization module to produce the actual operational decision (e.g., order quantity).

In the fine‑tuning stage, only a limited set of real observations is available, and the true parameters are unobserved. To overcome the lack of labels, the authors derive an MSE‑equivalent loss using a generalized Stein’s identity, which yields a tractable surrogate objective that aligns with minimizing the parameter estimation error. Moreover, they adopt Low‑Rank Adaptation (LoRA) to update only a small, low‑dimensional subset of the Transformer’s weights, drastically reducing the data requirement and computational burden.

The theoretical contribution is a comprehensive error decomposition. The total excess risk of the induced decisions is split into three components: (i) a pretraining domain gap, measuring the distributional mismatch between synthetic and real environments; (ii) a fine‑tuning generalization error, which scales as O(1/√N) where N is the number of simultaneous tasks, revealing an economies‑of‑scale effect; and (iii) an approximation error, reflecting the expressive limitation of the chosen model class. The analysis shows that whichever of the domain gap or the generalization error is larger will dominate the overall performance, providing a clear guideline on when to invest more effort in better synthetic data versus collecting more real observations. The approximation error decreases with model capacity, justifying the use of large Transformers when abundant synthetic data are available.

Empirically, the authors evaluate the approach on a multi‑product newsvendor problem with up to 10,000 SKUs. They compare (a) a baseline small neural network trained directly on real data, (b) a large Transformer trained only on synthetic data, (c) the full pretrain‑then‑fine‑tune pipeline, and (d) classical SAA and meta‑learning baselines. Results show that:

1. Synthetic‑data pretraining is essential for training a high‑capacity model when real data are scarce; the quality of the pretrained model hinges on how accurately the synthetic data reflect the true environment.
2. When domain knowledge is accurate, the pretrained model alone already achieves performance close to the oracle, and fine‑tuning yields only marginal gains.
3. When the synthetic data are misspecified, fine‑tuning becomes crucial; as N grows, the fine‑tuned model progressively corrects the bias inherited from pretraining and its decisions converge to the oracle benchmark.

The paper’s contributions can be summarized as follows:

• Introduction of a domain‑guided synthetic data generation pipeline for pretraining.
• Design of a task‑specific Transformer that acts as a parameter estimator and leverages attention for cross‑task transfer.
• Development of a Stein‑identity‑based loss for unlabeled fine‑tuning and the use of LoRA for data‑efficient adaptation.
• A novel non‑asymptotic error analysis that quantifies the interplay between pretraining, fine‑tuning, and model capacity, and reveals an economies‑of‑scale effect in the fine‑tuning stage.
• Extensive experiments confirming the theoretical insights and demonstrating substantial cost reductions in a realistic large‑scale newsvendor setting.

Overall, the work bridges the gap between LLM‑style pretraining‑fine‑tuning and practical operations‑management problems, offering both a solid theoretical foundation and a pragmatic framework that can be deployed in industries where data are scarce but decision spaces are massive.