ML Compass: Navigating Capability, Cost, and Compliance Trade-offs in AI Model Deployment
We study how organizations should select among competing AI models when user utility, deployment costs, and compliance requirements jointly matter. Widely used capability leaderboards do not translate directly into deployment decisions, creating a capability-deployment gap; to bridge it, we adopt a systems-level view of model selection and deployment, in which model choice is tied to application outcomes, operating constraints, and a feasible capability-cost frontier. We develop ML Compass, a unifying framework that treats model selection as a constrained optimization problem over this frontier. On the theory side, we characterize optimal model configurations under a parametric frontier and show that optimal internal measures exhibit a three-regime structure: some dimensions are pinned at compliance minima, some saturate at their maximum feasible levels, and the remainder take interior values governed by the curvature of the frontier. We derive comparative statics that quantify how budget changes, regulatory tightening, and technological progress propagate across capability dimensions and costs. On the implementation side, we propose a practical pipeline that (i) extracts low-dimensional internal measures from heterogeneous model descriptors, (ii) estimates an empirical frontier from capability and cost data, (iii) learns a user-or task-specific utility function from interaction-level outcome data, and (iv) uses these components to target capability-cost profiles and recommend models. We validate ML Compass with two case studies: a general-purpose conversational setting using the PRISM Alignment dataset and a healthcare setting using a custom dataset we build using HealthBench. In both environments, our framework produces recommendations-and corresponding deployment-aware leaderboards based on predicted deployment value under explicit constraints-that can differ materially from capability-only rankings, and clarifies how trade-offs between capability, cost, and safety shape optimal model choice.