Reinforcement learning has demonstrated remarkable success in complex decision-making tasks; however, its deployment in real-world systems reveals a critical limitation: learned policies frequently exhibit rapid, erratic control behaviors that optimize immediate rewards while incurring significant operational costs. This phenomenon manifests across diverse domains—from robotic manipulators experiencing mechanical stress due to abrupt movements  to building HVAC systems consuming excess energy through frequent equipment cycling .

The core challenge lies in the mismatch between RL’s objective function and real-world constraints. Standard RL formulations maximize cumulative rewards without considering the derivative structure of action sequences. While this approach succeeds in simulation environments, physical systems operate under thermodynamic, mechanical, and economic constraints that penalize high-frequency control variations. We address this challenge through three main contributions: (1) We introduce a third-order derivative penalty and compare first-order and second-order derivative penalties across control benchmarks. (2) We demonstrate that jerk minimization (third-order derivatives) provides optimal smoothness-performance trade-offs. (3) We validate our approach on the HVAC system, quantifying equipment longevity improvements and energy efficiency gains.

Our work establishes that incorporating higher-order action regularization enables RL deployment in energy-critical applications where smoothness directly impacts operational efficiency and system lifetime.

Related Work

Smooth control in RL has been addressed through regularization approaches that add penalty terms to reward functions. Early robotics work penalized first-order action differences , while recent studies have investigated second-order penalties in manipulation . However, systematic comparison across derivative orders remains limited, and higher-order penalties have received minimal attention despite their theoretical importance for mechanical systems.

Quantifying smoothness in continuous control varies across domains. Robotics commonly uses jerk (third derivative of position) due to its relationship with mechanical stress and energy consumption . Control theory emphasizes total variation and spectral properties , while recent RL work has primarily focused on action variance . Our work adopts jerk-based metrics as they capture both kinematic smoothness and practical concerns for physical deployment.

Building energy systems present unique opportunities for smooth control validation. HVAC systems benefit from gradual adjustments that maintain occupant comfort while minimizing equipment cycling . Studies demonstrate that excessive switching increases energy consumption compared to optimal smooth control . Recent work has begun applying RL to building management , but most implementations focus on reward engineering rather than fundamental smoothness constraints.

Problem formulation and methodology

Markov decision process with action history

We consider continuous control tasks as Markov Decision Processes (MDPs) defined by state space $`\mathcal{S}`$, action space $`\mathcal{A} \subset \mathbb{R}^d`$, transition dynamics $`P(s'|s, a)`$, and reward function $`r(s, a)`$. To maintain the Markov property while incorporating action smoothness constraints, we augment the state space with action history:

\begin{equation}
\tilde{s}_t = [s_t, a_{t-1}, a_{t-2}, a_{t-3}]
\end{equation}

This augmentation enables direct computation of action derivatives. The policy $`\pi_\theta(\tilde{s}_t)`$ can then access sufficient information for derivative-based regularization.

Higher-order derivative penalties

We augment the reward function with penalties targeting different orders of action derivatives:

\begin{align}
\text{First-order (velocity):} \quad &r'_t = r_t - \lambda_1 \|a_t - a_{t-1}\|^2 \label{eq:first_order}\\
\text{Second-order (acceleration):} \quad &r'_t = r_t - \lambda_2 \|a_t - 2a_{t-1} + a_{t-2}\|^2 \label{eq:second_order}\\
\text{Third-order (jerk):} \quad &r'_t = r_t - \lambda_3 \|a_t - 3a_{t-1} + 3a_{t-2} - a_{t-3}\|^2 \label{eq:third_order}
\end{align}

Norms are computed as $`\|x\|^2 = \sum_i x_i^2`$ with equal weighting across action dimensions. Higher-order penalties target increasingly sophisticated aspects of smoothness. Jerk minimization is particularly relevant for mechanical and thermal systems, as rapid acceleration changes induce stress, vibration, and energy inefficiency.

Experimental setup

Continuous control benchmarks: We evaluate across four OpenAI Gym environments : HalfCheetah-v4 (high-dimensional locomotion), Hopper-v4 (unstable balancing), Reacher-v4 (precise manipulation), and LunarLanderContinuous-v2 (fuel-efficient spacecraft control). All experiments use PPO with consistent hyperparameters across methods, training for 1M timesteps with results averaged over 5 random seeds. For penalty weights in equations [eq:first_order]–[eq:third_order], we use $`\lambda_1 = \lambda_2 = \lambda_3 = 0.1`$ across all environments, enabling direct comparison of derivative orders without confounding from differential scaling.

Building energy benchmark: We evaluate our methodology on a two-zone HVAC control environment, a custom-built open-source DollHouse, with SINDy-identified dynamics  from practical building data. The system controls temperature setpoints and damper positions while minimizing energy consumption and maintaining occupant comfort.

Results

Evaluation metrics

Smoothness quantification: We measure smoothness using jerk standard deviation: $`\sigma_{\text{jerk}} = \text{std}(\dddot{a}_t)`$, computed via third-order finite differences.

Energy efficiency: HVAC energy consumption follows standard building energy models, with particular focus on equipment switching frequency as a key efficiency indicator.

Continuous control benchmarks

Table 1 summarizes our continuous control results. Third-order derivative penalties consistently produce the smoothest policies, reducing jerk standard deviation by 78.8% in HalfCheetah, 77.3% in Hopper, 38.6% in Reacher, and 58.3% in LunarLander relative to the baseline. These gains come with only a modest performance cost in certain environments, indicating that higher-order regularization is the most effective approach to achieving smooth control. The trend across derivative orders is also clear: first-order penalties yield modest improvements, second-order penalties strengthen the effect, and third-order penalties consistently provide the largest reductions in action jerkiness across all environments.

Building energy management validation

style="width:60.0%" />

Figure 1 demonstrates the practical value of smooth control in building energy management. Third-order regularization achieves the optimal performance-smoothness trade-off, maintaining competitive reward while delivering superior smoothness properties.

HVAC short cycling significantly increases energy consumption. On-off control strategies can increase energy costs compared to continuous operation, with demand-limiting control methods achieving energy reduction ratios of 9.8% to 10.5%. Short cycling leads to increased energy consumption and higher energy bills due to systems working harder and less efficiently. Our smooth control approach reduces equipment switching events by 60%, aligning with established findings that frequent cycling imposes substantial energy penalties on HVAC systems.

Analysis and discussion

Why higher-order penalties excel

Our results reveal that third-order derivatives provide the most effective regularization for several reasons: (1) Physical alignment: Jerk minimization aligns with mechanical and thermal system constraints, where rapid acceleration changes cause stress and inefficiency. (2) Learning stability: Higher-order penalties reduce gradient noise during training, improving convergence quality. (3) Practical deployment: Smooth policies transfer better to real systems where actuator dynamics and sensor noise affect performance.

Energy efficiency mechanisms

The connection between smoothness and energy efficiency operates through multiple mechanisms: (1) Reduced switching losses: Each HVAC startup consumes significantly more power than steady-state operation. By reducing switching frequency, we eliminate unnecessary startup energy penalties. (2) Thermal efficiency: Gradual temperature changes maintain the system efficiency curves, whereas rapid setpoint changes force operation in suboptimal regimes. (3) Equipment longevity: HVAC components have finite switching cycles. Reducing daily switching events extends equipment life and reduces maintenance requirements.

Limitations and future work

Our work has several limitations that warrant discussion: (1) Hyperparameter selection: The penalty weights $`\lambda`$ require domain-specific tuning. Systematic methods for selecting appropriate penalty magnitudes across different applications remain an open challenge. (2) Performance trade-offs: While we demonstrate energy benefits in HVAC systems, the performance cost of smoothness constraints may not be justified in all domains. Clear guidelines for when smoothness should be prioritized over raw performance need development. (3) Limited scope: Energy management validation focuses on HVAC control; broader building systems (lighting, elevators) and other energy-critical domains require investigation.

Future work should develop adaptive penalty weighting schemes that adjust smoothness constraints based on system state and operational requirements, integrate these methods with model-predictive control for longer planning horizons, and establish principled criteria for hyperparameter selection across diverse applications.

Conclusion

We have demonstrated that higher-order action regularization, particularly third-order derivative penalties, provides an effective solution for deploying reinforcement learning in energy-critical applications. Our systematic evaluation across continuous control benchmarks establishes that jerk minimization achieves superior smoothness-performance trade-offs compared to lower-order methods.

The practical validation in building energy management demonstrates concrete operational value: 60% reduction in equipment switching events, aligning with established literature on HVAC energy efficiency. This represents actionable evidence that principled smoothness methods can bridge the gap between RL optimization and real-world deployment constraints.

As building automation and smart city technologies increasingly rely on RL-based control, incorporating higher-order action regularization becomes essential for sustainable, efficient, and economically viable deployments. Our work provides both theoretical foundations and practical validation for this critical capability.