Decomposable Reward Modeling and Realistic Environment Design for Reinforcement Learning-Based Forex Trading

Applying reinforcement learning (RL) to foreign exchange (Forex) trading remains challenging because realistic environments, well-defined reward functions, and expressive action spaces must be satisfied simultaneously, yet many prior studies rely on simplified simulators, single scalar rewards, and restricted action representations, limiting both interpretability and practical relevance. This paper presents a modular RL framework designed to address these limitations through three tightly integrated components: a friction-aware execution engine that enforces strict anti-lookahead semantics, with observations at time t, execution at time t+1, and mark-to-market at time t+1, while incorporating realistic costs such as spread, commission, slippage, rollover financing, and margin-triggered liquidation; a decomposable 11-component reward architecture with fixed weights and per-step diagnostic logging to enable systematic ablation and component-level attribution; and a 10-action discrete interface with legal-action masking that encodes explicit trading primitives while enforcing margin-aware feasibility constraints. Empirical evaluation on EURUSD focuses on learning dynamics rather than generalization and reveals strongly non-monotonic reward interactions, where additional penalties do not reliably improve outcomes; the full reward configuration achieves the highest training Sharpe (0.765) and cumulative return (57.09 percent). The expanded action space increases return but also turnover and reduces Sharpe relative to a conservative 3-action baseline, indicating a return-activity trade-off under a fixed training budget, while scaling-enabled variants consistently reduce drawdown, with the combined configuration achieving the strongest endpoint performance.

💡 Research Summary

The paper tackles three intertwined challenges that have long hindered the practical deployment of reinforcement learning (RL) in foreign‑exchange (Forex) markets: (1) realistic market execution, (2) interpretable and tunable reward design, and (3) expressive yet constraint‑aware action spaces. To address these, the authors build a modular, open‑source framework comprising (i) a friction‑aware execution engine, (ii) a decomposable 11‑component reward architecture, and (iii) a 10‑action discrete interface with legality masking.

The execution engine enforces strict anti‑look‑ahead semantics: the agent observes the close price at time t, its order is filled at the open of t + 1, and the position is marked‑to‑market at the close of t + 1. All major sources of trading friction are modeled explicitly—bid‑ask spread, commission, slippage, dynamic roll‑over financing (including the triple‑swap Wednesday rule), and margin‑triggered liquidation. By separating observation, execution, and settlement timestamps, the simulator eliminates hidden look‑ahead leakage and narrows the sim‑to‑real gap.

Reward is split into eleven independent terms (e.g., raw P&L, financing cost, spread cost, turnover penalty, drawdown penalty, margin‑usage cost, etc.). Each term has a fixed weight defined a priori, and per‑step values are logged. This design enables systematic ablation: researchers can toggle individual components, observe their direct impact on learning dynamics, and attribute performance changes to specific economic drivers. Experiments reveal strongly non‑monotonic interactions—adding more penalty terms does not guarantee higher Sharpe ratios or returns. The full‑reward configuration attains the highest training Sharpe of 0.765 and a cumulative return of 57.09 %.

The action space encodes ten discrete primitives that reflect real trading operations: open long, open short, close, partial reduction, pyramiding (adding to an existing winning position), martingale‑style scaling (adding after a loss), reversal, and others. A legality mask, computed from the current margin and position state, disables actions that would breach margin constraints, both during environment interaction and when constructing target values for learning. This ensures that the learned policy never proposes infeasible trades.

Three controlled experiment families are conducted on EUR/USD hourly data, each focusing on a specific research question. (RQ1) evaluates which reward components contribute useful learning signals; the full set wins, while progressive addition of penalties yields non‑monotonic Sharpe behavior. (RQ2) compares the 10‑action interface against a minimalist 3‑action baseline; the richer set improves raw return but raises turnover and reduces Sharpe, highlighting a return‑activity trade‑off under a fixed training budget. (RQ3) studies asymmetric scaling strategies (pyramiding vs. martingale); both scaling‑enabled variants lower drawdown relative to a no‑scaling baseline, with the combined configuration delivering the strongest endpoint performance.

Methodologically, the authors emphasize reproducibility: all experiments are driven by configuration snapshots, random seeds are fixed, and deterministic logging of reward components is provided. The environment is Gymnasium‑compatible, enabling easy integration with existing RL libraries.

Limitations include the focus on a single currency pair and a fixed historical training window, which may not capture regime shifts or multi‑asset interactions. Real‑time data latency, order‑book dynamics, and adaptive slippage are abstracted rather than fully simulated. The study also relies on value‑based DQN‑style learning; policy‑gradient methods (PPO, SAC) are not evaluated.

Future work is suggested along several axes: extending the framework to multi‑pair and multi‑asset portfolios, incorporating online learning with live data feeds, benchmarking against policy‑gradient algorithms, and enriching risk‑management modules (e.g., VaR or CVaR constraints).

In sum, the paper delivers a comprehensive system that simultaneously advances execution fidelity, reward transparency, and action feasibility for Forex RL. By open‑sourcing the components and providing detailed logging, it sets a new standard for reproducible, economically realistic RL research in algorithmic trading.