>_ Jorge Uc / projects← All projects
~/projects/deep-rl-trading-agent // REINFORCEMENT LEARNING
◆ Deep Reinforcement LearningPyTorchResearch experiment

Can an agent learn when to trade?

Deep RL Trading Agent investigates decision-making on historical financial data using a Dueling Deep Q-Network. I reproduced and adapted a reference implementation to compare 3-bar and 10-bar observation windows and explore how simulated transaction commissions affect rewards.

YNDX / HISTORICAL SIMULATION● OFFLINE
BUYCLOSEBUY
STATE → Q(·) → ACTIONILLUSTRATIVE / NOT RESULTS
METHODDueling DQNValue + advantage streams
WINDOWS03 / 10Historical price bars
MARKET DATAYNDX2015 and 2016
OUTPUT12 plots2 windows × 6 figures
01 / PROJECT CONTEXT

Trading as a sequential decision problem.

Instead of predicting a price in isolation, an RL agent learns a policy through repeated interactions with a simulated market environment. Its actions influence future rewards and expose the role of trading frictions.

01 / OBSERVATIONS

How much history matters?

Compare shorter and longer price-history windows to examine their impact on the behavior learned by the agent.

02 / TRANSACTION COSTS

What changes after commissions?

Examine simulated rewards under several commission assumptions, instead of treating trades as cost-free decisions.

03 / EVALUATION

What do reward plots show?

Interpret training and evaluation curves carefully without equating simulated reward with verified trading profits.

02 / RL ENVIRONMENT

States, actions, and rewards.

The environment turns historical stock quotes into an interactive decision process. This simplified representation outlines the environment used in the project.

STATE / Sₜ
[▥ ▥ ▥]

Price history + position state

Relative market features from a configurable number of historical bars, together with information about the simulated trading position.

ACTION / Aₜ
SKIPBUYCLOSE

Three available decisions

The policy selects an action at each step. These represent decisions in the simulation, not orders sent to a real exchange.

REWARD / Rₜ
Δprice − cost

Profit-sensitive feedback

Rewards depend on simulated market movements, positions and the transaction-cost configuration.

Research scope. Historical data and simulated execution do not reproduce every source of risk, slippage, liquidity or market impact encountered in real trading.
03 / MODEL ARCHITECTURE

Learning the value of decisions.

A Dueling DQN estimates action values through separate state-value and action-advantage pathways. The model is trained using replayed experience rather than a supervised label for the “correct” trade.

> agent.pipelineCONCEPTUAL ARCHITECTURE
INPUTMarket observationPrice features · Position features
↓
FEATURE REPRESENTATIONShared neural layersPyTorch network
↓
STREAM AState value V(s)How useful is this state?
STREAM BAdvantage A(s,a)How useful is each action?
↓
COMBINATIONQ(s,a) valuesChoose Skip / Buy / Close
Training uses an ε-greedy policy, experience replay and a target network. Diagram simplified from the reference implementation.
04 / EXPERIMENTAL DESIGN

One agent. Different assumptions.

I explored the impact of observation length and simulated trading fees on the learned policy. The original report documents two window configurations and multiple commission settings.

03 BARS

Short observation window

Represent the state with a shorter recent market history, reducing the amount of temporal context provided to the agent.

View 3-bar plots ↗
10 BARS

Long observation window

Provide a longer sequence of past observations to explore whether additional context changes simulated rewards.

View 10-bar plots ↗
COMMISSION SENSITIVITY

Small percentages, meaningful assumptions.

The saved figures include runs labeled no commission, 0.05%, 0.06% and 0.075%. These are plotted outputs from the original experiment, not a guarantee of statistically significant differences.

0%0.05%0.06%0.075%
05 / EXPERIMENTAL OUTPUTS

Explore the recorded reward plots.

Browse all 12 saved figures, organized by observation-window size. The original images are displayed at their natural proportions and can be opened at full size.

06 / RESULTS & INTERPRETATION

What the figures can—and cannot—tell us.

The experiment produced reward trajectories for distinct input windows and cost assumptions. Without a standardized out-of-sample benchmark, the plots should be treated as exploratory evidence rather than a profitability claim.

OBSERVATION WINDOW

Context changes the input

Three versus ten bars changes what the agent sees. Comparing outcomes is useful, but window length alone cannot explain every variation between training runs.

MARKET FRICTION

Commissions belong in the model

Trading decisions should be evaluated under transaction costs, which can materially affect any frequently trading strategy.

EVIDENCE QUALITY

A plot is not a backtest certificate

Reliable comparisons would also need chronological holdout data, repeated seeds, explicit return metrics and a buy-and-hold baseline.

07 / TECHNICAL IMPLEMENTATION

From price data to an evaluated policy.

The repository combines historical datasets, a custom market simulation, neural models, training scripts and reward visualizations. The code is adapted from an openly credited learning reference.

01 / DATA

Historical quote preprocessing

Read market observations and transform price information into state features for the RL environment.

PythonCSVTime series
02 / ENVIRONMENT

Custom trading simulation

Implement discrete position actions, state transitions and commission-aware simulated rewards.

RL environmentReward design
03 / TRAINING

DQN optimization

Train a Q-value estimator with a policy for exploration and mechanisms for more stable off-policy updates.

PyTorchReplay bufferTarget network
04 / ANALYSIS

Saved runs and result plots

Compare observation windows, simulated commission settings and recorded reward behavior across experiments.

ExperimentationVisualization
08 / PERSONAL RETROSPECTIVE

What I learned from this experiment.

The most useful outcome was not a particular trading signal: it was learning to design RL experiments, interpret reward curves and question the validity of financial-model evaluations.

BEYOND THE MODEL

A trained agent is not automatically a useful trading strategy. Reproducing and modifying this experiment helped me connect sequential decision-making with experiment design, evaluation methodology and the limits of historical simulations.

01 / REINFORCEMENT LEARNING

Reward design drives behavior

I saw why states, actions and rewards must be designed together. What an agent optimizes depends on how the simulation defines success.

MDPsRewardsPolicies
02 / DEEP LEARNING

Stability requires more than a network

Understanding Dueling DQN, epsilon-greedy exploration, replay buffers and target networks clarified how value-based RL differs from ordinary supervised learning.

Dueling DQNPyTorch
03 / EXPERIMENTAL DESIGN

Control what the agent receives

Testing 3-bar and 10-bar input windows demonstrated how seemingly small choices in state representation change the problem being learned.

Ablation-style comparisonTime series
04 / RESPONSIBLE EVALUATION

Historical success needs skepticism

Transaction fees, data chronology and baseline comparisons matter. Reward curves alone are insufficient evidence of generalization or investment performance.

BacktestingBiasLimitations
KEY TAKEAWAY

The real skill was learning how to evaluate an intelligent decision-maker—not just how to train one.

This project motivated a more rigorous view of sequential models: always distinguish simulated feedback from actual performance, isolate experimental variables and evaluate on genuinely unseen future periods.

09 / LIMITATIONS & NEXT STEPS

A useful experiment, not a live trading system.

The work reproduces and adapts a published implementation in an academic context. Its outputs are exploratory and should not be interpreted as investment recommendations or proof of profitable automated trading.

CURRENT LIMITATION

Reverse chronological split

The documented setup trained on 2016 data and evaluated on 2015. This is not a forward-in-time holdout test.

NEXT ITERATION

Chronological evaluation

Train on earlier periods, validate on a later period and reserve the latest unseen interval for a final test.

NEXT ITERATION

Benchmark & robustness

Report portfolio returns and drawdowns, compare against buy-and-hold, repeat random seeds and test greater realism in transaction assumptions.

SOURCE CODE & DOCUMENTATION

Explore the experiment on GitHub.

Code, original report, methodology and saved reward visualizations. The repository credits the source implementation and describes the study's limitations.

View repository ↗