Reinforcement Learning Trading Agents: Easy in Theory, Hard in Practice

RL vs. supervised learning

Supervised learning teaches "what's the right answer." RL teaches "how to try things."

Three core elements of RL

1. State
All observable information about the environment — in trading: prices, order book, technical indicators, current position.

2. Action
What the agent can do — buy / sell / hold / adjust position size.

3. Reward
The environment's feedback — usually realized P&L plus a risk adjustment.

Classic algorithms: Q-learning, Deep Q-Network (DQN), policy gradient, PPO.

The AlphaGo lesson — and where finance differs

AlphaGo beat Lee Sedol in 2016, an RL milestone. But Go has properties that markets don't:

AlphaGo could self-play millions of games. A trading agent cannot "self-trade" — the market only happens once.

Real-world RL trading applications

1. Optimal execution
Order slicing: how to break 100,000 shares into smaller orders to minimize market impact over a window. This is RL's most mature, most production-used trading application.

2. Market making quotes
Dynamically adjusting bid-ask quotes to balance inventory risk and spread income.

3. Portfolio rebalancing
When and how much to rebalance to maximize long-term cumulative Sharpe ratio.

4. High-frequency strategies
Microsecond decisions — but requires an extremely high-fidelity environment simulator.

Why RL is uniquely hard in finance

1. Sim-to-real gap
Profitable in simulation ≠ profitable in production. Market impact, opponent reactions, latency — none are accurately replicated by simulators.

2. Non-stationary environment
A strategy learned this year may break next year as market structure shifts.

3. Exploration costs real money
RL needs trial-and-error to learn. Real-market trial-and-error = real losses.

4. Reward engineering is hard
"Maximize return" is a terrible reward — agents will trend toward extreme leverage. Designing risk-balanced reward functions is itself a research area.

Important questions

Can a retail trader build an RL agent?
You can build something that "looks great in simulation." Building one that consistently profits in production is extremely hard. Toy versions can be built using the financial environments in OpenAI Gym to learn concepts.

Is RL really better than supervised learning?
Not always. Most trading problems are well-served by supervised models + rule-based execution. RL shines in long decision sequences with complex action spaces (like optimal execution).

Is ChatGPT's RLHF related to trading RL?
Same theoretical framework (both RL) but completely different goals. RLHF uses human preferences as reward signal, mainly to align language models.

Quiz

Q1. The core difference between RL and supervised learning?
A. RL doesn't use data　B. RL learns decisions via state-action-reward feedback loops; supervised learns mappings from labeled data
C. Supervised is newer　D. They're identical

Q2. RL's most mature trading application is:
A. Long-term holding　B. Optimal execution (order slicing, minimizing market impact)
C. Predicting tomorrow's direction　D. Stock picking

Q3. Why is RL harder in finance than in Go?
A. Go is more complex　B. Markets are non-stationary, rewards are delayed, simulators are imperfect, and your trades affect the environment
C. No GPUs　D. Too much data

Reference Answers

Q1: B　Q2: B　Q3: B

Further reading: Wikipedia: Reinforcement Learning　·　Wikipedia: AlphaGo　·　OpenAI Gym　·　Wikipedia: Algorithmic Trading — Optimal Execution

Educational content only — not investment advice. RL models carry significant production risk. Research-stage success doesn't imply profitability.