Machine Learning in Trading: From Features to Predictions

The supervised learning loop

Input features X → Model f → Predicted output Y

Example: use the last 5 days of returns, volume, and volatility (X) to predict tomorrow's direction (Y).

Common algorithms:

• Linear / logistic regression: simple and interpretable
• Random Forest: robust, handles nonlinearity
• Gradient boosting (XGBoost / LightGBM): the workhorse of finance
• Neural networks: high-dim data, images, text

On tabular financial data, gradient boosting trees usually beat neural networks — counterintuitive but repeatedly demonstrated.

Feature engineering: the lifeblood of the model

Features = the "clues" you feed the model. Good features matter more than fancy algorithms.

Common feature categories in finance:

Train / validation / test splits

The cardinal rule of ML: never train and evaluate on the same data.

Finance-specific rules:

• Time series must be split chronologically — never randomly
• Train (e.g., 2015–2020) → validate (2021) → test (2022–2024)
• Use walk-forward cross-validation for more realistic evaluation

Why? Random splits leak future data into the training set — the notorious look-ahead bias.

Metrics: don't measure accuracy, measure returns

A classic beginner mistake: evaluating a trading model on classification accuracy.

Problem: 80% accuracy sounds great — but if the model is only right on small moves and wrong on big ones, it can still lose money in production.

Better metrics:

• Sharpe Ratio: risk-adjusted return
• Max drawdown: deepest peak-to-trough loss
• Win rate × average payoff ratio: the economic essence
• Information Coefficient (IC): correlation between predictions and realized returns

Tools and environment

Beginner stack (all free):

• Python + Jupyter
• pandas / NumPy: data handling
• scikit-learn: classic ML
• PyTorch / TensorFlow: deep learning
• yfinance: free historical price data

Advanced: open-source backtesting frameworks like QuantConnect.

Three common traps

1. Overfitting
The model memorizes training noise; validation performance collapses. The #1 killer of ML trading.

2. Data leakage
Future information sneaks into features — e.g., using "forward-adjusted" prices without realizing they're only known after the fact.

3. Insufficient sample size
Daily data over 10 years gives only ~2,500 samples. Complex models need far more to avoid overfitting.

Important questions

Is deep learning always better than traditional ML?
No. On financial tabular data, gradient boosted trees regularly beat deep learning. Deep learning's edge shows up with text, images, or very complex time series structure.

Can I build a profitable model from free data?
You can build something that "looks profitable" — most are overfit. Real profitable models need: high-quality data + rigorous methodology + serious compute + risk management.

Do I need to retrain monthly?
Depends on regime change speed. HFT strategies may retrain daily; mid/low-frequency strategies monthly or quarterly. Retraining too often can itself introduce new overfitting.

Quiz

Q1. How should financial time series data be split into train/test?
A. Random sampling　B. Strictly chronological — future data must not be in the training set
C. By region　D. By industry

Q2. The #1 killer of ML trading is:
A. Lack of compute　B. Overfitting
C. Slow network　D. Software bugs

Q3. How should you evaluate a trading model?
A. Classification accuracy only　B. Composite metrics: Sharpe, max drawdown, IC, etc.
C. Win rate only　D. Total return only

Reference Answers

Q1: B　Q2: B　Q3: B

Further reading: Wikipedia: Supervised Learning　·　Wikipedia: Overfitting　·　scikit-learn Documentation　·　Investopedia: Sharpe Ratio

Educational content only — not investment advice. Historical model performance does not guarantee future results.