Three core data categories
1. Market data
Prices, volume, order book, trade prints.
- Free sources: Yahoo Finance (yfinance), Stooq, Quandl
- Paid: Bloomberg, Refinitiv, Polygon (mid-sized institutions)
2. Fundamentals
Filings, valuation ratios, industry metrics.
3. Alternative data
Non-traditional sources — anything reflecting business activity:
- Satellite imagery (counting cars in parking lots, oil tankers, crops)
- Credit card transaction aggregates
- Job posting counts
- App store downloads
- Web scraping (e-commerce prices, reviews)
- Social media sentiment
Hidden traps in market data
1. Stock split / dividend adjustments
"Forward-adjusted" vs. "back-adjusted" vs. "unadjusted" — three completely different price meanings. Mixing them produces severely distorted backtests.
2. Timezone alignment
US daily "close" is 16:00 EST. When comparing across markets, all data must be normalized to a common timezone.
3. Holidays
No trading on holidays — return calculations across holiday boundaries need special handling.
4. Delisted stocks
Backtests require survivorship-free datasets — using only currently-listed stocks inflates returns.
Hidden traps in fundamentals data
1. Point-in-time data
Today's view of historical EPS includes revisions, but the market only knew the unrevised version at the time. Use "as-known-then" EPS to avoid look-ahead bias.
2. Earnings publication lag
Q3 earnings are typically released late October to early November — you can't use end-of-September data to predict early-October prices.
3. Accounting reclassifications
GAAP and IFRS standards evolve; cross-year comparability needs care.
Alternative data: gold mine or trap?
Institutions spend billions on alternative data. But it's far from "AI's magic key":
Real example:
- A fund used satellite data counting Walmart parking lot cars → predicted quarterly revenue → built positions before earnings → excess returns for several months
- As more institutions adopted the same data source, alpha decayed to near zero
Core rule of alternative data: early adopters get alpha; followers get noise.
Required data cleaning steps
Raw data → handle missing → detect outliers → time-align → engineer features → normalize
Common issues:
- Missing values: holidays, halts, source failures — don't just fill with 0
- Outliers: flash crashes, fat-finger trades — whether to keep depends on strategy purpose
- Cross-source alignment: source A uses UTC, source B EST, source C local exchange time — must unify
- Deduplication: the same trade may appear in Level 1 and Level 2 feeds
"Four questions" for data quality
- Source: where does it come from? Primary (official) or secondary?
- Latency: real-time or delayed? By how much?
- Coverage: which securities and time periods? Survivorship issues?
- Revision history: revised? Can you get the original as-known-then version?
If any one fails — re-evaluate before feeding the model.
Important questions
What data quality can retail access?
Free Yahoo Finance is fine for learning but has survivorship, adjustment, and latency issues. Mid-tier paid sources (Polygon, Alpaca) are relatively affordable. Institutional-grade data starts in the tens of thousands per year.
Is alternative data worth pursuing?
For individuals or small teams, ROI is low — mainstream alt data is already saturated with institutional money. Unless you have a unique data source no one else has (your own e-commerce operations data, for example), not worth it.
How do I validate the data?
Basic moves: visualize random samples, compare against a second independent source, check missing-value distribution, sanity-check daily volume. Every engineering-rigorous research project starts by "looking at the data," not by "building the model."
Quiz
Q1. Which is true about price "adjustments"?
A. Adjustments don't matter B. Forward-adjusted, back-adjusted, and unadjusted have completely different meanings — mixing them severely distorts results
C. Always use unadjusted D. Adjustments are unrelated to splits
Q2. The value of point-in-time data is:
A. Faster compute B. Reflects "what was known at the time," avoiding look-ahead bias
C. Higher precision D. Unrelated to ML
Q3. Which is most accurate about alternative data alpha decay?
A. Alpha never decays B. Early adopters get alpha; as adoption grows, alpha decays to near zero
C. More adopters = more profit D. Unrelated to institutions
Reference Answers
Q1: B Q2: B Q3: B
Further reading: Wikipedia: Alternative Data (finance) · SEC EDGAR · FRED — St. Louis Fed Economic Data · Wikipedia: Survivorship Bias
Educational content only — not investment advice. Data quality directly affects trading decisions. Evaluate carefully before use.
Get the pre-trade checklist.
We are turning these guides into a searchable checklist for checking terms, rules and risk before you trade.