Case study · Event-driven trading systemBacktest run 2025-01 → 2025-12

Signal to trade in 30 seconds.
Then the edge disappears — and proving that was the point.

A system that ingests every Truth Social post by the U.S. president, has an LLM judge each one for market impact, and converts the judgment into sized, stop-protected paper trades. Built end-to-end solo — ingestion, two-stage classifier, backtest harness, live paper execution. The backtest's job was to find out if the edge is real before risking anything. It did its job.

Posts archived
25,125
6,688 in the 2025 window
Trades simulated
519
× 4 latency scenarios
Return @ 30s latency
+0.81%
on $100k · 5bps slippage
Out-of-sample return
−0.14%
post model-cutoff cohort
01 · The system

One LLM judgment drives both size and holding time

Instead of keyword rules (if "tariff" in post: sell), a two-stage classifier produces one structured judgment per post. A cheap model kills ~83% of posts as irrelevant; a stronger judge scores the survivors. Conviction sets position size (convex — only rare, concrete posts get real money), durability sets the exit timer, and hard + trailing stops guard every position.

Ingest
20s poll of Truth Social; text-hash dedup across restarts.
asyncio · SQLite
Judge
Haiku relevance filter → Sonnet judgment. Strict JSON, retry-once, fail-safe skip.
2-stage LLM · cached
Decide
conviction² × durability → size. Durability caps the hold: 5 min – 6 h.
pure functions
Execute
Paper orders, idempotent on post ID. 2% stop-loss, 1.5% trailing stop.
Alpaca paper API
Measure
Every trade replayed at 0 / 30 / 60 / 300s entry latency against minute bars.
backtest harness
« GREAT TIME TO BUY!!! » — posted 2025-04-09 13:37 UTC, minutes before the tariff pause
→ { "market_relevant": true, "direction": "bullish", "conviction": 9, "durability": "concrete", "instrument": "SPY" }
→ long SPY $8,100 · exit on trailing stop · +3.06%
02 · The funnel

6,688 posts in, 519 trades out

Every stage logs what it drops and why — no silent losses. The gap between "classified" and "posts in window" is an API credit outage the pipeline survived gracefully; between "candidates" and "trades" sit delisted tickers, FX pairs Alpaca can't price, and posts landing outside market hours.

03 · Results

The edge decays with every second of latency

Exactly as theory predicts for a crowded event trade: entering at the post timestamp captures +1.03%; waiting five minutes leaves +0.36%. A hobby stack lives in the 30–60s band. The equity curve tells the concentration story — two days carry nearly all of it.

Return by entry latency

% return on $100k notional · 519 trades each

P&L by conviction score

total $ per bucket @ 30s · conviction 4 = threshold

Cumulative P&L, 2025 · 30s latency

$ on fixed $100k per-trade equity · hover for daily detail

04 · The honest number

Split the year at the model's training cutoff

An LLM judging 2025 posts may simply remember 2025. So the harness splits results at the judge's training cutoff (2025-08-01). Trades the model could have memorized look profitable. Trades it had genuinely never seen do not. This is the finding.

Before cutoff · look-ahead risk
+1.16%
Sharpe 0.44 · hit 30.3% · 390 trades

The model's training data likely covers these events. Any "prediction" here can't be distinguished from recall.

After cutoff · unbiased
−0.14%
Sharpe −2.79 · hit 23.3% · 129 trades

On events the model had never seen, the strategy loses money. The apparent edge is largely memory, not foresight.

05 · The tape

Where the money actually lived

One trade — going long Elon Musk's X on a concrete deal post — earned 90% of the year's P&L. Winners were trades the market let run (trailing-stop exits averaged +0.24%); the 476 timer exits netted out negative. Losses stayed small by design: the worst trade lost $77 against a best of $735.

InstrumentSideDateConvDurabilityExitSizeP&LReturn
06 · Findings

What 519 trades taught

Concentration

The edge is 2 trades a month, not 500 a year. Conviction 8–9 + concrete posts made +$966 across 23 trades; the 439 conviction 4–6 trades lost money net. The sizing function knew this — the entry threshold didn't.

Latency

Two-thirds of the edge dies in five minutes. The decay curve confirms this is a speed game owned by co-located players. A hobby stack can observe the move; it can't reliably capture it.

Look-ahead

An LLM backtest over its own training window flatters itself. Splitting at the training cutoff turned +1.16% into −0.14%. Any LLM-driven strategy evaluated on pre-cutoff data needs this control.

Risk plumbing

The defensive layer worked. Stops capped every loss under $80 while letting the $735 winner run; malformed LLM responses, delisted tickers, unpriceable FX and a mid-run API credit outage were all absorbed and logged — the run completed anyway.

Verdict

Not tradeable as-is — and the system proved it for the price of a few takeaways. The realistic product is the classifier as a real-time alert engine: conviction-9 concrete calls delivered to a human ~30s after the post. The next test is live: unattended forward paper trading, where nothing can be memorized.