PQI — Polymarket Qualitative Intelligence

PQI is a solo Python system that identifies mispriced odds in Polymarket event-driven prediction markets using a 7-agent LLM council. Built across 47 documented sessions over 2 months: ~11,400 lines of Python, 83 commits, ~360KB of design docs, retrospectives, and phase plans. One author.

The system is deliberately advisor-only. The CLI prints ranked opportunities with two-sided probability, confidence scores, max ROI, advisory Kelly stake, and a written reason. A human decides whether to trade. Auto-execution is designed but blocked on its own pre-registered signal-quality test, which is currently failing.

The honest current state: measurement infrastructure is the durable product. The trading edge is mild positive but not validated. The council is structurally less diverse than its name implies, and that's the active engineering problem.

7 agents, each a biased system prompt designed for structured disagreement. The point of a council is disagreement, not consensus.

Rules Lawyer resolves on the oracle condition only, ignoring real-world outcome if the resolution criteria don't match. Receives resolution criteria pre-injected as the source of truth. Carries 2x static weight.

Favorite's Advocate and Underdog's Advocate are symmetric adversaries. Each receives a SIDE CONTEXT block identifying the favored vs underdog side from current price, then builds the strongest case for their assigned side.

Base-Rate Statistician anchors on historical base rates, ignoring current hype. Sentiment Analyst reads crowd psychology against current price. Domain Specialist brings field-specific technical nuance (CS map pools, LoL meta patches, etc.). The Contrarian hunts for black-swan scenarios that flip the board.

All 7 share a preamble enforcing grounding (analysis based only on the briefing document), an uncertainty floor (confidence capped at 25 if data is insufficient), and two-sided confidence as independent fields that can and should differ when evidence is asymmetric.

The council runs fully parallel via asyncio.gather. A quorum gate requires at least 5 of 7 verdicts or the run aborts. Per-agent retry with exponential backoff handles transient failures. The parser is failure-tolerant: a single agent failing doesn't kill the council.

Each analysis run assembles a briefing document from four data sources before the council sees anything.

Polymarket Gamma API provides the event and market catalog. A 3-pass category resolver with a persistent cache groups sibling markets (Moneyline, Map 1, Map 2) under one event, so the council evaluates the event as a whole.

Polymarket CLOB API fetches hourly price history for both YES and NO tokens in parallel. YES and NO sit on independent orderbooks and don't sum to 1.0. The price module computes exponential-decay weighted averages (half-life 2 days) and a linear-regression direction classifier (RISING / FALLING / STABLE) with sharp-move detection.

Tavily Search provides web context. An LLM "Search Architect" generates 5 typed queries per market with topic, time range, and domain constraints. Searches run in parallel via asyncio.gather, then a synthesis pass produces the briefing narrative.

PandaScore (esports markets only) provides structured match data: status, scheduled time, final scores, team form, head-to-head records. Polymarket question titles are fuzzy-matched to PandaScore matches using rapidfuzz.fuzz.token_set_ratio with a tunable threshold. Every failure path returns None and the pipeline continues without structured data. When structured data is available, a cross-source contamination tagger uses it as an authoritative anchor: if PandaScore says a match is scheduled or running, regex-detected past-tense scoreline patterns in web results get flagged as temporal contamination.

The council's 7 verdicts are aggregated into a two-sided confidence-weighted score:

yes_score = sum(prob_yes * conf_yes * weight) / sum(conf_yes * weight) no_score = sum(prob_no * conf_no * weight) / sum(conf_no * weight)

Agent weights are dynamic when enabled: computed from each agent's lean-rate deviation from the council average, with shrinkage and a non-negative clamp. The live weights reflect what the data shows about each agent's reliability.

The recommendation then passes through a 5-gate stack:

Price floor/ceiling filters out near-zero and near-resolved markets where execution risk outweighs ROI.

Contrarian gate catches cases where the council disagrees with the market at high confidence and high ROI.

Market-aligned gate labels opportunities by ROI band (HIGH / MODERATE / LOW MARGIN).

Banded EV gate rejects when the side price exceeds the Beta-binomial smoothed win rate for that price band. Bands are [0.09-0.50, 0.50-0.60, 0.60-0.70, 0.70-0.80, 0.80-0.95] with shrinkage (wins + K * global_wr) / (n + K), K=5. Dormant below 20 samples.

Advisory Kelly sizing uses f* = (bp - q) / b with banded win-rate inputs (not scalar), then applies a half-Kelly multiplier. Sizing and gating stay aligned on the same banded threshold. Half-Kelly is non-negotiable: "Dropping the safety scaler to chase variance is how small accounts die fast."

This is the part that outlives the strategy. If the council got scrapped tomorrow and replaced with a statistical model, all of this still applies.

Brier scores measure prediction calibration. Council Brier is 0.2176 vs a 0.25 chance baseline, a ~3 percentage point gap. Real but small.

Per-band realized rates with Wilson 95% confidence intervals show where the system actually performs. The [0.80, 0.95) band is best-EV (realized 0.938, +9.9pp vs mean price). The [0.50, 0.60) band is point-estimate negative-EV but at n=8, the CI is too wide to conclude anything.

Beta-binomial shrinkage with explicit pseudo-count K prevents small-sample bands from producing extreme win-rate estimates that would distort the EV gate.

Pre-event CLV (price at last hourly CLOB bucket before event start minus entry price) measures entry timing quality. Mean is -0.0167 with stddev 0.064. The 95% CI brackets zero, but the point estimate says we're paying ~1.7pp on entry on average.

Per-agent accuracy with confidence-weighted Brier shows which agents are actually contributing signal. This feeds the dynamic weight layer.

Reliability diagrams are generated as dated markdown reports, filterable by phase rollout tag, deliberately bypassing the DuckDB layer (which has a known STRUCT vs MAP inference bug on prompt flags).

DuckDB provides ad-hoc SQL analytics over the append-only JSONL history. Pre-defined views (history, verdicts, clv_by_row) mean any new question is a SQL query, not a Python script. pqi query "SELECT ..." from the CLI.

Equity curve with max drawdown and per-trade Sharpe via pqi history equity tracks cumulative performance over time.

The most important finding from Phase 0 was structural: the 7 agents are closer to 3 effective voices. 5 of 7 form a tight correlation cluster (pairwise Pearson r 0.80-0.92, with 9 of 21 pairs at r >= 0.85). Bull sits partially outside. Bear is structurally independent but its Brier is 0.2810, above the 0.25 chance baseline. Bear actively degrades council accuracy if equally weighted.

Phase 1 was a prompt-engineering rewrite of the shared preamble, gated behind a feature flag, with four pre-registered acceptance criteria that all had to pass simultaneously:

Cluster-pair r >= 0.85 fraction drops below 0.50

Per-agent YES/NO drift > 0.10 on at least 2 agents

Bear Brier moves toward 0.22 with independence preserved

Council Brier holds or improves from 0.2176

At n=78 resolved v2 verdicts: 0 of 4 passing. Cluster fraction pinned at 71% (worse). Max per-agent drift ~0.025 (no movement). Bear Brier drifted away to 0.3073. Council Brier regressed slightly to 0.2182.

The regression is stable, not a small-sample artifact. The plan is one more resolution batch to cross the formal n >= 100 trigger, then flip the flag off and write a retrospective on why the v2 preamble concentrated rather than diversified the cluster.

The discipline that mattered was not retuning the criteria. The experiment failed against its own pre-registered acceptance test, and the response is a revert and retrospective, not moved goalposts.

DuckDB over Postgres. An external proposal pushed Postgres + JSONB + Airflow + Redis. The actual problem was queryability (every metric required a Python script), not concurrency. DuckDB reads read_json_auto('history.jsonl') natively. Half a day of wire-up, zero ops cost. Postgres would have been a week plus ongoing ops for nothing the project needed.

Monolith-first, containers later. Phase 3 (deployment portability) is deliberately deferred because Phase 0 closed with verdict "inconclusive, not positive." No point spending a week on Docker until the edge validates.

Auto-trading deliberately blocked. py-clob-client (Polymarket trading SDK) is declared in dependencies but not imported anywhere. Tier A auto-BUY is gated on Phase 1's acceptance test. Tier B continuous listener is gated on a max-favorable-excursion study that hasn't run. Shipping auto-execution before the signal validates is named explicitly in the scaling assessment as "the fastest path from advisor to auto-loss generator."

Single LLM in production. Claude Haiku 4.5 in prod, Llama 3.1 8B as a local dev fallback. A multi-model MoE proposal was rejected in writing: "correlated hallucination is at the prompt level per the Option B shadow analysis; model-mixing multiplies failure modes without addressing the root cause."

Pydantic v2 as the contract layer. Every inter-module boundary is a model, never a raw dict. The rule is enforced in the project's context file: "never pass raw dicts between modules." This carries through to nested types, field validators, model validators, and backward-compatible parsing for schema evolution across 47 sessions.

Single LLM entrypoint. One chat() function in llm/client.py. Every other module is forbidden from importing anthropic or httpx for LLM calls. SDK auto-retry disabled; the module owns the retry policy (exponential backoff over 5xx including 529 OverloadedError, 429, connection errors, timeouts). Prompt caching uses explicit cache_control: ephemeral blocks with debug logging of cache-read token counts.

Session protocol for solo continuity. A context file (capped at 40k characters, enforced) plus a per-session handoff prompt reproduces the full mental model in ~10 minutes. When the context file grows, content extracts to docs. A per-session changelog and per-phase state documents complete the protocol. This is the same problem a team solves with standups and shared context; here it had to be built as infrastructure.