A/B Testing Rigor — When "Best Practice" Optimizations Hurt

A regional behavioral health nonprofit's WordPress portal was running through a Cloudflare proxy with Rocket Loader enabled. Conventional wisdom said disabling Rocket Loader on a heavy Elementor site should reduce TBT — that's the standard advice on most WordPress performance forums. Before committing, I ran an actual A/B test on production traffic with controlled cache state.

Result: disabling Rocket Loader cratered the metrics. TBT went from 100ms to 640ms; Lighthouse Performance score dropped from 60 to 43. The intervention got rolled back the same session, and Rocket Loader stays on until upstream JS payload is reduced enough to handle the unmask.

This is part of a broader testing methodology I apply to performance work: heavy Elementor sites can swing 17–26 seconds in single-run PSI LCP between identical loads, so I always run 3–5 PSI passes, drop the worst-LCP outlier, and report the median. Single-run Lighthouse numbers are diagnostic, not decision-grade, and PSI lab scores aren't a Google ranking input — Page Experience uses CrUX field data only, so the real goal is field-level passes, not 90+ scoreboards.

Stack: PageSpeed Insights, Chrome DevTools, CrUX field data analysis, Cloudflare configuration, Lighthouse CLI, controlled A/B methodology with cache-state isolation.

Outcome: Caught a TBT regression before deploying, saved the client from a downgrade dressed up as an upgrade. Methodology now applied as standard pre-flight on every performance engagement.

Tags: Performance optimization, A/B testing, Core Web Vitals, Cloudflare, Rocket Loader, methodology, evidence-driven