Where your LLM tokens actually go
The user's question is the smallest thing you send.
Most conversations about LLM cost start with the price list. Dollars per million tokens, this model against that one, which provider dropped prices last quarter. It's a reasonable place to start....
What does your AI feature do when the model is down?
We plan for the database failing. Almost nobody plans for the model.
For a long stretch of my career, the most useful question in a design review was also the most annoying one. What happens when this is down?
Not if. When. The...
There Is No Parameterised Prompt
We fixed injection attacks twenty years ago by separating the instruction from the data. That trick doesn't exist here.
If you built anything database-backed in the 2000s, you learned about SQL
injection the hard way, and then you learned the fix.
...
I Spent Years Shaving Milliseconds. Then We Added an LLM.
Most of the wait isn't the model. It's everything you built around it.
For a long stretch of my career, latency was the job.
On the VoIP platform, a few hundred milliseconds of delay in the signalling path
was the difference...
You Changed the Prompt. What Else Did It Break?
Almost every team shipping AI features changes prompts on instinct. Almost none
of them could tell you what the last change made worse.
Here's how a prompt usually gets changed in production.
Someone on the support team flags a bad...
