Latency budgets: what 800 ms actually costs
A field note from the PocketPatient latency work
The budget
A patient interview turn has a psychological budget around 1.5 seconds before the learner starts talking over the silence. With the evaluator running after the turn, the total request path had to fit under 2 seconds end to end. That single constraint split the architecture: a cheap model for turn-taking, a stronger one for grading.
What the budget cost us
- A routed-model design with two providers instead of one.
- An eval path that runs after the turn, not inside it.
- Instrumentation on every leg: inference, database, API — the pending claims in the PocketPatient ledger are exactly these measurements.
The lesson
Latency budgets are architecture decisions wearing number costumes. Write the budget down before you choose the stack.
Latency