LLM Infrastructure & FinOpsintermediate13 min · 9 steps
FinOps & Token Economics
Model where every token's cost comes from, capture the cheap levers first, and run AI spend like a cost-of-goods business.
heavy worktool calldatarequest
ProblemStep 1 of 9
The biggest saving is sitting unused
Datadog's 2026 State of AI Engineering: system prompts are about 69% of input tokens, yet only about 28% of calls use prompt caching. The largest, lowest-effort cost lever in most stacks is switched off.
And unlike classic SaaS, every AI request has a variable token cost. An AI product runs like a cost-of-goods business, so margin is now an engineering concern.
What changed
- +Input-token mix
- +Share of calls that cache
Lesson 4 of 4 in LLM Infrastructure & FinOps