Case Studies
Whole systems, end to end: enterprise RAG, coding agents, support automation, fraud, voice AI and more.
- lessons
- 20
- steps
- 159
- time
- 267 min
Enterprise RAG System
Design cited, permission-aware document Q&A over 500K files for 5,000 employees, one bottleneck at a time.
- Turn a vague ask into latency, accuracy, freshness and security targets
- Why vector-only retrieval misses, and how hybrid search plus reranking fixes it
- Enforce permissions inside retrieval, not after it
- Meet the 1-hour freshness and 500-user targets with streaming ingestion, replicas and caches
Conversational Support Agent
Automate 60% of 500K monthly B2B support tickets with intent routing, model routing and confidence-gated escalation.
- Translate support pain into latency, accuracy, escalation and CSAT targets
- Route by intent so each turn pulls only the sources and tools it needs
- Use model routing plus effort, and budget latency per path
- Gate every reply on safety and confidence, with context-rich escalation
Financial Analysis with Ensembles
Generate equity research reports in 25 minutes at 0.4% factual error by stacking self-consistency, mixture of agents, debate and a judge panel.
- Why zero-tolerance financial data rules out a single model pass
- Extract numbers with unanimous k5 self-consistency
- Verify every claim with a cross-family debate and consensus
- Route reports with quality gates: auto-publish vs human review
AI Code Assistant
Design an IDE assistant that completes code in under 200ms and runs multi-file refactors — by splitting it into a fast tier and a quality tier.
- Why one model can't serve both keystroke completions and refactors
- Split the system into a fast FIM tier and a quality agent tier
- Fill a 4K-token context budget by priority
- Verify generated code fail-fast before a developer sees it
Content Moderation at Scale
Moderate 10M posts a day with a four-tier cascade — filters, classifiers, an LLM and humans — so each tier only sees what the cheaper one couldn't decide.
- Why neither all-human nor all-LLM review scales
- Build a cascade where each tier escalates only what it can't decide
- Prioritise human review by severity, reach and confidence
- Defend the cheap tiers against evasion and read provenance
Real-Time AI Search
Answer “what's the sentiment on Tesla in the last hour?” with sourced facts under 3 seconds — streaming ingestion, a dual index and a small, fast synthesiser.
- Why batch-indexed, vector-only RAG fails live market questions
- Stream into two indexes with Kafka consumer groups
- Combine semantic and keyword search with RRF and a reranker
- Enforce freshness as an index filter and survive news spikes
Autonomous Coding Agent
Design an agent that turns “add rate limiting to every endpoint” into a tested, human-approved pull request for under $0.50.
- Why planning and coding run as separate agents on different models
- Tiered retrieval: summaries → symbols → full files for a 1,000-file repo
- A self-correction loop with hard limits, inside a sandbox
- Why cache hit rate, not token price, sets the cost per task
Multi-Tenant AI SaaS
Serve 500 competing enterprises from shared infrastructure with zero cross-tenant leakage, using tiered isolation and defense in depth.
- Why a single tenant filter is a single point of failure
- Tiered isolation: namespace, dedicated collection, dedicated pod
- The seven layers of defense in depth, from JWT to provider account
- Tag tenant_id at upload and design for deletion and audit
Customer Support Automation
Auto-resolve 60% of 2 million tickets a month with tool-grounded answers, and hand the risky ones to humans with full context.
- Why one chatbot answering everything makes false promises
- Three-tier routing on a multi-signal confidence score
- Tool-grounded answers plus a safety check before send
- Human takeover with a handoff package and logged reasons
Document Intelligence Pipeline
Extract parties, dates, obligations and termination terms from 50,000 contracts a month at 95%+ accuracy for under $0.50 each.
- Why OCR plus one big prompt fails on real contracts
- Route pages by difficulty: parser, parsing VLM, frontier VLM
- Section detection and parallel, specialized extractors
- Cross-field validation and field-level human review
Recommendation Engine with Explanations
Rank 50M users' movies with classic ML in under 200 ms, and spend the LLM only on cached, fact-checked explanations.
- Why an LLM should explain recommendations, not rank them
- The classic retrieve-then-rerank pipeline behind real-time recs
- Why the p95 budget turns into a cache-hit-rate target
- Cold start, hallucinated facts and privacy-safe caching
Regulatory Compliance Automation
Pre-screen pharma marketing against FDA rules claim by claim, with citations, recall-first thresholds and a full audit trail.
- Why claims, not whole documents, are the unit of review
- Why regulations belong in RAG, backed by a precedent index
- How recall-first thresholds turn AI into triage for lawyers
- What an audit-ready decision record contains
Voice AI Scribe for Healthcare
Turn a nurse's spoken encounter into a grounded SOAP note in the EHR, with a live transcript and the LLM kept off the real-time path.
- Why silence is a hazard for speech models
- Why HIPAA means a BAA, not necessarily on-prem ASR
- Splitting the latency budget: live transcript vs note after the visit
- Grounding, human review and the draft-to-signed diff
Real-Time Fraud Detection
Score 10M card transactions a day in under 100 ms with ML, decide with explainable rules, and explain declines with an LLM off the hot path.
- Why the 100 ms budget forces pre-computed features
- ML for scoring, rules for explainable decisions, and the gray zone
- Using an LLM to explain declines without touching the hot path
- Keeping up with fraud that changes every week
Enterprise Knowledge Management
A knowledge assistant for 10,000 consultants that answers only from documents each person may see, prefers current guidance, and admits when it knows nothing.
- Turn a vague ask into concrete constraints: scale, ACLs, citations, staleness, gaps
- Enforce permissions at retrieval, before the LLM sees anything
- Rank current guidance above old with a recency decay
- Detect knowledge gaps and conflicts instead of hallucinating
Production Computer-Use Agent
A computer-use agent closes 14,000 expense reports a week across three API-less tools, made safe by per-task microVMs, a risk-tiered action gate and a tamper-evident audit trail.
- Read computer-use benchmarks correctly: binary completion, not partial credit
- Gate every proposed action by risk: execute, confirm inline, or review out of flow
- Isolate each task in its own microVM with an audience-bound identity
- Defend against indirect prompt injection at the read layer
Multi-Tenant Fine-Tuning Platform
Serve 280 customers their own fine-tuned model from one shared 70B base plus per-tenant LoRA adapters, with an eval gate per tenant and no cross-tenant bleed.
- Why one full fine-tune per tenant breaks the unit economics
- Size the LoRA vs full fine-tuning trade-off: cost, artifact size, accuracy gap
- Budget GPU memory for adapters and stop noisy neighbors
- Gate every retrain on the tenant's own golden set and isolate tenants end to end
Eval-Gated CI/CD for an AI Product
Make every PR earn its merge button: golden sets, calibrated LLM judges with confidence intervals, and per-axis gates under a $40-per-PR budget.
- Why post-merge regression hunting failed and what the gate must satisfy
- Bound eval cost with stratified sampling, caching and nightly full runs
- Correct a biased LLM judge into an estimate with a confidence interval
- Gate per failure axis against main, not on one composite score
Customer-Specific Distillation Pipeline
Cut a $50K/month frontier bill to about $9.3K by distilling an 8B student on six months of production traces — with a payback under three months.
- Decide whether a workload is worth distilling at all
- Why real production traces and human-checked labels beat synthetic data
- Gate a student with a golden set, shadow traffic and a canary ramp
- Do the payback math: $96K upfront against ~$41K/month saved
Enterprise MCP Knowledge Agent
An agent that answers across Snowflake, Confluence, Jira and Slack through MCP — with audience-bound tokens, sandboxed STDIO, trust-tagged results and an audit chain.
- Why query sources live over MCP instead of crawling one vector index
- Bind every token to one server's audience and keep third-party tokens in a broker
- Layer defenses against indirect prompt injection: filter, trust-tag, gate
- Run it as a system of record: audit chain, per-identity limits, cost per query
Guide chapters · Case Studies
- Case Study: Enterprise RAG System8 min
- Case Study: Customer Support Conversational Agent7 min
- Case Study: Financial Analysis with Ensemble Verification5 min
- Case Study: AI Code Assistant7 min
- Case Study: Content Moderation at Scale6 min
- Case Study: Real-Time AI Search Engine4 min
- Case Study: Autonomous Coding Agent10 min
- Case Study: Multi-Tenant AI SaaS Platform7 min
- Case Study: AI-Powered Customer Support5 min
- Case Study: Document Intelligence Pipeline5 min
- Case Study: AI-Powered Recommendation Engine4 min
- Case Study: Regulatory Compliance Automation3 min
- Case Study: Voice AI Assistant for Healthcare7 min
- Case Study: Real-Time Fraud Detection3 min
- Case Study: Enterprise Knowledge Management3 min
- Case Study: Production Computer-Use Agent15 min
- Case Study: Multi-Tenant Fine-Tuning Platform14 min
- Case Study: Eval-Gated CI/CD for an AI Product13 min
- Case Study: Customer-Specific Distillation Pipeline12 min
- Case Study: Enterprise MCP Knowledge Agent16 min