Day 13: Self-Hosted AI Ceilings
We tried to build our own AI inference server. It didn't work the way we hoped. Here's what we learned.
Every startup building AI features hits the same wall eventually.
The product works. Users love it. And then the cloud AI bill arrives.
Per-token pricing sounds harmless at small scale. At real scale, thousands of requests a day, embedded in product features that run continuously, it becomes a variable cost that grows exactly as fast as your traction does.
The better the product performs, the more it costs to run.
So we tried to solve it ourselves.
We spun up a KVM virtual machine. 20-core Xeon, 128GB RAM. Installed Ollama, Open WebUI, exposed it via Cloudflare tunnel. CPU-only inference, no GPU.
It worked for light queries. Simple summarization, short classification tasks, quick lookups.
Then we pushed it harder.
Complex document analysis. Multi-step reasoning. Anything that required the model to actually think rather than pattern-match.
The server couldn't handle it. Response times jumped to 40-60 seconds. The experience broke. For a product feature, that's the same as not working at all.
So we made a different call.
We stayed on cloud APIs for anything latency-sensitive and complex. We kept the local server for lightweight, high-frequency tasks where speed requirements are low and volume is high.
Hybrid. Not pure cloud. Not pure local.
The lesson wasn't “self-hosted AI doesn't work.” The lesson was more precise than that:
CPU inference has a real ceiling. Below that ceiling it's a genuinely good economic decision. Above it, you're fighting physics, and physics wins.
Know your ceiling before you architect around an assumption.
The broader principle still holds:
The companies that will have durable margins in the AI era are the ones that understand their inference costs deeply, not the ones blindly routing everything through the same three providers, and not the ones overcorrecting into infrastructure they can't sustain.
The right answer is almost never all or nothing. It's knowing exactly where each approach breaks down.
Day 13 of my 30-day build-in-public series, sharing what I'm actually building at ViberNet and OpenLoop every day. Including the things that don't work.
Show your thinking. Especially when it's uncomfortable.