Every client that wants a local model — the chat interface, the coding assistant, the document classifier, the voice pipeline — talks to one address instead of reaching into the host running the models directly. LiteLLM is what sits at that address: an OpenAI-compatible proxy in front of Ollama, so nothing downstream has to know or care which model is actually answering.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
What it replaced
Nothing that was running. The alternative on the table was the absence of a gateway at all: every client pointed straight at Ollama, each one carrying its own base URL, its own idea of which model name to ask for, and no shared point to add a key, a limit, or a log. That was the actual starting state for the first client wired up, and it is the shape this component exists to avoid repeating for the second, third, and thirtieth.
Why this one
It speaks the API format every client already speaks. Open WebUI, Continue.dev, and everything else built against “OpenAI-compatible” work against LiteLLM with no client-side changes, which is the entire value of picking a proxy over writing one: the integration cost is a base URL, not a new SDK per tool.
The design on paper was considerably more ambitious than what actually shipped, and the gap between the two is the more interesting story.
The original plan routed by complexity: a fast small model for simple queries, a mid-size model for ordinary chat, a larger one for anything long or multi-step, and a cloud model as a fallback when the local tiers weren’t enough. None of that existed yet when the gateway actually got built, and the human scoped it down on the spot: “yes, add the queue. No anthropic access yet, we’ll just connect it to Ollama.” No cloud account, so no cloud tier — not deferred, just not built. The gateway that shipped routes one model to one backend.
That correction mattered because the runbook it was built from had drifted from reality in more places than the cloud tier. Checking what was actually running against what the design assumed found five models planned and one installed, a six-tier router sized for hardware that didn’t exist yet, and two ports the design called for with nothing listening on either. The gateway that got built was sized to what was measured on the host that day, not to what an earlier document had assumed.
How it is actually used here
Live behind the reverse proxy, reachable from the LAN and the mesh network but not exposed publicly — clients that need AI capability from outside the lab go through a separate public-facing interface rather than this API directly. Every service that talks to it gets its own scoped key rather than sharing one credential, so a key compromised in one client doesn’t carry privileges for every other one.
The shape of the configuration
client (chat UI, coding assistant, document classifier, ...)
-> LiteLLM, per-client scoped key
-> one model, one backend, loopback only
No complexity router in front of it, no fallback tier, no provider config pointing outward. The proxy’s own inventory of upstream providers is checked directly against the running process, not assumed from the config file on disk — the two have disagreed before.
What it cost
The scoping only works if the proxy is trusted to pass credentials through untouched, and an early version of the plan did not. The complexity router this build chose not to ship had a defect of its own before it was ever removed from scope: it re-authenticated every request internally with its own master key rather than forwarding whatever key the actual caller presented, which meant a deliberately restricted key would have been silently upgraded the moment a request passed through it. That component was never built here, so the bug never ran in production — but it is why per-caller key scoping only ships alongside a proxy path that has been checked to actually preserve the caller’s own credential end to end, not assumed to.
The other honest cost is narrower capability than the name on the runbook suggests. One model means no fallback if that model is a bad fit for a given request, and no cloud escape hatch for anything genuinely beyond local capacity. That is a scope decision, not a limitation nobody noticed.
What would change the decision
A second model backend, or a cloud account, are both changes to the input this was scoped against, not to the tool. LiteLLM already supports routing across many backends and providers; nothing here is a ceiling the software imposes. The day there’s more than one model worth routing between, the complexity router goes back on the table — and this time gets checked against a real caller-key test before it ships, rather than after.
If a genuinely different API shape were needed — something LiteLLM’s OpenAI-compatible translation can’t represent — that would be the point to look elsewhere. Nothing in the record points that direction today.
About LiteLLM
LiteLLM is an open source proxy and SDK that translates a wide range of model provider APIs into a single OpenAI-compatible interface, built and maintained by BerriAI. It is released under the MIT license and is commonly run as a lightweight gateway in front of self-hosted or cloud model backends.
- Website — litellm.ai
- Documentation — docs.litellm.ai
- Source — github.com/BerriAI/litellm
- Community — github.com/BerriAI/litellm/discussions