Model-Agnostic Doesn't Mean Models Are Interchangeable

Sooner or later, an agent product gets asked to run on a different model. Usually it’s for an ordinary reason. One customer standardises on Anthropic. Another is happy with whatever an aggregator like OpenRouter routes to. A third has confidentiality obligations and wants certain material to stay on a model running on their own hardware, behind vLLM or Ollama.

“Model-agnostic” is easy to claim and harder to mean. An adapter that converts one request format to another gets you about a fifth of the way. The rest is deciding what the system does when a model can’t do what the agent needs — and making sure it never pretends otherwise.

One wire format is the easy part

Pick one request shape and make every provider speak it. The OpenAI-compatible Chat Completions format is the obvious choice: most aggregators and self-hosted servers already expose it, and the major agent SDKs can be pointed at any compatible base URL. Native APIs can sit behind a thin translation layer.

There will be gaps. Some compatible endpoints can’t stream. A small adapter that turns a complete response into synthetic stream events keeps the rest of the harness from caring. Some return tool calls in slightly different shapes. Normalise them at the edge, once.

That’s the plumbing. It’s necessary, and it isn’t the hard part.

Each model should declare what it can do

The hard part is that models genuinely differ, and the harness has to know how. I’d give every configured model an explicit capability record rather than inferring capability from its name:

class Capabilities(BaseModel):
    context_window: int = 32_768
    tools: bool
    streaming: bool
    vision: bool

Then put a gate at dispatch — the last point before a request leaves the process — that checks every request against that record and the workspace’s policy:

def check(request, model, policy):
    if request.tools and not model.caps.tools:
        refuse("This model doesn't support tools.")
    if estimate(request) + request.max_output > model.caps.context_window:
        refuse("This request is larger than the model's context window.")
    if request.confidential and not model.endpoint.self_hosted:
        refuse("Confidential material can't be sent to this endpoint.")
    if model.id not in policy.allowed_models:
        refuse("This model isn't enabled for this workspace.")

Add the obvious others: no images to a non-vision model, no streaming from an endpoint that can’t, and no third-party inference at all when the workspace has turned it off. If the endpoint configuration changes in the middle of a run, stop the run and ask for a new one rather than finishing on a different model than it started.

Refuse; don’t fall back

The important word in that gate is refuse. There’s a strong temptation to “fall back” instead: the self-hosted endpoint is down, so quietly use the cloud one; the model can’t call tools, so run a tool-less version of the agent and hope the answer is close enough.

For confidential material, falling back to a cloud model is a data leak with extra steps. For capabilities, a tool-less agent isn’t a degraded version of the product. It’s a different product that happens to have the same name. In both cases the honest behaviour is a clear refusal that a person can act on: switch model, wait for the endpoint, or narrow the request.

Confidentiality should follow the content

The policy question “is this run confidential?” is subtler than it looks. I’d make it depend on what the run retrieved, not what it’s called or who started it. Once confidential content enters a run, the label sticks and follows it into follow-up turns, summaries, delegated child tasks, and generated artifacts. Metadata — names, counts, an empty search result — shouldn’t trigger it.

That stickiness has a consequence that’s easy to miss. Any auxiliary model call has to pass the same gate as the main one. A context summariser that “just uses a cheaper cloud model” is a perfect side door for confidential text. (More on that in the compaction post.)

Be honest about what “self-hosted” means, too. It’s a declaration an admin makes about an endpoint, not something the harness can attest over the network. A private proxy in front of a cloud model is third-party inference, whatever its hostname says. Custom endpoints deserve their own guard rails: HTTPS for anything external; plain HTTP and private addresses only for endpoints declared self-hosted; cloud metadata and link-local addresses blocked, including when they’re hidden inside IPv6 forms; the resolved address checked before the TLS handshake; redirects not followed; and changing an endpoint’s origin should require entering the key again, so a pasted URL can’t quietly redirect saved credentials.

Counting tokens without a tokeniser

If customers can plug in arbitrary models, you can’t load a tokeniser for each one. Estimate conservatively instead — dividing bytes by three, rather than the more typical four, leaves a reasonable margin — and use the same estimate everywhere: admission, history packing, and the dispatch gate. If those three disagree, a request can pass admission and then fail at dispatch, which is a miserable bug to track down.

The estimate also forces a product conversation. A large tool catalogue plus instructions can easily take tens of thousands of tokens before the user says anything. On a model with a 32k window, the agent has to start with a reduced set of tools and load others on demand, and it will be slower and more fragile. Tell the admin that when they configure the model, rather than letting them find out from users.

The harness owns retries and quirks

Run the provider SDK with its own retries turned off and own them in the harness. Classify failures — transient transport, rate limited, quota exhausted, unknown tool, repeated tool call, tool results without matching calls, unsupported parameter — and treat each differently. Cap any server-requested delay. An SDK that faithfully honours a multi-hour Retry-After inside a job with a ten-minute timeout is a classic way to get a spinner followed by an unexplained failure.

Handle model quirks in code, not in prompts. Some models write a tool call as JSON in their reply text instead of making a real call. A narrow, well-tested recovery path for that is better than a paragraph of pleading in the system prompt. If a mode means “no tools this turn”, enforce it by wrapping the tools so they refuse. A truncated completion should be detected from the response and either continued or clearly marked incomplete, never presented as a finished answer.

Put domain rules where models can’t negotiate

The model can vary. The rules about money shouldn’t. So they don’t belong in the prompt, where every model interprets them a little differently. They belong in typed code that every model’s output passes through:

class BoundFigure(BaseModel):
    symbol: str
    scope: Literal["standalone", "consolidated"]
    period: str
    metric: Literal["revenue", "net_profit", "eps"]
    annualised: bool = False   # only allowed in hypothetical scenarios, and disclosed

Never compare standalone with consolidated figures. Only annualise quarterly earnings in an explicitly hypothetical calculation, with the assumption disclosed. A cited source must be something the run actually retrieved. A report whose coverage or freshness can’t be verified is held back with a stated reason. When you switch models, none of this changes, and that’s exactly the property you want.

Extensions follow the same principle. If users can add their own calculators, run them only in the sandbox, never import their code into the API process, attach them only when they’re enabled and have passed their own tests, and remove them from the tool list entirely when the sandbox is unhealthy, rather than letting calls fail halfway through.

Be honest about what’s been proven

Two cautions. First, testing against a synthetic OpenAI-compatible server proves the plumbing, not the model. Before claiming support for a specific self-hosted stack, run the real agent against it, with real tools, at real context sizes. Second, the same discipline applies to non-model dependencies. A keyless search endpoint that starts returning bot challenges can send an agent into retry loops just as effectively as a flaky model. Pin versions, check what each dependency actually returned, and refuse clearly when it can’t do the job.

To me, model-agnostic doesn’t mean every model gives the same answer. It means every model gets the same contract — the same refusals, the same evidence rules, the same domain checks — and when one can’t meet it, the system says so instead of degrading quietly.