What Model Portability Actually Requires

Every company selling an AI gateway will tell you that an abstraction layer solves vendor lock-in. The claim is not wrong so much as incomplete, and the missing part is where the money goes.

An abstraction layer moves the provider's API behind a function call. That is the easy coupling to remove, and it is the only one most architectures actually remove. The couplings that decide whether you can genuinely change providers accumulate somewhere else entirely, and they accumulate quietly, over months, in code that nobody thinks of as provider-specific.

This is written from the position of having built the abstraction for our own product and then having to live inside it, which is a different vantage point from selling one.

Lock-in is not the API call

Ask what would actually have to change if you swapped providers tomorrow. In most systems the answer is not "the client library". It is five other things.

Call sites that name a model. A constant like MODEL = "some-provider-model" inside a feature is the canonical form, but the subtler version is a feature that branches: this provider for reasoning, that one for extraction. That is a routing decision made in the wrong layer, and it multiplies with every feature.
Prompts tuned to one provider's conventions. System-prompt placement, message shapes, how strictly the model follows a format instruction. Tune long enough and the prompt is a provider-specific artifact even though it contains no provider's name.
Tool and function-call formats. The wire shape of a tool definition and the shape of the response differ per provider. If your tool layer speaks one provider's dialect, your agent layer is not portable regardless of what sits underneath it.
Schemas, cache keys, and test assertions that assume a stable answer. This is the one that bites hardest and shows up last. A column named for a provider, a cache key that omits the model id, a test asserting an exact string: each encodes the assumption that the answer will not change. It will.
The contract you publish to your own clients. If your API, your mobile app, or your database schema cannot represent more than one provider, then the abstraction stops at your own boundary and the lock-in has simply moved outward.

None of those are fixed by putting a gateway in front of the call. They are fixed by a rule about what a feature is allowed to know.

The rule that does the work

A feature expresses a capability and its constraints. It never names a provider.

Concretely, feature code says something closer to: I need extraction over a document of this classification, in this language, for a user in this jurisdiction, with structured output, under this latency ceiling. It does not say which model. Something else answers that, and the answer is allowed to differ between users, countries, languages, tasks, data classifications, environments, individual requests, and over time.

That last clause is the test. If any code, schema, cache key, test assertion, or user-facing string in your system would break when the answer changes, you have found a coupling, and it does not matter that a gateway sits in the middle.

The layering that follows is unremarkable once the rule is in place:

Business logic, which knows capabilities and constraints.
An abstraction seam, which is the only thing business logic calls.
A selection layer, which answers "which model performs this task".
One adapter per provider, which is the only place a provider SDK may be imported.

The adapter containment rule is the enforceable part, and it is worth writing into review rather than into a wiki. If a provider SDK import can appear anywhere, it eventually appears everywhere, and the seam becomes decorative.

The failure mode nobody advertises: silent fallback

Here is the design error we think matters most, because it is presented as a feature.

A gateway is configured with several providers. One is missing a credential, or its adapter was never finished, or it is rate-limited. The gateway quietly serves the request from a different provider and returns a normal-looking answer.

That system now lies about where the data went. In a regulated context that is not a degraded experience, it is an incident: data with a classification that permitted one destination was sent to another, and nothing in the response says so. The audit record, if there is one, records a call that succeeded.

The correct behaviour is duller. Availability is explicit: a provider is available only if its credential is present and its adapter is implemented, and an unavailable provider reports itself unavailable rather than resolving to a neighbour. A fallback is legitimate only when it is a declared, policy-approved alternative that satisfies the same privacy, jurisdiction, and task constraints as the primary, and when the fallback is logged as a fallback and surfaced wherever it changes what the user gets. Failing the request is the right outcome when no authorized alternative exists.

This is also the distinction between resilience and substitution, and vendors blur it constantly. Resilience is planned. Substitution is what happens when nobody planned.

Provider resolution is not a policy engine

A note on honesty, because it is the part of this we would want a buyer to hold us to.

There is a large gap between resolving a provider and evaluating a policy. Resolution is a precedence order: an explicit pin wins, then a per-tenant or per-country rule, then a configured default. That is straightforward and most systems that claim intelligent routing have exactly this.

A policy engine evaluates the full constraint set, records which policy applied, and can explain afterwards why a given request went where it went. That is a different piece of software, and in our own platform it is a design target rather than a shipped component. We say so because a buyer in a regulated institution will eventually ask "why did this request go there", and a system that cannot answer should not have implied that it could.

If a vendor tells you their gateway does constraint-based routing, the follow-up question is: show me where the selection reason is recorded. If the answer is that provider and model are logged but the reason is not, you have resolution, not policy. That is fine. It should just be called what it is.

What portability costs

Portability is not free, and pretending otherwise is how architectures get sold and then abandoned.

You give up the edges. Provider-specific features that have no neutral equivalent either live in the adapter as an optional capability or do not get used. Some of them are genuinely useful.
You own an adapter per provider. Each is small. Each still needs contract tests, and each drifts when the provider ships a change.
Prompts get authored twice-ish. Neutral instruction at the top, provider adaptation in the adapter. The rule that keeps this sane is that a business rule lives in exactly one place: a rule duplicated across three provider-specific prompts drifts in three directions and cannot be reviewed.
Quality comparisons get harder. When the provider can change under you, "it got worse" needs an evaluation set to be a statement rather than a feeling.

What you buy for that is narrow and valuable: a provider change becomes a configuration change. Not a migration project, not a rewrite, not a renegotiation conducted from a position of having no alternative.

How we build it

For completeness, and because it is checkable rather than aspirational. The platform this company builds routes inference through a declarative provider registry with one adapter per provider. Several providers sit behind that one interface, including a self-hosted local deployment and an enterprise-hosted endpoint serving the same open-weight model family, which is what let us test the deployment-model argument rather than assert it. Providers that are declared but not implemented report themselves unavailable instead of resolving elsewhere. Usage and cost are metered per provider and model, and every call is written to an audit trail. The default test suite runs with no third-party credentials and no network access to any provider, which is the discipline that keeps the adapters honest.

The same reasoning applies one layer down, to the cloud rather than the model: our architecture standard requires business logic to depend on provider interfaces rather than on any cloud vendor's SDK, and confines those SDKs to infrastructure adapters. The principle is identical. The thing you cannot swap is the thing that eventually sets your terms.

Five questions for a vendor

If I swap providers, what changes besides configuration? Ask for the file list.
Where can a provider SDK be imported, and what enforces that?
What happens when a configured provider is unavailable? Listen for whether the answer contains the word "automatically".
Is the selection reason recorded, or only the selection?
Can your default test suite run with no provider credentials at all?

None of those require you to evaluate a model. All of them are answerable in a meeting, and the answers correlate well with whether the abstraction is real.

What model portability actually requires