LLM provider failover routes an eligible model request to an alternative when the preferred route cannot serve it. It can reduce dependence on one provider, but useful recovery also depends on the task, the replacement model, and what already happened in the agent workflow.
The planning question is concrete: if this model call cannot finish, what may the system safely do next?
Fall back only to an approved alternative; otherwise route the request for review.
Distinguish retry, rerouting, and recovery
A retry repeats a request, often after a delay. Failover changes the serving route. Workflow recovery determines whether the broader task can continue safely. Switching regions within one service and switching between providers are different designs; neither automatically restores agent state.
Set a bounded policy for waiting and attempting alternatives. Repeated attempts can add latency and cost, so define when the application should stop and report an incomplete result. These are design recommendations, not a description of AgentShelf's retry implementation.
Check the alternative before it is needed
An alternative endpoint must be approved for the work it receives. Check these questions with the system owner:
Scroll horizontally to see all columns.
| Check | Why it matters |
|---|---|
| Input compatibility | The replacement must accept the required context and input types. |
| Output and tool compatibility | Validate required fields and tool arguments rather than assuming matching behavior. |
| Quality | Run the task's evaluation cases on each approved candidate. |
| Data destination | Confirm where requests may be processed before widening a route. |
| Cost and capacity | An alternative has its own pricing and limits. |
| Operational visibility | Record which route was attempted and whether a usable result reached the application. |
A model that can answer ordinary questions may still be unsuitable for a document extraction step. Keep fallback eligibility tied to the task.
Example: recovering a draft without repeating an action
A fictional service company uses an agent to draft a follow-up after an inspection. The inspection record is read-only during drafting. If the preferred model is unavailable before producing a usable draft, the team permits an evaluated alternative to attempt that drafting step.
Sending the follow-up is a separate action requiring approval. If a later failure occurs after a send attempt, the application checks the delivery record before offering another attempt. Changing models is not evidence that the earlier send failed.
For an interrupted response, the interface should distinguish a partial draft from a finished result. Do not append a new model's output to partial text as if it were one coherent completion unless that behavior has been designed and verified.
Test failure states alongside ordinary requests
Use a controlled pilot to exercise:
- Preferred route unavailable before output begins.
- Alternative unable to satisfy the required output format.
- No approved alternative available.
- Response interrupted after partial text.
- Tool action with an uncertain completion state.
For each case, specify the visible user state, operator evidence, and permitted next action. A clear “not completed; review required” result can be the correct outcome. Track task completion separately from successful model calls, and review agent observability when deciding what evidence operators need.
In AgentShelf
AgentShelf's published architecture describes automatic failover across 7+ providers. The infrastructure page explains configured model access; availability depends on configuration, credentials, region, plan, and catalog. That positioning does not establish particular retry counts, uninterrupted streaming, automatic tool replay, or an availability SLA. Verify the supported path for your workflow before relying on a recovery behavior.
For the surrounding architecture, read what LLM infrastructure includes and what a gateway handles.