LLM infrastructure is the set of services and operating controls that lets an application use large language models reliably. For an AI agent, the model endpoint is one dependency within a larger system: the agent also needs an execution environment, permitted information, tools, and a way to report outcomes.
A useful starting point is to trace one task from the user to a checked result. This reveals who operates each dependency and where a failure could prevent useful work.
The model endpoint is one dependency; the runtime, data, and tools complete the system.
Separate model serving from agent execution
Model serving runs the model and returns generated output. Agent execution runs the application logic that assembles context, calls the model, invokes permitted tools, and decides what to do next. These can run in different environments with different owners.
A gateway can provide a common point of access to configured model endpoints. It does not by itself define the agent's job or decide which business actions a person must approve. For the broader system, see AI agent architecture.
Map the services and their owners
Use this planning table before comparing infrastructure offers. The entries describe responsibilities to investigate; they are not a universal feature checklist.
Scroll horizontally to see all columns.
| Layer | Question for the owner |
|---|---|
| Model access | Which models are available, with which credentials and regional restrictions? |
| Agent runtime | Who deploys the application, updates dependencies, and responds when execution fails? |
| Knowledge and tools | Which information can the agent read, and which actions may it request? |
| State and outputs | Where are conversation context, uploaded files, and finished results kept? |
| Reliability | What can recover automatically, and when must the user or operator intervene? |
| Usage and cost | Which requests are recorded, which budgets apply, and who investigates unexpected spending? |
Keep the model inventory separate from an agent's permission list. Adding a model should not silently expand access to customer records or allow a new tool action.
Example: a service request assistant
In this fictional pilot, a facilities team wants an assistant to classify incoming maintenance requests and prepare a suggested response. The model interprets free text; the application checks permitted categories and displays a draft. An operator approves any change to the maintenance record.
The team writes a small infrastructure brief:
- Input: Request text and the approved category guide; exclude unrelated personnel records.
- Execution: A named application owner maintains the runtime and its tool connection.
- Model choice: Test configured candidates against ambiguous requests and missing information.
- Output: A category recommendation, supporting evidence, and a draft response.
- Failure behavior: Show a pending review state when model access or the category guide is unavailable.
- Operating review: Inspect failed requests and spending alongside accepted recommendations.
Success means the operator can check the recommendation and understand a failure. A response from the model alone does not establish that the workflow succeeded.
Plan a pilot around observable outcomes
Choose representative cases before tuning infrastructure: a clear request, a request spanning two categories, a missing location, and an unavailable dependency. Record completion time, whether the draft is usable, and the cost of the whole task. Keep assumptions visible when usage records cover only part of the route.
Then decide whether your next gap is provider failover, deployment ownership, or the difference between a gateway and an agent platform.
In AgentShelf
The infrastructure overview describes managed services and configured model access. Provider availability depends on configuration, credentials, region, plan, and catalog. Review the developer runtime for supported execution options and governance and cost controls for monetary budgets and their coverage boundaries.