LLM proxying

LLM proxying is a process where agents send requests to an intermediate server rather than directly to an LLM provider. The intermediate server analyzes LLM requests and responses to provide additional features to agents and applications.

CoralOS has two distinct LLM proxies, for Coral Cloud and Coral Server. With your own Coral Server, you can configure provider keys directly or use the Coral Cloud LLM proxy as a convenience option.

Coral Server

The Coral server also offers an LLM proxy to the agents that it runs. Passing LLM traffic through the Coral server provides a number of benefits, such as:

• Automatic agent telemetry collection
• Eliminating the agent’s need to hold LLM provider keys
• Agent cost control
• Agent model and provider substitution
• LLM provider fallbacks
• LLM provider key cycling

Read more about how to configure agents to use the Coral server LLM proxy here and how to configure LLM proxies on the Coral server here.

Your own LLM provider keys

Configure providers such as OpenAI, Anthropic, or DeepSeek directly in your server’s config.toml. Coral Server holds the provider keys and routes each agent’s model requests to a matching provider.

[[llm-proxy.providers]]
name = "OpenAI"
format.type = "OpenAI"
api_key = "YOUR_OPENAI_API_KEY"
base_url = "https://api.openai.com"
models = ["gpt-4.1-mini", "gpt-5-mini"]

Load the file using CONFIG_FILE_PATH. See provider configuration for an Anthropic example and base URL composition when using a different SDK or compatible endpoint.

Coral Cloud

The Coral Cloud LLM proxy is an optional upstream provider for your self-hosted Coral Server. A Cloud API key automatically configures supported providers and models, with model usage charged through Coral Cloud.

[cloud]
api_key = "my Coral Cloud API key"

Your server, console, and agent sessions still run on your own infrastructure. Using this LLM proxy does not require using the Cloud-hosted server or console. See Cloud configuration.