Configuration Reference¶

Complete reference for all Olla configuration options.

Default Configuration

server:
  host: "localhost"
  port: 40114

proxy:
  engine: "olla"
  load_balancer: "least-connections"

discovery:
  model_discovery:
    enabled: true
    interval: 5m

logging:
  level: "info"
  format: "json"

Minimal Setup: Olla starts with sensible defaults - just run olla and it works!

Environment Variables: A curated set of settings support OLLA_ env var overrides (e.g., OLLA_SERVER_PORT=8080). See Environment Variables for the full list.

Configuration Structure¶

server:         # HTTP server configuration
proxy:          # Proxy engine settings
discovery:      # Endpoint discovery
model_registry: # Model management
translators:    # API translation (e.g., Anthropic ↔ OpenAI)
logging:        # Logging configuration
engineering:    # Debug features

Server Configuration¶

HTTP server and security settings.

Basic Settings¶

Example:

server:
  host: "0.0.0.0"
  port: 40114
  request_logging: true

Timeouts¶

Example:

server:
  read_timeout: 30s
  read_header_timeout: 10s
  write_timeout: 0s      # Required for streaming
  idle_timeout: 120s
  shutdown_timeout: 30s

Request Limits¶

Example:

server:
  request_limits:
    max_body_size: 52428800     # 50MB (typical production value)
    max_header_size: 524288     # 512KB

Rate Limits¶

Example:

server:
  rate_limits:
    global_requests_per_minute: 10000
    per_ip_requests_per_minute: 100
    health_requests_per_minute: 5000
    burst_size: 50
    cleanup_interval: 5m
    trust_proxy_headers: true
    trusted_proxy_cidrs:
      - "10.0.0.0/8"
      - "172.16.0.0/12"

CORS¶

Cross-Origin Resource Sharing settings. Only relevant when browser clients (OpenWebUI, custom dashboards) connect directly to Olla. Disabled by default; non-browser clients (curl, SDKs, coding agents) are unaffected regardless of this setting.

Example:

server:
  cors:
    enabled: true
    allowed_origins:
      - "http://localhost:3000"
    allow_credentials: true
    max_age: 600

Proxy Configuration¶

Proxy engine and request handling settings.

Basic Settings¶

Example:

proxy:
  engine: "olla"
  profile: "auto"
  load_balancer: "least-connections"

Connection Settings¶

Example:

proxy:
  connection_timeout: 45s
  connection_keep_alive: 30s
  response_timeout: 0s    # Disable for streaming
  read_timeout: 0s
  response_header_timeout: 180s  # allow slow on-demand model loads
  tls_handshake_timeout: 10s

Retry Behaviour¶

As of v0.0.16, the retry mechanism is automatic and built-in for connection failures. When a connection error occurs (e.g., connection refused, network unreachable, timeout), Olla will automatically:

1. Mark the failed endpoint as unhealthy
2. Try the next available healthy endpoint
3. Continue until a successful connection is made or all endpoints have been tried
4. Use exponential backoff for unhealthy endpoints to prevent overwhelming them

Note: The fields max_retries and retry_backoff that may still appear in your proxy: section are deprecated struct stubs. The retry: block (with enabled, on_connection_failure, max_attempts) that appears in the shipped config.yaml as a comment template has no corresponding struct in the config schema and is silently ignored if set. Retry behaviour is automatic and built-in; none of these fields have any effect.

Streaming Settings¶

Example:

proxy:
  stream_buffer_size: 8192

Profile Filtering¶

Control which inference profiles are loaded at startup. See Filter Concepts for pattern details.

Example:

proxy:
  profile_filter:
    include:
      - "ollama"        # Include Ollama
      - "openai*"       # Include all OpenAI variants
    exclude:
      - "*test*"        # Exclude test profiles
      - "*debug*"       # Exclude debug profiles

Sticky Sessions¶

KV-cache affinity routing for multi-turn LLM conversations. See Sticky Sessions for a full explanation.

Environment Variable: OLLA_PROXY_STICKY_SESSIONS_ENABLED (only enabled is exposed as an env var)

Example:

proxy:
  sticky_sessions:
    enabled: true               # opt-in
    idle_ttl_seconds: 600       # 10-min sliding window
    max_sessions: 10000         # LRU cap
    key_sources:
      - "session_header"        # X-Olla-Session-ID header
      - "prefix_hash"           # hash of messages prefix
      - "auth_header"           # hash of Authorization header
      # - "ip"                  # client IP (unreliable behind NAT)
    prefix_hash_bytes: 512

Discovery Configuration¶

Endpoint discovery and health checking.

Discovery Type¶

Example:

discovery:
  type: "static"
  refresh_interval: 10m

Static Endpoints¶

Endpoint Authentication (auth:)¶

Attaches credentials to requests forwarded from Olla to a backend. See Endpoint Authentication for the full guide.

${VAR} interpolation works on every value field. _file fields read and trim the file contents at startup. Setting both the inline field and its _file sibling is a fatal error, as is an unresolved ${VAR} with no default.

Bearer example:

discovery:
  static:
    endpoints:
      - url: "http://gpu-server:8000"
        name: "vllm-gpu"
        type: "vllm"
        auth:
          type: bearer
          token: "${VLLM_API_KEY}"

API key with custom header:

      - url: "http://custom-gw:9000"
        name: "custom-gw"
        type: "openai-compatible"
        auth:
          type: api_key
          key: "${CUSTOM_API_KEY}"
          header: "X-Api-Key"
        headers:
          X-Tenant-ID: "team-a"

Custom Outbound Headers (headers:)¶

headers: is a free-form map of header names to values. All entries are copied verbatim onto every request forwarded to that endpoint. auth: and headers: can coexist; the auth: block always wins for its own credential header. ${VAR} interpolation applies to values.

      - url: "http://custom-llm:9000"
        name: "custom"
        type: "openai-compatible"
        headers:
          X-Tenant-ID: "acme"
          X-Request-Source: "olla"

URL Configuration¶

The health_check_url and model_url fields are optional. When not specified, Olla uses profile-specific defaults based on the endpoint type:

Profile Defaults:

Both fields support:

1. Relative paths (recommended) - joined with the endpoint base URL:

   url: "http://localhost:8080/api/"
   health_check_url: "/health"     # Becomes: http://localhost:8080/api/health
   model_url: "/v1/models"         # Becomes: http://localhost:8080/api/v1/models
   

2. Absolute URLs - used as-is for external services:

   url: "http://localhost:11434"
   health_check_url: "http://monitoring.local:9090/health"  # Different host
   model_url: "http://registry.local/models"                # Different host
   

When using relative paths, any base path prefix in the endpoint URL is automatically preserved (e.g., http://localhost:8080/api/ + /v1/models = http://localhost:8080/api/v1/models).

Endpoint Model Filtering¶

Filter models at the endpoint level during discovery. See Filter Concepts for pattern syntax.

Path Preservation¶

The preserve_path field controls how Olla handles base paths in endpoint URLs during proxying. This is particularly important for endpoints that serve multiple services or use path-based routing.

Default Behaviour (preserve_path: false) When preserve_path is false (default), Olla strips the base path from the endpoint URL before proxying:

• Endpoint URL: http://localhost:8080/api/v1
• Request to Olla: /v1/chat/completions
• Proxied to: http://localhost:8080/v1/chat/completions (base path /api/v1 is replaced)

Path Preservation (preserve_path: true) When preserve_path is true, Olla preserves the base path:

• Endpoint URL: http://localhost:8080/api/v1
• Request to Olla: /v1/chat/completions
• Proxied to: http://localhost:8080/api/v1/v1/chat/completions (base path is preserved)

When to Use Path Preservation:

• APIs deployed behind path-based routers
• Services that require specific URL structures
• Multi-service endpoints using path differentiation

Docker Model Runner's shipped profile does not require preserve_path: use pathless endpoint URLs such as http://localhost:12434, and the profile forwards DMR routes such as /engines/v1/chat/completions and /anthropic/v1/messages unchanged.

Example:

discovery:
  static:
    endpoints:
      # Minimal configuration - uses profile defaults
      - url: "http://localhost:11434"
        name: "local-ollama"
        type: "ollama"
        priority: 100
        # health_check_url: "/" (default for ollama)
        # model_url: "/api/tags" (default for ollama)

      # Custom health check URL
      - url: "http://localhost:8080"
        name: "llamacpp-server"
        type: "llamacpp"
        priority: 90
        health_check_url: "/health"
        # model_url: "/v1/models" (default for llamacpp)

      # Endpoint with base path - URLs are preserved
      - url: "http://localhost:8080/api/"
        name: "vllm-gateway"
        type: "vllm"
        priority: 80
        # health_check_url: "/health" -> http://localhost:8080/api/health
        # model_url: "/v1/models" -> http://localhost:8080/api/v1/models

      # External health check on different host
      - url: "http://localhost:11434"
        name: "monitored-ollama"
        type: "ollama"
        health_check_url: "http://monitoring.local:9090/health/ollama"
        # Absolute URL used as-is

      # Docker Model Runner - pathless base URL, profile routes handle /engines/...
      - url: "http://localhost:12434"
        name: "docker-model-runner"
        type: "docker-model-runner"

      # Endpoint with model filtering
      - url: "http://remote:11434"
        name: "remote-ollama"
        type: "ollama"
        priority: 50
        check_interval: 60s
        model_filter:
          include:
            - "llama*"          # Only Llama models
            - "mistral*"        # And Mistral models

Model Discovery¶

Example:

discovery:
  model_discovery:
    enabled: true
    interval: 10m
    timeout: 30s
    concurrent_workers: 10
    retry_attempts: 3
    retry_backoff: 1s

Model Registry Configuration¶

Model management and unification settings.

Registry Type¶

Example:

model_registry:
  type: "memory"
  enable_unifier: true
  routing_strategy:
    type: strict  # Default: only route to endpoints with the model

Model Routing Strategy¶

Controls how requests are routed when models aren't available on all endpoints:

Example configurations:

# Production - strict routing
model_registry:
  routing_strategy:
    type: strict

# Development - optimistic with fallback
model_registry:
  routing_strategy:
    type: optimistic
    options:
      fallback_behavior: compatible_only

# Dynamic environments - discovery mode
model_registry:
  routing_strategy:
    type: discovery
    options:
      discovery_refresh_on_miss: true
      discovery_timeout: 2s

Unification Settings¶

Example:

model_registry:
  unification:
    enabled: true
    stale_threshold: 12h
    cleanup_interval: 15m
    cache_ttl: 10m

Custom Unification Rules¶

Example:

model_registry:
  unification:
    custom_rules:
      - platform: "ollama"
        name_patterns:
          "llama3.*": "llama3"
          "mistral.*": "mistral"
        family_overrides:
          "llama3": "meta-llama"

Model Aliases Configuration¶

Define virtual model names that map to platform-specific model names across different backends.

Model Alias Mapping¶

Each key is the virtual model name clients will use. Each value is a list of actual model names that backends may serve the model under. When a request matches an alias, Olla resolves endpoints for all listed model names and rewrites the request body to the correct name for the selected backend.

Example:

model_aliases:
  my-llama:
    - "llama3.1:8b"                          # Ollama
    - llama-3.1-8b-instruct                  # LM Studio
    - Meta-Llama-3.1-8B-Instruct.gguf        # llamacpp

  my-codegen:
    - "qwen2.5-coder:7b"                     # Ollama
    - qwen2.5-coder-7b-instruct              # LM Studio

Routing Response Headers¶

Routing decisions are exposed via response headers:

The routing strategy itself is configured under model_registry.routing_strategy. See Model Registry Configuration above.

Translators Configuration¶

API translation settings. Translators enable clients designed for one API format to work with backends that use a different format.

Anthropic Translation (v0.0.20+) Enabled by default. Still actively being improved -- please report any issues or feedback.

Anthropic Translator¶

The Anthropic translator enables Claude-compatible clients (Claude Code, OpenCode, Crush CLI) to work with OpenAI-compatible backends.

Two-Level Control: enabled + passthrough_enabled¶

The Anthropic translator uses a two-level configuration model:

1. enabled is the master switch. When false, the translator is completely disabled and the passthrough_enabled setting has no effect. It is true by default.
2. passthrough_enabled is the optimisation flag. It only takes effect when enabled: true.

When both are active, passthrough mode also requires that the backend profile declares native Anthropic support via api.anthropic_support.enabled: true. Both conditions must be true for passthrough to activate:

• translators.anthropic.passthrough_enabled: true (global configuration)
• Backend profile has api.anthropic_support.enabled: true (per-backend profile)

If either condition is false, Olla falls back to translation mode automatically.

Examples¶

Enable translator with passthrough (recommended for production):

translators:
  anthropic:
    enabled: true
    passthrough_enabled: true       # Forward directly to backends with native Anthropic support
    max_message_size: 10485760      # 10MB

Enable translator with translation only (useful for debugging/testing):

translators:
  anthropic:
    enabled: true
    passthrough_enabled: false      # Always translate Anthropic ↔ OpenAI format
    max_message_size: 10485760

Disable translator entirely:

translators:
  anthropic:
    enabled: false
    # passthrough_enabled has no effect when enabled=false
    passthrough_enabled: true

Performance Implications¶

Detecting the Active Mode¶

Check the X-Olla-Mode response header:

• X-Olla-Mode: passthrough -- passthrough mode was used
• Header absent -- translation mode was used

Inspector (Development Only)¶

Do not enable in production -- logs full request/response bodies including potentially sensitive user data.

See Anthropic Inspector for details.

Logging Configuration¶

Application logging settings.

Example:

logging:
  level: "info"
  format: "json"
  output: "stdout"

Log levels:

• debug: Detailed debugging information
• info: Normal operational messages
• warn: Warning conditions
• error: Error conditions only

Engineering Configuration¶

Debug and development features.

Example:

engineering:
  show_nerdstats: true

When enabled, displays:

• Memory allocation statistics
• Garbage collection metrics
• Goroutine counts
• Runtime information

Environment Variables¶

Only the variables listed below are recognised. They are each hand-mapped in internal/config/config.go; arbitrary OLLA_* names are not auto-derived from YAML paths. Anything you set that isn't in this list will be silently ignored.

When set, an environment variable overrides the value loaded from the YAML config.

Bootstrap and Runtime (read in main.go)¶

These control startup behaviour and are read before the YAML config is loaded.

Server¶

Proxy¶

Logging and Diagnostics¶

The variables below override the logging: config section and take effect after the YAML config is parsed. For the bootstrap logger (before config load), use OLLA_LOG_LEVEL and the OLLA_LOG_* file-rotation vars in the Bootstrap and Runtime table above.

Model Registry¶

Translators¶

Example¶

# Run on port 8080 with the Olla engine, profiling enabled, plain text logs for CI.
OLLA_SERVER_PORT=8080 \
OLLA_PROXY_ENGINE=olla \
OLLA_ENABLE_PROFILER=true \
OLLA_PRETTY_LOGS=false \
./olla

Duration Format¶

Duration values use Go duration syntax:

• s - seconds (e.g., 30s)
• m - minutes (e.g., 5m)
• h - hours (e.g., 2h)
• ms - milliseconds (e.g., 500ms)
• us - microseconds (e.g., 100us)

Examples:

• 30s - 30 seconds
• 5m - 5 minutes
• 1h30m - 1 hour 30 minutes
• 500ms - 500 milliseconds

Default Configuration¶

Complete default configuration:

server:
  host: "localhost"
  port: 40114
  read_timeout: 30s
  read_header_timeout: 10s
  write_timeout: 0s
  # idle_timeout: 0s  # Optional (0 = use read_timeout)
  shutdown_timeout: 10s
  request_logging: true
  request_limits:
    max_body_size: 104857600   # 100MB
    max_header_size: 1048576   # 1MB
  rate_limits:
    global_requests_per_minute: 1000
    per_ip_requests_per_minute: 100
    health_requests_per_minute: 1000
    burst_size: 50
    cleanup_interval: 5m
    trust_proxy_headers: false
    trusted_proxy_cidrs:
      - "127.0.0.0/8"
      - "10.0.0.0/8"
      - "172.16.0.0/12"
      - "192.168.0.0/16"

proxy:
  engine: "olla"
  profile: "auto"
  load_balancer: "least-connections"
  connection_timeout: 60s
  response_timeout: 15m
  read_timeout: 10m
  response_header_timeout: 30s
  connection_keep_alive: 30s
  tls_handshake_timeout: 10s
  # DEPRECATED as of v0.0.16 - retry is now automatic
  # max_retries: 3
  # retry_backoff: 1s
  stream_buffer_size: 8192

discovery:
  type: "static"
  refresh_interval: 30s
  model_discovery:
    enabled: true
    interval: 5m
    timeout: 30s
    concurrent_workers: 5
    retry_attempts: 3
    retry_backoff: 1s
  static:
    endpoints: []

model_registry:
  type: "memory"
  enable_unifier: true
  routing_strategy:
    type: "strict"
    options:
      fallback_behavior: "compatible_only"
      discovery_timeout: 2s
      discovery_refresh_on_miss: false
  unification:
    enabled: true
    stale_threshold: 24h
    cleanup_interval: 10m
    cache_ttl: 10m
    custom_rules: []

translators:
  anthropic:
    enabled: true
    passthrough_enabled: true
    max_message_size: 10485760   # 10MB
    inspector:
      enabled: false
      output_dir: "logs/inspector/anthropic"
      session_header: "X-Session-ID"

logging:
  level: "info"
  format: "json"
  output: "stdout"

engineering:
  show_nerdstats: false

Validation¶

Olla validates configuration on startup:

• Required fields are checked
• URLs must be valid
• Durations must parse correctly
• Endpoints must have unique names
• Ports must be in valid range (1-65535)
• CIDR blocks must be valid

Additionally, Olla's Validate() method catches dangerous zero or empty configuration values that would cause panics or silent failures at runtime. It runs after all config sources (file, environment overrides) have been merged, so the final state is what gets checked. The following conditions produce clear error messages at startup:

• proxy.engine is empty
• proxy.load_balancer is empty
• discovery.type is empty
• server.port is zero or negative
• When model_discovery.enabled is true: interval, concurrent_workers, or timeout is zero

Next Steps¶

• Configuration Examples - Common configurations
• Best Practices - Production recommendations
• Environment Variables - Override configuration