Configuration Reference¶

Complete reference for all Olla configuration options.

Default Configuration

server:
  host: "localhost"
  port: 40114

proxy:
  engine: "sherpa"
  load_balancer: "priority"

discovery:
  model_discovery:
    enabled: true
    interval: 5m

logging:
  level: "info"
  format: "json"

Minimal Setup: Olla starts with sensible defaults - just run olla and it works!

Environment Variables: All settings support OLLA_ prefix (e.g., OLLA_SERVER_PORT=8080)

Configuration Structure¶

server:         # HTTP server configuration
proxy:          # Proxy engine settings
discovery:      # Endpoint discovery
model_registry: # Model management
translators:    # API translation (e.g., Anthropic ↔ OpenAI)
logging:        # Logging configuration
engineering:    # Debug features

Server Configuration¶

HTTP server and security settings.

Basic Settings¶

Example:

server:
  host: "0.0.0.0"
  port: 40114
  request_logging: true

Timeouts¶

Example:

server:
  read_timeout: 30s
  write_timeout: 0s      # Required for streaming
  idle_timeout: 120s
  shutdown_timeout: 30s

Request Limits¶

Example:

server:
  request_limits:
    max_body_size: 104857600    # 100MB
    max_header_size: 1048576     # 1MB

Rate Limits¶

Example:

server:
  rate_limits:
    global_requests_per_minute: 10000
    per_ip_requests_per_minute: 100
    health_requests_per_minute: 5000
    burst_size: 50
    cleanup_interval: 5m
    trust_proxy_headers: true
    trusted_proxy_cidrs:
      - "10.0.0.0/8"
      - "172.16.0.0/12"

Proxy Configuration¶

Proxy engine and request handling settings.

Basic Settings¶

Example:

proxy:
  engine: "olla"
  profile: "auto"
  load_balancer: "least-connections"

Connection Settings¶

Example:

proxy:
  connection_timeout: 45s
  response_timeout: 0s    # Disable for streaming
  read_timeout: 0s

Retry Behaviour¶

As of v0.0.16, the retry mechanism is automatic and built-in for connection failures. When a connection error occurs (e.g., connection refused, network unreachable, timeout), Olla will automatically:

1. Mark the failed endpoint as unhealthy
2. Try the next available healthy endpoint
3. Continue until a successful connection is made or all endpoints have been tried
4. Use exponential backoff for unhealthy endpoints to prevent overwhelming them

Note: The fields max_retries and retry_backoff that may still appear in the configuration are deprecated and ignored. The retry behaviour is now automatic and cannot be configured.

Streaming Settings¶

Example:

proxy:
  stream_buffer_size: 8192

Profile Filtering¶

Control which inference profiles are loaded at startup. See Filter Concepts for pattern details.

Example:

proxy:
  profile_filter:
    include:
      - "ollama"        # Include Ollama
      - "openai*"       # Include all OpenAI variants
    exclude:
      - "*test*"        # Exclude test profiles
      - "*debug*"       # Exclude debug profiles

Discovery Configuration¶

Endpoint discovery and health checking.

Discovery Type¶

Example:

discovery:
  type: "static"
  refresh_interval: 10m

Static Endpoints¶

URL Configuration¶

The health_check_url and model_url fields are optional. When not specified, Olla uses profile-specific defaults based on the endpoint type:

Profile Defaults:

Both fields support:

1. Relative paths (recommended) - joined with the endpoint base URL:

   url: "http://localhost:8080/api/"
   health_check_url: "/health"     # Becomes: http://localhost:8080/api/health
   model_url: "/v1/models"         # Becomes: http://localhost:8080/api/v1/models
   

2. Absolute URLs - used as-is for external services:

   url: "http://localhost:11434"
   health_check_url: "http://monitoring.local:9090/health"  # Different host
   model_url: "http://registry.local/models"                # Different host
   

When using relative paths, any base path prefix in the endpoint URL is automatically preserved (e.g., http://localhost:8080/api/ + /v1/models = http://localhost:8080/api/v1/models).

Endpoint Model Filtering¶

Filter models at the endpoint level during discovery. See Filter Concepts for pattern syntax.

Path Preservation¶

The preserve_path field controls how Olla handles base paths in endpoint URLs during proxying. This is particularly important for endpoints that serve multiple services or use path-based routing.

Default Behaviour (preserve_path: false) When preserve_path is false (default), Olla strips the base path from the endpoint URL before proxying:

• Endpoint URL: http://localhost:8080/api/v1
• Request to Olla: /v1/chat/completions
• Proxied to: http://localhost:8080/v1/chat/completions (base path /api/v1 is replaced)

Path Preservation (preserve_path: true) When preserve_path is true, Olla preserves the base path:

• Endpoint URL: http://localhost:8080/api/v1
• Request to Olla: /v1/chat/completions
• Proxied to: http://localhost:8080/api/v1/v1/chat/completions (base path is preserved)

When to Use Path Preservation:

• Docker Model Runner endpoints with base paths
• APIs deployed behind path-based routers
• Services that require specific URL structures
• Multi-service endpoints using path differentiation

Example:

discovery:
  static:
    endpoints:
      # Minimal configuration - uses profile defaults
      - url: "http://localhost:11434"
        name: "local-ollama"
        type: "ollama"
        priority: 100
        # health_check_url: "/" (default for ollama)
        # model_url: "/api/tags" (default for ollama)

      # Custom health check URL
      - url: "http://localhost:8080"
        name: "llamacpp-server"
        type: "llamacpp"
        priority: 90
        health_check_url: "/health"
        # model_url: "/v1/models" (default for llamacpp)

      # Endpoint with base path - URLs are preserved
      - url: "http://localhost:8080/api/"
        name: "vllm-gateway"
        type: "vllm"
        priority: 80
        # health_check_url: "/health" -> http://localhost:8080/api/health
        # model_url: "/v1/models" -> http://localhost:8080/api/v1/models

      # External health check on different host
      - url: "http://localhost:11434"
        name: "monitored-ollama"
        type: "ollama"
        health_check_url: "http://monitoring.local:9090/health/ollama"
        # Absolute URL used as-is

      # Docker Model Runner with base path
      - url: "http://localhost:8080/api/models/llama"
        name: "docker-llama"
        type: "openai"
        preserve_path: true  # Keep /api/models/llama in requests

      # Endpoint with model filtering
      - url: "http://remote:11434"
        name: "remote-ollama"
        type: "ollama"
        priority: 50
        check_interval: 60s
        model_filter:
          include:
            - "llama*"          # Only Llama models
            - "mistral*"        # And Mistral models

Model Discovery¶

Example:

discovery:
  model_discovery:
    enabled: true
    interval: 10m
    timeout: 30s
    concurrent_workers: 10
    retry_attempts: 3
    retry_backoff: 5s

Model Registry Configuration¶

Model management and unification settings.

Registry Type¶

Example:

model_registry:
  type: "memory"
  enable_unifier: true
  routing_strategy:
    type: strict  # Default: only route to endpoints with the model

Model Routing Strategy¶

Controls how requests are routed when models aren't available on all endpoints:

Example configurations:

# Production - strict routing
model_registry:
  routing_strategy:
    type: strict

# Development - optimistic with fallback
model_registry:
  routing_strategy:
    type: optimistic
    options:
      fallback_behavior: compatible_only

# Dynamic environments - discovery mode
model_registry:
  routing_strategy:
    type: discovery
    options:
      discovery_refresh_on_miss: true
      discovery_timeout: 2s

Unification Settings¶

Example:

model_registry:
  unification:
    enabled: true
    stale_threshold: 12h
    cleanup_interval: 15m
    cache_ttl: 10m

Custom Unification Rules¶

Example:

model_registry:
  unification:
    custom_rules:
      - platform: "ollama"
        name_patterns:
          "llama3.*": "llama3"
          "mistral.*": "mistral"
        family_overrides:
          "llama3": "meta-llama"

Routing Configuration¶

Model routing strategy settings for handling requests when models aren't available on all endpoints.

Model Routing Strategy¶

Strategy Types¶

• strict: Only routes to endpoints known to have the model
• optimistic: Falls back to healthy endpoints when model not found
• discovery: Refreshes model discovery before routing decisions

Example:

routing:
  model_routing:
    type: strict
    options:
      fallback_behavior: compatible_only
      discovery_refresh_on_miss: false
      discovery_timeout: 2s

Response Headers¶

Routing decisions are exposed via response headers:

Translators Configuration¶

API translation settings. Translators enable clients designed for one API format to work with backends that use a different format.

Anthropic Translation (v0.0.20+) Enabled by default. Still actively being improved -- please report any issues or feedback.

Anthropic Translator¶

The Anthropic translator enables Claude-compatible clients (Claude Code, OpenCode, Crush CLI) to work with OpenAI-compatible backends.

Two-Level Control: enabled + passthrough_enabled¶

The Anthropic translator uses a two-level configuration model:

1. enabled is the master switch. When false, the translator is completely disabled and the passthrough_enabled setting has no effect. It is true by default.
2. passthrough_enabled is the optimisation flag. It only takes effect when enabled: true.

When both are active, passthrough mode also requires that the backend profile declares native Anthropic support via api.anthropic_support.enabled: true. Both conditions must be true for passthrough to activate:

• translators.anthropic.passthrough_enabled: true (global configuration)
• Backend profile has api.anthropic_support.enabled: true (per-backend profile)

If either condition is false, Olla falls back to translation mode automatically.

Examples¶

Enable translator with passthrough (recommended for production):

translators:
  anthropic:
    enabled: true
    passthrough_enabled: true       # Forward directly to backends with native Anthropic support
    max_message_size: 10485760      # 10MB

Enable translator with translation only (useful for debugging/testing):

translators:
  anthropic:
    enabled: true
    passthrough_enabled: false      # Always translate Anthropic ↔ OpenAI format
    max_message_size: 10485760

Disable translator entirely:

translators:
  anthropic:
    enabled: false
    # passthrough_enabled has no effect when enabled=false
    passthrough_enabled: true

Performance Implications¶

Detecting the Active Mode¶

Check the X-Olla-Mode response header:

• X-Olla-Mode: passthrough -- passthrough mode was used
• Header absent -- translation mode was used

Inspector (Development Only)¶

Do not enable in production -- logs full request/response bodies including potentially sensitive user data.

See Anthropic Inspector for details.

Logging Configuration¶

Application logging settings.

Example:

logging:
  level: "info"
  format: "json"
  output: "stdout"

Log levels:

• debug: Detailed debugging information
• info: Normal operational messages
• warn: Warning conditions
• error: Error conditions only

Engineering Configuration¶

Debug and development features.

Example:

engineering:
  show_nerdstats: true

When enabled, displays:

• Memory allocation statistics
• Garbage collection metrics
• Goroutine counts
• Runtime information

Environment Variables¶

All configuration can be overridden via environment variables.

Pattern: OLLA_<SECTION>_<KEY> in uppercase with underscores.

Examples:

# Server settings
OLLA_SERVER_HOST=0.0.0.0
OLLA_SERVER_PORT=8080
OLLA_SERVER_REQUEST_LOGGING=true

# Proxy settings
OLLA_PROXY_ENGINE=olla
OLLA_PROXY_LOAD_BALANCER=round-robin
OLLA_PROXY_PROFILE=auto

# Logging
OLLA_LOGGING_LEVEL=debug
OLLA_LOGGING_FORMAT=text

# Rate limits
OLLA_SERVER_RATE_LIMITS_GLOBAL_REQUESTS_PER_MINUTE=1000

Duration Format¶

Duration values use Go duration syntax:

• s - seconds (e.g., 30s)
• m - minutes (e.g., 5m)
• h - hours (e.g., 2h)
• ms - milliseconds (e.g., 500ms)
• us - microseconds (e.g., 100us)

Examples:

• 30s - 30 seconds
• 5m - 5 minutes
• 1h30m - 1 hour 30 minutes
• 500ms - 500 milliseconds

Default Configuration¶

Complete default configuration:

server:
  host: "localhost"
  port: 40114
  read_timeout: 30s
  write_timeout: 0s
  # idle_timeout: 0s  # Optional (0 = use read_timeout)
  shutdown_timeout: 10s
  request_logging: true
  request_limits:
    max_body_size: 104857600   # 100MB
    max_header_size: 1048576    # 1MB
  rate_limits:
    global_requests_per_minute: 1000
    per_ip_requests_per_minute: 100
    health_requests_per_minute: 1000
    burst_size: 50
    cleanup_interval: 5m
    trust_proxy_headers: false
    trusted_proxy_cidrs:
      - "127.0.0.0/8"
      - "10.0.0.0/8"
      - "172.16.0.0/12"
      - "192.168.0.0/16"

proxy:
  engine: "sherpa"
  profile: "auto"
  load_balancer: "priority"
  connection_timeout: 30s
  response_timeout: 10m
  read_timeout: 120s
  # DEPRECATED as of v0.0.16 - retry is now automatic
  # max_retries: 3
  # retry_backoff: 1s
  stream_buffer_size: 8192

discovery:
  type: "static"
  refresh_interval: 30s
  model_discovery:
    enabled: true
    interval: 5m
    timeout: 30s
    concurrent_workers: 5
    retry_attempts: 3
    retry_backoff: 5s
  static:
    endpoints: []

model_registry:
  type: "memory"
  enable_unifier: true
  routing_strategy:
    type: "strict"
    options:
      fallback_behavior: "compatible_only"
      discovery_timeout: 2s
      discovery_refresh_on_miss: false
  unification:
    enabled: true
    stale_threshold: 24h
    cleanup_interval: 10m
    cache_ttl: 10m
    custom_rules: []

translators:
  anthropic:
    enabled: true
    passthrough_enabled: true
    max_message_size: 10485760   # 10MB
    inspector:
      enabled: false
      output_dir: "logs/inspector/anthropic"
      session_header: "X-Session-ID"

logging:
  level: "info"
  format: "json"
  output: "stdout"

engineering:
  show_nerdstats: false

Validation¶

Olla validates configuration on startup:

• Required fields are checked
• URLs must be valid
• Durations must parse correctly
• Endpoints must have unique names
• Ports must be in valid range (1-65535)
• CIDR blocks must be valid

Next Steps¶

• Configuration Examples - Common configurations
• Best Practices - Production recommendations
• Environment Variables - Override configuration