Load Balancing¶

Default Configuration

proxy:
  load_balancer: "least-connections"  # least-connections, priority, or round-robin

Supported:

• least-connections (default) - Routes to least busy endpoint
• priority - Routes to highest priority endpoint
• round-robin - Cycles through endpoints evenly

Environment Variable: OLLA_PROXY_LOAD_BALANCER

Olla provides multiple load balancing strategies to distribute requests across backend endpoints efficiently. Each strategy has specific use cases and characteristics to optimise for different deployment scenarios.

For multi-turn LLM workloads, combine any strategy with Sticky Sessions to preserve KV-cache affinity across turns.

Overview¶

Load balancing determines which backend endpoint receives each incoming request. The strategy you choose affects:

• Request distribution - How evenly load spreads across endpoints
• Failover behaviour - How quickly traffic shifts from unhealthy endpoints
• Resource utilisation - How efficiently backend resources are used
• Response times - Latency based on endpoint selection

Available Strategies¶

Priority (Recommended)¶

Priority-based selection routes to the highest-priority healthy endpoint first.

How it works:

1. Endpoints assigned priority values (0-100, higher = preferred)
2. Selects from highest priority tier only
3. Within same priority, uses weighted random selection
4. Automatically falls back to lower priorities if higher ones fail

Best for:

• Production deployments with primary/backup servers
• GPU vs CPU server hierarchies
• Geographic or network-based preferences
• Cost-optimised deployments (prefer cheaper endpoints)

Configuration:

proxy:
  load_balancer: "priority"

discovery:
  static:
    endpoints:
      - url: "http://gpu-server:11434"
        name: "primary-gpu"
        type: "ollama"
        priority: 100  # Highest priority

      - url: "http://backup-gpu:11434"
        name: "backup-gpu"
        type: "ollama"
        priority: 90   # Second choice

      - url: "http://cpu-server:11434"
        name: "cpu-fallback"
        type: "ollama"
        priority: 50   # Last resort

Round Robin¶

Distributes requests evenly across all healthy endpoints in sequence.

How it works:

1. Maintains internal counter
2. Selects next endpoint in rotation
3. Skips unhealthy endpoints
4. Wraps around at end of list

Best for:

• Homogeneous server deployments
• Equal capacity endpoints
• Development and testing
• Simple load distribution

Configuration:

proxy:
  load_balancer: "round-robin"

discovery:
  static:
    endpoints:
      - url: "http://server1:11434"
        name: "ollama-1"
        type: "ollama"

      - url: "http://server2:11434"
        name: "ollama-2"
        type: "ollama"

      - url: "http://server3:11434"
        name: "ollama-3"
        type: "ollama"

Least Connections¶

Routes to the endpoint with the fewest active connections.

How it works:

1. Tracks active connections per endpoint
2. Selects endpoint with minimum connections
3. Updates count on connection start/end
4. Ideal for long-running requests

Best for:

• Variable request durations
• Streaming responses
• Mixed model sizes
• Preventing endpoint overload

Configuration:

proxy:
  load_balancer: "least-connections"

discovery:
  static:
    endpoints:
      - url: "http://fast-server:8000"
        name: "vllm-fast"
        type: "vllm"

      - url: "http://slow-server:8000"
        name: "vllm-slow"
        type: "vllm"

Strategy Comparison¶

Health-Aware Routing¶

All strategies respect endpoint health status:

Health States¶

Circuit Breaker Integration¶

Olla includes built-in circuit breaker functionality to prevent cascading failures. The circuit breaker automatically manages endpoint health without requiring configuration.

Circuit states:

• Closed - Normal operation, requests pass through
• Open - Endpoint marked unhealthy after consecutive failures
• Half-Open - Testing recovery with limited requests

Advanced Configurations¶

Multi-Tier Priority¶

Create sophisticated failover hierarchies:

discovery:
  static:
    endpoints:
      # Tier 1: Local GPU servers
      - url: "http://localhost:11434"
        name: "local-gpu"
        priority: 100

      - url: "http://192.168.1.10:11434"
        name: "lan-gpu"
        priority: 95

      # Tier 2: Cloud GPU servers
      - url: "https://gpu1.cloud.example.com"
        name: "cloud-gpu-1"
        priority: 80

      - url: "https://gpu2.cloud.example.com"
        name: "cloud-gpu-2"
        priority: 80

      # Tier 3: CPU fallbacks
      - url: "http://cpu-server:11434"
        name: "cpu-fallback"
        priority: 50

Model-Based Load Distribution¶

The load balancer works with model unification to distribute requests across endpoints that have the requested model available. When multiple endpoints have the same model, the configured load balancing strategy (priority, round-robin, or least-connections) determines which endpoint receives the request.

Geographic Distribution¶

Use priority for region-based routing:

endpoints:
  # US East - Primary
  - url: "https://us-east.example.com"
    name: "us-east-primary"
    priority: 100

  # US West - Secondary
  - url: "https://us-west.example.com"
    name: "us-west-secondary"
    priority: 90

  # EU - Tertiary
  - url: "https://eu.example.com"
    name: "eu-tertiary"
    priority: 70

Monitoring Load Distribution¶

Metrics Endpoints¶

Monitor load balancer effectiveness:

# Endpoint connection counts
curl http://localhost:40114/internal/status/endpoints

# Model routing statistics
curl http://localhost:40114/internal/stats/models

# Health status overview
curl http://localhost:40114/internal/health

Response Headers¶

Track routing decisions via headers:

X-Olla-Endpoint: gpu-server-1
X-Olla-Backend-Type: ollama
X-Olla-Request-ID: abc-123
X-Olla-Response-Time: 1.234s

Logging¶

Enable debug logging for routing decisions:

logging:
  level: debug

# Logs show:
# - Endpoint selection reasoning
# - Health state changes
# - Circuit breaker transitions
# - Connection tracking

Performance Tuning¶

Connection Pooling¶

The Olla proxy engine maintains per-endpoint connection pools with hardcoded defaults (max 100 idle connections, 50 connections per host, 25 idle connections per host, 90s idle timeout). These are not currently configurable via YAML.

Timeout Configuration¶

Balance reliability and responsiveness:

proxy:
  # Initial connection timeout
  connection_timeout: 10s

  # Request/response timeout
  response_timeout: 300s

  # Read timeout (response body)
  read_timeout: 10m

Buffering Strategies¶

Choose appropriate proxy profile:

proxy:
  profile: "auto"  # auto, streaming, or standard

  # Auto profile adapts based on:
  # - Response size
  # - Streaming detection
  # - Content type

Troubleshooting¶

Uneven Distribution¶

Issue: Requests not spreading evenly

Diagnosis:

# Check endpoint stats
curl http://localhost:40114/internal/status/endpoints | jq '.endpoints[] | {name, requests}'

Solutions:

• Verify all endpoints are healthy
• Check priority values aren't identical
• Review connection tracking for least-connections
• Ensure round-robin counter isn't stuck

Slow Failover¶

Issue: Takes too long to stop using failed endpoints

Solutions:

# Reduce per-endpoint check interval for faster detection
discovery:
  static:
    endpoints:
      - url: "http://backend:11434"
        check_interval: 5s   # Reduce for faster detection
        check_timeout: 2s    # Fail fast on timeouts

Connection Exhaustion¶

Issue: "Too many connections" errors

Solutions:

# Use least-connections to spread load
proxy:
  load_balancer: "least-connections"

The Olla proxy engine manages connection pool limits internally. Per-host limits are not currently configurable via YAML.

Best Practices¶

1. Match Strategy to Deployment¶

• Priority: Production with clear server tiers
• Round Robin: Development or identical servers
• Least Connections: Mixed workloads or streaming

2. Configure Health Checks¶

Health check settings are per-endpoint, not global:

discovery:
  static:
    endpoints:
      - url: "http://backend:11434"
        check_interval: 10s   # Balance between speed and load
        check_timeout: 5s     # Allow for model loading

There is no unhealthy_threshold config field. The circuit breaker (3 failures for the health checker, 5 transport failures for the proxy engine) provides equivalent flap protection automatically.

3. Set Appropriate Priorities¶

# Clear gaps between tiers
priority: 100  # Primary
priority: 75   # Secondary (clear gap)
priority: 50   # Tertiary (clear gap)
priority: 25   # Last resort

4. Monitor and Adjust¶

Regularly review:

• Request distribution patterns
• Endpoint utilisation
• Response time percentiles
• Error rates by endpoint

5. Plan for Failure¶

Always have:

• At least one backup endpoint
• Clear priority tiers
• Monitoring and alerting

Circuit breaker settings are hardcoded and do not require operator configuration.

Integration Examples¶

Kubernetes Service¶

apiVersion: v1
kind: Service
metadata:
  name: ollama-backends
spec:
  selector:
    app: ollama
  ports:
    - port: 11434
  type: ClusterIP
---
# Olla configuration
discovery:
  static:
    endpoints:
      - url: "http://ollama-backends:11434"
        name: "k8s-ollama"
        type: "ollama"

Docker Swarm¶

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama
    deploy:
      replicas: 3

  olla:
    image: thushan/olla
    environment:
      OLLA_PROXY_LOAD_BALANCER: "round-robin"

Next Steps¶

• Health Checking - Configure health monitoring
• Circuit Breaker - Prevent cascade failures
• Model Unification - Unified model routing
• Performance Tuning - Optimisation guide