Model Routing¶

Default Configuration

model_registry:
  routing_strategy:
    type: "strict"  # strict, optimistic, or discovery
    options:
      fallback_behavior: "compatible_only"  # compatible_only, all, or none
      discovery_timeout: 2s
      discovery_refresh_on_miss: false

Routing Strategies:

• strict (default) - Only routes to endpoints with the model
• optimistic - Routes with configurable fallback behavior
• discovery - Refreshes model catalog before routing

Fallback Behaviors:

• compatible_only (default) - Rejects if model not found
• all - Routes to any healthy endpoint
• none - Always rejects if model not found

Olla implements intelligent model routing strategies to handle scenarios where requested models aren't available on all endpoints.

Overview¶

When a request specifies a model (e.g., phi3.5:latest), Olla needs to determine which endpoints can handle that request. Not all endpoints have all models, and endpoints can become unhealthy during operation.

Ollama model names without a tag¶

Ollama lists a model pulled without a tag as name:latest and accepts name for it. Olla does the same: when no endpoint serves the exact name you asked for, a name with no tag is looked up as name:latest on Ollama endpoints. An exact match always wins, only Ollama endpoints answer to the untagged name, and the request is forwarded exactly as you wrote it. The lookup only goes one way, name to name:latest, because Ollama always lists models with a tag.

Routing Strategies¶

Strict Mode (Default)¶

Only routes requests to endpoints known to have the model.

Characteristics: - High reliability - requests only go to endpoints with the model - Fails fast when model unavailable - Returns 404 when model not found anywhere - Returns 503 when model only on unhealthy endpoints

Use Case: Production environments where predictability is critical.

model_registry:
  routing_strategy:
    type: strict

Optimistic Mode¶

Attempts to route to any healthy endpoint when the model isn't found.

Characteristics: - Higher availability - tries all healthy endpoints - May route to endpoints without the model - Configurable fallback behavior - Best effort approach

Use Case: Development environments or when models might be pulled on-demand.

model_registry:
  routing_strategy:
    type: optimistic
    options:
      fallback_behavior: compatible_only  # or "all", "none"

Discovery Mode¶

Refreshes model discovery before making routing decisions.

Characteristics: - Most accurate model availability - Adds latency for discovery refresh - Configurable timeout - Falls back to strict behavior after refresh

Use Case: When models are frequently added/removed from endpoints.

model_registry:
  routing_strategy:
    type: discovery
    options:
      discovery_refresh_on_miss: true
      discovery_timeout: 2s

Fallback Behavior¶

Controls what happens when the requested model isn't available on any healthy endpoint:

• compatible_only: Reject the request with 404 - prevents routing to endpoints that don't have the model
• all: Route to any healthy endpoint even if they don't have the model
• none: Never fall back, always reject with 404 if model not found

Requests Without a Model Name¶

Some requests name no model: a vision request sent to the unified route, TEI's native /rerank, /predict and /embed, whose bodies carry no model field. Olla narrows the candidate endpoints by what the request needs instead:

An endpoint stays in the candidate set when it hosts at least one single model carrying every required capability. A vision model on one endpoint and a tools model on another do not combine to satisfy a request needing both. A rerank or classification path takes precedence over the embeddings guess from the body, because no model is both an embedder and a reranker.

Capability tags are heuristic, so the filter only refuses in one case: models with the required capability exist, but every endpoint hosting one is down. A Busy or otherwise unhealthy host counts as down here, matching how Olla already picks endpoints. Olla then answers 503 with X-Olla-Routing-Reason: capability_unavailable (see Routing Reasons) rather than sending the request to hosts of another kind. When no model with the capability has been discovered at all, or a capable host is healthy but outside the candidates, the request falls back to every compatible endpoint as before. Requests that name a model skip this filter.

Outside that case capability filtering does not reject a request. If no model carries the capabilities, or a capable host is routable but not among the candidates, Olla uses the unfiltered candidates. A backend that cannot serve the request then answers for itself, as TEI does with 424 for a request sent to the wrong model kind.

The single-segment paths above (/embed, /rerank, /predict) match only that exact request path, so /api/embed (Ollama's embeddings route) is never read as TEI's /embed. A model's capabilities come from its type and name patterns; see Model Unification.

Named models skip capability filtering. When the request names a model, only the routing strategy above decides which endpoints are used, namely those hosting that model. A capability tag on some other model can't remove the hosts of the one you asked for.

Response Headers¶

Routing decisions are exposed via HTTP headers for observability:

X-Olla-Routing-Strategy: strict
X-Olla-Routing-Decision: routed
X-Olla-Routing-Reason: model_found

Status Codes and Routing Decisions¶

Different scenarios result in specific HTTP status codes and routing behaviors:

Strict Mode Behavior¶

Optimistic Mode Behavior¶

Discovery Mode Behavior¶

Routing Reasons¶

The X-Olla-Routing-Reason header provides detailed information about routing decisions:

Configuration Example¶

Complete routing configuration:

# Strict mode for production (default)
model_registry:
  type: "memory"
  enable_unifier: true
  routing_strategy:
    type: strict

# Optimistic with compatible fallback
model_registry:
  type: "memory"
  enable_unifier: true
  routing_strategy:
    type: optimistic
    options:
      fallback_behavior: compatible_only

# Discovery with timeout
model_registry:
  type: "memory"
  enable_unifier: true
  routing_strategy:
    type: discovery
    options:
      discovery_refresh_on_miss: true
      discovery_timeout: 2s
      fallback_behavior: none

Metrics and Monitoring¶

Routing decisions are tracked in metrics:

• Total routing decisions by strategy
• Success/failure rates per strategy
• Fallback usage statistics
• Discovery refresh latency

Access metrics via /internal/status endpoint.

Best Practices¶

1. Use strict mode in production for predictable behavior
2. Enable discovery mode when models change frequently
3. Monitor routing headers to understand request flow
4. Set appropriate timeouts for discovery mode
5. Choose fallback behavior carefully:
6. none or compatible_only for APIs that need model accuracy
7. all only when any endpoint can handle unknown models

Troubleshooting¶

Getting 404 for Known Models¶

Issue: Requests return 404 even though the model exists

Possible Causes: - Model only exists on unhealthy endpoints - Using compatible_only or none fallback when model isn't discovered yet - Model discovery hasn't run yet

Solutions: 1. Check endpoint health: curl http://localhost:40114/internal/status/endpoints 2. Verify model discovery: curl http://localhost:40114/internal/status/models 3. Try discovery mode with refresh:

routing_strategy:
  type: discovery
  options:
    discovery_refresh_on_miss: true

Getting 503 vs 404¶

Understanding the Difference: - 404: Model doesn't exist anywhere in the system - 503: Model exists but no healthy endpoints have it

How to Debug:

# Check all models in the system
curl http://localhost:40114/olla/models

# Check endpoint health
curl http://localhost:40114/internal/status/endpoints

# Look at routing headers
curl -I http://localhost:40114/olla/ollama/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "test-model", "messages": []}'

Unexpected Fallback Behavior¶

Issue: Requests going to endpoints without the model

Check Your Configuration:

# This allows fallback to any endpoint
fallback_behavior: "all"

# These prevent fallback
fallback_behavior: "none"        # Returns 404 for unknown models
fallback_behavior: "compatible_only"  # Returns 404 for unknown models