Text Embeddings Inference Integration¶

How TEI differs from LLM backends¶

TEI serves a single model per instance (an embedding, sequence-classification or reranker model) and does not expose an OpenAI-style /v1/models list. Instead, Olla discovers the served model from the /info endpoint (served_model_name, falling back to model_id) using a dedicated parser. Every native TEI endpoint is forwarded verbatim by the provider catch-all route; there are no per-endpoint handlers.

Chat and generation traffic does not reach a TEI endpoint for two reasons: the TEI profile declares no chat or completion paths, so those requests are never sent to TEI instances, and a request naming a chat model is routed by model name to the endpoints hosting that model. Model kinds and routing covers how embed, rerank and classify requests find the right instance.

Configuration¶

Basic Setup¶

Add a TEI instance to your Olla configuration:

discovery:
  static:
    endpoints:
      - url: "http://localhost:3000"
        name: "local-tei"
        type: "text-embeddings-inference"
        priority: 80
        model_url: "/info"
        health_check_url: "/health"
        check_interval: 5s
        check_timeout: 2s

Endpoints Supported¶

The following endpoints are forwarded verbatim to the TEI backend by the provider catch-all route:

Olla additionally serves an OpenAI-format model listing at /v1/models. It lists the models discovered via /info on every healthy TEI endpoint, not just one instance.

Usage Examples¶

Embeddings (OpenAI format)¶

curl -X POST http://localhost:40114/olla/tei/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Snowflake/snowflake-arctic-embed-l-v2.0",
    "input": "Hello, world!"
  }'

Native embeddings¶

curl -X POST http://localhost:40114/olla/tei/embed \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": ["Hello, world!", "How are you?"]
  }'

Reranking¶

curl -X POST http://localhost:40114/olla/tei/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What is machine learning?",
    "texts": [
      "Machine learning is a subset of artificial intelligence",
      "The weather today is sunny"
    ]
  }'

Server info and health¶

# Served model metadata (discovery path)
curl http://localhost:40114/olla/tei/info

# Health status
curl http://localhost:40114/olla/tei/health

# Prometheus metrics
curl http://localhost:40114/olla/tei/metrics

TEI Specifics¶

Single model per instance¶

Each TEI instance serves exactly one model. When Olla discovers the instance it registers that single model (the served_model_name, or model_id as a fallback). TEI itself ignores the model field and answers with the one model it loaded, but Olla's routing does not: it picks the backend by the requested name. Under the default strict model routing (model_registry.routing_strategy.type in config.yaml) a name Olla has not discovered returns 404, so use the model the instance reports in /info, or one of its aliases. Model-less TEI requests (/embed, /rerank, /predict and the like carry no model field) are routed by kind instead.

Model discovery via /info¶

Unlike LLM backends, TEI has no /v1/models listing. Olla's dedicated parser reads the served model from /info, so model_url should point at /info (the profile default).

Model kinds and routing¶

Olla reads each instance's kind from /info's model_type and gives the model a matching type and capabilities. None of them include text-generation.

An instance whose /info has no model_type (older TEI) or a kind Olla does not know is treated as an embedding model.

TEI's native request bodies carry no model field, so for those requests Olla picks the instance from the request path:

This applies to the unified route (/olla/proxy/rerank) and the provider route (/olla/tei/rerank) alike, so a fleet can mix embedding, reranker and classifier instances behind one Olla. A request that names a model (for example /v1/embeddings with "model": "...") is routed by that model name instead, and capability is not consulted.

If a model of the needed kind exists but every endpoint hosting one is unhealthy, Olla answers 503 with X-Olla-Routing-Reason: capability_unavailable rather than sending the request to instances of another kind. If no model of that kind has ever been discovered, Olla does not reject the request: it falls back to every compatible endpoint, as it does for other capability-based routing. TEI then answers 424 when the instance it reaches serves a different kind, for example /rerank sent to an embedding instance. The same 424 applies to a request you pin to a specific instance yourself.

The kind is also visible through Olla: /olla/models?type=reranker and ?type=classifier filter by it, and each model's capabilities list shows rerank and/or classification.

Starting a TEI Server¶

TEI is distributed as a container image and a standalone binary. The container exposes port 80; the binary listens on port 3000 by default.

Docker¶

model=sentence-transformers/all-MiniLM-L6-v2
volume=$PWD/data   # TEI caches weights under /data

docker run -p 3000:80 -v $volume:/data --pull always \
  ghcr.io/huggingface/text-embeddings-inference:cpu-1.9 --model-id $model
# GPU: use cuda-1.9 (or an arch tag such as 89-1.9 or hopper-1.9) with --gpus all

Point the Olla endpoint url at the published port (for example http://localhost:3000).

Monitoring¶

Prometheus Metrics¶

TEI exposes metrics at /metrics, reachable through Olla at /olla/tei/metrics:

scrape_configs:
  - job_name: 'tei'
    static_configs:
      - targets: ['localhost:40114']
    metrics_path: '/olla/tei/metrics'

Next Steps¶

• Profile Configuration - Customise TEI behaviour
• Model Unification - Understand model management
• Load Balancing - Scale with multiple TEI instances
• Monitoring - Set up Prometheus monitoring