Skip to content

Docker Model Runner API

Proxy endpoints for Docker Model Runner inference servers. Available through the /olla/dmr/ prefix.

Endpoints Overview

Method URI Description
GET /olla/dmr/engines/v1/models List available models
POST /olla/dmr/engines/v1/chat/completions Chat completion
POST /olla/dmr/engines/v1/completions Text completion
POST /olla/dmr/engines/v1/embeddings Generate embeddings
GET /olla/dmr/engines/llama.cpp/v1/models List models (llama.cpp engine)
POST /olla/dmr/engines/llama.cpp/v1/chat/completions Chat completion (llama.cpp engine)
GET /olla/dmr/engines/vllm/v1/models List models (vLLM engine)
POST /olla/dmr/engines/vllm/v1/chat/completions Chat completion (vLLM engine)

The /engines/v1/... paths use automatic engine selection. The explicit /engines/llama.cpp/v1/... and /engines/vllm/v1/... paths target a specific engine directly.

GET /olla/dmr/engines/v1/models

List models available on the Docker Model Runner instance.

Returns an empty data array when no models have been loaded yet. This is normal behaviour due to lazy model loading and does not indicate an unhealthy endpoint.

Request

curl -X GET http://localhost:40114/olla/dmr/engines/v1/models

Response

{
  "object": "list",
  "data": [
    {
      "id": "ai/smollm2",
      "object": "model",
      "created": 1734000000,
      "owned_by": "ai"
    },
    {
      "id": "ai/llama3.2",
      "object": "model",
      "created": 1734000001,
      "owned_by": "ai"
    }
  ]
}

POST /olla/dmr/engines/v1/chat/completions

OpenAI-compatible chat completion routed to the appropriate DMR engine.

Request

curl -X POST http://localhost:40114/olla/dmr/engines/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/smollm2",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "What is Docker Model Runner?"
      }
    ],
    "temperature": 0.7,
    "max_tokens": 300,
    "stream": false
  }'

Response

{
  "id": "chatcmpl-dmr-abc123",
  "object": "chat.completion",
  "created": 1734000000,
  "model": "ai/smollm2",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Docker Model Runner is Docker's built-in LLM inference server that ships with Docker Desktop 4.40+. It enables you to pull, run, and serve large language models directly from Docker Hub as OCI artifacts, using llama.cpp for GGUF models and vLLM for safetensors models."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 22,
    "completion_tokens": 54,
    "total_tokens": 76
  }
}

Streaming Response

When "stream": true:

data: {"id":"chatcmpl-dmr-abc123","object":"chat.completion.chunk","created":1734000000,"model":"ai/smollm2","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-dmr-abc123","object":"chat.completion.chunk","created":1734000000,"model":"ai/smollm2","choices":[{"index":0,"delta":{"content":"Docker"},"finish_reason":null}]}

...

data: {"id":"chatcmpl-dmr-abc123","object":"chat.completion.chunk","created":1734000001,"model":"ai/smollm2","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

POST /olla/dmr/engines/v1/completions

Text completion (non-chat) via DMR.

Request

curl -X POST http://localhost:40114/olla/dmr/engines/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/smollm2",
    "prompt": "Docker Model Runner is",
    "max_tokens": 100,
    "temperature": 0.8,
    "stream": false
  }'

Response

{
  "id": "cmpl-dmr-xyz789",
  "object": "text_completion",
  "created": 1734000000,
  "model": "ai/smollm2",
  "choices": [
    {
      "text": " Docker's built-in LLM inference server, enabling developers to run AI models locally using the same tools they use for containers.",
      "index": 0,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 5,
    "completion_tokens": 28,
    "total_tokens": 33
  }
}

POST /olla/dmr/engines/v1/embeddings

Generate embeddings using a DMR model that supports embeddings.

Request

curl -X POST http://localhost:40114/olla/dmr/engines/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/mxbai-embed-large",
    "input": "Docker Model Runner provides local LLM inference",
    "encoding_format": "float"
  }'

Response

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [0.0234, -0.0567, 0.0891, "..."]
    }
  ],
  "model": "ai/mxbai-embed-large",
  "usage": {
    "prompt_tokens": 8,
    "total_tokens": 8
  }
}

Explicit Engine Endpoints

Use these paths to target a specific inference engine, bypassing automatic engine selection.

llama.cpp Engine

# List models on llama.cpp engine
curl http://localhost:40114/olla/dmr/engines/llama.cpp/v1/models

# Chat completion via llama.cpp
curl -X POST http://localhost:40114/olla/dmr/engines/llama.cpp/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/smollm2",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 100
  }'

vLLM Engine

# List models on vLLM engine (Linux + NVIDIA only)
curl http://localhost:40114/olla/dmr/engines/vllm/v1/models

# Chat completion via vLLM
curl -X POST http://localhost:40114/olla/dmr/engines/vllm/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/llama3.1-8b",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 100
  }'

Model Naming

DMR uses a namespace/name format for all model identifiers:

Example Description
ai/smollm2 SmolLM2 from the ai namespace
ai/llama3.2 Llama 3.2 from the ai namespace
ai/phi4-mini Phi-4 Mini from the ai namespace

Always use the full namespace/name format in the model field of API requests.

Request Headers

All requests are forwarded with:

  • X-Olla-Request-ID - Unique request identifier
  • X-Forwarded-For - Client IP address
  • Custom headers from endpoint configuration

Response Headers

All responses include:

  • X-Olla-Endpoint - Backend endpoint name (e.g., local-dmr)
  • X-Olla-Model - Model used for the request
  • X-Olla-Backend-Type - Always docker-model-runner for these endpoints
  • X-Olla-Response-Time - Total processing time

Configuration Example

discovery:
  static:
    endpoints:
      - url: "http://localhost:12434"
        name: "local-dmr"
        type: "docker-model-runner"
        priority: 95
        model_url: "/engines/v1/models"
        health_check_url: "/engines/v1/models"
        check_interval: 10s
        check_timeout: 5s

See the Docker Model Runner Integration Guide for full configuration and setup instructions.