SGLang API¶

{
  "status": "healthy",
  "model_loaded": true,
  "radix_cache_ready": true,
  "radix_cache_usage_perc": 0.35,
  "num_requests_running": 3,
  "num_requests_waiting": 0,
  "speculative_decoding_enabled": true
}

{
  "object": "list",
  "data": [
    {
      "id": "meta-llama/Meta-Llama-3.1-8B-Instruct",
      "object": "model",
      "created": 1705334400,
      "owned_by": "sglang",
      "root": "meta-llama/Meta-Llama-3.1-8B-Instruct",
      "parent": null,
      "max_model_len": 131072,
      "permission": [],
      "radix_cache_size": 17179869184,
      "speculative_decoding": true
    }
  ]
}

{
  "id": "chatcmpl-sglang-abc123",
  "object": "chat.completion",
  "created": 1705334400,
  "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "RadixAttention in SGLang provides several key benefits over traditional attention mechanisms:\n\n1. **Advanced Prefix Caching**: Uses a tree-based structure to cache common prefixes more efficiently than block-based approaches, achieving 85-95% cache hit rates.\n\n2. **Memory Efficiency**: Reduces memory usage by 15-20% compared to PagedAttention through better sharing of computation across requests.\n\n3. **Higher Throughput**: Enables handling 150+ concurrent requests efficiently due to the intelligent caching strategy.\n\n4. **Faster Response Times**: Dramatically reduces time-to-first-token for requests with cached prefixes.\n\nThis makes SGLang particularly effective for applications with repetitive patterns like chatbots, code completion, and conversational AI."
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 42,
    "completion_tokens": 128,
    "total_tokens": 170
  },
  "metrics": {
    "time_to_first_token_ms": 35,
    "generation_time_ms": 850,
    "radix_cache_hit_rate": 0.92
  }
}

{
  "id": "cmpl-sglang-xyz789",
  "object": "text_completion",
  "created": 1705334400,
  "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
  "choices": [
    {
      "text": "\n\n1. **RadixAttention Efficiency**: More advanced prefix caching with tree-based attention patterns, achieving 85-95% cache hit rates compared to vLLM's 60-70%.\n\n2. **Higher Concurrency**: Supports 150+ concurrent requests versus vLLM's typical 100 limit due to better memory management.\n\n3. **Frontend Language**: Provides a flexible programming interface for complex LLM applications beyond simple API calls.\n\n4. **Speculative Decoding**: Enhanced performance through speculative execution, reducing latency by up to 30%.\n\n5. **Memory Efficiency**: Uses 15-20% less GPU memory while maintaining the same model quality.",
      "index": 0,
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 15,
    "completion_tokens": 142,
    "total_tokens": 157
  },
  "metrics": {
    "time_to_first_token_ms": 28,
    "generation_time_ms": 920,
    "radix_cache_hit_rate": 0.88
  }
}

{
  "text": "def fibonacci(n):\n    if n <= 1:\n        return n\n    else:\n        return fibonacci(n-1) + fibonacci(n-2)\n\n# Example usage:\nprint(fibonacci(10))  # Output: 55",
  "meta": {
    "id": "gen-sglang-def456",
    "rid": "conversation-789",
    "finish_reason": "stop",
    "input_tokens": 12,
    "output_tokens": 28,
    "completion_time_ms": 450,
    "radix_cache_hit": true
  }
}

{
  "responses": [
    {
      "text": "Translate to French: Hello world\n\nBonjour le monde",
      "meta": {
        "id": "batch-1-sglang-ghi789",
        "finish_reason": "stop",
        "input_tokens": 7,
        "output_tokens": 6,
        "completion_time_ms": 320
      }
    },
    {
      "text": "Translate to Spanish: Hello world\n\nHola mundo",
      "meta": {
        "id": "batch-2-sglang-jkl012",
        "finish_reason": "stop",
        "input_tokens": 7,
        "output_tokens": 5,
        "completion_time_ms": 315
      }
    },
    {
      "text": "Translate to German: Hello world\n\nHallo Welt",
      "meta": {
        "id": "batch-3-sglang-mno345",
        "finish_reason": "stop",
        "input_tokens": 7,
        "output_tokens": 5,
        "completion_time_ms": 310
      }
    }
  ],
  "batch_meta": {
    "total_requests": 3,
    "successful": 3,
    "total_time_ms": 895,
    "avg_radix_cache_hit_rate": 0.67
  }
}

{
  "text": "\n\n# Now create a memoized version:\n\ndef fibonacci_memo(n, memo={}):\n    if n in memo:\n        return memo[n]\n    if n <= 1:\n        memo[n] = n\n        return n\n    memo[n] = fibonacci_memo(n-1, memo) + fibonacci_memo(n-2, memo)\n    return memo[n]\n\n# This version is much more efficient for large n\nprint(fibonacci_memo(50))  # Much faster than the recursive version",
  "meta": {
    "id": "extend-sglang-pqr678",
    "rid": "conversation-789",
    "finish_reason": "stop",
    "input_tokens": 8,
    "output_tokens": 82,
    "completion_time_ms": 650,
    "radix_cache_hit": true,
    "context_length": 94
  }
}

{
  "id": "chatcmpl-vision-sglang-stu901",
  "object": "chat.completion",
  "created": 1705334400,
  "model": "llava-hf/llava-1.5-13b-hf",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "This diagram shows a microservices architecture pattern with the following key components:\n\n1. **API Gateway**: Acts as the entry point routing requests to appropriate services\n2. **Load Balancer**: Distributes traffic across multiple service instances\n3. **Service Mesh**: Handles inter-service communication with features like service discovery, load balancing, and monitoring\n4. **Container Orchestration**: Manages deployment and scaling of containerised services\n5. **Shared Data Storage**: Each microservice has its own database following the database-per-service pattern\n\nThis architecture promotes scalability, fault isolation, and independent deployment of services."
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 1024,
    "completion_tokens": 118,
    "total_tokens": 1142
  },
  "vision_metrics": {
    "image_processing_time_ms": 145,
    "vision_encoding_tokens": 1012
  }
}

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [0.0234, -0.0567, 0.0891, ...]
    }
  ],
  "model": "BAAI/bge-large-en-v1.5",
  "usage": {
    "prompt_tokens": 12,
    "total_tokens": 12
  }
}