Documentation

An OpenAI-compatible endpoint. Change two lines, keep your SDK.

The API is the chat completions wire format, implemented faithfully rather than approximately. Everything below is the reference for it: parameters, streaming, tools, structured output, caching, and the error and retry contract.

01Quickstart

Two changes to an existing integration

If your application already calls a chat completions endpoint, the migration is a base URL and a key. The samples below are the whole of it, in the three forms most people start from.

Get a key

Keys are issued per environment and carry the scope of the environment they belong to. Set the key as an environment variable rather than passing it inline: every sample here reads it from the process environment, because a key pasted into source is a key that ends up in a git history.

Write to sales with the environments you need and the shape of the workload, and we issue the keys. Rate limits are set from that description, so an accurate one saves you a 429 on your first busy afternoon.

Get an API key

First request

The only Seldon-specific line in any of these is the base URL. Everything else is the standard client you already have.

python

# pip install openaiimport osfrom openai import OpenAI client = OpenAI(    api_key=os.environ["SELDON_API_KEY"],    base_url="https://api.seldon.ai/v1",) response = client.chat.completions.create(    model="seldon-core-1",    messages=[        {"role": "system", "content": "You are a precise technical assistant."},        {"role": "user", "content": "Explain prefix caching in two sentences."},    ],) print(response.choices[0].message.content)print(response.usage.prompt_tokens, response.usage.completion_tokens)

typescript

// npm install openaiimport OpenAI from "openai"; const client = new OpenAI({  apiKey: process.env.SELDON_API_KEY,  baseURL: "https://api.seldon.ai/v1",}); const response = await client.chat.completions.create({  model: "seldon-core-1",  messages: [    { role: "system", content: "You are a precise technical assistant." },    { role: "user", content: "Explain prefix caching in two sentences." },  ],}); console.log(response.choices[0].message.content);

curl

curl https://api.seldon.ai/v1/chat/completions \  -H "Authorization: Bearer $SELDON_API_KEY" \  -H "Content-Type: application/json" \  -d '{    "model": "seldon-core-1",    "messages": [      {"role": "user", "content": "Explain prefix caching in two sentences."}    ],    "temperature": 0.2  }'

Endpoint

Base URL: https://api.seldon.ai/v1
Auth: Bearer token
Content type: application/json
Wire format: OpenAI chat completions

An Anthropic Messages format surface is available on the same host for applications built against that SDK. See the migration section for what does and does not carry across.

Routes implemented

POST /chat/completions
GET /models
GET /models/{id}

Deliberately small. A short surface implemented exactly is worth more than a wide one implemented approximately, and every route we add is a compatibility promise we then have to keep.

02Streaming

Server-sent events, with usage on the final chunk

Set stream to true and the response arrives as SSE frames. The important detail most integrations miss: token counts are absent from a streamed response unless you ask for them, and without them you cannot attribute cost per request.

python

stream = client.chat.completions.create(    model="seldon-core-1",    messages=[{"role": "user", "content": "Describe backpressure in one paragraph."}],    stream=True,    stream_options={"include_usage": True},) for chunk in stream:    # The usage chunk arrives last and carries an empty choices array,    # so guard on choices before indexing into it.    if chunk.choices and chunk.choices[0].delta.content:        print(chunk.choices[0].delta.content, end="", flush=True)     if chunk.usage:        print()        print("prompt", chunk.usage.prompt_tokens)        print("cached", chunk.usage.prompt_tokens_details.cached_tokens)        print("completion", chunk.usage.completion_tokens)

typescript

const stream = await client.chat.completions.create({  model: "seldon-core-1",  messages: [{ role: "user", content: "Describe backpressure in one paragraph." }],  stream: true,  stream_options: { include_usage: true },}); for await (const chunk of stream) {  const delta = chunk.choices[0]?.delta?.content;  if (delta) process.stdout.write(delta);   if (chunk.usage) {    const cached = chunk.usage.prompt_tokens_details?.cached_tokens ?? 0;    console.log("\n", chunk.usage.prompt_tokens, cached, chunk.usage.completion_tokens);  }}

Frame shape

Each frame is a line beginning with a data field carrying one JSON chunk object. The stream terminates with a literal DONE sentinel, which is not JSON and will throw if you parse it unconditionally.

Deltas are partial tokens, not whole words. Do not assume a chunk boundary is a safe place to split for display, and do not run per-chunk post-processing that assumes valid UTF-8 text in isolation.

Why stream

Beyond perceived latency, streaming sidesteps the gateway deadline that produces 504s on long non-streamed generations, and it lets you cancel a generation you no longer need. A cancelled stream stops billing at the tokens already produced.

03Chat completions

Every parameter, and what it actually does

These are the OpenAI chat completions parameters, because that is the contract we implement. Where our behaviour differs from the reference implementation, the difference is stated in the row rather than left to be discovered under load.

Parameter	Type	Default	Behaviour
model	string	required	Model id from the catalog below. An unknown id returns 404 rather than silently falling back to a default, because a silent downgrade is a cost and quality change you did not authorise.
messages	array	required	Ordered conversation. Each element carries a role of system, user, assistant, or tool, plus content. Assistant messages that requested a tool carry tool_calls, and each must be answered by a tool message with the matching tool_call_id before the next completion.
temperature	number, 0 to 2	1	Sampling temperature. Lower is more deterministic. Adjust this or top_p, not both: the two interact and tuning them together makes results hard to reason about. For extraction and classification work, 0 is usually correct.
top_p	number, 0 to 1	1	Nucleus sampling. Restricts sampling to the smallest set of tokens whose cumulative probability reaches top_p. An alternative to temperature rather than a companion to it.
max_tokens	integer	null	Upper bound on generated tokens, excluding the prompt. Null means the remaining context window. We also accept max_completion_tokens, which is the name OpenAI moved to; if both are present the smaller wins.
stream	boolean	false	Emit the response as server-sent events. Set stream_options.include_usage to receive a final chunk carrying token counts, which is otherwise unavailable on a streamed request.
stop	string or array	null	Up to four sequences at which generation halts. The stop sequence itself is not included in the returned content, and finish_reason comes back as stop rather than length.
seed	integer	null	Best-effort determinism. The same seed with identical parameters returns the same completion when the request lands on the same backend configuration. Batching and mixed-precision kernels make this a strong tendency rather than a guarantee, and no provider that tells you otherwise is being straight with you.
response_format	object	{"type": "text"}	Set to json_object for syntactically valid JSON with no shape guarantee, or to json_schema with strict enabled for output constrained to a supplied schema. See the structured output section for which to use.
tools	array	null	Tool definitions the model may call. Each has type function and a JSON Schema parameters object. Tool definitions are part of the cacheable prefix, so serialise them in a stable order.
tool_choice	string or object	"auto"	One of none, auto, or required, or an object naming a specific function. Use required when your application cannot proceed without a call, rather than instructing the model to always call a tool in prose.

Parameters outside this list follow one of two rules. Those that are meaningful but unimplemented here, such as logprobs and n, are rejected with a 400 rather than accepted and ignored, because silently dropping a parameter changes the response in a way the caller cannot detect. Those that are inert on this backend, such as user and the penalty parameters, are accepted so that an existing OpenAI request body does not have to be stripped before it is sent.

04Tool calling

The model asks, your code answers, the model continues

A tool call is a request for information, not an execution. The model returns a name and arguments, your application runs the function, and the result goes back as a tool message. Nothing runs on our side.

python

import json tools = [    {        "type": "function",        "function": {            "name": "get_invoice_status",            "description": "Look up the payment status of a single invoice.",            "parameters": {                "type": "object",                "properties": {                    "invoice_id": {                        "type": "string",                        "description": "Invoice reference, for example INV-4193.",                    },                },                "required": ["invoice_id"],                "additionalProperties": False,            },        },    },] messages = [{"role": "user", "content": "Has invoice INV-4193 been paid?"}] first = client.chat.completions.create(    model="seldon-core-1",    messages=messages,    tools=tools,    tool_choice="auto",) reply = first.choices[0].message if reply.tool_calls:    messages.append(reply)     for call in reply.tool_calls:        args = json.loads(call.function.arguments)        result = get_invoice_status(**args)         messages.append({            "role": "tool",            "tool_call_id": call.id,            "content": json.dumps(result),        })     second = client.chat.completions.create(        model="seldon-core-1",        messages=messages,        tools=tools,    )    print(second.choices[0].message.content)

typescript

const tools = [  {    type: "function" as const,    function: {      name: "get_invoice_status",      description: "Look up the payment status of a single invoice.",      parameters: {        type: "object",        properties: {          invoice_id: { type: "string", description: "Invoice reference." },        },        required: ["invoice_id"],        additionalProperties: false,      },    },  },]; const messages = [{ role: "user" as const, content: "Has invoice INV-4193 been paid?" }]; const first = await client.chat.completions.create({  model: "seldon-core-1",  messages,  tools,}); const reply = first.choices[0].message; for (const call of reply.tool_calls ?? []) {  const args = JSON.parse(call.function.arguments);  const result = await getInvoiceStatus(args);   messages.push(reply, {    role: "tool",    tool_call_id: call.id,    content: JSON.stringify(result),  });}

Getting this right

Answer every tool_call_id before the next completion. A missing tool result is a 400, not a silent skip, because the alternative is a model reasoning over a hole.
Append the assistant message that requested the calls, not just the results. Dropping it loses the model's own record of what it asked for.
Parse arguments defensively. They are model-generated JSON constrained to your schema, and a schema that permits nonsense will receive it.
Keep tool definitions byte-stable between requests. They sit in the cacheable prefix, and re-serialising them in a different key order costs you the whole cache hit.

05Structured output

Two modes, and only one of them guarantees a shape

json_object gives you syntactically valid JSON of unspecified structure. json_schema with strict enabled constrains decoding to your schema, so the parse cannot fail and the keys cannot drift. Use the second unless you have a reason not to.

python

schema = {    "name": "incident_summary",    "strict": True,    "schema": {        "type": "object",        "properties": {            "severity": {"type": "string", "enum": ["sev1", "sev2", "sev3"]},            "component": {"type": "string"},            "root_cause": {"type": "string"},            "impact_minutes": {"type": "integer"},        },        "required": ["severity", "component", "root_cause", "impact_minutes"],        "additionalProperties": False,    },} response = client.chat.completions.create(    model="seldon-core-1",    messages=[        {"role": "system", "content": "Extract a structured incident summary."},        {"role": "user", "content": transcript},    ],    response_format={"type": "json_schema", "json_schema": schema},    temperature=0,) summary = json.loads(response.choices[0].message.content)

typescript

const response = await client.chat.completions.create({  model: "seldon-core-1",  messages: [    { role: "system", content: "Extract a structured incident summary." },    { role: "user", content: transcript },  ],  response_format: {    type: "json_schema",    json_schema: {      name: "incident_summary",      strict: true,      schema: {        type: "object",        properties: {          severity: { type: "string", enum: ["sev1", "sev2", "sev3"] },          component: { type: "string" },          root_cause: { type: "string" },          impact_minutes: { type: "integer" },        },        required: ["severity", "component", "root_cause", "impact_minutes"],        additionalProperties: false,      },    },  },  temperature: 0,}); const summary = JSON.parse(response.choices[0].message.content ?? "{}");

Constraints worth knowing

Strict mode: Schema subset
Required keys: All of them
additionalProperties: Must be false
Recursion: Not supported

Strict mode accepts a subset of JSON Schema. Every property must appear in required, additionalProperties must be false on every object, and recursive references are rejected with a 422 naming the path. Optional fields are expressed as a union with null rather than by omission from required.

A caveat

Constrained decoding guarantees the shape, not the content. A schema-valid response can still be wrong, and the enum that forces a severity label will happily force the wrong one. Structured output removes a class of parsing failures, not the need to evaluate.

06Prompt caching

The cheapest token is one you have already paid to read

Cached input is billed at 10% of the input rate on every SKU in the catalog. On an agentic or retrieval workload with a large stable preamble, this is routinely the largest single line in the bill, and it is the one most often left untouched.

Caching is on by default and there is nothing to enable. The mechanism is a longest-common-prefix match: we hash the request from the first token forward and reuse the computed attention state for however much of the prefix matches a recent request. The moment a byte differs, the match stops, and every token after that point is billed as fresh input regardless of whether it appeared before.

That single property determines everything about how to structure a prompt. Stable content goes first, variable content goes last, and anything that changes per request must never sit in front of anything that does not. The ordering below is the whole technique.

cacheable ordering

messages = [    # Stable prefix. Identical bytes on every request, so this is a    # cache hit after the first call and bills at the cached rate.    {"role": "system", "content": SYSTEM_PROMPT},    {"role": "user", "content": STYLE_GUIDE},    {"role": "user", "content": ACCOUNT_POLICY},     # Variable tail. Everything from here down is fresh input, which    # is correct: it genuinely differs per request.    {"role": "user", "content": retrieved_documents},    {"role": "user", "content": question},]

what breaks it

# 1. A timestamp in the system prompt. This changes the first block,#    so nothing after it can match and the whole prompt bills as input.system = f"You are a support agent. The current time is {now}." # 2. Retrieved documents placed before the static instructions. The#    documents differ per request, so the instructions behind them are#    never reachable as a cached prefix.messages = [{"role": "user", "content": docs}, {"role": "system", "content": SYSTEM_PROMPT}] # 3. Tool definitions serialised from an unordered structure. Same#    tools, different key order, different bytes, no hit.tools = json.dumps(tool_map.values()) # 4. A session id or request id injected into the preamble for tracing.#    Put it in a header or a trailing message instead.

Cache entries are scoped to your account and expire on a short idle window, so a workload with steady traffic keeps its prefix warm and a workload with sporadic traffic will pay full input on the first request of each burst. There is no charge for writing an entry: unlike some vendors, a cache write is billed as ordinary input rather than at a premium multiple.

Cached input rates

Seldon Prime 1: $0.160 / 1M
Seldon Core 1: $0.120 / 1M
Seldon Vector 1: $0.012 / 1M
Seldon Horizon 1: $0.065 / 1M

Per million cached input tokens, against the standard input rate for the same SKU listed in the model ids section below. These are the rates you are billed at.

Worked example

A 12,000 token stable preamble on Seldon Core 1 costs $0.014 per request at the input rate and $0.0014 on a cache hit. At ten million requests a month, that difference is the whole argument for getting the prefix ordering right.

Model your own cache hit rate

07Rate limits and errors

Which failures are worth retrying

Half of these are permanent and retrying them is a way of turning one failed request into six. The retry column is the operative part of this table: only 429 and the 5xx class should ever be resent unchanged.

Status	Type	Meaning	Retry	Notes
400	invalid_request_error	Malformed JSON, an unknown parameter, or a messages array that fails validation.	Fix, then resend	Identical retries will fail identically. The error body names the offending field.
401	authentication_error	Missing, malformed, or revoked API key.	Do not retry	Check the Authorization header carries a Bearer prefix and that the key has not been rotated.
403	permission_error	Valid key without entitlement to this model or endpoint.	Do not retry	Usually a model outside your plan, or a region-restricted deployment.
404	not_found_error	Unknown model id or endpoint path.	Do not retry	List available ids from the models endpoint rather than hardcoding a string you have not verified.
413	request_too_large	Prompt plus max_tokens exceeds the model context window, or the body exceeds the transport limit.	Fix, then resend	Truncate the prompt or move to a longer-context SKU. Retrying unchanged wastes a round trip.
422	unprocessable_request	Structurally valid but semantically rejected, most often an unsupported JSON Schema construct.	Fix, then resend	Strict schemas reject recursive references and several keyword combinations. The body names the path.
429	rate_limit_error	Request or token rate exceeded for your key.	Retry with backoff	Honour Retry-After when present. Back off before it if the header is absent.
500	api_error	Unhandled fault on our side.	Retry with backoff	Safe to retry. If it persists past a few attempts, send us the request id from the response headers.
503	overloaded_error	Capacity saturated. The request was not processed.	Retry with backoff	Nothing was billed and nothing was generated, so a retry is a clean repeat rather than a duplicate.
504	timeout_error	The request exceeded the gateway deadline before completing.	Retry with backoff	Long non-streamed generations are the usual cause. Streaming avoids the deadline entirely.

Error bodies follow the OpenAI shape: an error object carrying type, message, and where applicable param and code. A 503 means the request was never processed, so nothing was billed and a retry is a clean repeat rather than a risk of duplicate work.

Backoff that does not make the outage worse

Exponential backoff with full jitter, capped, with a bounded attempt count. The jitter matters more than the exponent: a fleet of clients backing off on identical schedules re-synchronises into a thundering herd and produces exactly the load spike the backoff was supposed to relieve.

python

import random, timefrom openai import APIStatusError RETRYABLE = {429, 500, 502, 503, 504} def with_retry(fn, attempts=5, base=0.5, cap=30.0):    for attempt in range(attempts):        try:            return fn()        except APIStatusError as error:            if error.status_code not in RETRYABLE or attempt == attempts - 1:                raise             # Honour the server's own instruction when it gives one.            after = error.response.headers.get("retry-after")            if after:                delay = float(after)            else:                # Full jitter: uniform over the whole window, not the edge.                delay = random.uniform(0, min(cap, base * (2 ** attempt)))             time.sleep(delay)

Rate limit headers

x-ratelimit-limit-requests
x-ratelimit-remaining-requests
x-ratelimit-reset-requests
x-ratelimit-limit-tokens
x-ratelimit-remaining-tokens
x-ratelimit-reset-tokens
retry-after

Present on every response, not only on a 429. Limits are enforced on requests and on tokens independently, and a high-volume workload usually meets the token limit first. Reading the remaining-tokens header lets a client shed load before it starts collecting errors.

Request ids

Every response carries a request id header. Log it on failures. It is the difference between a support thread that resolves in one reply and one that starts with us asking you to reproduce the problem.

08Model ids

The strings to put in the model field

Ids are stable and are not aliased to a moving target. We will not silently repoint an id at a different model, because a change you did not authorise to the thing generating your output is not an upgrade, it is an incident.

Model id	Name	Context	Input / 1M	Cached / 1M	Output / 1M	Basis
seldon-prime-1	Seldon Prime 1	1.0M	$1.60	$0.160	$3.60	list
seldon-core-1	Seldon Core 1	1.0M	$1.20	$0.120	$3.60	list
seldon-vector-1	Seldon Vector 1	131K	$0.120	$0.012	$0.480	list
seldon-horizon-1	Seldon Horizon 1	1.0M	$0.650	$0.065	$3.20	list

Ids and rates are read directly from the catalog that drives the models page, so the two cannot disagree. Every rate is the list rate the endpoint bills at. The models endpoint returns this same list at runtime, and reading it there is better practice than hardcoding a string from a documentation page.

seldon-vector-1

Classification, extraction, routing, and the high-volume tail of an application. Most production traffic belongs here and is not sent here.

seldon-core-1

The default. General reasoning, drafting, summarisation, and tool-using agents that do not need the top tier.

seldon-prime-1

Hard multi-step reasoning and long-horizon agentic work where a weaker model fails rather than merely underperforms.

seldon-horizon-1

Whole-repository and whole-corpus work. Billed flat across the full window with no threshold surcharge and no retroactive repricing of the request.

09Migration

What carries over, and what does not

Compatibility claims are only useful when they name their boundary. Below is the boundary, on both surfaces. If something you depend on is in a right-hand column, the migration is larger than a base URL change and you should know that before you start.

From OpenAI

The closest migration available. Change base_url and the key, change the model id, and re-run your evaluation set. The third step is the one that takes the time.

Carries over

The SDK you already useThe official OpenAI Python and Node libraries work unmodified. Change base_url or baseURL and the API key, and leave the rest of your call sites alone.
Chat completions and streamingRequest and response shapes match, including choices, finish_reason, usage, and the server-sent event framing terminated by a DONE sentinel.
Tools and structured outputFunction tool definitions, tool_calls, tool_choice, json_object mode, and strict json_schema all use the same shapes.
Prompt caching, without the annotationsPrefix caching applies automatically. There is nothing to add to a request to enable it, and nothing to remove from an OpenAI request either.

Does not carry over

The Responses API and AssistantsWe implement chat completions. If your application has moved to the Responses API or is built on Assistants, threads, or the file and vector store endpoints, that is a rewrite rather than a base URL change.
Non-text modalitiesEmbeddings, audio, image generation, and moderation endpoints are not part of the catalog. Those calls keep going to whoever serves them for you today.
Fine-tuning and batch jobsThere is no fine-tuning job API and no asynchronous batch endpoint with a discounted rate. Dedicated capacity is the answer to sustained batch volume instead, and it is priced differently.
Model names and their behaviourModel ids differ, and so do the models. Prompts tuned hard against one vendor's model will need re-evaluating against ours, which is true of any provider switch and is the part most often underestimated.

From Anthropic

Supported through the Anthropic Messages format surface, so an existing SDK keeps working. The cost comparison needs more care here than the code does, because the tokenizers differ.

Carries over

The Messages wire formatThe Anthropic-format surface accepts the same request shape, so an SDK pointed at our endpoint keeps working, including content blocks and the system parameter as a top-level field rather than a message.
Tool usetool_use and tool_result blocks map onto the same underlying mechanism, and multi-turn tool loops behave the same way.
Streaming semanticsEvent-per-block streaming with content_block_delta and message_delta is preserved, including the final usage figures on message_delta.

Does not carry over

Explicit cache_control breakpointsAnthropic asks you to mark cache breakpoints on specific blocks. We cache on longest matching prefix automatically. Breakpoint annotations are accepted and ignored, which means a request tuned for Anthropic caching still caches here, but the tuning lever is different.
Extended thinking blocksThere is no thinking block type with its own budget parameter. Reasoning behaviour is a property of the model rather than a request-level toggle, so prompts that rely on a thinking budget will need reworking.
Token counts, and therefore your cost modelTokenizers differ. Anthropic's newer models produce materially more tokens for the same text than open-weight tokenizers do, so a per-token rate comparison that does not adjust for tokenizer inflation will understate the saving. Compare bills on the same corpus, not rate cards.
Anthropic-specific capabilitiesComputer use tools, citations, PDF input, and the message batches API have no equivalent here. If your application depends on one, that path stays where it is.

The honest version

Wire compatibility makes the integration cheap. It does not make the models identical, and no provider claiming a drop-in replacement is telling you the whole story. Prompts tuned hard against one model will behave differently against another, and the only way to find out how differently is to run your own evaluation set rather than to read a benchmark table. Budget for that work. It is usually a larger line item than the code change, and it is the step that determines whether the saving is real.

Bring your evaluation set, not your benchmark scores.

Send a representative sample of your prompts and we will run them across the catalog, report quality against your own criteria, and tell you plainly if the cheaper tier does not clear your bar.

Talk to an engineer See the catalog