Skip to main content
A model’s answers can change with the precision its weights are served in, and providers differ: one serves bf16, another fp8 or fp4. By default, NagaAI sends a request to any provider of the model. To keep a request on the precisions you trust, list them in routing.precision.

Before you begin

Check which precisions your model is served in. In GET /v1/models, the model’s precision lists the precisions its providers publish, such as ["bf16", "fp8"]. A model without precision has no provider that publishes one, so a requirement fails every request to it.

Require a precision

  1. Pick one or more values from the model’s precision. The routing object lists every value.
  2. Add routing.precision with the list.
  3. Send the request:
The response is a normal answer, and output_text holds the model’s text. Chat Completions and Messages take the field the same way, and so do image, video, audio, embedding, moderation, and rerank requests. In a form, the OpenAI SDKs send the list as repeated routing[precision][] parts.

Troubleshooting

400: no provider serves it in a listed precision

None of the model’s providers, and none of your fallback models, publishes a precision you listed. The request fails before any provider is called, and param is routing.precision:
Add a precision from the model’s precision, drop routing.precision, or add a fallback model that is served in a listed precision.

400: unknown precision

A value isn’t one NagaAI knows, such as fp7. Use a value from the model’s precision; The routing object lists every error of routing with its message.

See also