bf16, another fp8 or fp4. By default, NagaAI sends a request to any provider of the model. To keep a request on the precisions you trust, list them in routing.precision.
Before you begin
Check which precisions your model is served in. InGET /v1/models, the model’s precision lists the precisions its providers publish, such as ["bf16", "fp8"]. A model without precision has no provider that publishes one, so a requirement fails every request to it.
Require a precision
-
Pick one or more values from the model’s
precision. The routing object lists every value. -
Add
routing.precisionwith the list. -
Send the request:
output_text holds the model’s text. Chat Completions and Messages take the field the same way, and so do image, video, audio, embedding, moderation, and rerank requests. In a form, the OpenAI SDKs send the list as repeated routing[precision][] parts.
Troubleshooting
400: no provider serves it in a listed precision
None of the model’s providers, and none of your fallback models, publishes a precision you listed. The request fails before any provider is called, and param is routing.precision:
precision, drop routing.precision, or add a fallback model that is served in a listed precision.
400: unknown precision
A value isn’t one NagaAI knows, such as fp7. Use a value from the model’s precision; The routing object lists every error of routing with its message.
See also
- The routing object: every field of
routing. - Routing: why a requirement leaves fewer providers to try.
- Limit the price of a request: the other filter on providers.