Skip to content

LLM configuration

Whether using workflow.yaml (llms: blocks) or Python agents, the same routing idea applies: DataRobot LLM Gateway, a model deployment, NIM, or an external provider via LiteLLM.

In Python, each integration exposes the same helpers from its llm submodule—swap the import path only:

Integration Import from
LangGraph datarobot_genai.langgraph.llm
LlamaIndex datarobot_genai.llama_index.llm
CrewAI datarobot_genai.crewai.llm

Values are read from the process environment (and .env in the working directory when the runner loads it). For DataRobot-hosted routes, set DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN (see Example.ipynb).

DataRobot LLM Gateway

Use get_datarobot_gateway_llm() to always route through the gateway regardless of LLM_USE_DATAROBOT_LLM_GATEWAY. Calls go through the gateway URL derived from DATAROBOT_ENDPOINT; use DATAROBOT_API_TOKEN as the API key.

Pick a real gateway model id. Do not rely on the default model name for gateway use: get_datarobot_gateway_llm() falls back to LLM_DEFAULT_MODEL, which might be empty in the environment.

  1. Install the dr CLI plugin that lists models the account can use (get-datarobot-llms):
uv tool install git+https://github.com/carsongee/get-datarobot-llms.git
  1. With DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN set, list ids to pass as datarobot-model:
dr get-llms
  1. Pass a chosen id explicitly (LangGraph example):
from datarobot_genai.langgraph.llm import get_datarobot_gateway_llm

llm = get_datarobot_gateway_llm("datarobot/azure/gpt-5-nano-2025-08-07")

LLM deployment

Use get_datarobot_deployment_llm(deployment_id, ...) to send chat completions to a specific DataRobot deployment. The client uses DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN to build {endpoint}/deployments/{deployment_id}/chat/completions. Pass model_name and parameters like other helpers.

from datarobot_genai.langgraph.llm import get_datarobot_deployment_llm

llm = get_datarobot_deployment_llm("your-deployment-id")

NIM deployment

Use get_datarobot_nim_llm(nim_deployment_id, ...) for an NVIDIA NIM deployment hosted like a DataRobot deployment (same URL shape as get_datarobot_deployment_llm).

from datarobot_genai.langgraph.llm import get_datarobot_nim_llm

llm = get_datarobot_nim_llm("your-nim-deployment-id", model_name="optional-model-id")

External providers (LiteLLM)

Use get_external_llm() for LiteLLM to call external providers (for example OpenAI) using their environment variables (for example OPENAI_API_KEY). Do not rely on the datarobot/ prefix for model names in this mode.

from datarobot_genai.langgraph.llm import get_external_llm

llm = get_external_llm("gpt-4o-mini")

To reach this route via get_llm(), turn the gateway off and unset both LLM_DEPLOYMENT_ID and LLM_NIM_DEPLOYMENT_ID (see get_llm() (environment-driven routing)).

LLM provider fallback (router)

Use router fallback for a primary model/provider plus an ordered list of fallback models/providers. Calls start at primary and fail over automatically when the primary fails.

In workflow.yaml, this uses:

llms:
  datarobot_llm:
    _type: datarobot-llm-router

In Python, this uses get_router_llm(primary, fallbacks, router_settings) from the integration module (datarobot_genai.langgraph.llm, datarobot_genai.crewai.llm, or datarobot_genai.llama_index.llm). Use router_settings={"num_retries": <int>} to control retry count.

For the full agent-component checklist and copy-paste examples, see LLM provider fallback (router).

Extended reasoning

Pass reasoning=True on any get_*_llm() helper (or set reasoning: true under an llms: block in workflow.yaml) to enable provider-specific extended thinking. The library picks a default extra_body from the model name (Anthropic Sonnet, Opus, Gemini, OpenAI/Azure reasoning models) and omits temperature, which is incompatible with thinking on many providers.

Explicit extra_body in parameters or workflow YAML always wins; with reasoning=True only temperature is cleared in that case.

from datarobot_genai.langgraph.llm import get_datarobot_gateway_llm

llm = get_datarobot_gateway_llm(
    "datarobot/anthropic/claude-sonnet-4-6",
    reasoning=True,
)

Lower-level helpers live in datarobot_genai.core.llm_parameters (default_reasoning_extra_body, apply_reasoning_to_parameters).

get_llm() (environment-driven routing)

Prefer the explicit get_* helpers when the route is known. Use get_llm() as a single entry point for one code path steered entirely with configuration. It inspects settings and delegates to the same underlying helpers as those sections.

Routing order:

  1. Gateway if LLM_USE_DATAROBOT_LLM_GATEWAY=true (default).
  2. Else deployment if LLM_DEPLOYMENT_ID is set.
  3. Else NIM if LLM_NIM_DEPLOYMENT_ID is set.
  4. Else external (LiteLLM using provider-specific environment variables).

If both LLM_DEPLOYMENT_ID and LLM_NIM_DEPLOYMENT_ID are set with the gateway off, deployment wins.

These variables control get_llm() specifically:

Variable Role
LLM_USE_DATAROBOT_LLM_GATEWAY When true (default), use the DataRobot LLM Gateway.
LLM_DEPLOYMENT_ID When the gateway is off, use this LLM deployment chat endpoint.
LLM_NIM_DEPLOYMENT_ID When the gateway is off and no LLM deployment id is set, use this NIM deployment.
LLM_DEFAULT_MODEL Default model id when model_name is omitted on get_llm()

Every variable in that table is namespaced by the LLM instance it configures. LLM_ is the prefix for the default instance, which is named llm; an application that registers its own config under a different instance name reads <INSTANCE>_DEPLOYMENT_ID and friends instead. DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN are shared by every instance and are not namespaced.

Example (LangGraph; adjust the import for LlamaIndex or CrewAI):

from datarobot_genai.langgraph.llm import get_llm

llm = get_llm()  # optional: model_name="...", parameters={...}, streaming=True, reasoning=False

In workflow.yaml

The e2e samples usually declare one named LLM and reference it from workflow:

llms:
  datarobot_llm:
    _type: datarobot-llm-component

That component follows the same four outcomes using fields on the block (gateway flag, deployment ids, etc.) and the environment.