Skip to content

datarobot_genai.eval.judge

judge

Provider-compatible judge scoring.

A thin wrapper over nemo_evaluator.contrib.byob.judge.judge_score that fixes one cross-provider incompatibility: NeMo's judge always sends BOTH temperature and top_p in the chat payload (judge.judge_call hardcodes top_p with no way to suppress it). Anthropic models — Claude on Bedrock, Vertex, or the direct API — reject that combination with a hard 400:

"`temperature` and `top_p` cannot both be specified for this model.
 Please use only one."

The agent/target path in NeMo's own runner.py already guards this (if top_p is not None); only the judge path does not. We close the gap here without forking the library: NeMo honours sample.config["_judge_session"] as the request session, so we inject a session that drops the redundant top_p before the request leaves the process.

This is lossless. Every judge here runs at temperature=0.0 for determinism, and top_p has no effect at temperature 0 — so dropping it changes nothing on providers that accept both (OpenAI/Azure) while unblocking those that don't.

Usage — benchmarks import judge_score from here instead of from NeMo::

from datarobot_genai.eval.judge import judge_score

judge_score 

judge_score(sample: Any, *args: Any, **kwargs: Any) -> dict[str, Any]

nemo_evaluator judge_score with cross-provider payload sanitizing.

Drop-in replacement: identical signature and return value. It just installs the sanitizing session on sample.config before delegating, so the judge works against Anthropic/Bedrock models in addition to OpenAI/Azure.

Source code in datarobot_genai/eval/judge.py 
118
119
120
121
122
123
124
125
126
127
128
129
@functools.wraps(_judge_score)
def judge_score(sample: Any, *args: Any, **kwargs: Any) -> dict[str, Any]:
    """``nemo_evaluator`` ``judge_score`` with cross-provider payload sanitizing.

    Drop-in replacement: identical signature and return value. It just installs
    the sanitizing session on ``sample.config`` before delegating, so the judge
    works against Anthropic/Bedrock models in addition to OpenAI/Azure.
    """
    # setdefault: never clobber a session the caller deliberately supplied.
    sample.config.setdefault("_judge_session", _get_session())
    # cast: nemo_evaluator ships untyped, so its return is Any to mypy.
    return cast(dict[str, Any], _judge_score(sample, *args, **kwargs))