Skip to content

datarobot_genai.eval.benchmarks.answer_quality

answer_quality

Answer Quality — general-purpose response quality (LLM-as-judge).

The catch-all "is this a good answer?" benchmark. An LLM judge scores each response on a 1-5 Likert scale for helpfulness, coherence, and relevance to the question. Use it for open-ended tasks where there is no single correct answer.

Scoring (judge-based): likert_5 -> 1..5 mapped to 0.2..1.0; pass threshold is 0.5 (grade >= 3).

Dataset fields (user_datasets/*.json): input (required) the prompt sent to the agent notes (optional) extra grading criteria injected into the judge prompt

Configure the judge in the pipeline YAML's judge: block. See sibling judge-free benchmarks (answer_correctness, instruction_following, …) if you want to avoid a judge model entirely.

score 

score(sample: ScorerInput) -> dict[str, Any]

Likert-5 quality judge over the agent's free-form response.

Source code in datarobot_genai/eval/benchmarks/answer_quality.py 
72
73
74
75
76
77
78
79
80
81
82
83
84
85
@benchmark(  # type: ignore[untyped-decorator]
    name="answer_quality",
    dataset="cases.jsonl",  # placeholder; --dataset overrides at runtime
    prompt="{input}",
    endpoint_type="chat",
    extra={"judge": JUDGE},
)
@scorer  # type: ignore[untyped-decorator]
def score(sample: ScorerInput) -> dict[str, Any]:
    """Likert-5 quality judge over the agent's free-form response."""
    question = sample.metadata.get("input", "")
    criteria = sample.metadata.get("notes", "")
    result = judge_score(sample, template="likert_5", question=question, criteria=criteria)
    return _scored(result, "quality")