Skip to content

datarobot_genai.eval.benchmarks.faithfulness

faithfulness

Faithfulness / Groundedness — RAG hallucination check (LLM-as-judge).

For agents that answer from provided context. Each case supplies a context passage which is sent to the agent as part of the prompt (this is a black-box eval — the agent only knows what the prompt carries). A judge then decides whether the agent's answer is fully supported by that context (grounded) or introduces unsupported claims (hallucinated).

Scoring (judge-based): Reuses the built-in binary_qa template with the context injected into the grading criteria. GRADE C (grounded) -> 1.0, GRADE I (hallucinated) -> 0.0.

Dataset fields

input (required) the question asked of the agent context (required) the source passage — sent to the agent in the prompt AND used as the grounding reference for the judge notes (optional) extra grading guidance for the judge

Because the check is "supported by THIS context", a correct-but-ungrounded answer (true in the world, absent from the context) is intentionally scored as a hallucination — that is the RAG failure mode this benchmark exists to catch.

score 

score(sample: ScorerInput) -> dict[str, Any]

Judge whether the response is grounded in the case's context passage.

Source code in datarobot_genai/eval/benchmarks/faithfulness.py 
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
@benchmark(  # type: ignore[untyped-decorator]
    name="faithfulness",
    dataset="cases.jsonl",
    # The context is sent TO THE AGENT here (not just to the judge) — this is a
    # black-box eval, so the agent only knows what the prompt carries. The judge
    # then checks the answer against that same context. `context` is therefore a
    # required dataset field; a case without it fails to render.
    prompt=(
        "Use ONLY the following context to answer the question. "
        "If the context does not contain the answer, say so explicitly.\n\n"
        "Context:\n{context}\n\nQuestion: {input}"
    ),
    endpoint_type="chat",
    extra={"judge": JUDGE},
)
@scorer  # type: ignore[untyped-decorator]
def score(sample: ScorerInput) -> dict[str, Any]:
    """Judge whether the response is grounded in the case's context passage."""
    question = sample.metadata.get("input", "")
    context = sample.metadata.get("context", "")
    notes = sample.metadata.get("notes", "")
    result = judge_score(
        sample,
        template="binary_qa",
        question=question,
        criteria=_criteria(context, notes),
    )
    return _scored(result, "faithfulness")