Skip to content

datarobot_genai.eval.benchmarks.answer_correctness

answer_correctness

Answer Correctness — deterministic match against a known answer (judge-free).

The regression-testing workhorse: when you DO have ground-truth answers, check the agent's response against them without paying for a judge model. Fully deterministic, so results are reproducible run-to-run.

Scoring (judge-free): 1.0 on match, 0.0 otherwise, by match_mode: exact response equals reference after trimming whitespace normalized equal after lowercasing + stripping punctuation/extra spaces (the default — tolerant of formatting noise) contains normalized reference appears somewhere in the normalized response

Dataset fields

input (required) the prompt sent to the agent ideal_response (required) the ground-truth answer to match against match_mode (optional) exact | normalized | contains (default normalized)

A case with no ideal_response is scored inconclusive, not failed.

evaluate_response 

evaluate_response(response: str, metadata: dict[str, Any]) -> dict[str, Any]

Pure scoring logic — importable for unit tests, no judge, no I/O.

Source code in datarobot_genai/eval/benchmarks/answer_correctness.py 
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
def evaluate_response(response: str, metadata: dict[str, Any]) -> dict[str, Any]:
    """Pure scoring logic — importable for unit tests, no judge, no I/O."""
    reference = metadata.get("ideal_response")
    if reference is None or not str(reference).strip():
        return {"reason": "no ideal_response provided — cannot score"}

    mode = str(metadata.get("match_mode") or "normalized").lower()
    if mode not in _VALID_MODES:
        mode = "normalized"

    resp = response or ""
    ref = str(reference)
    if mode == "exact":
        ok = resp.strip() == ref.strip()
    elif mode == "contains":
        ok = _normalize(ref) in _normalize(resp)
    else:  # normalized
        ok = _normalize(resp) == _normalize(ref)

    value = 1.0 if ok else 0.0
    reason = f"{mode} match" if ok else f"no {mode} match"
    return {"score": value, "correctness": value, "reason": reason}