Skip to content

datarobot_genai.eval.benchmarks.instruction_following

instruction_following

Instruction Following — does the response obey explicit constraints (judge-free).

Checks structural constraints the prompt asked for: length limits, valid JSON, required/forbidden phrases, regex shape. Deterministic, so it is reproducible and needs no judge. Semantic constraints ("use a professional tone") are out of scope here — use answer_quality for those.

Scoring (judge-free): fraction of specified constraints satisfied, in [0, 1]. With the 0.5 pass threshold, a response must satisfy at least half its constraints to pass; the reason lists which ones failed.

Dataset fields

input (required) the prompt sent to the agent constraints (required) object with any of: max_words / min_words (int) word-count bounds max_chars (int) character-count upper bound must_be_json (bool) response must parse as JSON must_include (str | list[str]) substrings that must appear must_exclude (str | list[str]) substrings that must NOT appear regex (str) pattern that must match somewhere

A case with no constraints is scored inconclusive, not failed.

evaluate_response 

evaluate_response(response: str, metadata: dict[str, Any]) -> dict[str, Any]

Pure scoring logic — importable for unit tests, no judge, no I/O.

Source code in datarobot_genai/eval/benchmarks/instruction_following.py 
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
def evaluate_response(response: str, metadata: dict[str, Any]) -> dict[str, Any]:
    """Pure scoring logic — importable for unit tests, no judge, no I/O."""
    constraints = metadata.get("constraints") or {}
    if not constraints:
        return {"reason": "no constraints specified — cannot score"}

    resp = response or ""
    words = len(resp.split())
    checks: list[tuple[str, bool]] = []

    if "max_words" in constraints:
        try:
            limit = int(constraints["max_words"])
            checks.append((f"max_words<={limit} (got {words})", words <= limit))
        except (TypeError, ValueError):
            checks.append(("max_words (invalid value)", False))
    if "min_words" in constraints:
        try:
            limit = int(constraints["min_words"])
            checks.append((f"min_words>={limit} (got {words})", words >= limit))
        except (TypeError, ValueError):
            checks.append(("min_words (invalid value)", False))
    if "max_chars" in constraints:
        try:
            limit = int(constraints["max_chars"])
            checks.append((f"max_chars<={limit} (got {len(resp)})", len(resp) <= limit))
        except (TypeError, ValueError):
            checks.append(("max_chars (invalid value)", False))
    if constraints.get("must_be_json"):
        checks.append(("must_be_json", _is_json(resp)))
    for needle in _as_list(constraints.get("must_include")):
        checks.append((f"must_include '{needle}'", needle.lower() in resp.lower()))
    for needle in _as_list(constraints.get("must_exclude")):
        checks.append((f"must_exclude '{needle}'", needle.lower() not in resp.lower()))
    if constraints.get("regex"):
        pattern = str(constraints["regex"])
        try:
            checks.append((f"regex /{pattern}/", re.search(pattern, resp) is not None))
        except re.error:
            checks.append((f"regex /{pattern}/ (invalid pattern)", False))

    if not checks:
        return {"reason": "no recognized constraints — cannot score"}

    passed = sum(1 for _, ok in checks if ok)
    value = passed / len(checks)
    failed = [label for label, ok in checks if not ok]
    reason = (
        f"all {len(checks)} constraints satisfied"
        if not failed
        else f"{passed}/{len(checks)} satisfied; failed: " + "; ".join(failed)
    )
    return {"score": value, "instruction_following": value, "reason": reason}