fix(eval): count capitalized judge verdicts in final_response_match_v2 - #7715

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

• Related: fix(eval): avoid passing unknown alternate judge verdicts #7518 (changes the else branch under the fallback key; this PR leaves that branch alone)

2. Or, if no issue exists, describe the change:

Problem:

_parse_critique in final_response_match_v2.py compares the judge's label case-sensitively, and its capture group ([^"^\]^\s]*) cannot hold a space:

• Under is_the_agent_response_valid, a label in any case other than lowercase ("Valid", "Invalid", "True", "False") and "partially valid" return NOT_FOUND, so that sample is dropped from the majority vote. With five samples, two "valid" and three "Invalid" score 1.0 and pass. Label.PARTIALLY_VALID lists "partially valid", but the regex never lets it through.
• Under the fallback key is_the_agent_response_invalid, "True" and "Invalid" fall through to VALID: the judge says the response is invalid and it is scored as valid.

Solution:

Lowercase the label under both keys, keeping both regexes as they are. When the result is still NOT_FOUND, look for a label of several words separated by single spaces under is_the_agent_response_valid, so "partially valid" counts as INVALID. Every output that yields VALID or INVALID today once its labels are lowercased keeps that result.

Testing Plan

Unit Tests:

• I have added or updated unit tests for my change.
• All unit tests pass locally.

New cases in test_final_response_match_v2.py: capitalized labels, "partially valid", the fallback key with "True" and "Invalid", both keys at once in either order ("not valid" next to "is_the_agent_response_invalid": true), labels that must stay NOT_FOUND ("not valid", "va,lid", ",True"), and test_evaluate_invocations_majority_vote_over_judge_labels, which runs evaluate_invocations with a mocked judge sampled three times for three sets of responses. 7 of the 14 new cases fail on main; the other 7 guard behavior that main already gets right.

pytest tests/unittests/evaluation (locally, Windows, Python 3.12): 1149 passed (1135 on main at 25d1917, plus the 14 new cases).

Full suite: this repository's CI workflow, run on Linux in a fork with this branch merged into current main, passes all 16 jobs: pre-commit, mypy (3.11 to 3.13), and the unit, A2A v0.3 and MCP v2 tests (3.11 to 3.14) (run). The two "pass locally" boxes are unchecked because on Windows the full suite has failures that also occur on main (for example, gcloud.cmd where the tests expect gcloud).

Manual End-to-End (E2E) Tests:

The script below registers a judge in LLMRegistry that returns scripted verdicts, and runs FinalResponseMatchV2Evaluator.evaluate_invocations with 5 samples and a 0.8 threshold. No API key is needed.

import asyncio

from google.adk.evaluation.eval_case import Invocation
from google.adk.evaluation.eval_metrics import EvalMetric
from google.adk.evaluation.eval_metrics import JudgeModelOptions
from google.adk.evaluation.eval_metrics import LlmAsAJudgeCriterion
from google.adk.evaluation.final_response_match_v2 import FinalResponseMatchV2Evaluator
from google.adk.models.base_llm import BaseLlm
from google.adk.models.llm_response import LlmResponse
from google.adk.models.registry import LLMRegistry
from google.genai import types

VERDICTS = iter([])


class ScriptedJudge(BaseLlm):
  """Answers each judge request with the next scripted verdict."""

  model: str = "scripted-judge"

  @classmethod
  def supported_models(cls):
    return [r"scripted-judge"]

  async def generate_content_async(self, llm_request, stream=False):
    text = '{"reasoning": "r", "is_the_agent_response_valid": "%s"}'
    yield LlmResponse(
        content=types.Content(
            role="model", parts=[types.Part(text=text % next(VERDICTS))]
        )
    )


LLMRegistry.register(ScriptedJudge)


def invocation(text):
  return Invocation(
      user_content=types.Content(role="user", parts=[types.Part(text="q")]),
      final_response=types.Content(role="model", parts=[types.Part(text=text)]),
  )


async def run(verdicts):
  global VERDICTS
  VERDICTS = iter(verdicts)
  evaluator = FinalResponseMatchV2Evaluator(
      EvalMetric(
          metric_name="final_response_match_v2",
          criterion=LlmAsAJudgeCriterion(
              threshold=0.8,
              judge_model_options=JudgeModelOptions(
                  judge_model="scripted-judge", num_samples=len(verdicts)
              ),
          ),
      )
  )
  result = await evaluator.evaluate_invocations(
      [invocation("actual")], [invocation("reference")]
  )
  print(f"{verdicts} -> {result.overall_score} {result.overall_eval_status.name}")


async def main():
  await run(["valid", "valid", "invalid", "invalid", "invalid"])
  await run(["valid", "valid", "Invalid", "Invalid", "Invalid"])
  await run(["valid", "valid", "partially valid", "partially valid", "invalid"])


asyncio.run(main())

Checklist

• I have read the CONTRIBUTING.md document.
• I have performed a self-review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• I have added tests that prove my fix is effective or that my feature works.
• New and existing unit tests pass locally with my changes.
• I have manually tested my changes end-to-end.
• Any dependent changes have been merged and published in downstream modules.