This example demonstrates how to evaluate an LLM agent that uses filesystem tools via the Model Context Protocol (MCP) Agent system within the eval-protocol framework. The agent interacts with a Dockerized filesystem environment, and its success is evaluated based on the actual changes to the filesystem state.

• Task: Evaluate an LLM agent's ability to perform filesystem operations (e.g., move, list, create files) using MCP tools.
• Agent: The LLM acts as an agent, provided with filesystem tools (like list_directory, read_file, write_file, move_file) by the MCP agent system.
• Environment: A Dockerized mcp/filesystem server, managed by the RewardKitIntermediaryServer. Each evaluation rollout gets an isolated filesystem instance initialized from a template.
• Evaluation: The reward function (main.py) analyzes the final state of the filesystem (captured via an MCP tool like directory_tree) to determine if the agent successfully completed the task.
• Framework:
  • EvalProtocolIntermediaryServer (from eval_protocol.mcp_agent): Manages and orchestrates the Dockerized filesystem backend. Configured via an MCP agent YAML file.
  • eval-protocol CLI: Orchestrates the overall evaluation flow, including LLM interaction with the MCP agent and calling the reward function. Configured via this example's config.yaml.

examples/mcp_agent_filesystem_rl/
├── README.md                    # This file
├── config.yaml                  # Hydra configuration for this example (points to reward function, dataset, agent settings)
├── dataset.jsonl                # Dataset with filesystem tasks (e.g., move file_to_move.txt)
├── main.py                      # Main reward function (evaluate) that checks final filesystem state
├── test_example.py              # Basic sanity tests for this example's setup
├── user_simulator.py            # (Optional) For crafting multi-turn scenarios
├── templates/                   # Template directory structure for initializing filesystem instances
│   └── workspace/
│       ├── source_files/
│       │   └── important_document.txt
│       └── archive/
│           └── .gitkeep
└── outputs/                     # Generated by eval-protocol (evaluation results, logs)

# Relevant Root Files (optional):
# ├── mcp_agent_config.yaml      # Configuration for the intermediary server (if you create one)

1. Start the EvalProtocolIntermediaryServer (for example, python eval_protocol/mcp_agent/main.py --config path/to/your_mcp_agent_config.yaml). This configuration defines available backends like filesystem_rl_example.
2. The eval-protocol run command is executed with this example's config.yaml.
3. The EvaluationPipeline (configured for agent_type: mcp_agent in config.yaml): a. Connects to the RewardKitIntermediaryServer. b. For each task in dataset.jsonl, it requests the intermediary server to initialize a new filesystem_rl_example instance. The server uses the template from examples/mcp_agent_filesystem_rl/templates/workspace/ to set up an isolated Dockerized filesystem. c. The pipeline discovers available tools (e.g., list_directory, move_file) from this filesystem instance. d. The LLM is prompted with the task and the available tools. e. If the LLM requests a tool call, the pipeline sends it to the intermediary server, which executes it on the dedicated filesystem instance. This can happen multiple times in a rollout. f. After the LLM completes its actions (or a turn limit is reached), the pipeline uses a specified MCP tool (e.g., directory_tree, configured in config.yaml under agent.state_capture_tool) to get the final state of the filesystem instance. g. This final_filesystem_state is passed to the reward function in main.py.
4. The evaluate function in main.py compares this actual final state against the expected outcome for the task and returns a score.
5. The intermediary server cleans up the Dockerized filesystem instance.

• Docker installed and running.
• Python environment with eval-protocol and its dependencies installed.
• FIREWORKS_API_KEY environment variable set if using Fireworks models.

To run this example manually (for debugging or custom flows):

Step 1: Start the RewardKitIntermediaryServer In one terminal, from the root of the eval-protocol repository:

# Activate your virtual environment
source .venv/bin/activate

# Start the server with your MCP agent configuration
python eval_protocol/mcp_agent/main.py --config path/to/your_mcp_agent_config.yaml

Keep this server running.

Step 2: Run the Evaluation using eval-protocol CLI In another terminal, from the root of the eval-protocol repository:

# Activate your virtual environment
source .venv/bin/activate

# Run the evaluation for this example
python -m eval_protocol.cli run --config-path examples/mcp_agent_filesystem_rl --config-name config

Evaluation results, including scores, reasons, and detailed logs, will be saved in the outputs/ directory (by default, in a timestamped subdirectory within outputs/YYYY-MM-DD/).

• mcp_filesystem_rl_results.jsonl: Contains detailed evaluation results for each sample.
• preview_input_output_pairs.jsonl: Contains the conversation history (including tool calls and responses) and ground truth for each sample, useful for debugging agent behavior.

• examples/mcp_agent_filesystem_rl/config.yaml:
  • agent.type: mcp_agent: Specifies that an MCP agent is being used.
  • agent.config_path: "../../../mcp_agent_config.yaml": Path to your MCP agent server configuration.
  • agent.intermediary_server_url: URL of the running RewardKitIntermediaryServer.
  • agent.mcp_backend_ref: "filesystem_rl_example": Tells the pipeline which backend defined in mcp_agent_config.yaml to use for this example.
  • agent.state_capture_tool & agent.state_capture_args: Define which MCP tool (e.g., directory_tree) is used to get the final filesystem state for evaluation.
  • reward.function_path: Points to examples.mcp_agent_filesystem_rl.main.evaluate.

• Tasks: Modify dataset.jsonl to add or change filesystem tasks.
• Initial Filesystem State: Change the files and directories in examples/mcp_agent_filesystem_rl/templates/workspace/.
• Evaluation Logic: Update the evaluate function in main.py to change how success is measured.
• LLM / Agent Behavior: Adjust the system_prompt in config.yaml or the LLM model and generation parameters.
• Backend Tools: If mcp/filesystem is updated or a different filesystem server is used, ensure the tool names and arguments match.