This page provides an overview of and links to documentation for various examples demonstrating the capabilities of the Eval Protocol. All documentation for these examples is self-contained within the docs/ folder.
Many examples use Hydra for configuration. Please refer to the specific documentation page for each example for execution instructions.
-
Math with Format and Length Example:
- Combines accuracy, format compliance, and length rewards.
- View Documentation
-
APPS Coding Example:
- Illustrates evaluation for coding problems from the APPS dataset.
- View Documentation (New file to be created)
-
E2B (Code Execution Sandbox) Examples:
- Covers various E2B integration scenarios for sandboxed code execution.
- View Documentation
-
GCP Cloud Run Deployment Example:
- Shows how to deploy an Eval Protocol application on GCP Cloud Run.
- View Documentation (New file to be created)
-
Math Example (GSM8K):
- Focuses on evaluating math word problems using the GSM8K dataset.
- View Documentation
-
Math with Formatting Example:
- Extends math evaluation to handle specific formatting requirements.
- View Documentation (New file to be created)
-
Tool Calling Example:
- Demonstrates evaluation for models that use tools or function calls.
- View Documentation (New file to be created)
-
TRL Integration Example:
- Shows how to integrate eval-protocol functions with the TRL library.
- View Documentation
Note: The examples/metrics/ and examples/test_tasks/ directories in the root examples/ folder contain supporting resources and are not standalone documented examples here.
While the pages above cover specific examples, these general guides might also be useful:
- Reward Functions Overview (To be reviewed/updated)
- Basic Reward Function Concepts (To be reviewed/updated)
- Advanced Reward Function Concepts (To be reviewed/updated)
- See the Developer Guide for comprehensive information.
- Check the Tutorials for step-by-step guides.
- Refer to the API Reference for detailed documentation.