This directory contains the benchmark test suites and tooling for publishing to Hugging Face Hub.

datasets/
├── suites/                    # Source of truth (human-editable JSON)
│   └── wp-core-v1/
│       ├── execution/         # Code generation tests (one file per category)
│       │   ├── hooks.json
│       │   ├── rest-api.json
│       │   └── ...
│       └── knowledge/         # Multiple choice / short answer tests
│           ├── hooks.json
│           ├── rest-api.json
│           └── ...
├── data/                      # Generated Parquet for HF (gitignored)
│   └── test.parquet
├── export_dataset.py          # Converts suites → Parquet
└── README.md

The harness loads directly from suites/ JSON files:

# wp-bench.yaml
dataset:
  source: local
  name: wp-core-v1

1. Export to Parquet:

   python datasets/export_dataset.py

2. Upload to HF Hub:

   huggingface-cli upload WordPress/wp-bench-v1 datasets/data/

3. Users can then load:

   from datasets import load_dataset
   ds = load_dataset("WordPress/wp-bench-v1", split="test")

1. Create suites/<suite-name>/execution/ and knowledge/ directories
2. Add category JSON files (e.g., hooks.json, rest-api.json) to each directory
3. Follow the schema in existing suites
4. Run python datasets/export_dataset.py to include in Parquet export