Controlled measurement of how prompt format (markdown, plain text, prose, tabular), instruction count, and context length jointly affect large language model performance on two concrete, quantified tasks: instruction-following decay under increasing rule counts, and hallucination rates under increasing context length.

This work is currently under review. A full citation (arXiv ID, DOI) will be added here upon publication.

In the meantime, please cite the repository directly:

Eliav, N. (2026). VeyraBench: A Benchmark for Format, Scale, and Context Length Effects on LLM Instruction Adherence and Hallucination [Software]. Machine Human Intelligence Lab. Retrieved from https://gitlab.com/iNetanel/veyrabench

Working title of the associated paper (pre-publication): "Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models"

This repository contains a deterministic benchmark harness and a fully synthetic corpus designed to measure format sensitivity in large language models under controlled, reproducible conditions.

Experiment 1 (Instruction-Following Decay): Rules (N = 10, 20, 40, 80, 120, 160) are rendered in four formats (markdown, plain text, prose, tabular) and placed in either the system prompt or user turn (20 trials per cell). Every rule is programmatically checked against model responses. Output: adherence-vs-N decay curves per format per placement per model, measuring whether structured formatting and prompt placement both affect instruction compliance as rule count increases. Total: 960 calls/model.

Experiment 2 (Hallucination on Long Context): A deterministic ~512k-token synthetic corpus (the "Book of Veyra," 8,780 entities: solar systems, guilds, creatures) is rendered in four content-identical formats: markdown, plain text, prose, and tabular markdown. Probes (20 recall questions, 20 false-premise sycophancy tests, 20 absent-fact fabrication tests) are posed at six context rungs (2k, 16k, 64k, 128k, 256k, 512k tokens), with three needle depths (10%, 50%, 90%). The four-format design separates syntactic markup from structural organization, allowing testing whether structure per se or specific structural forms drive performance differences.

Both experiments are fully deterministic and reproducible from seed 42 (Experiment 1 cases, Experiment 2's Book of Veyra).

benchmark.py                 - Single-file benchmark harness (gen/run/score/analyze commands)
config.example.json          - Template configuration; copy to config.json and fill in your API endpoints
config.json                  - [NOT included in public release; contains internal infrastructure]
cases/
  exp1_cases.jsonl           - Generated Exp 1 rule sets and essay tasks (deterministic, seed=42)
  exp2_cases.jsonl           - Generated Exp 2 question/answer pairs (deterministic, seed=42)
book/
  book_markdown.txt          - Book of Veyra in markdown format
  book_plain.txt              - Book of Veyra in plain text format
  book_prose.txt              - Book of Veyra in prose format
  book_table.txt              - Book of Veyra in markdown table format
  meta.json                  - Entity count, token counts per format, answer key
results/
  raw_exp1_<model>.jsonl     - Raw Exp 1 API responses per model (append-only, resume-safe)
  raw_exp2_<model>.jsonl     - Raw Exp 2 API responses per model (append-only, resume-safe)
  scored_*.csv               - Exp 1 and Exp 2 results scored per case
  summary_*.csv              - Aggregate results per (model, format, rung, probe_type)
README.md                    - [This file]
REPRODUCING.md               - Step-by-step reproduction walkthrough for third parties
METHODOLOGY.md               - Full experimental design and analysis plan
LICENSE                      - MIT License (applies to benchmark.py)
CITATION.cff                 - Citation File Format metadata
requirements.txt             - Python dependencies (pinned versions)

pip install -r requirements.txt

Tested on Python 3.12.

cp config.example.json config.json

Open config.json and replace placeholder base URLs with your actual API endpoints:

• https://YOUR-ANTHROPIC-GATEWAY-HOST/... → your Anthropic-compatible gateway
• http://YOUR-OLLAMA-HOST:11434/v1 → your Ollama (or compatible) host

Set required environment variables:

export AI_GATEWAY_API_KEY="your-api-key"
export OLLAMA_API_KEY="your-ollama-key"  # if using local Ollama

See REPRODUCING.md for detailed configuration instructions.

Run a cheap test on a single model before committing to a full experiment:

python benchmark.py run --exp 1 --model sonnet-5 --limit 5
python benchmark.py run --exp 2 --model sonnet-5 --rungs 2k --limit 5

Scan the output for any API errors. (A few failed requests are normal and will be retried on the next run; see REPRODUCING.md for details.)

Full Experiment 2 runs can consume several thousand API calls per model. See REPRODUCING.md section "Cost and Time" for realistic estimates before attempting a full run.

python benchmark.py run --exp 1 --model sonnet-5
python benchmark.py run --exp 2 --model sonnet-5
# Repeat for other models...

python benchmark.py score
python benchmark.py analyze

The repository includes:

• Code and deterministic generation: benchmark.py, config.example.json, cases/, book/ — fully reproducible from seed 42.
• Precomputed results (data collection complete, all 5 models): results/raw_exp1_*.jsonl, results/raw_exp2_*.jsonl, results/scored_*.csv, results/summary_*.csv — these are the raw archived data for the paper and the derived analysis tables.
• Figures: figures/ — the exact plots used in the paper, plus figures/make_figures.py to regenerate them from results/.

To reproduce results yourself:

1. Follow the Quick Start steps above.
2. Run the experiments as described in REPRODUCING.md.
3. Raw outputs will be appended to results/raw_exp{1,2}_*.jsonl (resume-safe).
4. Run score and analyze to regenerate summary tables.

See REPRODUCING.md for a complete step-by-step guide, including:

• Detailed environment and configuration setup
• Determinism verification (byte-identical corpus regeneration)
• Cheap partial runs for verification before full scale
• Interpreting Experiment 2's anchor-set cross-rung methodology (critical for avoiding spurious conclusions)
• Troubleshooting and FAQ

See METHODOLOGY.md for the full experimental design:

• Detailed Experiment 1 and 2 specifications
• Five pre-registered hypotheses (H1–H5)
• Sampling strategy and token-count methodology
• The critical "anchor-set" technique for valid cross-rung comparisons in Experiment 2
• Analysis plan and evaluation metrics

This repository is dual-licensed:

• Code (benchmark.py and any other .py files): MIT License — see LICENSE file for full text.
• Generated data and content (book/, cases/, and results/ once completed): CC-BY-4.0 — reuse with attribution.

See LICENSE for details and copyright information.

1. Gemini Flash (3.5) content-leak issue (intermittent): One observed case contained a full essay response, a mangled tag remnant, and near-duplicate content — evidence of hidden reasoning leaking into visible output. This did not reproduce in three fresh attempts. No preventive fix was found. Mitigation: the score() function includes an automated has_duplicate_content() detector that flags responses with large-block duplicates. The analyze() command prints a per-model summary. Spot-check Gemini Flash output before trusting results — this is a low-frequency, non-deterministic, unresolved limitation.

2. Experiment 2 cross-rung comparison caveat: Each rung's main-set questions draw from a different non-overlapping set of entities (since needle depth is relative to that rung's own slice). At the current sample size (~20 questions per rung per probe type), a single hard or easy item can swing accuracy by 5–10+ percentage points. The only valid apples-to-apples cross-rung comparison is the anchor set — a fixed set of ~20 questions from the 2k rung, re-asked unchanged at every larger rung. Before citing any cross-rung improvement from the main-set summary tables, verify the corresponding anchor-set cells first. See METHODOLOGY.md section "Anchor-Set Cross-Check Methodology" and REPRODUCING.md section "Interpreting Exp 2 Cross-Rung Results Correctly" for full explanation and worked examples.

Author: Netanel Eliav, Machine Human Intelligence Lab (MHIL)

Email: netanel@mhil.org (primary) | inetanel@me.com (alternate)