Skip to content

feat: async + multi-source create_benchmark - #470

Merged
luke-e-schaefer merged 5 commits into
masterfrom
async-benchmark-create
Aug 11, 2026
Merged

feat: async + multi-source create_benchmark#470
luke-e-schaefer merged 5 commits into
masterfrom
async-benchmark-create

Conversation

@luke-e-schaefer

@luke-e-schaefer luke-e-schaefer commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Stacked on #467.

What & why

Brings the SDK's create_benchmark() in line with two server changes to POST /nucleus/benchmarks:

  1. Async — the server creates the benchmark in a "building" state and streams members in via a background job, responding 202 { benchmark_id, job_id } (removes the old ~50k item cap on slice/dataset-sourced benchmarks).
  2. Multi-source — members can come from any combination of item ids, slices, and datasets, unioned + de-duped server-side.

Changes

Async

  • create_benchmark() POSTs, reads { benchmark_id, job_id }, and by default blocks on the build job (reusing the existing AsyncJob poller) then returns the completed "ready" benchmark. Return type unchanged, so existing blocking callers are unaffected. A failed build raises JobError (the poller already treats Errored_Server/Errored_User/etc. as terminal).
  • wait_for_completion=False returns the "building" benchmark immediately for callers who poll themselves; verbose controls poll logging.
  • Benchmark gains a status field ("building" / "ready" / "failed"), parsed in from_json.

Multi-source

  • Adds plural slice_ids and dataset_ids params alongside the singular slice_id/dataset_id/item_ids/items. All sources combine (unioned + de-duped server-side).
  • Requirement relaxed from "exactly one source" to "at least one source".

Meta

  • Version → 0.20.0; CHANGELOG entry (Added: multi-source; Changed: async).
  • Tests (mock-based, no live API): async poll-then-fetch flow, wait_for_completion=False, status parsing, multi-source payload, and at-least-one-source validation.

Notes

  • black/pre-commit/pytest weren't run locally (not installed in this env); CI's format/lint/test jobs are the authority.

🤖 Generated with Claude Code

Greptile Summary

This PR updates create_benchmark() to match two server-side changes: the endpoint now responds 202 with {benchmark_id, job_id} (async build), and it accepts members from multiple sources in a single call. The client adapts by polling the build job via the existing AsyncJob poller before returning the completed benchmark, with a wait_for_completion=False escape hatch for non-blocking callers.

  • Async flow: create_benchmark POSTs, receives {benchmark_id, job_id}, blocks on AsyncJob.sleep_until_complete() by default, then re-fetches the benchmark. A missing job_id with wait_for_completion=True now raises ValueError explicitly (previously discussed thread already addressed).
  • Multi-source: adds slice_ids and dataset_ids list params alongside the singular equivalents; validation relaxed from "exactly one" to "at least one" source; all sources unioned server-side.
  • Benchmark dataclass: gains a status field ("building" / "ready" / "failed"), populated in from_json using a hardcoded "status" string instead of the existing STATUS_KEY constant.

Confidence Score: 5/5

  • Safe to merge. The async flow correctly threads the job poller through the existing AsyncJob infrastructure, the error path for a missing job_id is guarded, and the multi-source payload construction is straightforward. All new paths are covered by mock-based tests.
  • The core logic — POST, extract job_id, poll, re-fetch — is simple and delegates to well-tested existing infrastructure. The only findings are minor style inconsistencies (a hardcoded string key and a duplicate constant) that do not affect runtime behaviour.
  • No files require special attention.

Important Files Changed

Filename Overview
nucleus/init.py Adds slice_ids, dataset_ids, wait_for_completion, and verbose params to create_benchmark; switches from synchronous POST-and-parse to async POST → poll-job → re-fetch pattern; raises ValueError when job_id is absent with wait_for_completion=True. Logic is sound.
nucleus/benchmark.py Adds status field to the Benchmark dataclass and populates it in from_json using a hardcoded "status" string instead of the existing STATUS_KEY constant.
nucleus/constants.py Adds DATASET_IDS_KEY and SLICE_IDS_KEY. SLICE_IDS_KEY = "slice_ids" is a duplicate of the pre-existing DATASET_SLICES_KEY = "slice_ids" at line 49.
tests/test_benchmarks.py Adds tests for async poll-then-fetch, wait_for_completion=False, status parsing, multi-source payload assembly, and at-least-one-source validation. Coverage is comprehensive for the changed paths.
CHANGELOG.md Adds a well-documented v0.20.0 entry covering both the async change and the multi-source addition.
pyproject.toml Version bump from 0.19.1 to 0.20.0, consistent with the breaking-change-aligned changelog entry.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant NucleusClient
    participant Server
    participant AsyncJob

    Caller->>NucleusClient: "create_benchmark(name, slice_ids=[...], wait_for_completion=True)"
    NucleusClient->>NucleusClient: validate at least one source
    NucleusClient->>Server: "POST /benchmarks {name, slice_ids, ...}"
    Server-->>NucleusClient: "202 {benchmark_id, job_id}"
    NucleusClient->>AsyncJob: get_job(job_id).sleep_until_complete()
    loop poll until terminal state
        AsyncJob->>Server: "GET /jobs/{job_id}"
        Server-->>AsyncJob: "status (building → ready | failed)"
    end
    AsyncJob-->>NucleusClient: job complete
    NucleusClient->>Server: "GET /benchmarks/{benchmark_id}"
    Server-->>NucleusClient: "Benchmark (status="ready")"
    NucleusClient-->>Caller: "Benchmark(status="ready")"

    Note over Caller,NucleusClient: wait_for_completion=False skips polling,<br/>goes straight to GET /benchmarks/{id}
Loading

Reviews (6): Last reviewed commit: "P1" | Re-trigger Greptile

Comment thread nucleus/__init__.py Outdated
@luke-e-schaefer luke-e-schaefer changed the title feat: async create_benchmark (poll build job) feat: async + multi-source create_benchmark Jul 28, 2026
Base automatically changed from update-nuc-sdk-for-new-eval-stuff-pt1 to master August 11, 2026 14:24
luke-e-schaefer and others added 3 commits August 11, 2026 14:32
The server now creates a benchmark in a 'building' state and streams its
members in via a background job, responding 202 with {benchmark_id, job_id}
(this removes the previous item-count ceiling on slice/dataset-sourced
benchmarks). Update create_benchmark to match:

- POST, then by default poll the build job to completion (reusing AsyncJob)
  and return the ready benchmark. Return type is unchanged, so existing
  blocking callers are unaffected; a failed build raises JobError.
- Add wait_for_completion (default True) to return the 'building' benchmark
  immediately for callers that want to poll themselves.
- Benchmark now exposes `status` ('building' | 'ready' | 'failed').

Bumps to 0.20.0. Stacked on #467.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the server's multi-source benchmark creation: create_benchmark now
accepts plural slice_ids and dataset_ids alongside the singular slice_id/
dataset_id/item_ids/items, and members from all provided sources are unioned
and de-duplicated server-side. Requirement relaxed from "exactly one source"
to "at least one source".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
9f5ffd4 added the 'server returned no job_id' guard but deleted the
assignment it reads, so every wait_for_completion=True create_benchmark
raised NameError instead of polling.

@jaypsiri jaypsiri left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one thing but other than that this lgtm

Comment thread nucleus/__init__.py Outdated
@luke-e-schaefer
luke-e-schaefer merged commit 6dcfe19 into master Aug 11, 2026
9 checks passed
@luke-e-schaefer
luke-e-schaefer deleted the async-benchmark-create branch August 11, 2026 19:46
luke-e-schaefer added a commit that referenced this pull request Aug 13, 2026
Resolves two version-bump conflicts from #470 (v0.20.0):
- pyproject.toml: 0.19.2/0.20.0 -> 0.20.1
- CHANGELOG.md: keep both sections, retitle the weights entry to 0.20.1

nucleus/__init__.py auto-merged cleanly (master touched create_benchmark,
this branch adds the model-weights methods).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants