feat: async + multi-source create_benchmark - #470
Merged
Conversation
Base automatically changed from
update-nuc-sdk-for-new-eval-stuff-pt1
to
master
August 11, 2026 14:24
The server now creates a benchmark in a 'building' state and streams its
members in via a background job, responding 202 with {benchmark_id, job_id}
(this removes the previous item-count ceiling on slice/dataset-sourced
benchmarks). Update create_benchmark to match:
- POST, then by default poll the build job to completion (reusing AsyncJob)
and return the ready benchmark. Return type is unchanged, so existing
blocking callers are unaffected; a failed build raises JobError.
- Add wait_for_completion (default True) to return the 'building' benchmark
immediately for callers that want to poll themselves.
- Benchmark now exposes `status` ('building' | 'ready' | 'failed').
Bumps to 0.20.0. Stacked on #467.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the server's multi-source benchmark creation: create_benchmark now accepts plural slice_ids and dataset_ids alongside the singular slice_id/ dataset_id/item_ids/items, and members from all provided sources are unioned and de-duplicated server-side. Requirement relaxed from "exactly one source" to "at least one source". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
luke-e-schaefer
force-pushed
the
async-benchmark-create
branch
from
August 11, 2026 14:35
9f5ffd4 to
a15d1f2
Compare
9f5ffd4 added the 'server returned no job_id' guard but deleted the assignment it reads, so every wait_for_completion=True create_benchmark raised NameError instead of polling.
jaypsiri
approved these changes
Aug 11, 2026
jaypsiri
left a comment
There was a problem hiding this comment.
one thing but other than that this lgtm
luke-e-schaefer
added a commit
that referenced
this pull request
Aug 13, 2026
Resolves two version-bump conflicts from #470 (v0.20.0): - pyproject.toml: 0.19.2/0.20.0 -> 0.20.1 - CHANGELOG.md: keep both sections, retitle the weights entry to 0.20.1 nucleus/__init__.py auto-merged cleanly (master touched create_benchmark, this branch adds the model-weights methods). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #467.
What & why
Brings the SDK's
create_benchmark()in line with two server changes toPOST /nucleus/benchmarks:"building"state and streams members in via a background job, responding 202{ benchmark_id, job_id }(removes the old ~50k item cap on slice/dataset-sourced benchmarks).Changes
Async
create_benchmark()POSTs, reads{ benchmark_id, job_id }, and by default blocks on the build job (reusing the existingAsyncJobpoller) then returns the completed"ready"benchmark. Return type unchanged, so existing blocking callers are unaffected. A failed build raisesJobError(the poller already treatsErrored_Server/Errored_User/etc. as terminal).wait_for_completion=Falsereturns the"building"benchmark immediately for callers who poll themselves;verbosecontrols poll logging.Benchmarkgains astatusfield ("building"/"ready"/"failed"), parsed infrom_json.Multi-source
slice_idsanddataset_idsparams alongside the singularslice_id/dataset_id/item_ids/items. All sources combine (unioned + de-duped server-side).Meta
wait_for_completion=False,statusparsing, multi-source payload, and at-least-one-source validation.Notes
black/pre-commit/pytestweren't run locally (not installed in this env); CI's format/lint/test jobs are the authority.🤖 Generated with Claude Code
Greptile Summary
This PR updates
create_benchmark()to match two server-side changes: the endpoint now responds 202 with{benchmark_id, job_id}(async build), and it accepts members from multiple sources in a single call. The client adapts by polling the build job via the existingAsyncJobpoller before returning the completed benchmark, with await_for_completion=Falseescape hatch for non-blocking callers.create_benchmarkPOSTs, receives{benchmark_id, job_id}, blocks onAsyncJob.sleep_until_complete()by default, then re-fetches the benchmark. A missingjob_idwithwait_for_completion=Truenow raisesValueErrorexplicitly (previously discussed thread already addressed).slice_idsanddataset_idslist params alongside the singular equivalents; validation relaxed from "exactly one" to "at least one" source; all sources unioned server-side.Benchmarkdataclass: gains astatusfield ("building"/"ready"/"failed"), populated infrom_jsonusing a hardcoded"status"string instead of the existingSTATUS_KEYconstant.Confidence Score: 5/5
Important Files Changed
slice_ids,dataset_ids,wait_for_completion, andverboseparams tocreate_benchmark; switches from synchronous POST-and-parse to async POST → poll-job → re-fetch pattern; raisesValueErrorwhenjob_idis absent withwait_for_completion=True. Logic is sound.statusfield to theBenchmarkdataclass and populates it infrom_jsonusing a hardcoded"status"string instead of the existingSTATUS_KEYconstant.DATASET_IDS_KEYandSLICE_IDS_KEY.SLICE_IDS_KEY = "slice_ids"is a duplicate of the pre-existingDATASET_SLICES_KEY = "slice_ids"at line 49.wait_for_completion=False, status parsing, multi-source payload assembly, and at-least-one-source validation. Coverage is comprehensive for the changed paths.Sequence Diagram
sequenceDiagram participant Caller participant NucleusClient participant Server participant AsyncJob Caller->>NucleusClient: "create_benchmark(name, slice_ids=[...], wait_for_completion=True)" NucleusClient->>NucleusClient: validate at least one source NucleusClient->>Server: "POST /benchmarks {name, slice_ids, ...}" Server-->>NucleusClient: "202 {benchmark_id, job_id}" NucleusClient->>AsyncJob: get_job(job_id).sleep_until_complete() loop poll until terminal state AsyncJob->>Server: "GET /jobs/{job_id}" Server-->>AsyncJob: "status (building → ready | failed)" end AsyncJob-->>NucleusClient: job complete NucleusClient->>Server: "GET /benchmarks/{benchmark_id}" Server-->>NucleusClient: "Benchmark (status="ready")" NucleusClient-->>Caller: "Benchmark(status="ready")" Note over Caller,NucleusClient: wait_for_completion=False skips polling,<br/>goes straight to GET /benchmarks/{id}Reviews (6): Last reviewed commit: "P1" | Re-trigger Greptile