[DE-XXXX] Add NucleusClient.merge_model_runs() - #474
Draft
luke-e-schaefer wants to merge 1 commit into
Draft
Conversation
A benchmark evaluation names a single model run, and a benchmark's items may span several datasets. A model whose predictions were uploaded as separate runs — one per dataset, or one per inference batch — therefore had no single run covering the benchmark, and every uncovered item scored as a false negative. Merging the runs produces one run that does cover it, which can then be passed to create_benchmark_evaluation_v2(). The merge is a full union: predictions are copied, never deduplicated. Colliding annotation_ids are rewritten rather than dropped, and the response reports predictions_copied, predictions_ignored and annotation_ids_rewritten so nothing is lost silently. Wraps POST /v1/nucleus/modelRun/merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
NucleusClient.merge_model_runs(model_run_ids, name, *, model_id=None, metadata=None), wrappingPOST /v1/nucleus/modelRun/merge.Requires scaleapi PR https://github.com/scaleapi/scaleapi/pull/156919, which adds the endpoint.
Why
A benchmark evaluation names a single model run, and a benchmark's items may span several datasets. A model whose predictions were uploaded as separate runs — one per dataset, or one per inference batch — has no single run covering the benchmark, so every uncovered item scores as a false negative. Merge first, then pass the new run to
create_benchmark_evaluation_v2().Semantics
Full union — predictions are copied, never deduplicated, and the source runs are left untouched. If two source runs predict on the same item with the same
annotation_id, the colliding id is rewritten rather than dropped. The response reportspredictions_copied,predictions_ignoredandannotation_ids_rewrittenso nothing is lost silently.Verification
The test suite requires live API keys (
conftest.pyhard-asserts onNUCLEUS_PYTEST_API_KEY), so it could not be run locally, and there is no local backend to exercise the round trip against. Verified offline that the method builds the exact payload the server's Joi schema accepts (model_run_ids,name, optionalmodel_id/metadata), posts tomodelRun/merge, and raises on fewer than two distinct run ids before making a request.Note on the version bump
Bumped to 0.21.0. PR #473 also bumps
pyproject.toml/CHANGELOG.md; whichever lands second will need a trivial rebase on those two files.🤖 Generated with Claude Code