feat: hub search and metadata, trainer job handles, typed deploy options, endpoint diagnostics, Bedrock and S3 helpers - #6533

…s every deploy path

`deploy()` took `tags` only through `**kwargs`, so `inspect.signature` could
not validate it, and two deploy paths dropped it: the fine-tuned model package
path read only `self._tags`, and the Nova path did not read the kwarg at all.
The accumulated tags also had two shapes in one SDK. `add_tags` compared
`tag["Key"]` and raised `KeyError: 'Key'` for the `{"key": ..., "value": ...}`
form that the `sagemaker.core.resources` classes take, and raised `TypeError`
for a `sagemaker.core.shapes.Tag`, while the fine-tuned path converted the
stored tags back to lowercase by hand.

Add `tags` as a named `deploy()` parameter. Normalize every accepted form (a
`{key: value}` mapping, boto-style dicts, core-style dicts, `Tag` objects) once
in `format_tags`, and make `add_tags`, `tag_exists`, `_validate_new_tags`, and
`remove_tag_with_key` go through it. Merge the caller's tags into the builder
before dispatch so the built-model, inference-component, recommendation,
fine-tuned, restricted-package, and Nova paths all tag the endpoint, and
replace the two hand-written case conversions with one helper.

---
X-AI-Prompt: Why does ModelBuilder.deploy drop my tags on a fine-tuned model, and why does add_tags raise KeyError: 'Key'?
X-AI-Tool: claude-code

…isted hub content

`HubContent` had no `get_all`. The resource generator skips a List API when
the list summary lacks a member the Describe input requires, and
`HubContentInfo` omits `HubName`, the name the list call was made with. So
every caller paged `list_hub_contents` on the boto3 client by hand: six places
in the SDK and three scripts in the SageMaker agent skills repeat the same
loop. `HubContentInfo` also carries `OriginalCreationTime`, the model-level
catalog date, which `DescribeHubContent` does not return, and the `HubContent`
class had no field for it, so `ResourceIterator` dropped it. The same gap made
`HubContent.get_all_versions()` build each version without `hub_name`, so the
first `refresh()` failed with `Missing required parameter in input: "HubName"`.

Teach the generator two per-resource tables: list-input members to copy onto
every listed resource (`HubName` -> `hub_name`), and summary-only members to
keep as optional attributes (`OriginalCreationTime`). `ResourceIterator` takes
the copied members as `extra_init_data`. `resources.py` is the regenerated
output: `HubContent.get_all(hub_name, hub_content_type, name_contains,
max_schema_version, creation_time_before, creation_time_after, sort_by,
sort_order, session, region)`, the `original_creation_time` attribute, and the
`hub_name` binding on `get_all_versions()`.

---
X-AI-Prompt: Why does sagemaker-core have no HubContent.get_all, and why does get_all_versions() fail on refresh with a missing HubName?
X-AI-Tool: claude-code

… and document

A hub model's task, license, provider, size, context window, languages,
modalities, model type, capabilities, and recipes live in
`HubContentSearchKeywords` as prefixed strings (`@task:`, `@license:`,
`@recipe:finetuning_sft_lora`, ...) and in the `HubContentDocument` JSON. The
`HubContent` resource exposed the two raw attributes and nothing else, so
`jumpstart/search.py`, `recipe_utils.py`, and `training_session/_utils.py` each
split the keywords for the one field they use, and the SageMaker agent skills
repeat the full parse in four scripts. The hub writes at most 50 keywords per
model, so a model with many languages loses its `@model-size:` and
`@context-window:` keywords while the document still carries the values. No
consumer handled that.

Add `HubContentMixin` in `sagemaker.core.hub_content` and teach the resource
generator a `RESOURCE_MIXINS` table so `resources.py` emits
`class HubContent(HubContentMixin, Base)`. The mixin adds no model field and
no state. Its properties read `hub_content_search_keywords` and
`hub_content_document` at call time: a keyword is authoritative and the
document fills what the cap dropped, so the same resource answers before and
after `refresh()`. `HubContentRecipe` parses `@recipe:` keywords and
`RecipeCollection` entries into type, technique, and strategy, and the resource
answers `supports_technique("sft")` and `fine_tuning_techniques()` directly.

---
X-AI-Prompt: How do I read a hub model's license, size, and supported fine-tuning techniques from HubContent without parsing the search keywords myself?
X-AI-Tool: claude-code

The V3 structured-configuration example passes `use_spot_instances=True` and
`max_wait_time_in_seconds=7200` to `Compute`. `Compute` forbids extra fields,
so the example fails on 3.27.0 with a pydantic `extra_forbidden` error. The
spot flag is `Compute(enable_managed_spot_training=True)` and the wait limit
lives on `StoppingCondition(max_wait_time_in_seconds=7200)`, next to
`max_runtime_in_seconds`.

---
X-AI-Prompt: Why does the migration guide's Compute(use_spot_instances=True) example fail with extra_forbidden?
X-AI-Tool: claude-code

… API

The service indexes hub content for `Search(Resource="HubContent")` and
botocore 1.43.100 added the typed `SearchRecord.HubContent` record, but the
SDK never read it. A caller who wanted the public hub narrowed by license,
provider, task, or model size either walked every model with
`ListHubContents` or wrote the `Search` request, the `Metadata.<Field>` names,
the 20-filter and 1024-character limits, the paging, and the result parsing by
hand. The SageMaker agent skills carry a 955-line adapter for exactly this.

Add `HubContent.search(hub_name, filters, hub_content_type, session, region)`
to `HubContentMixin`. It builds the `And` expression of `HubName`,
`HubContentType`, and the caller's `HubContentFilter` entries, sorts by
`HubContentName` so the pages are stable, pages until the service returns no
`NextToken` or repeats one, and yields each record as a `HubContent` bound to
the session and region, with its keywords and document. `HubContentFilter`
holds one service filter. `HubContentFilter.metadata("license", "apache-2.0")`
maps a `HubContent` attribute name to its `Metadata.<Field>` name, and a
sequence of values selects the `In` operator. The filter checks the operator,
the value length, and the filter count against the service limits before the
request.

---
X-AI-Prompt: How do I search the SageMaker public hub for Apache-2.0 models from Meta without listing every model?
X-AI-Tool: claude-code

Commit 6e9df41 made `format_tags` normalize every tag through `_normalize_tag`,
which raised `TypeError` for a `(key, value)` tuple. `Processor(tags=[("Key",
"Value")])`, `ModelBiasMonitor(tags=...)`, and the feature processor
`to_pipeline(tags=[(key, value)])` callers pass pairs, and three unit tests in
`test_processing.py` and `test_clarify_model_monitoring.py` failed with it.
Accept a two-item tuple or list as one tag.

---
X-AI-Prompt: Why does Processor(tags=[("Key", "Value")]) raise TypeError after the deploy(tags=) change?
X-AI-Tool: claude-code

Callers that want training-loss values, not a PNG, set the MLflow tracking URI from
`training_job.mlflow_config.mlflow_resource_arn` and call `MlflowClient().get_metric_history`
with `training_job.mlflow_details.mlflow_run_id` themselves. The SDK repeated those lines in
`plot_training_metrics` and `get_available_metrics`, and each caller copied the recipe-specific
metric names (`total_loss` for LLMFT recipes, `train/loss` for Verl recipes).

`get_metric_history` returns `Dict[str, List[MetricPoint]]` sorted by step. It accepts a job
object or a job name, defaults to every logged metric, and resolves caller names through
`DEFAULT_METRIC_ALIASES` (`loss`, `val_loss`), first logged candidate wins. An unknown metric
raises `ValueError` that lists the metrics the run has. The tracking-URI and run-id resolution
now lives in one private helper that the three public functions share. The lazy `__getattr__`
in `sagemaker.train` exports `get_metric_history` and `MetricPoint`, and loses its duplicate
`plot_training_metrics` through `get_mlflow_url` branches.

---
X-AI-Prompt: How do I read the training loss history of a SageMaker training job as data instead of a plot?
X-AI-Tool: claude-code

…time

`verify_reward_function` accepted any dict as a sample, sent every sample in one invocation,
and selected the response format only through `is_nova`. The evaluation custom-scorer
contract differs: one invocation per sample, with `id`, `model_response`, `query`,
`response`, and `reference_answer` fields, and the `statusCode` and `body` envelope for Nova
too. The function and `RewardFunctionOutput` were importable only from
`sagemaker.train.common_utils.rlvr_reward_verifier`. `Evaluator.create` deployed a reward
Lambda from a file with `python3.9` (`EVALUATOR_DEFAULT_RUNTIME`) while
`CustomAgentLambda.create` carried its own `python3.12` literal.

`RewardSample` and `EvalRewardSample` are the typed inputs. The `envelope` option (`oss`,
`nova`, `eval_oss`, `eval_nova`) selects the request shape, the response shape, and the
sample model, and `None` keeps the `is_nova` behavior. The options live in
`RewardVerificationOptions`, which the function takes as an object or as keyword arguments,
so `verify_reward_function(reward_function, sample_data, is_nova=False)` still works.
`sagemaker.train` exports `verify_reward_function`, `RewardVerificationOptions`,
`RewardFunctionOutput`, `RewardSample`, and `EvalRewardSample`. `EVALUATOR_DEFAULT_RUNTIME`
is `python3.12`, `Evaluator.create(runtime=...)` reaches the Lambda create call, and
`CustomAgentLambda.create` defaults to the same constant.

---
X-AI-Prompt: How do I test my evaluation custom-scorer Lambda locally with the exact payload the evaluation sends, and why does Evaluator.create deploy python3.9?
X-AI-Tool: claude-code

`LLMAsJudgeEvaluator._validate_custom_metrics_json` only parsed the JSON. A metric whose
`instructions` had no `{{prompt}}` or `{{prediction}}`, an array of 11 metrics, a `ratingScale`
`definition` of 101 characters, or a `value` with neither or the two of `floatValue` and
`stringValue` reached the Bedrock evaluation job and failed there, after the pipeline started.

`LLMAsJudgeEvaluator.validate_custom_metrics(custom_metrics)` is a static method that returns
every violation as a message, an empty list means valid. It accepts the list or its JSON
string. The rules: 1 to 10 metrics, each with a `customMetricDefinition` that has a string
`name`, `instructions` of at most 5000 characters that contain `{{prompt}}` or `{{prediction}}`,
and an optional `ratingScale` whose entries have a `definition` of at most 100 characters and
a `value` with exactly one of `floatValue` or `stringValue`, one value type across the scale.
`_validate_custom_metrics_json` calls it and raises `ValueError` with the joined messages, so
`evaluate()` stops before any upload. Three existing tests passed `[{"name": "test"}]` as a
valid payload and now use a valid metric.

---
X-AI-Prompt: Can the SDK check my LLM-as-Judge custom metrics JSON before I start the evaluation job?
X-AI-Tool: claude-code

…ad metadata

`S3Downloader.read_bytes` downloaded the whole object through
`s3_resource.Bucket(bucket).download_fileobj`, so a caller that wanted the
first megabyte of a dataset, one line at a time, or the object size had to
drop to boto3. Fifteen SageMaker agent skill scripts split `s3://` URIs by
hand, eight call `get_object(Range=...)`, and four call `head_object`, each
without the `ExpectedBucketOwner` spot check that the SDK applies on the
default bucket.

Add `byte_range=(start, end)` to `read_bytes`, inclusive like an HTTP `Range`
header, served through `get_object`. Add `iter_lines`, which decodes
`Body.iter_lines()` and never holds the whole object, and `head`, which
returns a frozen `S3ObjectHead(content_length, last_modified, etag,
content_type)` from `head_object`. The three methods share `parse_s3_url`,
the session's `s3_client` with the same fallback as `Session.read_s3_file`,
and the `_get_account_id_if_default_bucket` spot check. A negative start or
an end before the start raises `ValueError` before any request.

---
X-AI-Prompt: How do I read the first megabyte of an S3 object or stream its lines with the SageMaker SDK instead of boto3?
X-AI-Tool: claude-code

`DatasetFormatDetector.validate_dataset` answered only `True` or `False` for
a local path. It stopped at the first line, named no format, read no `s3://`
source, and knew none of the Nova RLVR, PPO, and CPT shapes. Two SageMaker
agent skill scripts filled the gap with their own detectors (11 and 9 format
names, their own `s3://` parsing, their own `get_object(Range=)` call), and
the two disagree on names for the same shape.

Add `detect_dataset_format(source, options)` in `dataset_format_detector.py`.
It reads at most `sample_size_bytes` (1 MiB default) through
`S3Downloader.read_bytes(byte_range=)` for `s3://` or from the local file,
drops the last partial line, and matches every complete line against
`FORMAT_MATCHERS`: the 7 SDK JSON schemas through `jsonschema`, plus
`nova_rlvr`, `nova_ppo`, and `nova_cpt` as small pure functions. The frozen
`FormatDetectionResult` carries the format, the matched-line ratio, the
sampled line count, `"<line>: <reason>"` entries for the lines that failed,
and every candidate with its ratio. `strict=True` demands a full match.
`strict=False` picks the best candidate above a 0.5 ratio. `.parquet`,
`.csv`, and `.json` keep the detector's whole-file checks.
`SKILL_FORMAT_ALIASES` maps the 19 skill names onto the matcher names.
Export the function, the options, and the result from `sagemaker.ai_registry`.

---
X-AI-Prompt: How do I find out which fine-tuning format a JSONL dataset in S3 uses before I create a DataSet from it?
X-AI-Tool: claude-code

…ndle

Problem: the skill templates call bedrock and bedrock-runtime by hand around BedrockModelBuilder. get_bedrock_candidates.py pages list_foundation_models and list_inference_profiles and joins the two on models[].modelArn. deploy-oss-bedrock.py loops invoke_model for 12 minutes on ModelNotReadyException after the import job completes. deploy-nova-bedrock.py calls converse with the deployment ARN. The SDK opens three Bedrock clients and lists nothing and invokes nothing. deploy() returns a response dict, so the caller picks the ARNs out of it by hand.

Solution: add sagemaker.serve.bedrock_deployment. list_invocable_foundation_models() pages ListFoundationModels filtered by output modality and ListInferenceProfiles filtered to SYSTEM_DEFINED, then joins profile ids to model ids. It keeps a model when inferenceTypesSupported holds ON_DEMAND or an active cross-region profile routes to it. BedrockDeployment.wait_until_ready() probes Converse for a custom model deployment and InvokeModel for an imported model. It retries only ModelNotReadyException and raises a TimeoutError that names the ARN. invoke() does a JSON round trip, and converse() passes inferenceConfig and system through. BedrockModelBuilder.deploy() keeps its return value and sets deployment on every successful path (importedModelArn on the OSS path, modelArn plus customModelDeploymentArn on the Nova path). latest_deployment() raises ValueError before the first deploy(). Field names come from the botocore 1.43.118 bedrock and bedrock-runtime service models.

Tests: sagemaker-serve/tests/unit/test_bedrock_deployment.py fails on master with ModuleNotFoundError: No module named 'sagemaker.serve.bedrock_deployment'. On this branch test_bedrock_model_builder.py and test_bedrock_deployment.py report 124 passed.

---
X-AI-Prompt: Can BedrockModelBuilder list the Bedrock models I can invoke, wait until an import is ready, and invoke it without raw boto3 calls?
X-AI-Tool: claude-code

ModelBuilder held two private copies of the same EC2 lookup. `_get_instance_resources` and `_infer_accelerator_count_from_instance_type` each created an EC2 client, stripped the `ml.` prefix with `str.replace`, called `describe_instance_types` for one name, and read a different slice of the response. No public SDK function listed the SageMaker instance type names or described their hardware, so every caller outside ModelBuilder wrote its own EC2 call.

`sagemaker.core.instance_types` gains `InstanceSpec`, `list_instance_types`, `describe_instance_type`, and `describe_instance_types`. `list_instance_types` reads the `ProductionVariantInstanceType` enum of the loaded SageMaker service model and makes no request. `describe_instance_types` strips the `ml.` prefix, sends at most 100 names per EC2 request (the `RequestInstanceTypeList` limit), reads every page, and maps vCPUs, memory, GPU count, GPU memory, and GPU name. EC2 rejects a whole request with `InvalidInstanceType` when one name does not exist, so the function then asks for each name alone. An unknown type gets `gpu_count=0`, `None` elsewhere, and a warning. Every other `ClientError` propagates.

The two ModelBuilder helpers call `describe_instance_type` with the session's `boto_session` and keep their signatures and fallbacks: `(None, None)` and `None` on an unknown type or a failed call. The existing `test_get_instance_resources` stub now carries `InstanceType`, which the real EC2 response always holds and the mapper keys on. The module joins the `sagemaker-core` API reference.

---
X-AI-Prompt: Can sagemaker-core list and describe SageMaker instance types, so ModelBuilder drops its two private EC2 lookups?
X-AI-Tool: claude-code

`DescribeModelPackageGroup` reports a missing group as a `ValidationException`
whose message ends in `does not exist`, not as a `ResourceNotFound` error, and
`ModelPackageGroup` offered only the generated `get` and `create`. So every
"use the group, or create it" flow parsed `ClientError` codes by hand. Ten
fine-tuning templates in the SageMaker agent skills wrap `create` in a `try`
block and match `e.response["Error"]["Code"]` against
`("ResourceInUse", "ValidationException")` to fall back to `get`, which also
treats an unrelated validation error as "the group exists".

Add `sagemaker.core.model_package_group.ModelPackageGroupMixin` and attach it
to the generated class through the `RESOURCE_MIXINS` table, so `resources.py`
emits `class ModelPackageGroup(ModelPackageGroupMixin, Base)`.
`get_or_create(name, spec=None, session=None, region=None)` calls the
generated `get`, and on a `ValidationException` whose message says the group
is absent, or on a code that starts with `ResourceNotFound`, calls the
generated `create` with the set fields of a `ModelPackageGroupSpec`
(description, tags, managed configuration). Every other error propagates. The
mixin adds no model field and no state, and the returned group keeps the
session and region it was loaded with.

---
X-AI-Prompt: Can ModelPackageGroup get or create a group in one call instead of parsing ClientError codes?
X-AI-Tool: claude-code

…cessor._start_new

`Processor._start_new` built `ProcessingJob(**transformed)` from the create
request. The job carried no `processing_job_arn`, no `processing_job_status`,
and no client context, so `processor.latest_job.refresh()` built its client
from the process default session and polled the default region (aws#5796). Four
processing templates in the SageMaker agent skills call
`describe_processing_job` on a boto3 client after `run()` to read the ARN.

After the submit through the configured session, bind that session and its
region to the job with `_set_client_context` and describe the job once with
`refresh()`, so the returned job has its ARN and status and every later
`refresh()`, `wait()`, and `stop()` talks to the account and region the job
was created in. The `PipelineSession` path still returns the captured step
arguments with no describe call. A `LocalSession` job exists only in the
session's `LocalSagemakerClient`, so that path returns the request-built job
as before and `run()` keeps polling it through the local client.

---
X-AI-Prompt: Why does processor.latest_job have no ARN and refresh() poll the wrong region after run()?
X-AI-Tool: claude-code

A generated `wait()` raised `FailedStatusError` only for a status that contains
`failed`. `Stopped` is a terminal state too, so `TrainingJob.wait()` returned on
a stopped job as if the job completed, and the caller had to read
`training_job_status` after the wait to tell the two apart. Three training
templates in the SageMaker agent skills do that and raise on `Stopped`
themselves.

Add `raise_on_stopped: bool = False` after `timeout` to `WAIT_METHOD_TEMPLATE`
and a `STOPPED_STATUS_ERROR_TEMPLATE` block that the generator formats with the
same failure-reason reference as the failed block. With
`wait(raise_on_stopped=True)` a status that contains `stopped` raises
`FailedStatusError` with the resource type, the status, and the failure reason.
The default keeps the old return. Regenerated `resources.py`: all 15 `wait()`
methods take the parameter.

---
X-AI-Prompt: Can TrainingJob.wait() raise when the job is stopped instead of returning like it completed?
X-AI-Tool: claude-code

When an endpoint misbehaves, an operator reads the endpoint status and
failure reason, the per-variant instance type and counts, the 11 CloudWatch
metrics of the endpoint dashboard, and the latest lines of the
`/aws/sagemaker/Endpoints/<name>` log group. `Endpoint` exposed none of that
beyond `refresh()`, so the SageMaker agent skill `collect_diagnostics.py`
rebuilt the collection with raw boto3 clients from a region string. That
script reads `InstanceType` from `ProductionVariantSummary`, where
`DescribeEndpoint` never returns it, so the instance type is always `None`.
It fetches one unpaged page of 100 log events, and it loses the session a
caller bound the resource with.

Add `EndpointMixin` in `sagemaker.core.endpoint` and register it in the
generator's `RESOURCE_MIXINS` table, so `resources.py` emits
`class Endpoint(EndpointMixin, Base)`. `diagnostics()` refreshes the
resource, reads the instance type per variant from
`EndpointConfig.get(endpoint_config_name)` matched by variant name, sends
the 11 dashboard queries in one `GetMetricData` call keyed
`"<MetricName>/<Stat>"`, and collects the log lines. A CloudWatch, Logs, or
endpoint config failure lands in `errors` instead of raising. Only
`DescribeEndpoint` raises. `recent_logs()` pages `FilterLogEvents` until
the next token is absent, the page is empty, or the limit is reached, and
formats `"<ISO timestamp> <log stream> <message>"` lines. The CloudWatch and
Logs clients come from the session, region, and config the resource is bound
to, so `Endpoint.get(name, session=..., region=...)` reads the metrics and
logs of that account and region. The mixin adds no model field and no state.

---
X-AI-Prompt: How do I get the status, metrics, and recent logs of a SageMaker endpoint from the Endpoint resource without writing my own boto3 calls?
X-AI-Tool: claude-code

The `codestyle-doc-tests (sagemaker-core)` hook fails on this branch and on
`master`. `black --check` rejects `resources.py` because the import block ends
with two newlines and `LOGGER_STRING` opened with a third, so the generator
emitted two blank lines before `logger = ...` where black allows one.
`pydocstyle` reports `D100: Missing docstring in public module` on the two
generated data modules, `config_schema.py` and `code_injection/shape_dag.py`.

Drop the leading newline from `LOGGER_STRING`, and make the generator write a
module docstring at the top of the two data modules. Regenerate the three
files, so the next botocore sync keeps them clean.

---
X-AI-Prompt: Why does the codestyle-doc-tests hook fail with black-check and D100 on generated files, and how do I fix it in the generator?
X-AI-Tool: claude-code

…tation

Three review points on the shared tag helpers and `deploy()`:

- `format_tags({"n": 1})` gave `{"Value": "1"}` while `format_tags([("n", 1)])`
  gave `{"Value": 1}`, so the output type depended on the input shape. Every
  representation now comes out with `str` keys and values. A `PipelineVariable`
  stays intact for the request serializer.
- `deploy(tags=...)` merged the call's tags into the builder for good, so a
  second `deploy()` on the same builder (a retry, another endpoint) carried the
  first call's tags. The merge now lasts for the call. `deploy()` validates,
  merges, runs the path selection in `_deploy_with_options`, and restores the
  builder's tags on return and on raise.
- `_validate_new_tags` returns the merged list in boto form on purpose, so
  `ModelBuilder.add_tags`, its one caller, stores one shape whatever shape it
  received. The docstring now says so.

---
X-AI-Prompt: Can you address the review comments about tag value types and deploy() tags persisting across calls?
X-AI-Tool: claude-code