Repository navigation
[Bug]: Kubernetes: with proxy_jump.port set, runs submitted together can fail with no capacity #4463
Description
Steps to reproduce
-
Configure the
kubernetesbackend with a fixed jump pod port:contexts: - name: my-cluster proxy_jump: port: 30122
-
In a project whose jump pod doesn't exist yet, submit several runs at once, e.g., three
dstack apply -din a row.
Actual
One of the runs fails with failed_to_start_due_to_no_capacity:
(422) Reason: Unprocessable Entity ... Service "dstack-main-ssh-jump-pod-service" is invalid: spec.ports[0].nodePort: Invalid value: 30122: provided port is already allocated
Root cause:
_create_jump_pod_service_if_not_exists()reads the Service and creates it if it's missing. It tolerates only409from a concurrent create.- With a fixed
nodePort, creating a Service that already exists returns422(port already allocated), not409.
Expected
All runs start.
Notes
Only affects the first runs of a project on clusters with proxy_jump.port set.
Fix: if the Service create fails with 422, re-read the Service and continue if it exists. Checked with kubectl: a duplicate create returns 422 with a fixed nodePort and 409 without one.
Activity
ShivanshShukla commentedon Nov 10, 2026
Hi @peterschmidt85! I'd love to pick this up.
This is a great catch regarding the K8s API semantics. Since the Control Plane validates the
nodePortallocation before evaluating the duplicate object conflict, it throws the422 Unprocessable Entityinstead of the expected409 Conflictwhen a race condition occurs during concurrentdstack apply -druns.My Execution Plan:
I will update
_create_jump_pod_service_if_not_exists()to catch the422exception alongside the409. To ensure we don't accidentally swallow legitimate422validation errors, I will add a quick fallback to read the Service again upon catching the422. If the Service exists, we can safely assume it was a concurrent creation and proceed.Let me know if I'm good to open a PR for this!