[Bug]: Kubernetes: run plans list a busy instance on the node dstack tried, not the node its pod runs on #4392

Steps to reproduce

1. Configure the kubernetes backend with a cluster that has two nodes with the same GPU, for example one A100 80GB each, and a fleet with nodes: 0...

2. Use the first node's GPU outside dstack, for example with another workload, so that only the second node has a free GPU.

3. Run a task that keeps the GPU busy:

   type: task
   name: busy
   image: dstackai/base:0.15-base-ubuntu24.04
   commands:
     - sleep 180
   resources:
     gpu: A100

4. While it runs, check dstack ps -v and dstack offer --gpu A100 (or the plan of another dstack apply).

Actual behaviour

dstack tries the first node's offer (trying node-a in the server log), and Kubernetes schedules the pod on the second node. dstack ps -v shows the right node:

 NAME  BACKEND     RESOURCES                                    INSTANCE TYPE  PRICE  STATUS
 busy  kubernetes  cpu=x86:1 mem=1GB disk=10GB gpu=A100:80GB:1  node-b         $0     running

But dstack offer and dstack apply list the run's busy instance on the first node:

 #  BACKEND     RESOURCES                                    INSTANCE TYPE  PRICE
 1  kubernetes  cpu=x86:1 gpu=A100:80GB:1                    node-a         $0
 2  kubernetes  cpu=x86:1 gpu=A100:80GB:1                    node-b         $0
 3  kubernetes  cpu=x86:1 mem=1GB disk=10GB gpu=A100:80GB:1  node-a         $0     busy

Expected behaviour

The busy instance is listed on node-b, as in dstack ps -v.

dstack version

master ead2cb470 and 0.22.4.

Notes

Root cause

1. When a job is assigned, its instance stores the offer dstack tried (1,`` 2).```` The pod can run on another node: its node affinity selects every node with a matching GPU, not the node of the offer.
2. After the pod is scheduled, instances/check.py````` saves the provisioning data that update_provisioning_data() rebuilds from the pod's node, but leaves the stored offer as it was.
3. Plans list busy and idle instances from the stored offer (_get_offers_from_instances()), while dstack ps and dstack fleet use the provisioning data (instance_model_to_instance()).

Only the display is affected: Kubernetes instances end with their jobs, so the stored offer isn't used to place other runs.

Suggested fix

In _get_offers_from_instances(), take the instance type from the provisioning data when it's set, as instance_model_to_instance() does.

Tested

Reproduced on k3s v1.31. node-b is a real node with nvidia.com/gpu: 1 advertised and the GPU Operator's nvidia.com/gpu.product label. node-a is simulated with kwok, with the same label and the nvidia.com/gpu taint GKE adds to GPU nodes, and its GPU is held by another pod. With the suggested fix, both dstack offer and the dstack apply plan list the busy instance on node-b. The tests in src/tests/_internal/server/services/runs, routers/test_runs.py, routers/test_fleets.py and core/backends/kubernetes pass.

It first showed up on GKE with two A100 nodes: a busy run was listed on the node dstack tried, while its pod ran on the other one.

Generated by Claude Code