multikernel: manage LLC way isolation through device tree - #6

Thanks for working on this. Feedback is mostly about the device tree modelling rather than the implementation, since the DT shape is the part that becomes ABI and is hardest to change later.

llc-way-mask is an allocation, not a resource

Every other resource in this device tree is a request that the host answers. memory-bytes is a size, and the host picks the range and writes memory-base back into the generated instance DTB (dts.c). llc-way-mask inverts that: the DTB author performs the allocation, and the kernel is reduced to validating it. Nothing can repack or defragment masks afterwards, because the caller owns the layout.

The patch already half-admits the mask is not the allocated thing. The genuinely scarce resource is the CLOSID, which is why linux,resctrl-closid has to exist as host-generated write-back with a "must not be supplied in an input DTB" rule attached to it.

Related problems with the property as specified:

No domain dimension. An L3 CBM is per-domain (per package on Intel, per CCX on AMD), but llc-way-mask is a single global mask applied to every domain, and mk_instance_reserve_llc_ways() checks disjointness globally across all instances. On a two-socket box, two instances pinned to different sockets are forbidden from using the same ways even though that is perfectly safe, and an instance with CPUs only on socket 0 still consumes ways in socket 1's mask.

The grant can be revoked under a running instance. Everything here is gated on resctrl_mounted. umount /dev/resctrl calls rdt_kill_sb(), which does resctrl_arch_reset_all_ctrls() on every alloc-capable resource and then rmdir_all_sub(). So the spawn kernel's CBM is reset to the full mask and its CLOSID is freed for reallocation, while it keeps running with that CLOSID in PQR_ASSOC. No other resource in this DT can be reclaimed out from under an instance like that.

The pool is not actually reserved. resctrl_multikernel_set_pool_mask() narrows only the default group. Pre-existing groups keep their masks, and an admin can create a new shareable group overlapping the pool at any time.

It is Intel CAT ABI in the DT. LSB-first CBM numbering, the contiguity rule, cbm_len, arch_has_sparse_bitmasks, and CDP index doubling all leak into what the author has to write. It does not translate to MPAM.

It is runtime policy frozen at boot. There is no way to retune an instance's cache share short of teardown and recreate.

Suggested device tree shape

Express demand, and derive the domain set from the CPUs the instance already owns. There is no new property for domains: the set of L3 domains an instance can fill is exactly the set its CPUs live in, so stating it separately can only create contradictions.

What you write:

/dts-v1/;
/ {
    compatible = "multikernel-v1";

    resources {
        cpus = <0x2 0x3 0x4 0x5 0xa 0xb 0xc>;
        memory-base = /bits/ 64 <0x100000000>;
        memory-bytes = /bits/ 64 <0x40000000>;
        llc-ways = <8>;                 /* per L3 domain, reserved for the pool */
    };

    instances {
        web-server {
            id = <1>;
            resources {
                cpus = <0x2 0x3>;       /* socket 0 */
                memory-bytes = <0x20000000>;
                llc-ways = <4>;
            };
        };

        database {
            id = <2>;
            resources {
                cpus = <0xa 0xb>;       /* socket 1 */
                memory-bytes = <0x20000000>;
                llc-ways = <4>;
            };
        };

        cache-tier {
            id = <3>;
            resources {
                cpus = <0x4 0xc>;       /* spans both sockets */
                memory-bytes = <0x10000000>;
                llc-ways = <2>;
            };
        };
    };
};

What the host generates, per instance. llc-cbm is a list of <cache-id cbm> pairs, one pair per L3 domain the instance's CPUs occupy. On a 16-way machine with the pool resolved to 0xff00:

/* web-server */
resources {
    memory-base = <0x1 0x00000000>;
    memory-bytes = <0x0 0x20000000>;
    cpus = <0x0 0x2 0x0 0x3>;
    llc-ways = <0x4>;
    llc-cbm = <0x0 0x0f00>;
    llc-closid = <0x1>;
};

/* database */
resources {
    memory-base = <0x1 0x20000000>;
    memory-bytes = <0x0 0x20000000>;
    cpus = <0x0 0xa 0x0 0xb>;
    llc-ways = <0x4>;
    llc-cbm = <0x1 0x0f00>;
    llc-closid = <0x2>;
};

/* cache-tier, two domains, one CLOSID */
resources {
    memory-base = <0x1 0x40000000>;
    memory-bytes = <0x0 0x10000000>;
    cpus = <0x0 0x4 0x0 0xc>;
    llc-ways = <0x2>;
    llc-cbm = <0x0 0x3000 0x1 0x3000>;
    llc-closid = <0x3>;
};

Three properties total: llc-ways in, llc-cbm and llc-closid out. They sit in resources next to memory-base, which is already a host-resolved property living beside its request, so no separate node and no new "internal property" convention. llc-cbm/llc-closid are to llc-ways exactly what memory-base is to memory-bytes.

Note that web-server and database both hold 0x0f00 on different sockets. That configuration is impossible under the current global mask and global disjointness check, and it is the common case on a multi-socket host. A multi-domain instance is still one CLOSID with several domain entries, which is precisely what a schemata line expresses (L3:0=3000;1=3000).

What this implies for the resctrl side

The host keeps a per-domain free mask over the pool and carves a contiguous run per domain (lowest-fit, so allocations pack toward the pool's low edge). All domains are carved before anything is committed, so a shortfall on the second socket does not strand ways on the first.

Deriving the domain set must not use get_cpu_cacheinfo_id(): an instance's CPUs are offlined on the host, and free_cache_attributes() drops the cacheinfo on the way down. The persistent topology id (cpu_data(cpu).topo.llc_id on x86) is the same apicid >> index_msb value resctrl uses for hdr.id and survives offline.

That in turn suggests a much smaller resctrl surface than this patch takes. Instead of resctrl_multikernel_create_group() calling rdtgroup_mkdir_ctrl_mon() and then re-scanning rdt_all_groups by name string to recover the group it just created, multikernel needs roughly: query L3 caps, reserve the pool mask, allocate/free a CLOSID, and program a per-domain CBM set on that CLOSID. No kernfs driving from inside the kernel, and no requirement that resctrl be mounted before a DTB upload works. The pool reservation is the one genuinely new resctrl concept, a mask that schemata writes and mkdir refuse to hand out, and that is a far smaller and more upstreamable ask than the current approach.

Domains an instance does not occupy can be filled with the host complement mask. No CPU on that socket ever carries that CLOSID, so the value is inert and costs zero pool ways. This is also why instance CLOSIDs should stay shareable: RDT_MODE_EXCLUSIVE would force real ways to be burned on sockets the instance never touches.

Two things fall out on the x86 side:

• The CBM writes in mk_park_identity() can go away, along with resctrl_l3_mask and resctrl_l3_cdp in struct mk_spawn_context. The asm only needs to write the CLOSID into MSR_IA32_PQR_ASSOC. One field instead of three, one instruction sequence instead of roughly 28 lines.
• The domain_setup_ctrlval() change should stay. It is genuinely necessary: if the host has donated every CPU in a socket, resctrl on the host has no domain object there and cannot program those registers at all, so the spawn kernel has to do it as its first CPU comes online. Doing it from C with a per-domain value from the DT is strictly better than doing it from asm with a single global mask.

Smaller things, independent of the above

• resctrl_closid == 0 doubles as "none" in struct mk_spawn_context and in the parking asm (testl %r10d, %r10d; jz). It works only because CLOSID 0 happens to be reserved. An explicit valid flag would be clearer.
• The -ERANGE returns in domain_setup_ctrlval() leave hw_dom->ctrl_val allocated and already assigned.
• mk_instance_reserve_llc_ways() walks mk_instance_list for overlap. Is the instance list lock held by all callers at that point?