This placeholder link points to the main Kubernetes repository. It should be updated to point to the relevant integration tests within the OpenPERouter repository or removed if not yet implemented.

Instead of configuring ISIS in the underlay, we could also have an ISIS CRD. From the underlay configuration, + the SRV6 configuration, we could then generate ISIS CRs. But that seems more complex with few gains?

If we allow only a single ISIS process then we can hardcode the name of the process, as well

I think we should have a discriminated union in the Underlay.

You'd then select either EPVN or ISIS, and provide a configuration for the chosen protocol.

This seems simpler, and aligned with k8s api best practices.

imo, is legit ISIS to have ISIS as part of the underlay configuration. It's where it effectively belogns.

On the other hand, I would maybe question moving the evpn / srv6 bits elsewhere if they don't really represent underlay configuration.

I would try to understand if we can threat isis and srv6 orthogonally, allowing to:

• use bgp to exchange locator reacheability
• use isis to exchange vtep reacheability

I don't think that's something that's possible / makes sense.

The locator /48s must be announced through the entire routing domain, including transit nodes, and that's the job of an IGP (ISIS). Otherwise, you'd have to add bgp sessions to every single node of the routing domain.
On the other hand, the actual provider edge nodes (the tunnel edges) must transmit meta information that fits inside BGP message and not inside ISIS, such as RT / route-distinguiser.

** I don't quite understand what the End.X routes are for, yet, but ISIS also carries info about End.X behavior.

# FRR:
I>* fd00:0:12:e002::/64 [115/0] is directly connected, to-l1-1-1, seg6local uA nh6 fe80::a8c1:abff:fea8:a61b, weight 1, 00:09:02
# Kernel:
fd00:0:12:e002::/64 nhid 32  encap seg6local action End.X nh6 fe80::a8c1:abff:fea8:a61b flavors next-csid lblen 32 nflen 16 dev to-l1-1-1 proto isis metric 20 pref medium
# show isis database detail
l1-1-pe.00-00             287   0x00000002  0x4cd1     925    0/0/0
  Protocols Supported: IPv4, IPv6
  Area Address: 49.0001
  Hostname: l1-1-pe
  TE Router ID: 172.31.1.12
  Router Capability: 172.31.1.12 , D:0, S:0
    SR Algorithm:
      0: SPF
    SRv6: O:0
  Extended Reachability: 1720.3100.1011.44 (Metric: 10)
    SRv6 Lan End.X SID: fd00:0:12:e002::, Algorithm: SPF, Weight: 0, Endpoint Behavior: uA, Flags: B:0, S:0, P:0 Neighbor-ID: 1720.3100.1011
        SRv6 SID Structure Locator Block length: 32, Locator Node length: 16, Function length: 16, Argument length: 0, 
  Extended Reachability: 1720.3100.1012.47 (Metric: 10)
    SRv6 Lan End.X SID: fd00:0:12:e003::, Algorithm: SPF, Weight: 0, Endpoint Behavior: uA, Flags: B:0, S:0, P:0 Neighbor-ID: 1720.3100.1013
        SRv6 SID Structure Locator Block length: 32, Locator Node length: 16, Function length: 16, Argument length: 0, 
  IPv4 Interface Address: 172.31.1.12
  Extended IP Reachability: 10.1.2.0/24 (Metric: 10)
  Extended IP Reachability: 10.1.3.0/24 (Metric: 10)
  Extended IP Reachability: 172.31.1.12/32 (Metric: 10)
  IPv6 Reachability: fd00:0:12::/48 (Metric: 0)
  IPv6 Reachability: fc00::2:172:31:1:12/128 (Metric: 10)
  SRv6 Locator: fd00:0:12::/48 (Metric: 0) standard
    Sub-TLVs:
      SRv6 End SID Endpoint Behavior: uN, SID value: fd00:0:12::
        Sub-Sub-TLVs:
          SRv6 SID Structure Locator Block length: 32, Locator Node length: 16, Function length: 16, Argument length: 0, 

BGP advertisement example:

l1-1-pe# show bgp ipv4 vpn 
BGP table version is 2, local router ID is 172.31.1.12, vrf id 0
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 172.31.1.12:100
 *>  192.168.0.0/31   0.0.0.0@3<               0         32768 ?
    UN=0.0.0.0 EC{65000:100} label=917504 sid=fd00:0:12:: sid_structure=[32,16,16,0] type=bgp, subtype=5
 *>  192.168.1.0/24   192.168.0.1@3<           0             0 65001 ?
    UN=192.168.0.1 EC{65000:100} label=917504 sid=fd00:0:12:: sid_structure=[32,16,16,0] type=bgp, subtype=5
Route Distinguisher: 172.31.1.12:101
 *>  192.0.2.0/26     192.0.3.1@4<             0             0 65002 ?
    UN=192.0.3.1 EC{65000:101} label=917520 sid=fd00:0:12:: sid_structure=[32,16,16,0] type=bgp, subtype=5
 *>  192.0.3.0/31     0.0.0.0@4<               0         32768 ?
    UN=0.0.0.0 EC{65000:101} label=917520 sid=fd00:0:12:: sid_structure=[32,16,16,0] type=bgp, subtype=5
Route Distinguisher: 172.31.1.22:100
 *>i 192.168.0.2/31   fc00::2:172:31:1:22
                                               0    100      0 ?
    UN=fc00::2:172:31:1:22 EC{65000:100} label=917504 sid=fd00:0:22:: sid_structure=[32,16,16,0] type=bgp, subtype=0
 *>i 192.168.2.0/24   fc00::2:172:31:1:22
                                               0    100      0 65001 ?
    UN=fc00::2:172:31:1:22 EC{65000:100} label=917504 sid=fd00:0:22:: sid_structure=[32,16,16,0] type=bgp, subtype=0
Route Distinguisher: 172.31.1.22:101
 *>i 192.0.2.64/26    fc00::2:172:31:1:22
                                               0    100      0 65002 ?
    UN=fc00::2:172:31:1:22 EC{65000:101} label=917520 sid=fd00:0:22:: sid_structure=[32,16,16,0] type=bgp, subtype=0
 *>i 192.0.3.2/31     fc00::2:172:31:1:22
                                               0    100      0 ?
    UN=fc00::2:172:31:1:22 EC{65000:101} label=917520 sid=fd00:0:22:: sid_structure=[32,16,16,0] type=bgp, subtype=0
Route Distinguisher: 172.31.1.32:100
 *>i 192.168.0.4/31   fc00::2:172:31:1:32
                                               0    100      0 ?
    UN=fc00::2:172:31:1:32 EC{65000:100} label=917504 sid=fd00:0:32:: sid_structure=[32,16,16,0] type=bgp, subtype=0
 *>i 192.168.3.0/24   fc00::2:172:31:1:32
                                               0    100      0 65001 ?
    UN=fc00::2:172:31:1:32 EC{65000:100} label=917504 sid=fd00:0:32:: sid_structure=[32,16,16,0] type=bgp, subtype=0
Route Distinguisher: 172.31.1.32:101
 *>i 192.0.2.128/26   fc00::2:172:31:1:32
                                               0    100      0 65002 ?
    UN=fc00::2:172:31:1:32 EC{65000:101} label=917520 sid=fd00:0:32:: sid_structure=[32,16,16,0] type=bgp, subtype=0
 *>i 192.0.3.4/31     fc00::2:172:31:1:32
                                               0    100      0 ?
    UN=fc00::2:172:31:1:32 EC{65000:101} label=917520 sid=fd00:0:32:: sid_structure=[32,16,16,0] type=bgp, subtype=0

Displayed 12 routes and 12 total paths

not sure I get your point.

You could just have a network statement advertising the reacheability of the locator. And bgp doesn't require you to have sessions with all the nodes, as it's not for evpn.

Similrarly, we could use isis to advertise the reacheability of the vtep assigned to the loopback

You could just have a network statement advertising the reacheability of the locator. And bgp doesn't require you to have sessions with all the node

Sure, but how do the transit nodes know where your locator is? That's ISIS' job:

PE-A <-> transit a <-> transit b <-> transit c <-> PE-B

ISIS takes care of announcing the locator prefix to all nodes in the domain, so that all 5 of them know where the locator prefix is. Otherwise, if you announce it via BGP, you must have BGP sessions spanning between all nodes (as in a datacenter that uses eBGP unnumbered between all nodes). Otherwise, you'd blackhole your traffic.

Also, ISIS is responsible for the End.X route:

FRR:

I>* fd00:0:10:e000::/64 [115/0] is directly connected, eth1, seg6local uA nh6 fe80::a8c1:abff:fe21:97a0, eth1, weight 1, 00:31:48

Linux:

fd00:0:10:e000::/64 nhid 52  encap seg6local action End.X nh6 fe80::a8c1:abff:fe21:97a0 flavors next-csid lblen 32 nflen 16 dev eth1 proto isis metric 20 pref medium

Even though I have to admit that I've got no clue atm what it's used for.

Similrarly, we could use isis to advertise the reacheability of the vtep assigned to the loopback
I am using ISIS to advertise the loopback IP

From all of the documents and examples I can find, I think the standard is pretty much:

• use ISIS to advertise the locator prefix
• use BGP to advertise SRv6 functions = the specific /128 for decapsulation

I would go as far as to say - because we use ISIS, and otherwise we need another way to announce the BGP update source across the network:

• use ISIS to advertise the BGP update-source and SRv6 source-address (meaning: IPs on the loopback)

We could have tons of other esoteric combinations, but shouldn't we stick to what seems to be the established standard? In theory, we also wouldn't need BGP at all, we could just inject announce the /128 manually and announce them via an IGP ourselves, or do everything with static routes ;-)

So I'd stick to Cisco's way of doing it: https://www.segment-routing.net/images/20250311-srv6-usid-frr-netdev-0x19.pdf

Or, way shorter: what's our target first user going to use - and we should stick with that :-)

Despite the fact that as you wrote, it's much easier to find references to the srv6 - isis combo, I think that supporting BGP is a nice addition. A good reference is https://datatracker.ietf.org/doc/draft-hss-srv6ops-srv6-routing-planes/

If I read that correctly:

   * Locator Advertisements:

   Green locators are advertised as standard IPv6 prefixes (AFI-2, SAFI-
   1) and are tagged with the extended color community [RFC4360]
   corresponding to Green.

I am fine with starting with isis only, but I'd like to write the API in such a way that we'll be able to add BGP support afterwards

and iiuc, it's how cilium works too (https://www.youtube.com/watch?v=k_2nMzxi1Hk)

https://github.com/fedepaol/srv6lab/blob/main/pe1/frr.conf#L62

note to myself: Modify the RFE's constraints: SRv6 can only be configured if ISIS OR BGP is configured

or to their route reflectors (we should also mention the RR enhancement here #218

According to the description in #218

In cloud environments where the ToR does not support EVPN, east/west traffic cannot be distributed via the fabric. This enhancement adds an internal RR so EVPN routes are reflected between router pods and VXLAN data plane goes directly node-to-node, avoiding the ToR.

It seems that enhancement is for east-west traffic, so wouldn't apply to L3VPN?

I rephrased the sentence to:

They want to span ISIS all the way to the Kubernetes cluster and want
to peer iBGP between their tunnel edge nodes (and potentially Route Reflectors)
and the Kubernetes nodes (for iBGP support, see

Although iBGP edge nodes are kind of agnostic to the fact that they are talking to an RR or not, the RR is the only device with special syntax / behavior

not sure we might want to tie this more to srv6?

do you mean the name?

yes

vni though is very evpn specific. We can call it networkIdentifier or something like that

The field is actually called Assigned Number subfield: https://datatracker.ietf.org/doc/html/rfc4364#section-4.2
What about: RDAssignedNumber?

do we want to declare what addressfamily to enable for each neighbor?

that brings us back to the discussion that we had on the IPv6 PR, doesn't it? I thin the Neighbor entry needs an AddressFamily field

yep I agree

#395
I will take this up once the IPv6 PR merged

not sure I get your point.

You could just have a network statement advertising the reacheability of the locator. And bgp doesn't require you to have sessions with all the nodes, as it's not for evpn.

Similrarly, we could use isis to advertise the reacheability of the vtep assigned to the loopback

in l3vni (and evpn in general) we don't override the source and we use the interface ip. I think there's value, I just wonder if we should rethink that too

yes

either is fine imo, the direction we are taking is to do cel whenever we can

should we have a user story about exposing workloads to the outside world using their IPs, with an srv6 backend ?

unsure if that's covered by the first user story below.

the first story covers reaching workloads via service endpoints from the outside. I think that's analogous to what L3VNI does. or am I missing something?

question: is L2 possible ? i.e. can you implement EVPN using ISIS and srv6 ?

it's possible but not currently implemented in frr (and some bits are also missing in the kernel, specifically the l2 evpn part). We should be open to that

OK, so whatever API we end up with must accept that scenario.

The problem is that it's in theory possible to do any of the following (non-exhaustive list):

• L2 EVPN using ISIS and srv6
• L3 EVPN using ISIS and srv6
• L2VPN using ISIS and srv6
• L2 EVPN using BGP and srv6
• L3 EVPN using BGP and srv6
• L2VPN using BGP and srv6

I'm a bit afraid that if we keep in mind every potential future implementation, we overengineer an API which is currently in the alpha stage and is by definition in a transient state.

Any system that is successful needs to grow and change as new use cases emerge or existing ones change. 

https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-changes
https://kubernetes.io/docs/reference/using-api/#api-versioning

Also isn't the OpenPERouter purposefully a subset of the FRR config? Otherwise, we could just expose the FRR config verbatim to the enduser, in API constructs ;-)

This RFE is designed in a way that ISIS is independent (on an API level) from SRv6.
SRv6 in a first implementation depends on ISIS.
What this doesn't contain, yet, is e.g. a selector under SRV6 for the underlay protocol, ISIS or BGP, but it can be extended.

An L3VPN by definition uses SRv6 (not EVPN), that's why I named the resource that way (and not SRv6VPN or something). An L2VPN resource can be added later, when/if support lands.

On the other hand, I believe that the EVPN configuration or current L3VNI / L2VNI resources would have to be extended to have a selector for either VXLAN or for SRv6 as the tunneling mechanism. This would, afaict, reuse the segmentrouting locator and encapsulation source in FRR, which is mirrored in this RFE by the SRV6 config block.

If we want to account for all of the different possible configurations, I feel that we have to "explode" the underlay config into different CRDs and work with selectors.

Where the current RFE fails is how do we have EVPN and L3VPN both use SRv6 for tunneling and then select different protocols for the underlay:

• EVPN via BGP for the overlay, SRv6 for tunneling, BGP for the underlay
• L3VPN via BGP for the overlay, SRv6 for tunneling, ISIS for the underlay

But are designs with such mixed uses cases, all configured at the same time in a single cluster, really realistic?

So in an ideal API with maximum flexibility, shouldn't we have something that breaks up resources configuratoin elements along conceptual lines, a little bit along the ideas of ISO/OSI or network layers:

layer 0:
CRD for host configuration: adds IP addresses to the loopbacks, brings interfaces into the OpenPERouter; maybe configures sysctls, etc

layer 1:
CRDs for the underlay routing protocols: BGP, ISIS, OSPF configuration

layer2:
CRD for VXLAN tunnels: selects the tunnel source IP, e.g.
CRD for SRv6: configures the segment routing block in the FRR configuration, set's the locator prefix and
SRv6 tunnel source, and once support is there in FRR, configures e.g. encap reduced

layer 3:
CRD for the overlay routing protocol: BGP configuration

layer4:
L3VNI and L2VNI
L3VPN and L2VPN

either integrated in layer 4:
L3VNI and L2VNI resources must select: host configuration CRD; VXLAN or SRv6 tunnel protocol; one of the underlay protocols; and the overlay protocol
L3VPN and L2VPN resources that can select: host configuration; SRv6 tunnel protocol; one of the underlay protocols; BGP configuration
Or having an integration layer with a crd that then selects everythig that has to go together

And then we'd add checks to avoid incompatible configuration options. But that's super complex and doesn't fit the current API (nor its philosophy of being small and nimble)

I'm a bit afraid that if we keep in mind every potential future implementation, we overengineer an API which is currently in the alpha stage and is by definition in a transient state.

I agree. We should let the dust settle and then see what we missed, while trying to make this as good as we can

will we be able to implement srv6 using BGP ? or only ISIS ?

yep. added to the RFE, but that's not the immediate goal

why is this an explicit goal ? trying to understand if srv6 requires ISIS as control plane, or not.

while it might work with bgp too (see the thread below, https://github.com/openperouter/openperouter/pull/316/changes#r3123018997) isis is the most popular / documented way to advertise srv6 locators

do you elaborate the arithmetic to calculate the ISIS net somewhere ?

The existing IPAM functions (VTEPIp, RouterID) operate on CIDRs ... I think we'll need something different here. Especially interested in how we offset the locator.

Not in this RFE, but in the prototype.

type ISISSystemID [6]byte

// IncrementSystemID takes an ISIS SystemID and an offset and returns the result of the sum of both.
func IncrementSystemID(si ISISSystemID, offset int) (ISISSystemID, error) {
	carry := offset
	for i := range 6 {
		if carry == 0 {
			break
		}
		idx := len(si) - 1 - i
		res := int(si[idx]) + carry
		carry = res / 256
		si[idx] = byte(res % 256)
	}
	if carry > 0 {
		return si, fmt.Errorf("overflow while incrementing SystemID %s with offset %d", si, offset)
	}
	return si, nil
}

And for the locator :
I think it's the only thing in the entire prototype that I let Claude dot, because every time I look at that precise bit, it's super late and I'm too tired to understand the math behind it :-P

According to the unit tests, this here works, though:

// OffsetIPv6Prefix adds nodeIndex to the network portion of an IPv6 prefix.
// For example, with prefix "fd00:0:11::/48", nodeIndex 2, and prefixLen 48,
// it returns "fd00:0:13::/48" (0x11 + 2 = 0x13).
// Note: written by Claude, verify.
func OffsetIPv6Prefix(prefix string, nodeIndex, prefixLen int) (string, error) {
	ip, ipNet, err := net.ParseCIDR(prefix)
	if err != nil {
		return "", fmt.Errorf("failed to parse prefix %s: %w", prefix, err)
	}

	if ip.To4() != nil {
		return "", fmt.Errorf("prefix %s is not IPv6", prefix)
	}

	ipBytes := ip.To16()
	if ipBytes == nil {
		return "", fmt.Errorf("failed to convert IP to IPv6")
	}

	// Calculate how many bytes and bits we need to modify
	numBytes := prefixLen / 8
	remainingBits := prefixLen % 8

	// Convert the network portion to a big integer and add nodeIndex
	var networkValue uint64 = 0
	for i := 0; i < numBytes && i < 8; i++ {
		networkValue = (networkValue << 8) | uint64(ipBytes[i])
	}

	// Handle remaining bits if prefixLen is not a multiple of 8
	if remainingBits > 0 && numBytes < 8 {
		networkValue = (networkValue << uint(remainingBits)) | (uint64(ipBytes[numBytes]) >> uint(8-remainingBits))
	}

	// Add the node index
	networkValue += uint64(nodeIndex)

	// Write the value back to the IP bytes
	if remainingBits > 0 && numBytes < 8 {
		// Extract the remaining bits and put them back
		mask := byte((1 << uint(8-remainingBits)) - 1)
		hostPortion := ipBytes[numBytes] & mask
		ipBytes[numBytes] = byte((networkValue&((1<<uint(remainingBits))-1))<<uint(8-remainingBits)) | hostPortion
		networkValue >>= uint(remainingBits)
		numBytes--
	}

	// Write back the full bytes
	for i := numBytes - 1; i >= 0; i-- {
		ipBytes[i] = byte(networkValue & 0xff)
		networkValue >>= 8
	}

	// Get the original prefix length from the input
	ones, _ := ipNet.Mask.Size()

	return fmt.Sprintf("%s/%d", ipBytes.String(), ones), nil
}

this is the part I find extremely confusing. so evpn in the API stands for the combination of EVPN (control plane) + vxlan (data plane). That's what the current nomenclature is doing.

Then we have the isis configuration, which, afaiu just instructs the router on how to configure ISIS to reach the TOR (BGP would still be used to exchange the routes ...).

Finally, we have the srv6 configuration, which configures this non VXLAN data-plane.

IIUC, the EVPN config is actually just the VXLAN data-plane configuration (which I think could/should be renamed).

To make everything more explicit, I would:

• implement a discriminated union in the UnderlaySpec CRD to define the control plane used. It could either be VXLAN, or, SRv6 (which you're adding now). This means we should rename EVPN to VXLAN.
• add another discriminated union meant to represent the connection to the ToR. It would default to BGP. With your proposal, you'd add ISIS. Maybe in the future we'll want / need to add OSPF. Maybe call it ToRConnectionProtocol

I'm aware I might be making this more complex, but when I read this proposal I didn't understand the purpose of ISIS in it at all. This would - imho - help make that clearer.

Grr ... I knew I was missing something.

I'm now aware my suggestion would not work for the L2VNI exposed via L3VPN. L2VNIs always require VXLAN. I lost that detail.

It can't be a discriminated union. But, the model could work like:

• L2VNIs will always be carried by VXLAN
• L3 VPN will be carried by either VXLAN OR SRv6.

The latter can be represented in a union - meaning, the attribute name could be something along the lines of L3 VPN data-plane .

L2VNIs are always carried by VXLAN via IPv4
L3VNIs are always carried by VXLAN via IPv4
L3VPN is always carried by IPv6/SRv6 via IPv6.

Within the terminology of the current version of this RFE. Federico didn't specifically like the L3VPN terminology, and I'm o.k. with finding a better name for it. But L3VPN here means SRv6

this is the part I find extremely confusing. so evpn in the API stands for the combination of EVPN (control plane) + vxlan (data plane). That's what the current nomenclature is doing.

Yeah. But when you look at the EVPN field, what it really does is, it just either configures an Ipv4 address on the lound loopback. Or, it moves an interfaces into the namespace and picks the IPv4 address of that interface as the VXLAN tunnel source. It's not doing anything that is, per se, EVPN related. Indeed, I think there should be a configuration section for the tunnel source, and either it moves an interface into the ns, or it assigns IPv4 and/or IPv6 addresses from source CIDRS. Any feature that requires tunneling (such as EVPN and SRv6) should then pick the tunnel source addresses from there. Even when EVPN will eventually support IPv6 endpoints, there's no need to have different IPv6 addresses for the VXLAN and SRv6 tunnel sources. They can simply share the same IP.

In routing, using the loopbacks is also generally super important. E.g., in ISIS, a good practice is to announce only the loopback IPs into the network, and then establish BGP sessions between the loopbacks.

This means we should rename EVPN to VXLAN.

I think we shoul rename it to something related to tunnel or loopback (not verbatim, but something that conveys that meaning), because that's what it does ... picking and configuring an IP address for dependent services that can then use those IPs.

I was about to add this change here, but then I thought that it's overkill: the current version of this RFE tries to fit the feature within the scope of the current API. Otherwise, I'd have to rename the EVPN field, meaning that the RFE and the implementation would become more complex. Fortunately, SRv6 is IPv6 only and EVPN is IPv4 only right now, so it's possible to make the current RFE work, and then redesign this in #341 ?