Alternative CNI Plugins

Alternative CNI Plugins#

This document provides guidance on using different CNI plugins that have been tested with AMD Network Operator as alternatives to the AMD Host Device CNI plugin. These CNI plugins can be used with NetworkAttachmentDefinition to address specific networking use cases.

The AMD Network Operator supports multiple CNI plugins through NetworkAttachmentDefinition configurations, below are the few CNIs that we were tested:

  • SR-IOV CNI: For high-performance networking using Virtual Functions (VFs)

  • RDMA CNI: For RDMA workloads requiring network namespace isolation

  • SBR CNI: For Source-Based Routing to enable policy-based routing for secondary networks

SR-IOV CNI Plugin#

The SR-IOV CNI plugin enables moving SR-IOV Virtual Functions (VFs) directly to pod network namespaces, providing high-performance networking with minimal overhead.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-nad
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/resourceName: amd.com/vnic
spec:
  config: |-
    {
      "cniVersion": "0.3.1",
      "name": "sriov-nad",
      "type": "sriov",
      "ipam": {
        "type": "whereabouts",
        "range": "51.3.1.0/24"
      }
    }

RDMA CNI Plugin#

The RDMA CNI plugin provides network namespace isolation for RDMA workloads in containerized environments. It must be chained with other CNI plugins that handle the actual interface movement.

SR-IOV + RDMA + Tuning Example#

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-rdma-nad
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/resourceName: amd.com/vnic
spec:
  config: |-
    {
      "cniVersion": "1.0.0",
      "name": "sriov-rdma-nad",
      "plugins": [
        {
          "type": "sriov",
          "spoofchk": "off",
          "ipam": {
            "type": "whereabouts",
            "range": "51.1.1.0/24",
            "exclude": [
              "51.1.1.3/32",
              "51.1.1.4/32"
            ]
          }
        },
        {
          "type": "tuning",
          "sysctl": {
            "net.ipv4.conf.all.arp_announce": "2",
            "net.ipv4.conf.all.arp_filter": "1",
            "net.ipv4.conf.all.arp_ignore": "1",
            "net.ipv4.conf.all.rp_filter": "0",
            "net.ipv4.conf.all.accept_local": "1"
          },
          "mtu": 4220
        },
        {
          "type": "rdma"
        }
      ]
    }

Verification#

To verify that the RDMA CNI plugin is functioning correctly, run rdma link show inside the workload pod. The output should display only the RDMA device assigned to the pod in the exclusive mode, as shown below:

root@workload:/tmp# rdma link show
link ionic_0/1 state ACTIVE physical_state LINK_UP netdev net1
root@workload:/tmp#

In contrast, on a system configured in RDMA shared mode, the same command will list all available RDMA devices:

root@rccl-app-5f8b8dbddb-fgr6s:/tmp# rdma link show
link roceo3/1 state ACTIVE physical_state LINK_UP netdev net1
link rocep132s0/1 state ACTIVE physical_state LINK_UP
link rocep33s0f0/1 state ACTIVE physical_state LINK_UP
link rocep33s0f1/1 state ACTIVE physical_state LINK_UP
link ionic_0/1 state ACTIVE physical_state LINK_UP
root@rccl-app-5f8b8dbddb-fgr6s:/tmp#

SBR (Source-Based Routing) CNI Plugin#

The SBR CNI plugin enables Source-Based Routing (also known as policy-based routing) for secondary networks. This allows traffic originating from specific source IP addresses or interfaces to use designated routing tables and gateways, which is essential for multi-homed pods that need to route traffic through specific interfaces based on the source address.

AMD Host Device + SBR Example#

The SBR plugin can be used with the AMD Host Device CNI to enable proper source-based routing for secondary networks. Routing is opt-in: set both nexthopNetAddrOffset and routeDstPrefixLen on amd-host-device so a route is injected into the static IPAM result that SBR consumes. See Policy route injection.

Migration from v1.2.x: previous releases injected an implicit default gateway for /31 links even when those fields were absent. That behaviour is removed. Existing /31 + SBR NADs must add "nexthopNetAddrOffset": 0 and "routeDstPrefixLen": 0 to keep a default route via the peer.

With route injection (supernet route)#

For leaf/spine fabrics where host interfaces use /25 and the broader fabric is a /19 supernet:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: vf-amd-host-device-sbr-nad
  annotations:
    k8s.v1.cni.cncf.io/resourceName: amd.com/vnic
spec:
  config: |-
    {
      "cniVersion": "0.3.1",
      "name": "vf-amd-host-device-sbr-nad",
      "plugins": [
        {
          "type": "amd-host-device",
          "nexthopNetAddrOffset": 1,
          "routeDstPrefixLen": 19
        },
        { "type": "sbr" }
      ]
    }

Without route injection (basic chaining)#

When no routing is needed — the interface moves to the pod with its IPs. Without a gateway or injected routes in the IPAM result, SBR does not create a policy routing table; you will only have the kernel’s connected subnet route in the main table:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: vf-amd-host-device-sbr-basic-nad
  annotations:
    k8s.v1.cni.cncf.io/resourceName: amd.com/vnic
spec:
  config: |-
    {
      "cniVersion": "0.3.1",
      "name": "vf-amd-host-device-sbr-basic-nad",
      "plugins": [
        {
          "type": "amd-host-device"
        },
        { "type": "sbr" }
      ]
    }

See AMD Host Device CNI — Configuration Reference for all field combinations and their effects.

How SBR Works#

When the SBR plugin is invoked:

  1. It reads the IPAM result from the previous CNI plugin in the chain

  2. For each IP address with a gateway or routes, it creates:

    • A new routing table (using a unique table ID)

    • A default route in that table pointing to the specified gateway (if present)

    • Any explicit routes from the IPAM routes array (e.g., a supernet route injected by amd-host-device)

    • An IP rule that directs packets with that source address to use the new routing table

  3. This ensures that traffic originating from the secondary interface is routed through the correct gateway or destination

Verification#

To verify SBR configuration inside a pod:

# Show routing tables
ip route show table all

/31 point-to-point example (with routeDstPrefixLen: 0)#

root@workload-app-647dc5f6fc-pvpw2:/tmp# ip route show table all
default via 192.168.4.9 dev net1 table 100
192.168.4.8/31 dev net1 table 100 proto kernel scope link src 192.168.4.8
default via 192.168.2.9 dev net2 table 102
192.168.2.8/31 dev net2 table 102 proto kernel scope link src 192.168.2.8
default via 192.168.1.9 dev net3 table 104
192.168.1.8/31 dev net3 table 104 proto kernel scope link src 192.168.1.8
default via 192.168.6.9 dev net4 table 106
192.168.6.8/31 dev net4 table 106 proto kernel scope link src 192.168.6.8
default via 192.168.5.9 dev net5 table 108
192.168.5.8/31 dev net5 table 108 proto kernel scope link src 192.168.5.8
default via 192.168.7.9 dev net6 table 111
192.168.7.8/31 dev net6 table 111 proto kernel scope link src 192.168.7.8
default via 192.168.8.9 dev net7 table 114
192.168.8.8/31 dev net7 table 114 proto kernel scope link src 192.168.8.8
default via 192.168.3.9 dev net8 table 116
192.168.3.8/31 dev net8 table 116 proto kernel scope link src 192.168.3.8

# Show IP rules
ip rule show

# Example output showing SBR rules:
root@workload-app-647dc5f6fc-pvpw2:/tmp# ip rule show
0:      from all lookup local
32758:  from 192.168.3.8 lookup 116
32759:  from 192.168.8.8 lookup 114
32760:  from 192.168.7.8 lookup 111
32761:  from 192.168.5.8 lookup 108
32762:  from 192.168.6.8 lookup 106
32763:  from 192.168.1.8 lookup 104
32764:  from 192.168.2.8 lookup 102
32765:  from 192.168.4.8 lookup 100
32766:  from all lookup main
32767:  from all lookup default

Supernet route example (with nexthopNetAddrOffset: 1, routeDstPrefixLen: 19, NIC at 10.1.2.130/25)#

root@workload-app:/tmp# ip route show table 100
10.1.0.0/19 via 10.1.2.129 dev net1
10.1.2.128/25 dev net1 proto kernel scope link src 10.1.2.130

The first line is the injected supernet route: destination 10.1.0.0/19 (interface IP masked to /19), gateway 10.1.2.129 (network address 10.1.2.128 + offset 1). The second is the connected /25 subnet route from static IPAM.

To ping a NIC interface between two pods, you can use the following command:

ping <IP_ADDRESS> -I <interface-name or ip-address>

root@workload-app-647dc5f6fc-pvpw2:/tmp# ping 192.168.1.14 -I 192.168.3.8
PING 192.168.1.14 (192.168.1.14) from 192.168.3.8 : 56(84) bytes of data.
64 bytes from 192.168.1.14: icmp_seq=1 ttl=61 time=0.167 ms
64 bytes from 192.168.1.14: icmp_seq=2 ttl=61 time=0.185 ms
^C
--- 192.168.1.14 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1018ms
rtt min/avg/max/mdev = 0.167/0.176/0.185/0.009 ms
root@workload-app-647dc5f6fc-pvpw2:/tmp#

Using RDMA Tools with SBR#

When using RDMA performance testing tools (such as ib_write_bw, ib_read_bw, etc.) in environments with SBR configured, you must bind to the correct source IP address to ensure traffic uses the appropriate routing table. Use the -B or --bind_source_ip flag to specify the source IP address.

For example, to run ib_write_bw with source IP binding:

# On the server side (node1)
root@workload-app:/tmp# ib_write_bw -d roce_ai1 -B 192.168.4.8

# On the client side (node2) - connecting to node1
root@workload-app:/tmp# ib_write_bw -d ionic_0 -B 192.168.2.8 192.168.4.8

  • Important: Without the -B flag, RDMA tools may not correctly utilize the source-based routing rules, which can result in connectivity issues or traffic being routed through the wrong interface. Always specify the source IP address when working with SBR-configured secondary networks.

Contents