AMD Host Device CNI Plugin#
The AMD Host Device CNI plugin is a specialized Container Network Interface (CNI) plugin that moves Physical Function (PF) or Virtual Function (VF) network interfaces from the host directly into pod network namespaces while preserving their IP addresses. This approach provides pods with direct access to high-performance network interfaces while maintaining network configuration consistency.
Key Features#
Interface Movement and IP Preservation#
Direct PF/VF Movement: Moves entire Physical or Virtual Function interfaces from host to pod namespace
IP Address Preservation: Captures and preserves existing IP addresses (both IPv4 and IPv6 addresses) from the host interface and passes them to the static IPAM configuration
Policy route injection (opt-in): set both
nexthopNetAddrOffsetandrouteDstPrefixLenon the NAD to inject a route into the static IPAM result for SBR. See Policy route injection. Earlier releases implicitly added a gateway for/31links; that behaviour is removed. Restore a default route via the peer with"nexthopNetAddrOffset": 0and"routeDstPrefixLen": 0IP Address and State Persistence: IP addresses and the interface state are retained on the host interface even after workload deletion
Configuration#
NetworkAttachmentDefinition#
Separate NAD should be created for each resource type: nic and vnic
NAD for nic#
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: amd-host-device-nad-nic
annotations:
k8s.v1.cni.cncf.io/resourceName: amd.com/nic
spec:
config: '{
"name": "amd-host-device-nad-nic",
"cniVersion": "0.3.1",
"type": "amd-host-device"
}'
NAD for vnic#
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: amd-host-device-nad-vnic
annotations:
k8s.v1.cni.cncf.io/resourceName: amd.com/vnic
spec:
config: '{
"name": "amd-host-device-nad-vnic",
"cniVersion": "0.3.1",
"type": "amd-host-device"
}'
For detailed information on how this resource is allocated and how the CNI is invoked, please refer to the integration flow documentation.
Configuration Reference#
Field |
Type |
Required |
Description |
|---|---|---|---|
|
string |
yes |
Must be |
|
string |
yes |
CNI spec version (e.g. |
|
integer |
no |
Offset from subnet network address to derive gateway IP. Ignored on |
|
integer (0–32) |
no |
Prefix length for route destination. Must be less than interface prefix. |
Route Injection (opt-in)#
When both nexthopNetAddrOffset and routeDstPrefixLen are set, the plugin computes a route per IPv4 address on the host interface and injects it into the static IPAM routes array. When chained with the SBR CNI plugin, SBR installs the route into a per-source policy routing table. For the full derivation algorithm and edge cases, see Policy Route Injection.
Field value combinations:
|
|
Effect |
|---|---|---|
(absent) |
(absent) |
No route injected. Interface moves to pod with IPs only. |
|
|
Supernet route: |
|
|
Default route: |
|
|
Default route on /31: |
|
|
/31 supernet: |
Key behaviors:
Both fields must be set together — setting only one is a no-op (logged as warning)
routeDstPrefixLen: 0means a default route (0.0.0.0/0) — routes all non-local traffic via the computed gatewayOn
/31links,nexthopNetAddrOffsetis ignored. The gateway is the kernel peer when one is present, otherwise the XOR of the local addressWhen the interface prefix length is 30 or less (
/30,/24,/16, and so on), offset0(the subnet address) and the all-ones offset (the broadcast address) are skipped. Use1for the first usable host
NAD with supernet route (leaf/spine fabric)#
For environments with /25 host interfaces and a /19 supernet across the fabric (e.g., spine/leaf datacenter topologies), this NAD injects a supernet route so pods can reach peers behind other leaf switches. For a NIC at 10.1.2.130/25, this installs 10.1.0.0/19 via 10.1.2.129 in the SBR policy table.
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: amd-host-device-sbr-nad
annotations:
k8s.v1.cni.cncf.io/resourceName: amd.com/nic
spec:
config: |-
{
"cniVersion": "0.3.1",
"name": "amd-host-device-sbr-nad",
"plugins": [
{
"type": "amd-host-device",
"nexthopNetAddrOffset": 1,
"routeDstPrefixLen": 19
},
{ "type": "sbr" }
]
}
NAD with default route (non-/31)#
Routes all secondary-network traffic via the first host in the subnet. For a NIC at 10.1.2.130/25, this installs 0.0.0.0/0 via 10.1.2.129.
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: amd-host-device-sbr-default-nad
annotations:
k8s.v1.cni.cncf.io/resourceName: amd.com/nic
spec:
config: |-
{
"cniVersion": "0.3.1",
"name": "amd-host-device-sbr-default-nad",
"plugins": [
{
"type": "amd-host-device",
"nexthopNetAddrOffset": 1,
"routeDstPrefixLen": 0
},
{ "type": "sbr" }
]
}
NAD with default route (/31 — migration from v1.2.x)#
Replaces the implicit /31 gateway that was automatically injected in v1.2.x. For a NIC at 192.168.4.8/31, this installs 0.0.0.0/0 via 192.168.4.9 (XOR peer). The offset value is ignored on /31.
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: amd-host-device-sbr-p2p-nad
annotations:
k8s.v1.cni.cncf.io/resourceName: amd.com/nic
spec:
config: |-
{
"cniVersion": "0.3.1",
"name": "amd-host-device-sbr-p2p-nad",
"plugins": [
{
"type": "amd-host-device",
"nexthopNetAddrOffset": 0,
"routeDstPrefixLen": 0
},
{ "type": "sbr" }
]
}
The basic NAD examples shown earlier (without nexthopNetAddrOffset and routeDstPrefixLen) continue to work unchanged — the interface moves to the pod with its IPs and no routes are injected.
Verifying Route Injection#
When route injection is configured with SBR, verify the injected route appears in the SBR policy table inside the pod. Using the supernet NAD example above (NIC at 10.1.2.130/25, nexthopNetAddrOffset: 1, routeDstPrefixLen: 19):
root@workload-app:/tmp# ip route show table 100
10.1.0.0/19 via 10.1.2.129 dev net1
10.1.2.128/25 dev net1 proto kernel scope link src 10.1.2.130
root@workload-app:/tmp# ip rule show
0: from all lookup local
32765: from 10.1.2.130 lookup 100
32766: from all lookup main
32767: from all lookup default
The first line in table 100 (10.1.0.0/19 via 10.1.2.129) is the injected supernet route — the gateway 10.1.2.129 is the network address (10.1.2.128) plus the offset (1), and the destination 10.1.0.0/19 is the interface IP masked to /19. The second line is the connected /25 subnet route added by static IPAM. The ip rule ensures traffic sourced from the secondary IP (10.1.2.130) uses this table.
Verification#
This section demonstrates how to verify that a RoCE (RDMA over Converged Ethernet) device is correctly allocated to a pod and moved from the host namespace into the pod namespace.
On the host (Before Allocation)#
Check the RoCE device using rdma and ibv_devices:
root@genoa4:~# rdma link show rocep68s0/1
link rocep68s0/1 state ACTIVE physical_state LINK_UP netdev enp68s0
root@genoa4:~# ibv_devices | grep rocep68s0
rocep68s0 069081fffe2c4f90
root@genoa4:~#
Check the associated Ethernet interface:
root@genoa4:~# ifconfig enp68s0
enp68s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 55.1.1.56 netmask 255.255.255.0 broadcast 55.1.1.255
inet6 fe80::690:81ff:fe2c:4f90 prefixlen 64 scopeid 0x20<link>
ether 04:90:81:2c:4f:90 txqueuelen 1000 (Ethernet)
RX packets 630705 bytes 70505656 (70.5 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
On the Workload Pod (After Allocation)#
Once the RoCE device is allocated to a pod, the device and interface are moved out of the host namespace and become visible inside the pod.
Check inside the pod:
root@workload-app-nic-679fb76687-wbhlg:/tmp# rdma link show rocep68s0/1
link rocep68s0/1 state ACTIVE physical_state LINK_UP netdev enp68s0
root@workload-app-nic-df886b98c-v5glk:/tmp# ibv_devices
device node GUID
------ ----------------
rocep68s0 069081fffe2c4f90
root@workload-app-nic-df886b98c-v5glk:/tmp#
root@workload-app-nic-679fb76687-wbhlg:/tmp# ifconfig enp68s0
enp68s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 55.1.1.56 netmask 255.255.255.0 broadcast 55.1.1.255
inet6 fe80::690:81ff:fe2c:4f90 prefixlen 64 scopeid 0x20<link>
ether 04:90:81:2c:4f:90 txqueuelen 1000 (Ethernet)
RX packets 631105 bytes 70548756 (70.5 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
On the Host (After Allocation)#
After allocation, the ethernet interface is no longer present in the host namespace:
root@genoa4:~# ifconfig enp68s0
enp68s0: error fetching interface information: Device not found
root@genoa4:~#