AMD Instinct MI250 / MI250X

The AMD Instinct™ MI250 is a data-center OAM-form-factor GPU. This document provides MI250-specific prerequisites, health checks, validation steps, and performance acceptance criteria. It also applies to the AMD Instinct™ MI250X, which shares the same CDNA 2 (gfx90a) OAM platform and acceptance criteria; MI250X-specific differences are noted inline.

Overview

The AMD Instinct MI250 brings the second-generation CDNA architecture to an OCP Accelerator Module (OAM) form factor purpose-built for HPC and large-scale AI training. Each MI250 packages two Graphics Compute Dies (GCDs) under a single OAM, each GCD presenting 110 CUs with Matrix Core technology and 64 GB of HBM2e memory at up to 1.6 TB/s, for a combined 128 GB and 3.2 TB/s per OAM. The two GCDs on an OAM are linked by a high-bandwidth on-package AMD Infinity Fabric™ interconnect, and each OAM exposes additional xGMI ports for direct GPU-to-GPU connectivity across a 4-OAM all-to-all mesh. A typical qualified configuration hosts 4 MI250 OAMs (8 GCDs total) per node.

The MI250 is built on the CDNA 2 architecture (gfx90a) in an OCP Accelerator Module (OAM) form factor. Each MI250 OAM hosts two Graphics Compute Dies (GCDs), each enumerated as an independent GPU by ROCm tools, with 128 GB of HBM2e memory per OAM (64 GB per GCD). GPUs are connected to each other and to the host CPUs through AMD Infinity Fabric™ (xGMI).

The MI250X is the higher-performance variant of the same CDNA 2 (gfx90a) OAM platform and is validated using the criteria in this document. It powers exascale-class supercomputers such as Frontier and LUMI. MI250X reference deployments commonly use an 8-OAM (16-GCD) node topology; scale the per-node GCD counts in the commands below accordingly (for example, -g 16 for RCCL and mpiexec -n 16 for BabelStream on an 8-OAM node). MI250X also shares the MI250 PCI vendor:device ID (1002:740c).

• MI250 Product Page

• MI250X Product Page

• MI200 Series Microarchitecture

• MI200 Series Data Sheet

System requirements

Operating system support

For the most up-to-date information on supported operating systems and distributions, refer to the official ROCm documentation:

ROCm System Requirements - Supported Distributions

Note

ROCm docs is the single source of truth for supported versions, distribution compatibility, and required dependencies for the ROCm toolkit.

For general NUMA and OS-level tuning that applies to all AMD Instinct hosts, see OS tuning. MI250-specific BIOS settings are in the BIOS settings section below.

GPU identification

All MI250 GCDs (PCI vendor:device 1002:740c) should appear in lspci output. On a fully populated 4-OAM MI250 platform you should see 8 GCD entries (2 per OAM):

sudo lspci -d 1002:740c

Expected output example:

0000:11:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran (rev 01)
0000:14:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran (rev 01)
0000:32:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran (rev 01)
0000:35:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran (rev 01)
0000:8e:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran (rev 01)
0000:93:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran (rev 01)
0000:ae:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran (rev 01)
0000:b3:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran (rev 01)

The 8 GCDs are paired by OAM (e.g. 11+14 are the two GCDs on one OAM, 32+35 on the next, and so on). Same-OAM GCD pairs are connected by a high-bandwidth on-package link, while cross-OAM connectivity uses external xGMI ports in a 4-OAM all-to-all mesh.

BIOS settings

For maximum MI250 GPU performance on systems with AMD EPYC™ 7003-series processors (codename “Milan”) and AMI System BIOS, the following BIOS settings have been validated. These settings should be set as default values in the system BIOS. Analogous settings for other non-AMI BIOS providers can be configured similarly. On systems with Intel processors, some settings may not apply.

Note

The BIOS setting locations and names may vary by hardware and BIOS vendor. Consult your system documentation for details.

BIOS setting location	Parameter	Value	Comments
Advanced / PCI subsystem settings	Above 4G decoding	Enabled	GPU large BAR support.
Advanced / PCI subsystem settings	SR-IOV support	Disabled	Disable single root IO virtualization.
AMD CBS / CPU common options	Global C-state control	Auto	Global C-states.
AMD CBS / CPU common options	CCD/Core/Thread enablement	Accept	May be necessary to enable the SMT control menu.
AMD CBS / CPU common options / performance	SMT control	Disable	Set to Auto if the primary application is not compute-bound.
AMD CBS / DF common options / memory addressing	NUMA nodes per socket	NPS 1, 2, or 4	See NPS memory configuration below.
AMD CBS / DF common options / memory addressing	Memory interleaving	Auto	Depends on NPS setting.
AMD CBS / DF common options / link	4-link xGMI max speed	18 Gbps	Set to highest rate supported by the CPU.
AMD CBS / NBIO common options	IOMMU	Disabled
AMD CBS / NBIO common options	PCIe ten bit tag support	Auto
AMD CBS / NBIO common options	Preferred IO	Bus
AMD CBS / NBIO common options	Preferred IO bus	Use lspci to find PCIe device ID
AMD CBS / NBIO common options	Enhanced Preferred IO mode	Enabled
AMD CBS / NBIO common options / SMU common options	Determinism control	Manual
AMD CBS / NBIO common options / SMU common options	Determinism slider	Power
AMD CBS / NBIO common options / SMU common options	cTDP control	Manual	Set cTDP to the maximum supported by the installed CPU.
AMD CBS / NBIO common options / SMU common options	cTDP	280	Value in watts.
AMD CBS / NBIO common options / SMU common options	Package power limit control	Manual	Set to the maximum supported by the installed CPU.
AMD CBS / NBIO common options / SMU common options	Package power limit	280	Value in watts.
AMD CBS / NBIO common options / SMU common options	xGMI link width control	Manual	Set AMD CPU xGMI width to 16 bits.
AMD CBS / NBIO common options / SMU common options	xGMI force link width	2	0: x2 / 1: x8 / 2: x16
AMD CBS / NBIO common options / SMU common options	xGMI force link width control	Force
AMD CBS / NBIO common options / SMU common options	APBDIS	1	Disable DF P-states.
AMD CBS / NBIO common options / SMU common options	DF C-states	Enabled
AMD CBS / NBIO common options / SMU common options	Fixed SOC P-state	P0
AMD CBS / UMC common options / DDR4 common options	Enforce POR	Accept
AMD CBS / UMC common options / DDR4 common options / Enforce POR	Overclock	Enabled
AMD CBS / UMC common options / DDR4 common options / Enforce POR	Memory clock speed	1600 MHz	Set to max speed if using 3200 MHz DIMMs.
AMD CBS / UMC common options / DDR4 common options / DRAM controller configuration / DRAM power options	Power down enable	Disabled	RAM power down.
AMD CBS / security	TSME	Disabled	Memory encryption.

NBIO link clock frequency

The NBIOs (4× per AMD EPYC™ processor) are the serializers/deserializers (SerDes) that convert and prepare I/O signals for the processor’s 128 external I/O interface lanes (32 per NBIO). The NBIO link clock frequency (LCLK) controls the speed of the internal bus connecting NBIO silicon to the data fabric. All data between the processor and PCIe lanes flows through the data fabric at this frequency, so it must be forced to the maximum for optimal PCIe performance.

For AMD EPYC™ 7003-series processors, configuring all NBIOs to Enhanced Preferred I/O mode (see BIOS table above) is sufficient to enable the highest LCLK frequency. The AMD-IOPM-UTIL utility required for EPYC 7002 (MI100) is not needed on EPYC 7003 systems.

NPS memory configuration

For the number of NUMA nodes per socket (NPS), follow the guidance of the HPC Tuning Guide for AMD EPYC™ 7003 Series Processors for the optimal host configuration. For most HPC workloads, NPS4 is the recommended value.

Acceptance criteria

The MI250 system acceptance process validates that the platform is correctly configured, stable, and performing to expectations. Follow the sequence: Prerequisites → Basic Health Checks → System Validation → Performance Benchmarks.

System acceptance process

1. Prerequisites validation - Ensure all system requirements and dependencies are met

2. Basic health checks - Verify hardware detection and basic system health

3. System validation - Conduct comprehensive stress testing and qualification

4. Performance benchmarks - Validate compute, memory, and interconnect performance

The system is accepted when all criteria below are successfully validated.

Prerequisites validation

Ensure all system requirements are met before proceeding with validation. See the Prerequisites documentation and System setup for more details.

• ✅ Supported operating system version installed

• ✅ Compatible ROCm version installed (verify: cat /opt/rocm/.info/version); see the ROCm System Requirements for the current supported version matrix

• ✅ BIOS configured per MI250 BIOS settings (EPYC 7003-specific table above)

• ✅ Required kernel parameters present: pci=realloc=off iommu=pt

• ✅ Minimum 1T system memory available

• ✅ Latest applicable firmware applied consistently across nodes

• ✅ ROCm Validation Suite (RVS) installed

Basic health checks

These checks ensure fundamental system health and proper GPU detection. For detailed procedures, see Health Checks.

Test	Command	Pass/Fail criteria
Check OS distribution	cat /etc/os-release	Pass: OS version listed in compatibility matrix Fail: Otherwise
Check kernel boot arguments	cat /proc/cmdline	Pass: Contains pci=realloc=off iommu=pt Fail: Otherwise
Check for driver errors	sudo dmesg -T | grep amdgpu | grep -i error	Pass: Null Fail: Errors reported
Check available memory	lsmem | grep "Total online memory"	Pass: ≥ 1T Fail: Less than 1T
Check GPU presence	sudo lspci -d 1002:740c	Pass: 8 MI250 GCDs found Fail: Otherwise
Check GPU link speed and width	sudo lspci -d 1002:740c -vvv | grep -e DevSta -e LnkSta	Pass: Speed PCIe Gen 4 (16 GT/s), width x16, no FatalErr+ Fail: Otherwise
Monitor utilization metrics	amd-smi monitor -putm	Pass: Idle metrics as specified Fail: Otherwise
Check system kernel logs for errors	sudo dmesg -T | grep -i 'error|warn|fail|exception'	Pass: Null Fail: Otherwise

System validation

Comprehensive validation ensures system stability under load. For detailed procedures, see System Validation.

Test	Command	Pass/Fail criteria
Compute/GPU properties	rvs -c ${RVS_CONF}/gpup_single.conf	Pass: All GCDs listed with no errors Fail: Missing GCDs or errors
GPU stress test (GST)	rvs -c ${RVS_CONF}/MI250/gst_single.conf	Pass: met: TRUE in logs Fail: Target GFLOP/s not met
Input energy delay product (IET)	rvs -c ${RVS_CONF}/MI250/iet_single.conf	Pass: met: TRUE for all actions Fail: Otherwise
Memory test (MEM)	rvs -c ${RVS_CONF}/mem.conf -l mem.txt	Pass: All tests passed; bandwidth ≥ 1050 GB/s per GCD Fail: Any test failed or low bandwidth
PCIe bandwidth benchmark (PEBB)	rvs -c ${RVS_CONF}/MI250/pebb_single.conf	Pass: All distances and bandwidths displayed Fail: Missing data
PCIe qualification tool (PEQT)	rvs -c ${RVS_CONF}/peqt_single.conf	Pass: All actions true Fail: Otherwise
P2P benchmark and qualification tool (PBQT)	rvs -c ${RVS_CONF}/pbqt_single.conf	Pass: peers:true lines and non-zero throughput across all xGMI peers Fail: Otherwise

Performance benchmarks

Performance validation ensures the system meets MI250 specifications. For detailed procedures, see Performance Benchmarking.

TransferBench all-to-all

Command: TransferBench a2a

Pass: ≥ 800 GB/s aggregate

Fail: otherwise

TransferBench peer-to-peer

Command: TransferBench p2p

Test	Pass criteria
UniDir	≥ 30 GB/s
BiDir	≥ 55 GB/s

Fail: otherwise

RCCL Allreduce

Command: build/all_reduce_perf -b 8 -e 8G -f 2 -g 8

Pass: ≥ 125 GB/s busbw (peak, at 8 GiB message size)

Fail: otherwise

rocBLAS FP32

Command: rocblas-bench (see code block below)

rocblas-bench -f gemm \
  -r s -m 4000 -n 4000 -k 4000 \
  --lda 4000 --ldb 4000 --ldc 4000 \
  --transposeA N --transposeB T

Pass: ≥ 30 TFLOPS per GCD

Fail: otherwise

BabelStream

Command: mpiexec -n 8 wrapper.sh

Kernel	Threshold (MB/s)
Copy	≥ 1,200,000
Mul	≥ 1,200,000
Add	≥ 1,100,000
Triad	≥ 1,100,000
Dot	≥ 1,200,000

Fail: otherwise