AMD Instinct MI300A

AMD Instinct MI300A#

The AMD Instinct™ MI300A is a data-center Accelerated Processing Unit (APU) that integrates AMD “Zen 4” CPU cores and CDNA 3 GPU compute dies on a single package with unified HBM3 memory. This document provides MI300A-specific prerequisites, health checks, validation steps, and performance acceptance criteria.

Overview#

The MI300A is built on the CDNA 3 architecture (gfx942) and combines CPU and GPU compute dies sharing a single coherent pool of 128 GB HBM3 per APU. Unlike discrete OAM accelerators, MI300A platforms are vendor-defined; a typical qualified configuration hosts 4 MI300A APUs per node.

System requirements#

Operating system support#

For the most up-to-date information on supported operating systems and distributions, refer to the official ROCm documentation:

ROCm System Requirements - Supported Distributions

Note

ROCm docs is the single source of truth for supported versions, distribution compatibility, and required dependencies for the ROCm toolkit.

For BIOS, IOMMU, transparent hugepages, NUMA, and OS-level tuning that applies to all AMD Instinct hosts, see BIOS settings, OS tuning, and Kernel parameters. MI300A requires a Linux kernel that supports “Zen 4” (≥ 5.18 recommended).

GPU identification#

All MI300A APUs (PCI vendor:device 1002:74a0) should appear in lspci output:

sudo lspci -d 1002:74a0

Expected output example:

0000:01:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI300A]
0001:01:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI300A]
0002:01:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI300A]
0003:01:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI300A]

APU-specific OS tuning#

Because the MI300A integrates CPU and GPU on a single package with shared HBM3, several OS-level settings that are optional on discrete OAM systems are required or strongly recommended here.

Transparent huge pages#

The transparent_hugepage=always kernel parameter is required on MI300A. The unified CPU+GPU shared-memory model depends on transparent huge pages for correct and performant operation. Add it to GRUB_CMDLINE_LINUX and update GRUB as described in Kernel Parameters.

To verify the setting is active after boot:

cat /sys/kernel/mm/transparent_hugepage/enabled

The output should show [always] as the active selection.

Increase allocatable GPU memory (amdttm)#

By default, only 96 GiB of the 128 GiB HBM per APU is allocatable via HIP device allocations. To make the full 128 GiB available, load the amdttm module with updated page limits. The values correspond to the number of 4 KiB pages.

Check the current limit:

cat /sys/module/amdttm/parameters/pages_limit

To set the full 128 GiB across all four APUs on a single-node system (total 512 GiB), run:

sudo modprobe amdttm pages_limit=134217728 page_pool_size=134217728
sudo modprobe amdgpu

Note

For a two-socket node with two APUs, halve the values: pages_limit=67108864 page_pool_size=67108864. The per-APU limit is 33554432 pages × number of APUs on the node.

To make the setting persistent, add it to /etc/modprobe.d/amdttm.conf:

options amdttm pages_limit=134217728
options amdttm page_pool_size=134217728

If the filesystem is not yet mounted when the kernel driver loads (for example, on early-boot initrd), pass the values as boot parameters instead:

amdttm.pages_limit=134217728 amdttm.page_pool_size=134217728

Verify the new setting:

cat /sys/module/amdttm/parameters/pages_limit

Limit GPU memory allocation percentage#

Many AI applications were originally developed for discrete GPUs and probe available memory until exhaustion. On the MI300A APU, where CPU and GPU share the same HBM pool, this can starve the OS of memory.

Use the following environment variables to bound how much of the shared HBM the GPU side can claim:

export GPU_MAX_ALLOC_PERCENT=<value>
export GPU_SINGLE_ALLOC_PERCENT=<value>

Setting either variable to 100 allows the GPU to allocate all free memory. Before doing so, account for expected CPU workload and OS usage. For example, on a 128 GiB system where the OS requires 8 GiB, setting GPU_MAX_ALLOC_PERCENT=100 authorizes a single workload to allocate up to 120 GiB. Any over-allocation is handled by the kernel’s out-of-memory (OOM) policies unless swap space is configured.

Add these variables to /etc/environment or the user shell environment.

Enable memory compaction#

The MI300A APU dynamically shares memory between the CPU and GPU. Without periodic memory compaction, fragmentation accumulates and application performance degrades over time. Compaction is enabled by default on Ubuntu but disabled by default on RHEL-based distributions.

Enable proactive compaction:

echo 20 > /proc/sys/vm/compaction_proactiveness
echo 1 > /proc/sys/vm/compact_unevictable_allowed

Acceptance criteria#

The MI300A system acceptance process validates that the platform is correctly configured, stable, and performing to expectations. Follow the sequence: Prerequisites → Basic Health Checks → System Validation → Performance Benchmarks.

System acceptance process#

  1. Prerequisites validation - Ensure all system requirements and dependencies are met

  2. Basic health checks - Verify hardware detection and basic system health

  3. System validation - Conduct comprehensive stress testing and qualification

  4. Performance benchmarks - Validate compute, memory, and interconnect performance

The system is accepted when all criteria below are successfully validated.

Prerequisites validation#

Ensure all system requirements are met before proceeding with validation. See the Prerequisites documentation and System setup for more details.

  • ✅ Supported operating system version installed with kernel ≥ 5.18 (Zen 4 support)

  • ✅ Compatible ROCm version installed (verify: cat /opt/rocm/.info/version); see the ROCm System Requirements for the current supported version matrix

  • ✅ BIOS configured per BIOS settings, with MI300A-specific values per platform vendor (IOMMU off, memory interleaving, NPS)

  • ✅ Required kernel parameters present: pci=realloc=off transparent_hugepage=always numa_balancing=disable

  • ✅ Sysctl tunings applied: vm.compaction_proactiveness=20 and vm.compact_unevictable_allowed=1 (required on RHEL-based systems; enabled by default on Ubuntu) — see Enable memory compaction

  • ✅ amdttm page pool configured for full 128 GiB HBM access per APU — see Increase allocatable GPU memory

  • ✅ Environment variables (where applicable):

  • ✅ Minimum 4 × 128 GB = 512 GB unified HBM3 visible to the OS usable host-visible memory (note: MI300A’s HBM is unified with CPU)

  • ✅ Latest applicable firmware applied consistently across nodes

  • ✅ ROCm Validation Suite (RVS) installed

Basic health checks#

These checks ensure fundamental system health and proper APU detection. For detailed procedures, see Health Checks.

Test

Command

Pass/Fail criteria

Check OS distribution

cat /etc/os-release

Pass: OS version listed in compatibility matrix; kernel ≥ 5.18
Fail: Otherwise

Check kernel boot arguments

cat /proc/cmdline

Pass: Contains pci=realloc=off transparent_hugepage=always numa_balancing=disable
Fail: Missing any required param

Verify IOMMU is off (MI300A-specific)

sudo apt install acpica-tools && sudo acpidump | grep IVRS && sudo acpidump | grep DMAR

Pass: Both commands return no output (IOMMU disabled)
Fail: Either command returns output

Check for driver errors

sudo dmesg -T | grep amdgpu | grep -i error

Pass: Null
Fail: Errors reported

Check available memory

lsmem | grep "Total online memory"

Pass: ≥ 4 × 128 GB = 512 GB unified HBM3 visible to the OS
Fail: Less than 4 × 128 GB = 512 GB unified HBM3 visible to the OS

Check GPU presence

sudo lspci -d 1002:74a0

Pass: 4 MI300A APUs found
Fail: Otherwise

Check GPU link speed and width

sudo lspci -d 1002:74a0 -vvv | grep -e DevSta -e LnkSta

Pass: Speed PCIe Gen 5 (32 GT/s), width x16, no FatalErr+
Fail: Otherwise

Monitor utilization metrics

amd-smi monitor -putm

Pass: Idle metrics as specified
Fail: Otherwise

Check system kernel logs for errors

sudo dmesg -T | grep -i 'error|warn|fail|exception'

Pass: Null
Fail: Otherwise

System validation#

Comprehensive validation ensures system stability under load. For detailed procedures, see System Validation.

Test

Command

Pass/Fail criteria

Compute/GPU properties

rvs -c ${RVS_CONF}/gpup_single.conf

Pass: All APUs listed with no errors
Fail: Missing APUs or errors

GPU stress test (GST)

rvs -c ${RVS_CONF}/MI300A/gst_single.conf

Pass: met: TRUE in logs
Fail: Target GFLOP/s not met

Input energy delay product (IET)

rvs -c ${RVS_CONF}/MI300A/iet_single.conf

Pass: met: TRUE for all actions
Fail: Otherwise

Memory test (MEM)

rvs -c ${RVS_CONF}/mem.conf -l mem.txt

Pass: All tests passed; bandwidth ≥ 2.0 TB/s per APU
Fail: Any test failed or low bandwidth

PCIe bandwidth benchmark (PEBB)

rvs -c ${RVS_CONF}/MI300A/pebb_single.conf

Pass: All distances and bandwidths displayed
Fail: Missing data

PCIe qualification tool (PEQT)

rvs -c ${RVS_CONF}/peqt_single.conf

Pass: All actions true
Fail: Otherwise

P2P benchmark and qualification tool (PBQT)

rvs -c ${RVS_CONF}/pbqt_single.conf

Pass: peers:true lines and non-zero throughput across all xGMI peers
Fail: Otherwise

Performance benchmarks#

Performance validation ensures the system meets MI300A specifications. For detailed procedures, see Performance Benchmarking.

TransferBench all-to-all

Command: TransferBench a2a

Pass: ≥ 700 GB/s aggregate

Fail: otherwise

TransferBench peer-to-peer

Command: TransferBench p2p

Test

Pass criteria

UniDir

≥ 80 GB/s

BiDir

≥ 155 GB/s

Fail: otherwise

RCCL Allreduce

Command: build/all_reduce_perf -b 8 -e 8G -f 2 -g 4

Pass: ≥ 230 GB/s busbw (peak, at 8 GiB message size)

Fail: otherwise

rocBLAS FP32

Command: rocblas-bench (see code block below)
rocblas-bench -f gemm \
  -r s -m 4000 -n 4000 -k 4000 \
  --lda 4000 --ldb 4000 --ldc 4000 \
  --transposeA N --transposeB T

Pass: ≥ 60 TFLOPS per APU

Fail: otherwise

BabelStream

Command: mpiexec -n 4 wrapper.sh

Kernel

Threshold (MB/s)

Copy

≥ 2,900,000

Mul

≥ 3,000,000

Add

≥ 3,250,000

Triad

≥ 3,250,000

Dot

≥ 2,200,000

Fail: otherwise

Contents