# AMD Instinct MI325X

The AMD Instinct™ MI325X is a high-performance GPU accelerator designed for AI, HPC, and demanding workloads. This document provides a comprehensive overview of MI325X-specific requirements, specifications, and acceptance testing criteria.

## Overview

The AMD Instinct MI325X is a high-performance GPU accelerator designed for AI, HPC, and demanding workloads. The MI325X platform utilizes a Universal Baseboard (UBB 2.0) configuration that hosts 8 AMD Instinct MI325X OAM (OCP Accelerator Module) accelerators with a total of 2TB of HBM3 memory.

Each MI325X GPU features multiple chiplet design with 8 XCDs (Accelerator Complex Dies) and 256GB of HBM3 memory per accelerator. It utilizes AMD's CDNA 3 architecture with fully-meshed Infinity Fabric™ connectivity between accelerators.

- **[MI352X Product Page](https://www.amd.com/en/products/accelerators/instinct/mi300/mi325x.html)**
- **[MI325X Data Sheet](https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/product-briefs/instinct-mi325x-datasheet.pdf)**
- **[MI300 Series Microarchitecture](https://instinct.docs.amd.com/latest/gpu-arch/mi300.html)**

## System Requirements

### Operating System Support

For the most up-to-date information on supported operating systems and distributions, please refer to the official ROCm documentation:

[ROCm System Requirements - Supported Distributions](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html#supported-distributions)

```{note}
[ROCm docs](https://rocm.docs.amd.com) is the single source of truth for supported versions, distribution compatibility, and required dependencies for the ROCm toolkit.
```

### GPU Identification

All MI325X GPUs (device ID 1002:74a5) should appear in `lspci` output:

```bash
sudo lspci -d 1002:74a5
```

Expected output example:

```bash
05:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Device 74a5
26:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Device 74a5
46:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Device 74a5
65:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Device 74a5
85:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Device 74a5
a6:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Device 74a5
c6:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Device 74a5
e5:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Device 74a5
```

## Acceptance Criteria

The MI325X system acceptance process is designed to validate that your system is operating correctly and meets all performance specifications. Users are expected to step through the validation guides in sequence to ensure comprehensive system verification.

### System Acceptance Process

1. **[Prerequisites Validation](#prerequisites-validation)** - Ensure all system requirements and dependencies are met
2. **[Basic Health Checks](#basic-health-checks)** - Verify hardware detection and basic system health
3. **[System Validation](#system-validation)** - Conduct comprehensive stress testing and qualification
4. **[Performance Benchmarks](#performance-benchmarks)** - Validate compute, memory, and interconnect performance

The system is accepted when all criteria below are successfully validated:

### Prerequisites Validation

Ensure all system requirements are met before proceeding with validation. See the [Prerequisites documentation](../common/prerequisites.md) and [System setup](../common/system-setup.md) for more details.

- ✅ Supported operating system version installed
- ✅ ROCm 6.3.2 or later installed
- ✅ System manufacturer compatibility verified
- ✅ All required dependencies installed

### Basic Health Checks

These checks ensure fundamental system health and proper GPU detection. For detailed procedures, see [Health Checks](../common/health-checks.md).

| Test | Command | Pass/Fail criteria |
|------|---------|-------------------|
| [Check OS distribution](../common/health-checks.md#check-os-distribution) |`cat /etc/os-release` | **Pass**: OS version listed in compatibility matrix<br>**Fail**: Otherwise |
| [Check kernel boot arguments](../common/health-checks.md#check-kernel-boot-arguments) | `cat /proc/cmdline` | **Pass**: Contains `pci=realloc=off`, `pci=bfsort`, `iommu=pt`, `numa_balancing=disable`, `modprobe.blacklist=amdgpu` (+ `intel_iommu=on` if Intel CPU)<br>**Fail**: Otherwise |
| [Check available memory](../common/health-checks.md#check-for-available-system-memory) | `free -h` | **Pass**: 2.5T or more<br>**Fail**: Less than 2.5T |
| [Check GPU presence](../common/health-checks.md#check-gpu-presence) | `sudo lspci -d 1002:74a5` | **Pass**: All 8 GPUs found<br>**Fail**: Otherwise |
| [Check GPU link speed and width](../common/health-checks.md#check-gpu-pcie-bus-link-speed-and-width) | `sudo lspci -d 1002:74a5 -vvv \| grep -e DevSta -e LnkSta` | **Pass**: Speed 32GT/s, width `x16`, no `FatalErr+`<br>**Fail**: Otherwise |
| [Monitor utilization metrics](../common/health-checks.md#monitor-utilization-metrics) | `amd-smi monitor -putm` | **Pass**: Idle metrics as specified<br>**Fail**: Otherwise |
| [Check system kernel logs for errors](../common/health-checks.md#check-system-kernel-logs) | `sudo dmesg -T \| grep -i 'error\|warn\|fail\|exception'` | **Pass**: Null<br>**Fail**: Otherwise |

### System Validation

Comprehensive validation ensures system stability under load. For detailed procedures, see [System Validation](../common/system-validation.md).

| Test | Command | Pass/Fail criteria |
|------|---------|-------------------|
| [Compute/GPU properties](../common/system-validation.md#gpu-properties) | `rvs -c ${RVS_CONF}/gpup_single.conf` | **Pass**: All GPUs listed with no errors<br>**Fail**: Missing GPUs or errors |
| [GPU stress test (GST](../common/system-validation.md#gpu-stress-test) | `rvs -c ${RVS_CONF}/MI325X/gst_single.conf` | **Pass**: `met: TRUE` in logs<br>**Fail**: Target GFLOP/s not met |
| [Input energy delay product (IET](../common/system-validation.md#input-energy-delay-product) | `rvs -c ${RVS_CONF}/MI325X/iet_single.conf` | **Pass**: `met: TRUE` for all actions<br>**Fail**: Otherwise |
| [Memory test (MEM](../common/system-validation.md#mem) | `rvs -c ${RVS_CONF}/mem.conf -l mem.txt` | **Pass**: All tests passed; bandwidth ~2TB/s<br>**Fail**: Any test failed or low bandwidth |
| [PCIe bandwidth benchmark (PEBB](../common/system-validation.md#pcie-bandwidth-benchmark) | `rvs -c ${RVS_CONF}/MI325X/pebb_single.conf` | **Pass**: All distances and bandwidths displayed<br>**Fail**: Missing data |
| [PCIe qualification tool (PEQT](../common/system-validation.md#pcie-qualification-tool) | `rvs -c ${RVS_CONF}/peqt_single.conf` | **Pass**: All actions true<br>**Fail**: Otherwise |
| [P2P benchmark and qualification tool (PBQT](../common/system-validation.md#p2p-benchmark-and-qualification-tool) | `rvs -c ${RVS_CONF}/pbqt_single.conf` | **Pass**: `peers:true` lines and non-zero throughput<br>**Fail**: Otherwise |

### Performance Benchmarks

Performance validation ensures the system meets MI325X specifications. For detailed procedures, see the [Performance Benchmarking](../common/system-validation.md#performance-benchmarking).

:::{card} Command: `TransferBench a2a`
[TransferBench all-to-all](../common/system-validation.md#transferbench)
^^^
**Pass:** ≥ 32.9 GB/s
+++
**Fail:** otherwise
:::

:::{card} Command: `TransferBench p2p`
[TransferBench peer-to-peer](../common/system-validation.md#transferbench)
^^^

| Test | Pass Criteria |
|------|--------------|
| UniDir | ≥ 33.9 GB/s |
| BiDir | ≥ 43.9 GB/s |

+++
**Fail:** otherwise
:::

:::{card} Command: `TransferBench example.cfg`
[TransferBench config tests (1–6](../common/system-validation.md#transferbench)
^^^

| **Test** | **Pass Criteria** |
|------|--------------|
| Test 1 | ≥ 47.1 GB/s |
| Test 2 | ≥ 48.4 GB/s |
| Test 3 | ≥ 31.9 (0→1), ≥ 38.9 (1→0) GB/s |
| Test 4 | ≥ 1264 GB/s |
| Test 5 | N/A (GPU validation) |
| Test 6 | ≥ 48.6 GB/s |

+++
**Fail:** otherwise
:::

:::{card} Command: `build/all_reduce_perf -b 8 -e 8G -f 2 -g 8`
[RCCL Allreduce](../common/system-validation.md#rccl-allreduce)
^^^
**Pass:** ≥ 304 GB/s
+++
**Fail:** otherwise
:::

:::{card} Command:
[rocBLAS FP32](../common/system-validation.md#rocblas-gemm-benchmarks)
^^^

```bash
rocblas-bench -f gemm \
  -r s -m 4000 -n 4000 -k 4000 \
  --lda 4000 --ldb 4000 --ldc 4000 \
  --transposeA N --transposeB T
```

**Pass:** ≥ 94100 GFLOPS  
+++
**Fail:** otherwise
:::

:::{card} Command:
[rocBLAS BF16](../common/system-validation.md#rocblas-gemm-benchmarks)
^^^

```bash
rocblas-bench -f gemm_strided_batched_ex \
  --transposeA N --transposeB T \
  -m 1024 -n 2048 -k 512 \
  --a_type h --lda 1024 --stride_a 4096 \
  --b_type h --ldb 2048 --stride_b 4096 \
  --c_type s --ldc 1024 --stride_c 2097152 \
  --d_type s --ldd 1024 --stride_d 2097152 \
  --compute_type s \
  --alpha 1.1 --beta 1 \
  --batch_count 5
```

**Pass:** ≥ 130600 GFLOPS
+++
**Fail:** otherwise
:::

:::{card} Command:
[rocBLAS INT8](../common/system-validation.md#rocblas-gemm-benchmarks)
^^^

```bash
rocblas-bench -f gemm_strided_batched_ex \
  --transposeA N --transposeB T \
  -m 1024 -n 2048 -k 512 \
  --a_type i8_r --lda 1024 --stride_a 4096 \
  --b_type i8_r --ldb 2048 --stride_b 4096 \
  --c_type i32_r --ldc 1024 --stride_c 2097152 \
  --d_type i32_r --ldd 1024 --stride_d 2097152 \
  --compute_type i32_r \
  --alpha 1.1 --beta 1 \
  --batch_count 5
```

**Pass:** ≥ 162700 GFLOPS
+++
**Fail:** otherwise
:::