# Release Notes ## v1.5.2 - **ROCm 10 Libraries** - Device Metrics Exporter is now based on ROCm 10 libraries - **Configurable Metric Collection Filter (Performance)** - GPU Agent now derives a server-side collection filter from the enabled metric configuration, skipping collectors that no configured metric or label requires (clock, XGMI status/stats, process, VRAM usage, violation, PCIe stats, activity). This reduces per-scrape collection cost on nodes that enable only a subset of metrics. - ECC and PCIe status are always collected because GPU health validation depends on them. The filter is re-derived on the 3-second config auto-reload. - **Runtime Toggle for Debug Endpoints (`CommonConfig.Debug.EnableAPI`)** - A new `CommonConfig.Debug.EnableAPI` config option (default **false**) gates the debug surfaces: the pprof/expvar profiling endpoints on the metrics HTTP port and the error-injection (`SetError`) gRPC API. ### Issues Fixed - **CVE remediation** ### Known Issues - **PCIe bandwidth metric adds ~1s per GPU to scrape latency on MI250** - On MI250 nodes, reading PCIe bandwidth takes about 1 second per GPU because the kernel `pcie_bw` sysfs sampler blocks for ~1s per read. On multi-GPU nodes this serialized cost can dominate scrape latency and, in dense configurations, lead to empty or delayed metric responses. - **Workaround**: Disable the PCIe bandwidth metric in the ConfigMap on affected MI250 nodes if scrape latency is a concern. ### Platform Support ROCm 6.2 or later, MI2xx, MI3xx ROCm 7.13 or later, MI350P, Radeon AI ## v1.5.1 - **New Platform Support** - MI350P and Radeon AI platforms are now supported - **SUSE Linux Enterprise Server (SLES) Support** - Device Metrics Exporter now supports deployment on SLES 15 and SLES 16 GPU worker nodes on vanilla Kubernetes. - The AMD GPU Operator is listed as **SUSE Ready** for SLES 15 and SLES 16 in the [SUSE Partner Software Catalog](https://www.suse.com/pcsc/viewVersionPage?versionID=26965). - **SR-IOV Exporter Image** - New image `device-metrics-exporter`, tag `sriov-v1.0.0` (also available as a Debian package), for MI-series SR-IOV/GIM hypervisor hosts. See [MxGPU-Virtualization releases](https://github.com/amd/MxGPU-Virtualization/releases) - **Helm Chart: Configurable Exporter Arguments** - The Helm chart now supports passing arbitrary arguments to the exporter at deploy time - New deployment options available: - `--exit-on-rocpctl-error`: Exit instead of auto-disabling profiler metrics after consecutive errors - `--exit-on-agent-down`: Exit when the GPU agent is unreachable or reports a GPU error. **Note:** affects health monitoring continuity during pod restart window > The image ships with recommended defaults built in; Helm chart args allow selective overrides per deployment environment. - **Kubernetes Event on Profiler Auto-Disable** - The exporter now emits a Kubernetes event when it automatically disables profiler metrics as part of failure mitigation, making the state change visible without requiring log inspection ### Issues Fixed - **CPER-based health detection enabled for MI350P and Radeon AI** - Added null pointer and out-of-bounds guards in the CPER code path; CPER-based health detection is now functional on MI350P and Radeon AI platforms - **`GPU_AFID_ERRORS` re-enabled for MI350P and Radeon AI** - The `GPU_AFID_ERRORS` field is now functional on MI350P and Radeon AI platforms - **GPU Agent no longer holds `/dev/kfd` open for its lifetime** - GPU Agent previously kept `/dev/kfd` open permanently, blocking GPU partition changes and driver upgrades without a full service restart. The file descriptor is now released when not in use - **Kubernetes watchers no longer start unconditionally** - Node and pod watchers previously started at boot regardless of whether they were needed. They now start lazily based on `HealthService.Enable` and `ExtraPodLabels` configuration, and their state changes are now visible via Kubernetes Events - **`GPU_CPER_MAX_AGE` added to filter stale CPER records** - A new `HealthThresholds` field lets deployments ignore fatal CPER records older than a configured age when determining GPU health, avoiding stale records marking a healthy GPU as unhealthy. See [Kubernetes configuration](configuration/configmap.md) - **`gfx_activity` idle uptick on Radeon AI resolved** - On Radeon AI platforms, `gfx_activity` now shows true idle values. ### Platform Support ROCm 6.2 or later, MI2xx, MI3xx ROCm 7.13 or later, MI350P, Radeon AI ## v1.5.1-beta - **New Platform Support** - MI350P and Radeon AI platforms are now supported - **Helm Chart: Configurable Exporter Arguments** - The Helm chart now supports passing arbitrary arguments to the exporter at deploy time - New deployment options available: - `--exit-on-rocpctl-error`: Exit instead of auto-disabling profiler metrics after consecutive errors - `--exit-on-agent-down`: Exit when the GPU agent is unreachable or reports a GPU error. **Note:** affects health monitoring continuity during pod restart window > The image ships with recommended defaults built in; Helm chart args allow selective overrides per deployment environment. ### Known Issues - **CPER-based health detection is disabled** - CPER-based health detection is not functional in this release for MI350P and Radeon AI platforms - **AFID field is disabled** - The `GPU_AFID_ERRORS` field is disabled in this release for MI350P and Radeon AI platforms - **`gfx_activity` idle uptick on Radeon AI** - On Radeon AI platforms, `gfx_activity` may show a spurious uptick when the GPU is idle. This is a cosmetic issue and does not indicate actual GPU activity ### Platform Support ROCm 6.2 or later, MI2xx, MI3xx ROCm 7.13 or later, MI350P, Radeon AI ## v1.5.0 - **Unix Domain Socket For IPC** - The GPU Agent now uses Unix Domain Socket (`/var/run/gpuagent.sock`) for communication with the metrics exporter - Secure and Improved performance Lower latency compared to TCP/IP connections - **New Metrics** - `GPU_PROCESS_CU_OCCUPANCY` to report Compute Unit occupancy for each process using the GPU, with `process_id` label to differentiate between processes - `GPU_ECC_DEFERRED_*` for each ECC supported block - **Profiler Configuration Enhancement** - `SamplingInterval` to set sampling window for the profiler metrics - Default value is 1000 microseconds (1 millisecond) - **Configurable Health Polling Rate** - Added `PollingRate` field to `HealthService` configuration in `CommonConfig` - Supports duration formats: 30s, 5m, 1h, 1d, 23h10m15s - Default: 30 seconds, Min: 30 seconds, Max: 24 hours - **Slinky** - Slinky job reporting is now supported, with labels providing both Kubernetes and Slurm job information - **KFD_PROCESS_ID Label Now Optional** - The `KFD_PROCESS_ID` label is no longer mandatory and is disabled by default to reduce metric cardinality - This change helps optimize Prometheus storage and query performance for deployments that don't require process-level tracking - For users who need this label, simply add it to the ConfigMap configuration. See [Configuration Documentation](configuration/configmap.md) for details ### Known Issues - **Power metrics intermittently missing for individual GPUs on MI350X (SPX/NPS1 mode)** - `gpu_package_power` and `gpu_power_usage` may be absent for one or more GPUs on MI350X nodes in SPX/NPS1 mode. The affected GPU is present in all other metrics (temperature, VRAM, clocks) in the same response. - **Root cause**: If the first metrics poll after exporter startup returns `0` for a GPU's power (which occurs transiently on MI350X when PMFW/SMU telemetry is not yet populated), the exporter suppresses that GPU's power metrics until the exporter is restarted. - **Workaround**: Restart the exporter pod after the GPU has been running for a few seconds so the first poll sees non-zero power. ### Platform Support ROCm 6.2 or later, MI2xx, MI3xx ## v1.4.2 - **New Label** - `KFD_PROCESS_ID` label will now report the process ID using the respective GPU. This enables baremetal debian deployments to have job information where no scheduler is used. - `DEPLOYMENT_MODE` label to specify the GPU operating environment - **New Field** - `GPU_AFID_ERRORS` field added to report RAS events associated AMD Field Identifier (AFID) list, More details can be found at [Metrics List page](configuration/metricslist.md#afid-error-metrics) - More Info on AMD Field ID and next steps are https://docs.amd.com/r/en-US/AMD_Field_ID_70122_v1.0/AFID-Event-List - Violation Metrics 1.8 version fields - Current percentage and Per Compute Core violation metrics available for - `GPU_VIOLATION_PROCESSOR_HOT_RESIDENCY_PERCENTAGE` - `GPU_VIOLATION_PPT_RESIDENCY_PERCENTAGE` - `GPU_VIOLATION_SOCKET_THERMAL_RESIDENCY_PERCENTAGE` - `GPU_VIOLATION_VR_THERMAL_RESIDENCY_PERCENTAGE` - `GPU_VIOLATION_HBM_THERMAL_RESIDENCY_PERCENTAGE` - `GPU_VIOLATION_GFX_CLOCK_BELOW_HOST_LIMIT_POWER_ACCUMULATED` - `GPU_VIOLATION_GFX_CLOCK_BELOW_HOST_LIMIT_THERMAL_ACCUMULATED` - `GPU_VIOLATION_LOW_UTILIZATION_ACCUMULATED` - `GPU_VIOLATION_GFX_CLOCK_BELOW_HOST_LIMIT_TOTAL_ACCUMULATED` - `GPU_VIOLATION_GFX_CLOCK_BELOW_HOST_LIMIT_POWER_PERCENTAGE` - `GPU_VIOLATION_GFX_CLOCK_BELOW_HOST_LIMIT_THERMAL_PERCENTAGE` - `GPU_VIOLATION_LOW_UTILIZATION_PERCENTAGE` - `GPU_VIOLATION_GFX_CLOCK_BELOW_HOST_LIMIT_TOTAL_PERCENTAGE` - Clock Metrics Added `GPU_MIN_CLOCK`, `GPU_MAX_CLOCK` - **Label Value Change** - Clock type was not normalized in previous releases; now clock type label values are normalized without the `GPU_CLOCK_TYPE_` prefix. More details can be found at [Metrics list page](./configuration/metricslist.md#clock-metrics) ### Platform Support ROCm 6.2 or later, MI2xx, MI3xx ## v1.4.1.2 ### Issues Fixed - **Security Vulnerability Fix (CVE-2026-0481, AMD-SB-6031)** - CVE ID: CVE-2026-0481 - CVSS 3.1 Base Score: 9.2 (Critical) - CWE-1327: Binding to an Unrestricted IP Address - `gpuagent.service` internal service port was hosted on all interfaces instead of only localhost - The internal service is now only hosted on localhost and is configurable through service files. More details can be found at [Installation section](installation/deb-package.rst) - **Workaround for prior version** - Add firewall rules in front of the service to block inbound connections, allowing only traffic originating from localhost (127.0.0.1). ## v1.4.1 - **Configurable Resource Limits** - Exporter Pod resource limits can now be configured through the Helm chart - Default limits are set if not specified in the Helm chart - Debian systemd service file is now set with default resource limits - Beta: DRA (Dynamic Resource Allocation) device support — exporter can detect and report DRA-allocated GPU devices from the device plugin/pod resources. - **Profiler Failure Handling** - Profiler metrics are sensitive to hardware failures. To better handle potential system disruption from coredumps or profiler failures, the exporter will automatically disable profiler metrics to maintain critical exporter functionality and server stability: - **Coredump**: Profiler metrics will be disabled immediately upon detection - **Non-crash errors**: If the profiler metrics read encounters 3 consecutive non-crash errors, it will be automatically disabled - **Re-enabling**: Profiler metrics can be re-enabled by restarting/deleting the exporter pod on that node/server - **_Note_** Profiler metrics are prefixed by `gpu_prof_` in [metrics list](./configuration/metricslist.md) ### Platform Support ROCm 7.1.1 MI2xx, MI3xx; Kubernetes 1.34+ (required only for DRA beta feature) ## v1.4.0.1 ### Issues Fixed - **Security Vulnerability Fix (CVE-2026-0481, AMD-SB-6031)** - CVE ID: CVE-2026-0481 - CVSS 3.1 Base Score: 9.2 (Critical) - CWE-1327: Binding to an Unrestricted IP Address - `gpuagent.service` internal service port was hosted on all interfaces instead of only localhost - The internal service is now only hosted on localhost and is configurable through service files. More details can be found at [Installation section](installation/deb-package.rst) ## v1.4.0 - **MI35x Platform Support** - Exporter now supports MI35x platform with parity with latest supported fields. - **Mask Unsupported Fields** - Platform-specific unsupported fields (amd-smi marked as N/A) will not be exported. Boot logs will indicate which fields are supported by the platform (logged once during startup). - **New Profiler Fields** - New fields are added for better understanding of the application - **Deprecated Fields Notice** - Following fields are deprecated from 6.14.14 driver onwards - GPU_MMA_ACTIVITY - GPU_JPEG_ACTIVITY - GPU_VCN_ACTIVITY - These fields are replaced by following fields - GPU_VCN_BUSY_INSTANTANEOUS - GPU_JPEG_BUSY_INSTANTANEOUS ### Platform Support ROCm 7.0 MI2xx, MI3xx ### Issues Fixed - fixed metric naming discrepancies between config field and exported field. The following prometheus fields name are being changed. - one config field is being renamed which would require updating the config.json from the released branch for `pcie_nac_received_count` -> `pcie_nack_received_count` | S.No | Old Field Name | New Field Name | |------|-------------------------------------------------|----------------------------------------------------| | 1 | xgmi_neighbor_0_nop_tx | gpu_xgmi_nbr_0_nop_tx | | 2 | xgmi_neighbor_1_nop_tx | gpu_xgmi_nbr_1_nop_tx | | 3 | xgmi_neighbor_0_request_tx | gpu_xgmi_nbr_0_req_tx | | 4 | xgmi_neighbor_0_response_tx | gpu_xgmi_nbr_0_resp_tx | | 5 | xgmi_neighbor_1_response_tx | gpu_xgmi_nbr_1_resp_tx | | 6 | xgmi_neighbor_0_beats_tx | gpu_xgmi_nbr_0_beats_tx | | 7 | xgmi_neighbor_1_beats_tx | gpu_xgmi_nbr_1_beats_tx | | 8 | xgmi_neighbor_0_tx_throughput | gpu_xgmi_nbr_0_tx_thrput | | 9 | xgmi_neighbor_1_tx_throughput | gpu_xgmi_nbr_1_tx_thrput | | 10 | xgmi_neighbor_2_tx_throughput | gpu_xgmi_nbr_2_tx_thrput | | 11 | xgmi_neighbor_3_tx_throughput | gpu_xgmi_nbr_3_tx_thrput | | 12 | xgmi_neighbor_4_tx_throughput | gpu_xgmi_nbr_4_tx_thrput | | 13 | xgmi_neighbor_5_tx_throughput | gpu_xgmi_nbr_5_tx_thrput | | 14 | gpu_violation_vr_thermal_tracking_accumulated | gpu_violation_vr_thermal_residency_accumulated | | 15 | pcie_nac_received_count | pcie_nack_received_count | | 16 | gpu_violation_proc_hot_residency_accumulated | gpu_violation_processor_hot_residency_accumulated | | 17 | gpu_violation_soc_thermal_residency_accumulated | gpu_violation_socket_thermal_residency_accumulated | ## v1.3.1 ### Release Highlights - **New Metric Fields** - GPU_GFX_BUSY_INSTANTANEOUS, GPU_VC_BUSY_INSTANTANEOUS, GPU_JPEG_BUSY_INSTANTANEOUS are added to represent partition activities at more granular level. - GPU_GFX_ACTIVITY is only applicable for unpartitioned systems, user must rely on the new BUSY_INSTANTANEOUS fields on partitioned systems. - **Health Service Config** - Health services can be disabled through configmap - **Profiler Metrics Default Config Change** - The previous release of exporter i.e. v1.3.0's ConfigMap present under example directory had Profiler Metrics enabled by default. Now, this is set to be disabled by default from v1.3.1 onwards, because profiling is generally needed only by application developers. If needed, please enable it through the ConfigMap and make sure that there is no other Exporter instance or another tool running ROCm profiler at the same time. - **Notice: Exporter Handling of Unsupported Platform Fields (Upcoming Major Release)** - Current Behavior: The exporter sets unsupported platform-specific field metrics to 0. - Upcoming Change: In the next major release, the exporter will omit unsupported fields (e.g., those marked as N/A in amd-smi) instead of exporting them as 0. - Logging: Detailed logs will indicate which fields are unsupported, allowing users to verify platform compatibility. ## v1.3.0 ### Release Highlights - **K8s Extra Pod Labels** - Adds more granular Pod level details as labels meta data through configmap `ExtraPodLabels` - **Support for Singularity Installation** - Exporter can now be deployed on HPC systems through singularity. - **Performance Metrics** - Adds more profiler related metrics on supported platforms, with toggle functionality through configmap `ProfilerMetrics` - **Custom Prefix for Exporter** - Adds more flexibility to add custom prefix to better identify AMD GPU on multi cluster deployment, through configmap `CommonConfig` ### Platform Support ROCm 6.4.x MI3xx ## v1.2.1 ### Release Highlights - **Prometheus Service Monitor** - Easy integration with Prometheus Operator - **K8s Toleration and Selector** - Added capability to add tolerations and nodeSelector during helm install ### Platform Support ROCm 6.3.x ## v1.2.0 ### Release Highlights - **GPU Health Monitoring** - Real-time health checks via **metrics exporter** - With **Kubernetes Device Plugin** for automatic removal of unhealthy GPUs from compute node schedulable resources - Customizable health thresholds via K8s ConfigMaps ### Platform Support ROCm 6.3.x ## v1.1.0 ### Platform Support ROCm 6.3.x ## v1.0.0 ### Release Highlights - **GPU Metrics Exporter for Prometheus** - Real-time metrics exporter for GPU MI platforms. ### Platform Support ROCm 6.2.x