# Prometheus Integration with Metrics Exporter

The AMD Network Operator integrates with Prometheus to enable monitoring of Network metrics across the Kubernetes cluster. Metrics exposed by the Device Metrics Exporter can be automatically discovered and scraped by Prometheus through the creation of a ServiceMonitor resource.

Prometheus integration is managed via the **ServiceMonitor** configuration in the NetworkConfig Custom Resource (CR). When enabled, the operator automatically creates a ServiceMonitor tailored to the metrics exported by the Device Metrics Exporter. The integration supports various authentication and authorization methods, including Bearer Tokens and mutual TLS (mTLS), providing flexibility to accommodate different security requirements.

## Prerequisites

Before enabling Prometheus integration, ensure you have:

- A running instance of the Prometheus Operator in your Kubernetes cluster.
- The Device Metrics Exporter enabled in your Network Operator deployment.
- Properly configured kube-rbac-proxy in the NetworkConfig CR if the exporter endpoint is protected (Optional).

The AMD Network Operator relies on the ServiceMonitor CRD being available to inject the CRs when enabled. This CRD is installed by the Prometheus Operator.

## NetworkConfig Configuration

To integrate Prometheus, configure the following section in the NetworkConfig CR under `metricsExporter.prometheus.serviceMonitor`:

```yaml
metricsExporter:
  enable: true
  prometheus:
    serviceMonitor:
      enable: true
      interval: "60s" # Scrape frequency
      attachMetadata:
        node: true
      honorLabels: false
      honorTimestamps: true
      labels:
        release: prometheus-operator # Prometheus release label for target discovery
```

- **enable**: Enable or disable Prometheus ServiceMonitor creation.
- **interval**: Frequency at which Prometheus scrapes metrics (e.g., "30s", "1m"). Defaulted to the interval configured in Prometheus global scope.
- **attachMetadata.node**: Attaches node metadata to discovered targets.
- **honorLabels**: Retain scraped metric labels over the target labels if conflicts arise.
- **honorTimestamps**: Retain timestamps from scraped metrics.
- **labels**: Custom labels added to the ServiceMonitor to facilitate Prometheus discovery.

## Authentication and TLS Options

The ServiceMonitor configuration supports various authentication and security methods for secure metrics collection:

```yaml
metricsExporter:
    prometheus:
        serviceMonitor:
            enable: true
            bearerTokenFile: "/var/run/secrets/kubernetes.io/serviceaccount/token" # Deprecated
            authorization:
                credentials:
                    name: metrics-token
                    key: token
            tlsConfig:
                insecureSkipVerify: false
                serverName: metrics-server.example.com
                ca:
                    configMap:
                        name: server-ca
                        key: ca.crt
                cert:
                    secret:
                        name: prometheus-client-cert
                        key: tls.crt
                keySecret:
                    name: prometheus-client-cert
                    key: tls.key
```

- **bearerTokenFile**: (Deprecated) Path to a file containing the bearer token for authentication. Retained for legacy use case. Use authorization block instead to pass tokens.
- **authorization**: Configures token-based authorization. Reference to the token stored in a Kubernetes Secret
- **tlsConfig**: Configures TLS for secure connections:
  - **insecureSkipVerify**: When true, skips certificate verification (not recommended for production)
  - **serverName**: Server name used for certificate validation
  - **ca**: ConfigMap containing the CA certificate for server verification
  - **cert**: Secret containing the client certificate for mTLS
  - **keySecret**: Secret containing the client key for mTLS
  - **caFile/certFile/keyFile**: File equivalents for certificates/keys mounted in Prometheus pod.

These options allow secure metrics collection from AMD Device Metrics Exporter endpoints that are protected by the kube-rbac-proxy sidecar for authentication/authorization.

## Accessing Metrics with Prometheus

Upon applying the NetworkConfig with the correct settings, the Network Operator automatically:

- Deploys the ServiceMonitor resource in the Network Operator namespace.
- Sets the required labels and namespace selectors in ServiceMonitor CR for Prometheus discovery.

After the **ServiceMonitor** is deployed, Prometheus automatically begins scraping metrics. Verify the integration by accessing the Prometheus UI and navigating to the "Targets" page. Your Device Metrics Exporter should appear as a healthy target. The ServiceMonitor object is deployed in the operator namespace and Prometheus must be configured to look for ServiceMonitor objects in this namespace. These two options in the Prometheus CR control ServiceMonitor discovery:

- **serviceAccountNamespaceSelector**: Allows selecting namespaces to search for ServiceMonitor objects. An empty value selects all namespaces.
- **serviceAccountSelector**: Specifies which ServiceAccounts to select in the selected namespace. The `labels` in NetworkConfig CR is added to the ServiceMonitor metadata and can be selected here.

These selectors help Prometheus identify the correct ServiceMonitor to use in the AMD Network Operator namespace and begin metrics scraping.

## Using with device-metrics-exporter Grafana Dashboards

The [ROCm/device-metrics-exporter](https://github.com/ROCm/device-metrics-exporter) repository includes Grafana dashboards designed to visualize the exported metrics, particularly focusing on job-level or pod-level Network usage. These dashboards rely on specific labels exported by the metrics exporter, such as:

- `pod`: The name of the workload Pod currently utilizing the AINIC.
- `job_id`: An identifier for the job associated with the workload Pod.

### The `pod` Label Conflict

When Prometheus scrapes targets defined by a `ServiceMonitor`, it automatically attaches labels to the metrics based on the target's metadata. One such label is `pod`, which identifies the Pod being scraped (in this case, the metrics exporter Pod itself).

This creates a conflict:

1. **Exporter Metric Label:** `pod="<workload-pod-name>"` (Indicates the actual AINIC user)
2. **Prometheus Target Label:** `pod="<metrics-exporter-pod-name>"` (Indicates the source of the metric)

### Solution 1: `honorLabels: true` (Default)

To ensure the Grafana dashboards function correctly by using the workload pod name, the `ServiceMonitor` created by the Network Operator needs to prioritize the labels coming directly from the metrics exporter over the labels added by Prometheus during the scrape.

This is achieved by setting `honorLabels: true` in the `ServiceMonitor` configuration within the `NetworkConfig`. **This is the default setting in the Network Operator.**

```yaml
# Example NetworkConfig snippet
spec:
  metricsExporter:
    prometheus:
      serviceMonitor:
        enable: true
        # honorLabels defaults to true, ensuring exporter's 'pod' label is kept
        # honorLabels: true 
        # ... other ServiceMonitor settings
```

**Important:** For this to work, the `device-metrics-exporter` must actually be exporting the `pod` label, which typically only happens when a workload is actively using the AINIC on that node. If no workload is present, the `pod` label might be missing from the metric, and the dashboards might not display data as expected for that specific AINIC/node.

### Solution 2: Relabeling

An alternative approach is to use Prometheus relabeling rules within the `ServiceMonitor` definition. This allows you to explicitly handle the conflicting `pod` label added by Prometheus.

You can rename the Prometheus-added `pod` label (identifying the exporter pod) to something else (e.g., `exporter_pod`) and then drop the original `pod` label added by Prometheus. This prevents the conflict and ensures the `pod` label from the exporter (identifying the workload) is the only one present on the final ingested metric.

Add the following `relabelings` to your `ServiceMonitor` configuration in the `NetworkConfig`:

```yaml
# Example NetworkConfig snippet
spec:
  metricsExporter:
    prometheus:
      serviceMonitor:
        enable: true
        honorLabels: false # Must be false if using relabeling to preserve exporter_pod
        relabelings:
          # Rename the Prometheus-added 'pod' label to 'exporter_pod'
          - sourceLabels: [pod]
            targetLabel: exporter_pod
            action: replace
            regex: (.*)
            replacement: $1
          # Drop the Prometheus-added 'pod' label to avoid conflict
          - action: labeldrop
            regex: pod
        # ... other ServiceMonitor settings
```

This method explicitly resolves the conflict by manipulating the labels before ingestion, ensuring the `pod` label always refers to the workload pod as intended by the `device-metrics-exporter`.

## Conclusion

The AMD Network Operator provides native support for Prometheus integration, simplifying Network monitoring and alerting within Kubernetes clusters. By configuring the NetworkConfig CR, you can manage Network metrics collection tailored to your requirements and preferences.
For more detailed configuration guidance, refer to the example scenarios [README](https://github.com/rocm/network-operator/blob/main/example/).