# Developer Guide

This guide provides information for developers who want to contribute to or modify the AMD GPU Operator.

```{warning}
This project is not ready yet to accept the external developers commits.
```

## Prerequisites

- Go v1.20 (due to [open issues](https://github.com/golang/go/issues/65637) with Go v1.21 or v1.22)
- Docker
- Kubernetes cluster (v1.29.0+) or OpenShift (4.16+)
- `kubectl` or `oc` CLI tool configured to access your cluster

## Development Environment Setup

- Install Helm:

```bash
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
```

For alternative installation methods, refer to the [Helm Official Website](https://helm.sh/docs/intro/install/).

- Install Helmify:
  - Download the released binary from the [Helmify GitHub release page](https://github.com/arttor/helmify/releases/tag/v0.4.13), unpack it, and move it to your `PATH`.

- Clone the repository:

```bash
git clone https://github.com/ROCm/gpu-operator.git
cd gpu-operator
```

- (Optional) Set up a local Docker registry. If you want to build and host container images locally, you can set up a local Docker registry:

```bash
docker run -d -p 5000:5000 --name registry registry:latest
```

- Modify the registry-related variables in the `Makefile`:
  - `DOCKER_REGISTRY`: Set to `localhost:5000` for local development, or your preferred registry
  - `IMAGE_NAME`: Set to `rocm/gpu-operator`
  - `IMAGE_TAG`: Set as needed (e.g., `v1.0.0` or `latest`)

- Compile the project:

 ```bash
 make
 ```

This will generate the basic YAML files for CRD, build controller images, build Helm charts and build OpenShift OLM bundle.

- (Optional) Run specific make target:
  - Run `make docker/shell` to build and attach to a container with build environment configured
  - Run `make <specific target>` within the container to execute specific make target.

- Build and push the AMD GPU Operator image:

```bash
make docker-build
make docker-push
```

> Note: If you're using a remote registry that requires authentication, ensure you've logged in using `docker login` before pushing.

### Building the controller manager image for a specific architecture

`make docker-build` runs `$(CONTAINER_ENGINE) buildx build` and is architecture-configurable through the `PLATFORM` variable, which defaults to `linux/amd64`. Set it to a single platform, or a comma-separated list for a multi-arch build. This works with `docker buildx` and with `podman` 4.0+ (which aliases `podman buildx build` to `podman build`).

```bash
# Default: builds a linux/amd64 image and loads it into the local image store
make docker-build

# Build a single arm64 image
make docker-build PLATFORM=linux/arm64
```

`buildx` populates the `TARGETOS`/`TARGETARCH` build arguments per platform, which drive both the Go cross-compilation (`GOOS`/`GOARCH`) and the architecture-specific `kubectl` download. With no override, `make docker-build` produces a `linux/amd64` image, unchanged from the previous default.

A single-platform build is loaded into the local image store automatically. A **multi-platform** `PLATFORM` (e.g. `linux/amd64,linux/arm64`) cannot be loaded locally — buildx must push the resulting manifest list straight to a registry, which the `docker-build` target does not do. For that case, run buildx directly with `--push`.

The utils container image build follows the same convention through the same `PLATFORM` variable:

```bash
make docker-build-utils                  # linux/amd64
make docker-build-utils PLATFORM=linux/arm64
```

> Note: When building for an architecture different from your host, register QEMU emulation once per host so the runtime stage can execute:
>
> ```bash
> docker run --privileged --rm tonistiigi/binfmt --install arm64,ppc64le
> ```

- Generate Helm charts:
  - For vanilla Kubernetes: `make helm`
  - For OpenShift: `OPENSHIFT=1 make helm`

- Check `Makefile` help message for more options:

```bash
make help
```

## Running Tests

Running e2e requires a Kubernetes cluster, please prepare your Kubernetes cluster ready for running the e2e tests, as well as configure the kubeconfig file at ```~/.kube/config``` for kubectl and helm toolkits to get access to your cluster. The e2e test cases will deploy the Operator to your cluster and run the test cases.

To run the e2e tests:

```bash
make e2e
```

To run e2e tests with a specific Helm chart:

```bash
make e2e GPU_OPERATOR_CHART="path to helm chart"
```

To run e2e test only:

```bash
make -C tests/e2e # run e2e tests only
```

## GPU Operator E2E Tests

The `tests/k8s-e2e/` directory contains an e2e test suite that installs the GPU Operator via Helm and verifies metrics and health. Tests run against a live Kubernetes cluster.

### Prerequisites

- A running Kubernetes cluster with at least one AMD GPU node
- `kubectl` configured (`~/.kube/config` or a custom kubeconfig)
- Docker (to build the test runner image)

### Test runner image

```bash
docker build -t gpu-op-k8s-e2e:latest -f tests/k8s-e2e/Dockerfile.e2e tests/k8s-e2e/
```

### Running tests

#### Full install + verify + teardown

Pass the helm chart as a local directory path (the `helm-charts-k8s/` directory in the repository root) or an OCI/repo reference if publishing to a registry:

```bash
docker run --rm \
  -v /path/to/kubeconfig:/kubeconfig:ro \
  -v /path/to/gpu-operator/helm-charts-k8s:/helm-charts:ro \
  gpu-op-k8s-e2e:latest \
  -kubeconfig /kubeconfig \
  -operatorchart /helm-charts \
  -operatortag v1.5.1 \
  -test.timeout 60m
```

#### Verify only (pre-deployed cluster)

```bash
docker run --rm -v /path/to/kubeconfig:/kubeconfig:ro \
  gpu-op-k8s-e2e:latest \
  -kubeconfig /kubeconfig -existing \
  -check.f 'TestOp010|TestOp020|TestOp030|TestOp040|TestOp050|TestOp060|TestOp065|TestOp070' \
  -test.timeout 30m
```

#### Using make

```bash
# Full install+verify+teardown
make -C tests/k8s-e2e all KUBECONFIG=/path/to/kubeconfig OPERATOR_TAG=v1.5.1

# Verify only (pre-deployed)
make -C tests/k8s-e2e verify KUBECONFIG=/path/to/kubeconfig
```

### Common flags

| Flag | Default | Description |
| --- | --- | --- |
| `-kubeconfig` | `~/.kube/config` | Path to kubeconfig |
| `-operatorchart` | OCI registry chart | GPU Operator helm chart (OCI ref or local path) |
| `-operatortag` | `v1.4.1` | GPU Operator chart version |
| `-namespace` | `kube-amd-gpu` | Kubernetes namespace |
| `-existing` | `false` | Skip install/teardown — verify only against pre-deployed cluster |
| `-noteardown` | `false` | Skip teardown after tests (leave operator installed) |
| `-helmset` | _(none)_ | Extra helm `--set` override (repeatable) |
| `-check.f` | _(all)_ | Regex filter for test names (gocheck syntax) |
| `-test.timeout` | `30m` | Overall test timeout |

## Creating a Pull Request

1. Fork the repository on GitHub.
2. Create a new branch for your changes.
3. Make your changes and commit them with clear, descriptive commit messages.
4. Push your changes to your fork.
5. Create a pull request against the main repository.

Please ensure your code follows our coding standards and includes appropriate tests.

## Build Documentation Website Locally

- Download mkdocs utilities

```bash
python3 -m pip install mkdocs
```

- Build the website

```bash
cd docs
python3 -m mkdocs build
```

- Deploy the website

```bash
python3 -m mkdocs serve --dev-addr localhost:2345
```

- The local docs website will dynmically update as changes are made to markdown docs.