Building#
Prerequisites#
Rust — install via rustup; the repo’s
rust-toolchain.tomlpins the required version automaticallyprotobuf-compiler —
sudo apt install protobuf-compiler(Debian/Ubuntu) orsudo dnf install protobuf-compiler(Fedora/RHEL)Linux (tested on Ubuntu 22.04+, Fedora 38+)
Build#
cargo build # debug build
cargo build --release # optimized build
All crates build in one invocation — no separate steps needed.
MPI plugin (optional)#
Open MPI jobs use --mpi=pmix, which loads spur_mpi_pmix.so on compute
nodes at runtime (spurd itself does not link libpmix).
For production clusters, use a release or nightly tarball from GitHub
Releases (or your artifactory mirror). Both ship lib/spur/spur_mpi_pmix.so
when built with BUILD_MPI_PLUGIN=1 (see .github/workflows/release.yml
and nightly.yml). See Manual Deployment (systemd) for the full MPI deploy
and upgrade steps.
For local development, build the plugin when libpmix development packages are
installed (pkg-config pmix must succeed):
cargo build --release -p spur-mpi-pmix
sudo install -D target/release/spur_mpi_pmix.so /usr/lib/spur/spur_mpi_pmix.so
Install the resulting spur_mpi_pmix.so on every agent, using the same
glibc as the running spurd. If pkg-config pmix is missing on agents but
system PMIx headers exist, compile on the agent with gcc -fPIC -shared and
the system PMIx -I / libpmix.so path (see Manual Deployment (systemd)).
spurd launches multi-rank --mpi=pmix jobs via per-rank fork and
PMIx_server_setup_fork (see Manual Deployment (systemd)). Agents need
Open MPI libraries matching the application build; mpirun is not used for
bootstrap.
Without libpmix, the crate still builds a stub plugin that fails at load time with
an actionable error. Container/nightly images include a functional plugin via
BUILD_MPI_PLUGIN=1 (see Dockerfile and .github/workflows/nightly.yml).
Native-host MPI E2E tests (pytest -m mpi) additionally require mpicc on
the test nodes and SPUR_TEST_MPI_PLUGIN (or the default plugin path under
target/release/../lib/spur/).
Running Tests#
cargo test
All tests are self-contained. No external services needed (no database, no network, no GPU). The E2E suite below is the only one that requires actual hardware (nodes with SSH access, optionally GPUs), and it is ignored by default.
Important
The E2E suites do not support parallel test execution. Do not use pytest-xdist.
End-to-End Tests (Native-Host)#
The native-host E2E suite lives in tests/native_host/e2e/ and uses pytest. It SSHes into pre-provisioned nodes, deploys Spur, runs tests, and tears down the cluster after each test. Build the release binaries first (cargo build --release).
Prerequisites#
Python 3.11+ with
pip install -r tests/requirements.txtPre-provisioned nodes accessible via SSH (password, key, or ssh-agent)
Container tests require
squashfs-toolson the runner and all nodesGPU tests require GPU hardware on the nodes, plus a Python venv with PyTorch (auto-provisioned if
SPUR_TEST_GPU_VENVis unset)GPU test scripts (
gpu_test.hip,distributed_test.py,inference_test.py) live intests/native_host/e2e/fixtures/and are shipped to nodes by the harness
Environment Variables#
Variable |
Description |
Example |
|---|---|---|
|
Comma-separated list of node IPs/hostnames. First node becomes the controller. |
|
|
SSH username for all nodes. |
|
|
SSH password. If neither password nor key is set, ssh-agent is used. |
|
|
Path to SSH private key. If neither password nor key is set, ssh-agent is used. |
|
|
Path to release binaries on the test runner. Defaults to |
|
|
Fixed remote path for binaries on nodes. If set, not cleaned up (useful for CI + AppArmor). If unset, an ephemeral temp path is used and cleaned up after the session. |
|
|
Port for spurctld. Defaults to |
|
|
Port for spurd. Defaults to |
|
|
Path to a pre-existing Python venv (on nodes) with PyTorch. If unset, the GPU tests provision a fresh venv automatically. |
|
|
PyPI index URL for installing PyTorch (used when auto-provisioning the venv). Defaults to |
|
Node Setup#
AppArmor (Ubuntu 24.04+) — Container tests need unprivileged user namespaces, which AppArmor restricts by default. The recommended approach is to set SPUR_TEST_REMOTE_BIN_DIR to a fixed path and provision an AppArmor profile for spurd:
export SPUR_TEST_REMOTE_BIN_DIR=/tmp/spur-e2e-bin
# Provision AppArmor profile on each node:
IFS=',' read -ra NODES <<< "$SPUR_TEST_NODES"
SPURD_PATH="${SPUR_TEST_REMOTE_BIN_DIR}/spurd"
PROFILE="abi <abi/4.0>,
profile spur-e2e ${SPURD_PATH} flags=(unconfined) {
userns,
}"
for node in "${NODES[@]}"; do
ssh "${SPUR_TEST_SSH_USER:-vm}@${node}" \
"echo '${PROFILE}' | sudo apparmor_parser -r"
done
In CI, set SPUR_TEST_REMOTE_BIN_DIR to a fixed path and provision the profile once during node setup. The directory is not cleaned up when this variable is set, so binaries persist between runs.
If you do not set SPUR_TEST_REMOTE_BIN_DIR, binaries go into an ephemeral temp path (cleaned up after each pytest session). Since the path is unpredictable, you cannot provision an AppArmor profile for it. In that case, disable the restriction on test nodes instead:
# On each node (persists across reboots):
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
echo kernel.apparmor_restrict_unprivileged_userns=0 | sudo tee /etc/sysctl.d/99-spur-userns.conf
Running the Tests#
export SPUR_TEST_NODES=10.0.1.10,10.0.1.11,10.0.1.12
# Run the full native-host suite
pytest tests/native_host/e2e/ -v
# Run a specific test
pytest tests/native_host/e2e/test_single_node.py::TestJobLifecycle::test_job_cancel -v
Tests that require more nodes than provided, or missing GPU/container prerequisites, are automatically skipped.
End-to-End Tests (Kubernetes)#
The K8s E2E suite lives in tests/k8s/e2e/ and uses pytest with the Kubernetes Python client. It deploys Spur’s controller (StatefulSet), operator (Deployment), and SpurJob CRD into a Kubernetes cluster, then submits SpurJobs and verifies their lifecycle.
You need a running Kubernetes cluster with:
kubectl access — a valid
KUBECONFIGpointing to the clusterA Spur container image available to the cluster (either in a registry or pre-loaded via
ctr -n k8s.io images import)RBAC permissions to create namespaces, CRDs, StatefulSets, Deployments, and Pods
Prerequisites#
Python 3.11+ with
pip install -r tests/requirements.txtkubectlaccess to the clusterA Spur container image available to the cluster
Environment Variables#
Export these so the test process can read them:
Variable |
Description |
Example |
|---|---|---|
|
Path to kubeconfig file. Defaults to |
|
|
Container image for the controller and operator. Defaults to |
|
|
Kubernetes namespace for the test run. Defaults to |
|
Setup#
Build and load the image:
# Build the Spur container image
docker build --target runtime -t spur:ci .
# If running a local cluster (e.g. kind):
kind load docker-image spur:ci
# If running on native-host K8s nodes, load via containerd:
docker save spur:ci -o /tmp/spur-ci.tar
# On each node:
sudo ctr -n k8s.io images import /tmp/spur-ci.tar
The harness applies manifests from tests/k8s/e2e/manifests/ (CRD, RBAC, controller, operator). Production-oriented samples live under examples/k8s/. You do not need a separate kubectl apply before pytest unless you are debugging RBAC outside the suite.
Running the Tests#
export SPUR_CI_IMAGE=spur:ci
export SPUR_TEST_NS=spur-ci-local
# Run the full K8s suite
pytest tests/k8s/e2e/ -v
# Run SpurJob lifecycle tests only
pytest tests/k8s/e2e/test_spurjob.py -v
# Run Raft HA tests only
pytest tests/k8s/e2e/test_raft_ha.py -v
Cleanup#
The session k8s_suite fixture creates one namespace per pytest run. Class-scoped fixtures deploy the controller and operator once per test class (SpurJob lifecycle vs Raft HA); an autouse fixture removes SpurJobs between tests. Session teardown deletes the cross-namespace auxiliary namespace ({SPUR_TEST_NS}-user1) first, then the primary namespace and the SpurJob CRD.
If pytest exits abnormally (SIGKILL, node reboot, etc.), namespaces may be left behind. Clean up stale namespaces manually:
# List test namespaces (includes cross-ns suffix -user1)
kubectl get ns | grep spur-ci
# Delete auxiliary namespace first if both exist
kubectl delete ns "${SPUR_TEST_NS}-user1" --ignore-not-found
kubectl delete ns <namespace-name>