Partitioning

Partitioning#

Partitions (Slurm’s queues) group nodes so jobs can target a subset of the cluster with -p/--partition. A node joins a partition if it matches either the partition’s hostlist or its label selector. The two mechanisms are independent and combine as a union, so you can mix them.

Label-Based Membership (Recommended)#

A partition’s selector is a set of key = value label pairs. A node joins the partition when it carries all of those labels. Nodes declare their own labels when the agent starts:

spurd -D --controller http://10.44.0.1:6817 \
    --label pool=gpu --label rack=a

Repeat --label once per label to set several. The SPUR_NODE_LABELS environment variable sets a single label only (it is not comma-split), so setting multiple labels requires repeated --label flags.

Define the partition by selector:

[[partitions]]
name = "gpu"
selector = { pool = "gpu" }

Every node started with pool=gpu joins the gpu partition, regardless of its hostname.

Why this is the recommended approach: partition membership is decoupled from hostnames. Adding, replacing, renaming, or scaling nodes needs no change to the controller config, the node simply advertises its labels on registration. This suits dynamic or elastic clusters and heterogeneous hardware, where grouping by role (pool=gpu, tier=spot) is clearer and more durable than enumerating hostnames.

Hostlist-Based Membership#

A partition’s nodes field is a Slurm-style hostlist pattern, matched by exact hostname after expansion. It is not a regular expression.

[[partitions]]
name = "cpu"
nodes = "cpu-node-[1-64]"

Supported forms:

Bracket ranges and lists: node[001-003], node[1,3,5-7]
Comma-joined patterns: gpu[01-04],cpu[01-02]

The special value ALL (case-insensitive) is not expanded; it short-circuits to match every node.

Hostlists are convenient for small, static clusters with stable naming.

Combining Both#

Set both fields on one partition and membership is the union: a node joins if it matches the hostlist or the selector.

[[partitions]]
name = "mixed"
nodes = "gpu-node-4"
selector = { rack = "a" }

Here gpu-node-4 joins via the hostlist, and every node labeled rack=a joins via the selector.

The same dual-path matching applies to [[nodes]] config blocks (which set features and weight; see Manual Deployment (systemd)), keyed by names (hostlist) or selector (labels).

Applying Config Changes#

After editing spur.conf, apply the changes to a running controller without a restart:

scontrol reconfigure

reconfigure re-reads spur.conf and makes the file authoritative: runtime-only changes not reflected in the file are overwritten.

Applied live (no restart): [[partitions]] (created, updated, or deleted to match the file), [[nodes]] features and weight, licenses, burst_buffer, controller-side [hooks] (prolog_slurmctld, epilog_slurmctld), [notifications], [federation], [power] suspend/resume commands, [admission] mode, and the [scheduler] tunables complete_wait_secs and resv_overrun_minutes. Node-side hooks (the per-node prolog/epilog run by spurd), the device registry, and memlock are read by the node agent at its own startup; reconfigure does not reach compute nodes, so those need a spurd restart.

Restart-only: settings baked in when the daemon starts — listen addresses and ports ([controller], [metrics], [rest_api]), the accounting database ([accounting]), Raft identity and peers, first_job_id, auth.jwt_key (swapping the node-token signing key live would immediately invalidate every outstanding node token), and the scheduler loop cadence (interval_secs, max_jobs_per_cycle, topology). reconfigure reads these but does not apply them; a full controller restart is required. This mirrors Slurm, where a documented subset of parameters (ports, StateSaveLocation, AuthType, plugin set) also require a daemon restart.

Leader-only, in an HA cluster. reconfigure is handled by the Raft leader and swaps only the leader’s in-memory config; no Raft log entry carries the new config, so follower controllers keep the config they loaded at startup until they restart (in Kubernetes they re-read the same ConfigMap on restart). Partition changes still replicate through the partition write-ahead log, but followers recompute node membership from their own (pre-reconfigure) node config. Do not rely on reconfigured non-partition state surviving an immediate failover; roll the controllers to converge them.

Verifying Membership#

# Nodes grouped by partition
sinfo -o "%P %N"

# Partitions each node belongs to
sinfo -N -o "%N %P"