Upgrading Spur#
This page covers upgrading Spur binaries, whether on a single host or across a whole
cluster. It describes the spur self-update self-updater, re-running install.sh,
and the Ansible playbooks that upgrade a running cluster with or without an outage.
Note
There are two upgrade scopes: upgrading the binaries on a single host and
upgrading a whole cluster. There is no in-process hot-swap that preserves
running jobs inside a single daemon process. spur self-update only swaps the
binaries on disk — it does not restart the daemons. The drain-aware, jobs-preserving
cluster path is the Ansible rolling_upgrade.yml playbook (see below).
Preemption accounting compatibility#
The preemption-outcome feature adds a preemption_usage column to accounting
job history and PostgreSQL checkpoint, aggregate, and cause tables.
The accounting service creates these automatically; its database role needs the
same schema-update permissions as for other migrations. Back up the database
and controller state before upgrading. Counting starts with the upgrade:
earlier preemptions are not counted.
Upgrade every controller before relying on preemption totals. Older controllers
can read the new log and snapshot entries but do not update the counts or hold
finished jobs for export. While an older controller leads, it can evict a
finished job whose latest counts were never exported; those counts are lost
from the sreport preemptions totals for good, and any of its observations
still open there are closed as ended_other / unknown.
Rollback. An older controller drops the preemption counts from its next snapshot: live totals, open observations, and per-job counts reset, and finished jobs held for export can be evicted before their counts reach PostgreSQL. The PostgreSQL tables and rollups already written stay in place and are ignored by the older controller; upgrading again resumes counting from zero in Raft while keeping the existing rollups. A job that lives across the rollback and the re-upgrade restarts its counts from zero, and its new counts reach the rollups only once they exceed what was already exported for it.
Rollups survive job-history cleanup and fairshare resets. Accounting outages
keep affected finished jobs in the controller, up to 24 hours past
terminal_job_retention_secs, which can increase controller memory and
snapshot size. See Preemption outcome export.
Account QOS allow-list compatibility#
Account QOS allow-lists (sacctmgr add/modify account ... qos=) add an
allowed_qos column to the accounts table, created automatically at
startup. Upgrade every controller before setting one. An older controller
ignores the field: sacctmgr reports the account as added or modified, but no
list is stored.
Rollback. An older controller leaves the column in place but does not read
it. Users who have no qos= of their own stop inheriting the account’s list
and may again use any QOS. Lists set directly on users are unaffected.
Single-Host: spur self-update#
spur self-update downloads the latest release and replaces the spur, spurctld,
spurd, spurstepd, and spurauthd binaries on the current host. It is a
single-host convenience: it does not restart daemons and gives no drain or quorum
protection, so it is not a substitute for the cluster playbooks below.
Check whether an update is available:
spur version --check
update available: 0.14.0 → v0.15.0
Run `spur self-update` to install.
Install the update:
spur self-update
spur update is an alias for spur self-update. Add --nightly to pull from the
nightly channel instead of the latest stable release:
spur self-update --nightly
Both aliases accept --help (or -h) without checking for updates or changing
files. Unknown options and positional arguments are rejected before an update
starts. spur update always means a binary upgrade; use spur control update
to change a job or node.
The updater downloads the release tarball, verifies it against its published SHA256
checksum, then replaces each binary atomically: the current binary is renamed to
<name>.spur-old as a backup, the new binary is copied into place, and the backups are
deleted on success. If any copy fails, the .spur-old backup is restored. The install
directory is auto-detected as wherever the running spur binary already lives.
After a successful update the CLI prints:
Updated spur to v0.15.0
Note: Restart running daemons (spurctld, spurd, spurauthd) to use the new version.
Warning
spur self-update never restarts daemons. Running spurctld and spurd
processes keep executing the old binary until you restart them yourself:
sudo systemctl restart spurctld spurd
If you run spurauthd, restart it too.
The [update] config block#
The optional [update] block in spur.conf controls the daemon startup update check.
Its fields are:
Field |
Default |
Effect |
|---|---|---|
|
|
Check the GitHub releases API when the daemon starts and log if an update exists. |
|
|
Download and install an available update automatically. Even when |
|
|
Release channel to check: |
|
|
Directory for the update-check cache (1-hour TTL). |
Note
Even with auto_update = true, a new binary on disk does not take effect until the
daemon is restarted. The config block applies to spurctld; spurd never
auto-installs updates.
See Configuration Reference (spur.conf) for the full [update] field reference.
Single-Host: Re-running install.sh#
Re-running the one-line installer upgrades an existing single-host install in place. It
downloads the requested release, verifies its checksum, and copies the binaries over the
install directory (default ~/.local/bin):
curl -fsSL https://raw.githubusercontent.com/ROCm/spur/main/install.sh | bash
Pass nightly or a pinned vX.Y.Z to select a specific release, or set
INSTALL_DIR to install elsewhere:
curl -fsSL https://raw.githubusercontent.com/ROCm/spur/main/install.sh | bash -s -- nightly
curl -fsSL https://raw.githubusercontent.com/ROCm/spur/main/install.sh | bash -s -- v0.15.0
Like spur self-update, install.sh does not manage systemd units or restart
daemons. Restart spurctld and spurd yourself after re-installing.
Cluster Upgrades with Ansible#
For a multi-node cluster, the Ansible toolkit is the recommended upgrade path. Two
playbooks are supported; both reuse the same install, config, and health-check roles as
deploy.yml, so their behavior stays consistent.
Rebuild all binaries from the same source tree together — they share a Raft write-ahead-log schema, and mixing binaries from different builds can leave a controller unable to parse a log written by a differently-versioned peer:
cargo build --release -p spur-cli -p spurctld -p spurd -p spur-stepd -p spurauthd
Binaries roll out by content, not version string: Ansible compares checksums, so an unchanged re-run is a near no-op.
Full convergence (deploy.yml)#
deploy.yml is the simplest upgrade: it re-installs binaries and restarts every daemon
on every host in one play. This causes a brief cluster-wide outage — in-flight jobs are
disrupted, and in an HA setup all controllers bounce together, briefly losing the Raft
leader. It is non-destructive by default (state is preserved). Use it for topology changes
or when a short blip is acceptable:
ansible-playbook playbooks/deploy.yml -i inventory/hosts.ini -e spur_binary_src=/path/to/target/release
Note
Spur has no online Raft membership change. Adding, removing, or reordering a
controller fails early unless you also pass -e spur_wipe_state=true (a Raft
reinit that wipes state). Compute agents are not Raft members, so they can be added or
removed freely without a wipe.
Rolling upgrade (rolling_upgrade.yml)#
rolling_upgrade.yml is the seamless, no-full-outage path: it upgrades one host at a
time so the cluster keeps scheduling and running jobs throughout.
ansible-playbook playbooks/rolling_upgrade.yml -i inventory/hosts.ini -e spur_binary_src=/path/to/target/release
The playbook proceeds in order:
Guard rails. Abort if
spur_wipe_state=true(never wipe Raft mid-upgrade), ifspur_transport=wireguard(not yet supported by this playbook), or if the cluster is not already healthy (spur nodesmust return success).Upgrade controllers one at a time (
serial: 1, no failures tolerated), preserving Raft quorum. On a multi-controller cluster the playbook probes each controller first and upgrades the Raft leader last (see Controller order: the leader goes last). Each controller’s binaries are force-reinstalled and the daemon restarted; the “wait for Raft leader” step is the health gate before moving to the next controller. The existingspur.confis preserved unless you pass-e spur_overwrite_conf=true.Drain and upgrade agents in batches. For each agent: drain it from the controller (
spur node drain <node> --reason "ansible rolling upgrade"), wait until its state isDRAINEDorDOWN(running jobs finish first — drain never force-kills), swap the binary, restartspurd, wait for the node to re-register, then resume it (scontrol update NodeName=<node> State=RESUME).spur_in_place_upgrade=truechanges this step: the node is never drained and never resumed, so it stays schedulable for the whole swap. See Non-disruptive in-place upgrade.Verify. Submit a real test job to confirm the upgraded cluster schedules work.
Until every controller is upgraded, a node removed with spur node remove or
scontrol delete NodeName= may register again; see keeping a removed node out.
The rolling upgrade is controlled with these -e flags:
Flag |
Default |
Effect |
|---|---|---|
|
(unset) |
Directory of pre-built |
|
|
Agents upgraded per batch. Controllers are always upgraded one at a time regardless. |
|
|
Skip agents unreachable over SSH instead of aborting. |
|
|
Leave a still-busy node on its current binary and continue, rather than aborting. |
|
|
Kill running jobs and containers on a busy node and upgrade it anyway. Affected jobs
are marked |
|
|
Non-disruptive mode: never drain any agent, and upgrade a busy node’s |
|
|
Re-render |
Note
A larger spur_rolling_batch_size upgrades faster but drains more capacity at once. A
single-controller cluster still has a short outage while its own controller restarts —
true zero-downtime requires an HA quorum of 3 or more controllers.
Non-disruptive in-place upgrade#
ansible-playbook playbooks/rolling_upgrade.yml -i inventory/hosts.ini \
-e spur_binary_src=/path/to/target/release -e spur_in_place_upgrade=true
Every other busy-node strategy drains the node first and resumes it afterwards, which makes
it unschedulable for the length of the swap. spur_in_place_upgrade=true issues neither,
so the controller’s view of the node never changes and an operator’s own pre-existing drain
is never cleared.
A running job is carried across by spurstepd, the per-job supervisor that lives outside
spurd’s process lifetime: spurd restarts, the new binary rediscovers the live
session and re-adopts it, and the payload is never signalled. The mode still refuses a busy
node that has no spurstepd installed yet — trusting a supervisor that is not there would
be unsafe — and is mutually exclusive with spur_skip_busy_agents and
spur_force_upgrade_busy_agents.
The trade-off runs the opposite way from the drained modes. Because nothing stops the
scheduler, a new job can be dispatched onto the node during the few seconds spurd is
down and may fail to launch. Already-running supervised work is unaffected.
Two further consequences of not draining:
spur_rolling_batch_sizebehaves differently. Elsewhere it trades speed for drained capacity; here nothing drains, so a batch of N instead puts N agents’spurddown simultaneously while all N stay schedulable, widening the dispatch window above N-fold rather than serialising it. Keep the default of1.An agent unreachable over SSH is left schedulable and not upgraded. Every other mode still drains it, because the drain is a controller-side call that works on a host Ansible cannot reach. That is usually desirable — a node whose
spurdis healthy but whose SSH is down should keep taking work — but a genuinely dead node is flagged not-responding (taking no new work) once its heartbeats go stale, then markedDOWNonce the leader confirms it is unreachable (see Node Liveness).
Note
sbatch work — plain, containerised and multi-node — rides the swap
untouched. A buffered srun client reconnects after a controller restart
while its allocation remains active, making up to 30 attempts with a two-second
backoff (roughly a minute, plus connection and RPC time). If the controller
returns within that retry window, the client resumes waiting for the step
or retrieves its recorded result. A longer outage can exhaust the retries
and leave a live payload without its client; scancel reaps it cleanly.
This is more likely on a single-controller cluster, where there is no peer
to fail over to. See Surviving a Controller Restart for output retention and
replay limits.
Controller order: the leader goes last#
On a multi-controller cluster rolling_upgrade.yml probes each controller before touching
any of them and upgrades the Raft leader last. This applies to every busy-node mode, not
only spur_in_place_upgrade: it is gated solely on there being more than one controller
and no explicit spur_controller_upgrade_order.
This matters because of how a mixed-version cluster replicates. WalOperation is an
externally tagged enum with no tolerant unknown-variant path, and the Raft transport
deserializes an entire AppendEntries payload in one call. If a release adds a WAL
variant and the new build reaches the leader first, every still-old follower rejects the
appends carrying that variant and stops replicating until it is itself upgraded. On a
three-controller cluster that costs quorum. While the leader is still on the old build it
cannot emit a new variant at all, which is what closes the window.
Ping’s advertised capabilities are the leader’s: a follower queries the leader
before answering, narrowing its own list to whatever the leader also supports, so a
typed step’s capability check reflects whichever controller is actually in charge, not
whichever one srun happens to reach. A follower that cannot reach a leader
advertises none, rather than guessing. Until every controller in the cluster is
upgraded, avoid the per-GPU options and typed steps these capabilities gate.
Detection is a read-only GET to each controller’s own loopback metrics listener
(http://127.0.0.1:6822/metrics/nodes), which answers 200 on the leader and 503
on a follower. It never blocks the upgrade: if no controller answers as leader, or more than
one does, the playbook warns and falls back to inventory order. Pass
-e spur_controller_upgrade_order=ctl-2,ctl-0,ctl-1 to override it entirely.
Warning
Leader-last shrinks the mixed-version window but cannot remove it. If leadership moves
onto an already-upgraded controller mid-run and something then writes a new WAL variant —
scancel and preemption are the usual triggers — a not-yet-upgraded follower stalls.
Quorum survives, and the stalled follower catches up as soon as it is upgraded, so this
is degraded-but-self-healing rather than an outage. The stalled follower logs nothing
itself; the rejection is only visible on the leader.
Safe Upgrade Order#
Follow this order for any cluster upgrade:
Rebuild all binaries together from the same source tree — they share a Raft WAL schema and must stay version-matched.
Upgrade controllers before agents. Both playbooks do this automatically, one controller at a time to preserve quorum.
Drain agents before swapping binaries. The rolling playbook drains automatically; a running job blocks the swap unless you force it. The exception is
spur_in_place_upgrade=true, which deliberately never drains and relies onspurstepdto carry running jobs across instead (see Non-disruptive in-place upgrade).Never wipe state during an upgrade. Keep the default
spur_wipe_state=false. Wiping resets the Raft job-id counter and destroys accounting history.HA membership is fixed at init. You can roll new binaries onto the existing controller set freely, but changing which hosts are controllers requires
deploy.yml -e spur_wipe_state=true.Only roll forward across an accounting schema migration. The first upgraded controller applies pending PostgreSQL migrations on startup, so upgrade controllers before agents and let each migration finish before the next controller starts. One such migration widens the
jobsjob-id columns to 64-bit: it rewrites the table under anACCESS EXCLUSIVElock, so its duration scales with the row count and accounting writes block until it completes. Rolling back to a pre-migration controller is not supported — an older controller reads the widened columns as 32-bit and its accounting queries fail against a migrated database. Take a database backup before upgrading if you need a recovery path.Roll forward, not back. The Raft log gains entries — new job states, pending reasons, operations — as Spur evolves, and a controller replaying a log written by a newer build cannot parse them. A newer controller reads older logs fine, so the supported recovery from a bad upgrade is to roll forward, not to reinstall the previous version over a log the new one has already written.
Warning
Upgrading to the release that introduces spurstepd requires an empty
cluster. Drain every node and let all running jobs finish, or cancel them,
before swapping binaries. Sessions written by the previous build are not
adopted, the two builds disagree on where a step’s processes live, and a
mixed-version cluster is not supported across this upgrade — a new controller
dispatching to a not-yet-upgraded agent tears the job down. Upgrade every
controller and agent in the same maintenance window.
spurstepd is a new binary and must be installed next to spurd on every
compute node. The agent resolves it beside its own executable and does not
search $PATH, so if it is missing every job launch on that node fails.
spur_mpi_pmix.so is versioned against the binaries that load it, and this
release bumps that version. Replace the plugin in [mpi].plugin_dir in the
same window, or --mpi=pmix jobs fail with unsupported MPI plugin API
version. The plugin is now loaded by spurstepd rather than spurd, for
steps as well as jobs.
Note
Once this release is in place, restarting spurd no longer kills the work
on that node: jobs, srun steps and held allocations run under supervisors
that outlive the agent, and the restarted agent re-adopts them. This requires
KillMode=process in the spurd unit — the systemd default,
control-group, kills the whole cgroup on stop and takes the supervisors
with it. See Manual Deployment (systemd) for the unit file and for the two launches
that remain unsupervised. Draining first is still the recommended order
because it keeps new work off a node mid-swap.
Behavior Changes Between Releases#
Some releases change how an existing spur.conf behaves without that file being
edited. Review these before rolling binaries out.
Job resource enforcement ([cgroup])#
The release introducing the [cgroup] section changed what spurd writes for
a config that has no [cgroup] section. The first two can affect jobs that
ran fine before the upgrade; the rest relax an existing bound.
Control file |
Before |
After |
Effect |
|---|---|---|---|
|
unset for |
the memory the scheduler allocated |
A ``–mem-per-cpu`` job that ran unbounded is now capped, and is OOM-killed if it overruns. Only the per-node request was read before, which those jobs do not set. |
|
unset |
equal to |
Memory-heavy jobs stall in reclaim before the OOM kill, costing throughput. |
|
CFS quota sized from |
unset |
Relaxed: the cpuset is the CPU bound, as in Slurm. |
|
sized from |
sized from the node allocation |
Relaxed: a multi-task job gets all the cores it was granted, not one task’s worth. |
To keep the previous behavior, put this in spur.conf on every compute node
before restarting spurd:
[cgroup]
cpu_quota = true # restore the CFS quota
spurd reads [cgroup] only at startup, so this has to be in place before the
restart — scontrol reconfigure will not apply it afterwards.
Note
Neither memory change has a knob of its own: memory.max and memory.high
are both written whenever constrain_ram_space is on, matching Slurm’s
cgroup/v2 plugin. Setting constrain_ram_space = false restores the old
behaviour but drops the memory ceiling for every job, not just the
--mem-per-cpu ones. Prefer raising allowed_ram_percent (for example
125), which gives jobs headroom above their allocation and starts reclaim
there, over turning memory constraints off.
Rolling back is safe with the section left in place: older binaries do not know
[cgroup] and ignore it.
Job device access (constrain_devices)#
The release introducing constrain_devices attaches a default-deny BPF device
filter to every job cgroup, and it defaults to on — including for a
spur.conf with no [cgroup] section, or one written before the field
existed.
Effect: a batch payload can open only the device nodes its allocation granted,
plus the base pseudo-devices and a built-in host-infrastructure set. A job that had
been reaching a GPU it was not allocated — by re-exporting ROCR_VISIBLE_DEVICES,
or because it never read the variable — now fails that open with EPERM.
A job can also be denied a host device node its allocation never covered, and
that is the risk to plan the rollout around. Some device nodes belong to the node
rather than to any job, so no allocation hands them out and nothing puts them in the
allow-list. The main case is RDMA/InfiniBand verbs under /dev/infiniband/ —
needed by MPI, and by NCCL or RCCL over InfiniBand, which makes this reach
non-GPU jobs. These are allowed by default, so the common cases keep working with
no configuration; see What the device filter allows without an allocation for the full implicit set.
The vendor control nodes a GPU runtime initializes through, such as /dev/nvidiactl
and /dev/nvidia-uvm, reach a job through its allocation’s CDI device edits. A node
configured through GRES rather than a CDI spec does not list them as devices, so a GPU
job there fails with EPERM on those nodes until they are named in
extra_device_paths.
A site whose jobs open some other host device node will still see EPERM. Name
those paths rather than turning the filter off:
[cgroup]
extra_device_paths = ["/dev/some-site-device"]
Roll the agent out node by node and watch job output for EPERM on device paths.
To defer the change entirely:
[cgroup]
constrain_devices = false
Installing the filter needs CAP_BPF and CAP_NET_ADMIN (or CAP_SYS_ADMIN);
a cgroup-device program is a net-admin program type, so CAP_BPF alone is not enough.
An agent without them logs device filter not installed and runs the job with no
device isolation, so an unprivileged spurd behaves as before — unless required = true, which
turns that degradation into a refused launch.
Job spool folders (/var/spool/spur/job<id>)#
spurd keeps each job’s generated scripts and step output in a per-job spool
folder. When spurd runs as root, that folder is now owned by root with the job’s
group and mode 0750, as Slurm’s batch directory is, instead of belonging to the
job’s user. spurd creates every file in it through an open directory handle and
replaces an existing file rather than rewriting it in place.
Effect on jobs:
A job can still read its own scripts in the folder, but can no longer create, rename or remove entries there. A script that wrote files next to itself (
$(dirname "$0")) must write to its work dir or$TMPDIRinstead.srunsteps thatspurdwraps in generated scripts (several tasks per node,--label,--mpi=pmix) keep those scripts in the spool folder, not in a.spur_step_<id>folder in the work dir. Such steps now also run when root cannot write the work dir, for example on an NFS export withroot_squash. A job’s containers, its batch job’s and any a step builds, see the spool folder mounted read-only at the same path, which is how a step inside one reaches its scripts.A root
spurdno longer falls back to$TMPDIR/spurwhen it cannot create the spool folder. The failure drains the node, like any other fault in the spool tree.
A folder created by an older spurd for a job still running across the upgrade is
taken over by root the next time the new agent writes to it, so no drain is needed.
Spool root ownership. With a root spurd, /var/spool/spur should be owned by
root and not writable by group or others; spurd logs a warning at startup when it is
not. If spurctld runs as an unprivileged user on the same node and shares that
directory, move its state out first; see spool root ownership.
Reserved step IDs aligned to Slurm#
This release changes the integer IDs of the reserved batch, extern, and interactive
steps to Slurm’s exact slurm.h sentinels (SLURM_BATCH_SCRIPT,
SLURM_EXTERN_CONT, SLURM_INTERACTIVE_STEP). These IDs are persisted in the Raft
snapshot, embedded in each node’s stepd runtime state (session directory names, launch
descriptors, PMIx namespaces), and carried on the gRPC wire, so old and new builds
disagree on which integer means which step.
Warning
This upgrade requires an empty cluster. Drain every node and let all running jobs finish, or cancel them, before swapping binaries, and upgrade every controller and agent in the same window. A job in flight across the upgrade has its reserved step recorded under the old ID: the new build no longer recognizes it as batch/extern/interactive, so it reports the raw integer and its supervisor and cgroup are not reaped correctly. Rolling back is not supported once a new controller has written a snapshot carrying the new IDs.
On a drained cluster the change is invisible: user-facing output already renders these
steps as batch/extern/interactive rather than the integer, so no scripts or
CLI output change.
Job and usage visibility (private_data)#
Earlier releases pinned identified callers with no way to turn it off:
v0.9 to v0.14 pinned the gRPC job list (
squeue,sprio) to the caller’s own jobs, Operators included through v0.12.v0.11 to v0.14 pinned
scontrol show assoc_mgrto the caller’s own scopes.v0.13 and v0.14 also pinned REST
/jobsand returned not found for another user’s job fromscontrol show job,scontrol show step,sstat, and REST/job/{id}.
The release introducing [auth] private_data makes this opt-in and defaults to Slurm’s behavior: every user sees every job and every assoc_mgr scope.
Effect: with an unchanged spur.conf, a User now sees other tenants’ jobs
and steps, including their command (the first non-comment line of the batch
script), working directory, and output paths. A step command such as
srun --jobid against another user’s job now fails with a permission error
instead of “job not found”. To keep the previous visibility, set this on every
controller before upgrading:
[auth]
private_data = ["jobs", "usage"]
Warning
rolling_upgrade.yml and deploy.yml regenerate spur.conf from the
toolkit template, which has no [auth] section, and restart the controllers.
On a toolkit-managed cluster, every job is visible from that restart until you
re-add private_data and restart every controller again. Every later run of
either playbook drops it again.
A request from a User that names another user no longer falls back to the caller’s
own jobs. In v0.14, squeue -u bob, sprio -u bob, and REST
/jobs?user=bob returned the caller’s own jobs, and scancel -u bob
cancelled them. Under "jobs" these now return nothing and cancel nothing, as
in Slurm. Without private_data they list bob’s jobs.
Without private_data, scancel -p, -A, or -n also selects other
users’ jobs. Each one the caller does not own is refused and reported on
stderr.
A cluster that already sets "jobs" now also hides other users’ sacct
history and sreport job sizes. A cluster that already sets "usage" now
also hides other users’ sshare and sreport utilization figures. sprio
stays with "jobs". Set the categories only where that narrower view is what
you want, and upgrade spur-cli together with spurctld.
scontrol reconfigure applies private_data on the leader only; restart every
controller after changing it. During a rolling upgrade, visibility depends on which
controller serves the request, but it is never wider than the new configuration
allows.
spurctld refuses to start on a category it does not support. Agents and the CLI
ignore the field, so a spur.conf naming a category from a newer release still
loads on them.
Rolling back to v0.13 or v0.14 is safe with the field left in place: those releases
ignore it and pin Users to their own jobs and assoc_mgr scopes regardless. v0.12 and
earlier also pin Operators to their own job list, but leave REST /jobs and
scontrol show job unpinned.
scancel exit status on a refused cancel#
This release makes scancel exit 1 when the controller refuses a cancel or signal
(for example, cancelling a job you do not own). It previously always exited 0 and only
printed to stderr. Scripts that call scancel and branch on its exit code now see the
failure. A job that is already finished or unknown still does not fail a plain cancel (or
-s KILL), and --quiet/-Q hides the report but not the exit status.
Job preemption policy (preempt_type)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The release introducing scheduler.preempt_type makes preemption opt-in
and narrows what makes a running job eligible. Both changes apply to a
spur.conf that is not edited.
Behavior |
Before |
After |
Effect |
|---|---|---|---|
Whether preemption runs |
On whenever the pending job’s partition set a non- |
Off unless |
A cluster that preempted before stops preempting entirely until the new field is set. |
What makes a job eligible |
An effective-priority gap of more than 2× |
The pending job’s QOS lists the victim’s QOS in |
Preemption no longer fires without a QOS allow-list. Fair-share, job
age, and |
To restore preemption, enable the engine in spur.conf on the controller:
[scheduler]
preempt_type = "qos_priority"
then give each preempting QOS an allow-list and a higher priority than the QOS it should displace:
sacctmgr modify qos name=burst set priority=100
sacctmgr modify qos name=normal set priority=10000 preempt=burst
Both are live: preempt_type applies on scontrol reconfigure, and QOS
changes take effect without a restart. A victim still needs a resolved action
that is not off: its QOS preemptmode if set, else the most aggressive
of its partitions’ preempt_mode. See Accounting, Accounts, Users, and QOS for
the full decision table.
A pending job’s own partition preempt_mode no longer gates whether it may
preempt. Previously a job sitting in an off partition could never trigger
preemption; now only the victim’s scope decides. Partitions used to hold
pending work away from preemption need their victims protected instead.
QOS ``preemptmode=off`` is now a real protection, not a no-op. It used to
mean “no QOS override, defer to the partition”; it now always wins over the
partition, including to protect a job the partition would otherwise let
through. Every pre-upgrade row holding the literal off behaved as unset
under the old resolution rule, so a one-time migration resets those rows back
to unset — they keep deferring to their partition exactly as before. Only a
QOS an admin sets to off after upgrading gains the new protection.
Leaving preempt_type at its "none" default is safe but silent: nothing
logs that preemption was skipped, so verify with a test job rather than
assuming the old behavior carried over.
Reaping abandoned interactive allocations (inactive_limit_secs)#
This release turns scheduler.inactive_limit_secs on by default (300),
including for a spur.conf with no such line. It defaulted to 0 (off).
Behavior |
Before |
After |
Effect |
|---|---|---|---|
An |
Held its nodes until the time limit |
Reaped once its client has been silent for 300 seconds; the job ends
|
A non-PTY standalone ``srun`` whose client is gone no longer runs to
completion, for example |
Which allocations a set limit reaps |
Every interactive allocation |
Only allocations whose client sends keepalives: |
The K8s operator, third-party API clients that do not send keepalives, and CLIs older than v0.9 stay exempt. |
A session that outlives its user’s token under |
Ran to its time limit |
Reaped about 300 seconds after the token expires, even with its client still running |
Issue user tokens that outlive the longest interactive time limit. A token’s lifetime counts from when it was minted. |
To keep the previous behavior, set the limit to 0 in spur.conf on the
controller before upgrading:
[scheduler]
inactive_limit_secs = 0
The field is live and applies on scontrol reconfigure. Jobs submitted while
an older controller ran carry no keepalive mark, so they are reaped only if
their client pings the upgraded controller. scontrol show config prints the
value in effect as InactiveLimit. See Reaping abandoned interactive allocations (inactive_limit_secs).
GPU partition identities (CPX, QPX, DPX)#
This release changes the allocation id (stable_id) of partitions 1 and up of a
GPU that runs in a partitioned compute mode. Partition 0 and GPUs in SPX mode keep
their ids. The id is the GPU allocation key in the Raft log and in the job state of
spurd. Old log entries still deserialize.
The ids change when spurd regenerates the CDI spec (spur k8s up), or at a
restart on a node that has no /etc/cdi/amd.json. A binary swap alone does not
change them while that file exists.
Note
spurd carries a running job’s GPU ids across this change, both at a
restart and when it rewrites the CDI spec: it saves the last
controller-acknowledged inventory to disk and recognizes the old encoding
against it, or, on the first restart after upgrade with no saved inventory
yet, recognizes the old encoding directly. When the node re-registers, the
controller remaps the job’s allocation to the new ids. Upgrade the
controllers before the agents: an older controller does not remap
allocations when a node re-registers, and an upgraded agent then rejects
dispatches with
controller-allocated GPUs are not in this node's gpu inventory.
With the controllers upgraded first, jobs keep running, no
dispatch rejection or JobHoldMaxRequeue hold results, and draining
nodes is no longer required for the GPU id change. See
Changing a GPU’s partition mode for the separate, still-destructive case
of actually switching a GPU’s partition mode.
Bounded controller shutdown (shutdown_grace_secs)#
After SIGTERM, spurctld now waits at most [controller] shutdown_grace_secs
(default 10s) for in-flight RPCs to drain, then flushes accounting and exits. Previously it
waited on every in-flight RPC with no bound, and an srun step holds its RPC open for the
step’s whole runtime, so restarting a controller while a step ran (for example during
rolling_upgrade.yml) hung until systemd’s TimeoutStopSec killed it, skipping the
accounting flush.
A step still running when the grace period ends keeps running on its node. A buffered
srun re-attaches to it if it reconnects to the restarted controller within its retry
window, and otherwise exits with an error while the step runs on; see
Surviving a Controller Restart. srun --pty follows a different path and is not covered.
Accounting for requests still in flight at that point is best-effort: anything they write
after the flush is lost.
The default suits most sites. Keep it well below heartbeat_timeout_secs: the controller
answers no heartbeats while it drains, so a grace within about 35 seconds of that timeout
lets a full drain flag nodes not-responding and mark operator-managed nodes, which it cannot
probe, DOWN. 0 restores the unbounded wait,
which can run past that timeout too, and which a repeated SIGTERM or SIGINT does not
end; only SIGKILL does. spurctld warns at startup in either case. Keep the grace,
plus up to 5 seconds for the accounting flush, below TimeoutStopSec. A step that
finishes during the drain does not end it, so raising the value mostly lengthens the
restart. A new value takes effect at the next start.
Confirmed node DOWN and the not-responding flag#
The leader no longer marks a node DOWN as soon as its heartbeats pass
controller.heartbeat_timeout_secs. It flags the node not-responding (* in sinfo),
which keeps new work off it, and asks the node’s agent directly. See Node Liveness.
A dead node goes
DOWNone pass later. Its agent must fail to answer on two health passes at leastcontroller.health_tick_secsapart, soNODE_FAILand the requeue of a--requeuejob come about one pass later than before. Nodes managed by the Kubernetes operator are not probed and keep the timeout rule.A node whose heartbeats cannot reach a controller is no longer marked
DOWN. If its agent answers, it stays up and flagged, keeping its running jobs, until its heartbeats resume. Such an agent often cannot report job completions either, so its finished jobs can stay running. The leader warns about the node every 10 minutes and the newspur_nodes_silent_answeringgauge counts it.spur_nodes_not_respondingcounts every node the leader is not hearing,DOWNones included, so alert on the new gauge.A mass silence marks no node
DOWN. The suspects in a pass are the nodes that fail their probe, operator-managed nodes past the heartbeat timeout, and, in a pass with either of those, nodes that have missed a heartbeat. When they exceedcontroller.max_stale_node_fraction(default0.5) of the watched nodes, rounded down but never below one node (so a single silent node never trips it), the leader assumes the fault is its own and marks none of themDOWN. A real mass failure then needsscontrol update nodename=<nodes> state=down. Alert onspur_nodes_liveness_breaker_tripped, or set the fraction to1to turn the check off.Node state strings can carry the flag.
sinfo%tand%Tadd a trailing*(idle*,down*),scontrol show nodeadds+NOT_RESPONDINGtoState=, andspur healthstars the node. ADOWNnode the leader has not heard from since shows asdown*, which covers every node markedDOWNfor not responding. Scripts that comparesinfo -h -o %ttodownoridlemust strip a trailing*.Mixed versions do not show the flag. It is a new field in the node reply, so an
sinfoorscontrolfrom an earlier release shows no*or+NOT_RESPONDINGand lacks-d,-r, and-t no_respond. A new client against an older controller shows no*either, andsinfo -dand-t no_respondmatch no node, because an older controller never sets the flag.
No configuration change is needed. The flag lives only in the leader’s memory, so no Raft
entry is added and rolling back is safe; an older controller ignores
max_stale_node_fraction.
Stopping a requeued job’s run on a recovered node#
When a node is marked Down after missing heartbeats, its jobs end NODE_FAIL
and a job submitted with --requeue goes back to the queue, but the job’s
nodes are not told to stop when they go Down. From this release, the controller stops that run when a
node heartbeats again, even while the job is still waiting in the queue. Before,
a node kept the run and its GPUs until the job ended or ran again, and refused
every job placed on those GPUs in the meantime.
Stopping the run on heartbeat needs both an upgraded controller and an upgraded
agent on that node, which reports the run it holds; with either side from an
earlier release, the old behavior stays. The recovery hold that keeps a returning
node DOWN until it reports a clean job list applies to earlier agents too: one
that still reports a job the controller ended is held, and drained at the cap if
that job does not stop. A node drained at the cap is admin-locked, with reason
Kill task failed (JobId=...) or Agent job list unavailable, and takes no work
until an operator runs scontrol update nodename=<node> state=resume, so scripts that
watch sinfo -R or resume nodes automatically should expect these reasons. Only
upgraded agents report running_jobs_incomplete when they cannot list their jobs
in full, though. An earlier agent sends an empty list when its job map is busy,
which reads as clean, so its hold can end while an ended run is still going. A job
entry with run attempt 0 (an agent too old to report the run sends 0 for every
job) is judged like any other: it holds the node if the controller ended the job or
placed it on other nodes. The exception is a job back in the queue: its entry is
neither stopped (a cancel without an attempt could hit the job’s next launch) nor held
for, so it reads as clean. An upgraded agent also stops the old run’s
processes, including its srun steps, when the job is placed back on the same node,
which an earlier agent could miss for a run without a cgroup
([cgroup] enabled = false, or spurd cannot create cgroups, for example when not
running as root).
Before accepting a replacement attempt, the agent fences every older attempt
it can discover in its tracked sessions, persisted session descriptors, and
/sys/fs/cgroup/spur/job_<id>_<attempt> directories. This includes attempts
with reused step IDs after a restart and orphan cgroups without descriptors.
It kills old workloads and removes their complete cgroup trees, including the
parent job cgroup. Same or newer attempts and other jobs are left alone.
Batch launches and allocation registration are refused if an old cgroup cannot
be removed or an old session descriptor cannot be read; the old reservation is
not replaced. Step startup is serialized with this cleanup so an old step
cannot recreate a cgroup during replacement.
Completed steps keep their recorded exit status when a replacement reclaims
their retained sessions. An unreadable exit record also refuses replacement.
Kill waits and the time-limit completing state (kill_wait_secs, timeout_kill_wait_secs)#
This release makes the wait between SIGTERM and SIGKILL configurable with two
settings. scheduler.kill_wait_secs (default 5) covers scancel,
scontrol requeue, the final kill of a preemption, idle-fill reclaim,
reservation end, the stop of a finished batch script’s other nodes and the other
graceful stops. scheduler.timeout_kill_wait_secs
(default 30) covers the stops that end TIMEOUT: the time limit and the
inactivity limit. Both changes apply to a spur.conf that is not edited, and
with the defaults most waits are the ones a job had before.
scontrol show config prints both values in effect as KillWait and
TimeoutKillWait (in seconds, floor applied); against a controller older than
this release it omits them rather than showing a misleading 0.
Behavior |
Before |
After |
Effect |
|---|---|---|---|
|
SIGKILL 5 seconds after SIGTERM |
SIGKILL up to |
Unchanged by default. Raise |
A batch job at its time limit |
Stayed |
Moves to |
|
An idle interactive allocation reaped by |
Stayed |
Normally ends |
The allocation is freed sooner than before. |
A requeued job (requeue-mode preemption, |
Held out of the queue for |
Held for the larger of that and |
Unchanged by default. Raising |
An unsupervised |
SIGKILL after 2 seconds |
SIGKILL after the full kill wait that applies: 5 seconds on cancel, 30 at a time limit |
A step that handles SIGTERM now has time to shut down cleanly, where a stop reaches it (kill wait note). |
Keep complete_wait_secs above the larger of the two waits plus
[cgroup] cleanup_timeout_secs (60 seconds by default). A node reports only
after the job is killed and its cgroup cleanup finishes, so a shorter
complete_wait_secs force-finishes the job while healthy nodes are still
working, and every node that has not reported is drained. The controller logs a
warning on startup and on reconfigure. The warning uses the controller’s own
[cgroup] cleanup_timeout_secs, so set it there to the largest value any
node uses. kill_wait_secs and timeout_kill_wait_secs both accept
0-65533, and a value below 5 acts as 5. See
Configuration Reference (spur.conf) for the fields, including a
known issue with requeue: a requeued job frees its nodes while the old run is
still in its kill wait, so keep kill_wait_secs low if that matters with
requeue-mode preemption.
Upgrade the leader last. Every time limit and every inactivity-limit
expiry now writes a new Raft log entry, JobTimeLimitExpired. A controller
that predates it cannot parse the entry and stops replicating until it is
upgraded, so the order in Controller order: the leader goes last applies, and a rollback
after the entry has been written is not supported: roll forward.
Time-limited jobs during the rolling upgrade. The controller sends the kill
wait with each cancel, and an agent that predates it ignores the value and
waits a fixed 5 seconds. Controllers are upgraded before agents, so a job that
reaches its time limit on a node that is not yet upgraded gets 5 seconds
between SIGTERM and SIGKILL instead of the 30 it had before, which can be too
short for a checkpoint. Upgrade the agents promptly, or drain the nodes still
on the old build before their jobs’ limits arrive. Once a node is upgraded the
job gets the full timeout_kill_wait_secs.
Preemption grace time (GraceTime)#
This release makes the per-QOS GraceTime take effect and adds a per-partition
GraceTime (see preemption grace time). A victim’s
grace is its QOS value if set, else the highest value among the partitions it runs
in, else 0. Nothing is set by default, so nothing changes until an administrator sets
one.
Upgrade every controller before setting a partition ``GraceTime``. The
partition’s grace_time is a new field in the partition entries of the Raft
log and snapshots, including the entry that
scontrol update partition GraceTime= writes. A controller that predates it ignores the field, so a
change applied while a follower is not upgraded is lost on that follower, which
would lack the grace if it became leader, until it replays from a new snapshot.
This is already required before setting a QOS GraceTime (see below).
Upgrade the leader last. The first preemption that starts a grace writes a
new Raft log entry, JobPreemptSignaled. A controller that predates it
cannot parse the entry and stops replicating until it is upgraded, so the order
in Controller order: the leader goes last applies, and a rollback after the entry has been
written is not supported: roll forward. The entry is written only when a
GraceTime greater than 0 is in use.
Upgrade controllers and agents before setting ``GraceTime``. A node whose
spurd predates this release does not advertise the signal-job
capability. A victim on such a node gets no grace and is preempted at once, and
the controller logs a warning naming the node, so setting GraceTime before
the agents are upgraded silently does nothing for jobs on those nodes.
Kubernetes operator nodes get no grace. Victims on nodes managed by
spur-k8s are preempted at once whatever the QOS or partition GraceTime.
The grace_time column is added to the QOS table automatically with
ADD COLUMN IF NOT EXISTS, and a controller that predates this release
ignores it.
Preemption handoff#
Cancel and requeue preemption now hold the nodes they free for the job that
triggered the eviction, so other pending jobs no longer take them in the gap
(see Configuration Reference (spur.conf)). No configuration change is needed
and no new Raft entry type is added: the requeue entry gains an optional
handoff field that a controller predating it ignores.
During a rolling upgrade a controller that predates this release does not hold the nodes, so the protection applies only once the leader runs the new build. Victim selection also counts nodes that a draining job has already released, which changes ordinary preemption as well.
Partition resource limits#
This release adds per-partition resource caps and memory limits (see
Partition Resource Limits). Existing spur.conf files
behave as before: with no limit set, nothing changes, and
enforce_part_limits keeps its meaning for wall time.
Jobs already running when you upgrade have no recorded partition. Once a cap
is set, each one is charged to the highest-priority_tier partition it
requested, which is the partition it ran in unless it listed several.
Warning
Upgrade every controller replica before you configure any cap or memory
knob, whether in spur.conf, with scontrol, or through the API.
An older replica ignores the new fields, so it diverges from the leader:
It schedules without the caps if it becomes leader.
It ignores the adjusted request a
def_mem_*ormax_mem_*knob records when a job starts, so its copy of the job keeps the original CPU and memory request.It cannot load a Raft snapshot holding a job blocked by a cap, whose
PartitionTresLimitpending reason it does not know, either on restart or when the leader sends one to catch it up.
Rolling back. Once a cap has blocked a job, a controller from an earlier release cannot load the snapshot. To downgrade:
Remove every cap and memory knob (
spur.confplusscontrol reconfigure, orscontrol update).Wait until no job carries the
PartitionTresLimitreason. A job keeps its last pending reason after it ends, so this includes finished jobs: wait until no pending job shows the reason, then at leastcontroller.terminal_job_retention_secs(default one hour) more, so finished jobs that had it are evicted.Wait for a new Raft snapshot: on every controller,
<state_dir>/raft/snapshot.jsonmust be newer than the end of the previous step. A snapshot is taken every 5000 Raft log entries, so on a quiet cluster this can take a while; job submissions and completions move it along.Downgrade the controllers.
Configured node list (admission.node_names)#
This release adds the optional [admission] node_names list (see
Configuration-driven node removal). Only scontrol reconfigure writes its new Raft
log entry, NodeInventoryUpdate, and only when the list in the edited file
differs from the saved one, or when no list is saved yet and the file the
controller started with has one. A removed field counts as no list. Every
scontrol reconfigure first waits for earlier Raft writes to apply, so on a
heavily loaded controller it can fail with a timeout; rerun it.
Upgrade every controller before running scontrol reconfigure with a list. A
controller that predates NodeInventoryUpdate cannot parse it and stops
replicating until it is upgraded, and it ignores the list in a snapshot. If the
controllers that predate it are a majority, the entry cannot commit and every
later write stalls. Adding node_names to the file during a rolling upgrade is
safe only until the next scontrol reconfigure, even one made for a partition
change, so add the field after every controller is upgraded. Restarts and agent
registrations never write it. A rollback after the entry has been written is not
supported because older controllers cannot replay it: roll forward.