Monitoring & Controlling Jobs#
Once jobs are submitted, you inspect the queue, the nodes, and the accounting
history with a handful of commands, and you steer running jobs with cancel,
hold, and update operations. This page covers viewing the queue (spur queue),
nodes and partitions (spur nodes), accounting history (spur history),
live job stats (spur stat), detailed records (spur show), and the
controls that change a job’s state.
Note
In spur queue (squeue) and spur nodes (sinfo), -h means
--noheader, not help. spur history (sacct) uses -n for the
same purpose.
View the Queue — squeue#
spur queue (Slurm squeue) lists jobs currently in the system. With no
arguments it shows every active job, or only your own if your site sets
private_data = ["jobs"] (see Job and usage visibility (private_data)).
spur queue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1 gpu train-ll alice R 2:14 2 node[01-02]
2 default hello bob PD 0:00 1 (Priority)
The default columns are JOBID PARTITION NAME USER ST TIME NODES
NODELIST(REASON). --long/-l adds the full STATE and TIME_LIMIT.
Common flags:
Long |
Short |
Description |
|---|---|---|
|
|
Show only this user’s jobs. |
|
Show only your own jobs (same as |
|
|
|
Filter by partition. |
|
|
Filter by state codes or names (comma list). |
|
|
Show only these job IDs (comma list). A step suffix ( |
|
|
Filter by account. |
|
|
Filter by job name. |
|
|
Filter by QOS (comma-separated list). |
|
|
Filter by reservation (comma-separated list). |
|
|
Custom column format (see below). |
|
|
Long form; adds STATE and TIME_LIMIT. |
|
|
Omit the header line. |
Note
--me looks up your username on the machine you run it from. If your
login token names a different user than your local account, pass -u
with the token’s name instead: --me would list the local account’s
jobs, or nothing when your site sets private_data = ["jobs"].
When --states is omitted the default filter is PENDING, RUNNING,
SUSPENDED, COMPLETING.
--format/-o uses %[.][-][width]<letter> fields. The resolved letters:
Letter |
Column |
Letter |
Column |
|---|---|---|---|
|
JOBID |
|
CPUS |
|
NAME |
|
NODELIST |
|
USER |
|
NODELIST(REASON) |
|
ACCOUNT |
|
REASON |
|
PARTITION |
|
PRIORITY |
|
QOS |
|
WORK_DIR |
|
ST (short code) |
|
COMMAND |
|
STATE (full) |
|
RESERVATION |
|
TIME |
|
START_TIME |
|
TIME_LIMIT |
|
SUBMIT_TIME |
|
NODES |
|
END_TIME |
|
SCHEDNODES |
|
TIME_LEFT |
|
GRES |
|
PRIORITY |
|
BORROWED |
%p and %Q both render the integer priority. Slurm splits these (%p
is a normalized float, %Q the integer); Spur exposes the integer under both.
spur queue -u alice -t R
squeue --me -t PD
squeue -p gpu -o "%.18i %.9P %.8T %.10M %R"
squeue --states=PD,R --noheader
Verbose field names — squeue -O/--Format#
-O/--Format selects columns by field name instead of %-letter.
Fields are comma-separated, each type[:[.][size][suffix]]:
sizeis a minimum width (default20); unlike-o’s., it never truncates.A leading
.right-justifies the column. The default is left-justified, the opposite of-o.suffixis arbitrary text appended after the field, e.g. a separator.
Field names are case-insensitive and render the same data as their %-letter
equivalents. -O and -o/--format are mutually exclusive; passing both
is an error, matching Slurm.
squeue -O "JobID:10,Partition,State,QOS"
squeue -O "JobID:.18,Priority,Reason"
Supported field names: Account, ArrayJobID, Command, Comment,
EndTime, GRES, JobID, Name, NodeList, NumCPUs,
NumNodes, Partition, Priority, PriorityLong, QOS,
Reason, ReasonList, Reservation, SchedNodes, StartTime,
State, StateCompact, SubmitTime, TimeLeft, TimeLimit,
TimeUsed, UserName, WorkDir.
Slurm exposes many more -O fields that Spur has no backing data for
(Licenses, Dependency, Nice, Reboot, most of the tres-*
family, federation fields, job-step fields, and others). Requesting an
unsupported name is an error listing the supported fields, rather than the
blank column Slurm prints. This is a deliberate divergence: fail fast on a
typo instead of silently emitting an empty column. Priority renders Spur’s
integer priority (same as PriorityLong); Spur does not compute Slurm’s
normalized 0.0-1.0 float.
Projected Start Times — squeue --start#
--start answers “when will my job run, and where” for the whole queue at
once, rather than one job at a time through scontrol show job:
$ squeue --start
JOBID PARTITION NAME USER ST START_TIME NODES SCHEDNODES NODELIST(REASON)
1024 gpu train.sh alice PD 2026-08-27T17:09:56 2 node[07-08] (Resources)
1031 gpu eval.sh bob PD N/A 1 (null) (Priority)
It restricts the view to pending jobs and orders them by projected start,
soonest first; jobs with no reserved slot report N/A and sort last. Each of
those defaults is overridable by passing -t, -S, or -o explicitly.
%S and %e carry the same projection in any format string, so
squeue -o "%.18i %.19S %.19e" works without --start.
Note
Projections come from the backfill pass on the controller’s leader, and assume every running job uses its full time limit. A job that finishes early moves every projection behind it earlier.
Job state codes. The ST column uses these short codes:
Code |
State |
Code |
State |
|---|---|---|---|
|
PENDING |
|
CANCELLED |
|
RUNNING |
|
TIMEOUT |
|
COMPLETING |
|
NODE_FAIL |
|
COMPLETED |
|
PREEMPTED |
|
FAILED |
|
SUSPENDED |
|
DEADLINE |
|
OUT_OF_MEMORY |
For a pending job the NODELIST(REASON) column shows why it is waiting.
Common reasons include Priority (waiting its turn), Resources (waiting
for nodes to free up), Dependency (waiting on another job), Reservation
(waiting for a reservation window), Preempted (requeue-preempted and held
until its eligibility window reopens), JobArrayTaskLimit (held back by the
array’s own %N concurrency cap, see Job Arrays),
PartitionTresLimit (held back by a partition’s per-node resource cap; the
column shows a description with the numbers, see
Partition Resource Limits), and various QOS or association
limit reasons (QOSMax*, AssocMax*, AssocGrp*).
View Nodes & Partitions — sinfo#
spur nodes (Slurm sinfo) shows partition and node state. The default is a
partition-oriented view; -N switches to one line per node.
spur nodes
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
default* up infinite 4 idle node[01-04]
gpu up 1-00:00:00 2 mix node[05-06]
Flags:
Long |
Short |
Description |
|---|---|---|
|
|
Filter by partition. |
|
|
Filter by node state (comma-separated; a node in any listed state
matches). Accepts the base states, the display-only labels |
|
|
Only nodes the controller is not hearing from. Combines with |
|
|
Only nodes the controller is hearing from. Combines with |
|
|
Filter by node. |
|
|
One line per node instead of per partition. |
|
|
List unavailable nodes (down/drain/draining/error) grouped by reason. |
|
|
Custom column format. |
|
|
Long form; adds CPUS and MEMORY. |
|
|
Omit the header line. |
Default columns (partition view): PARTITION AVAIL TIMELIMIT NODES STATE
NODELIST; -l adds CPUS and MEMORY. Node view (-N): NODELIST NODES
PARTITION STATE CPUS MEMORY GRES.
sinfo -N -o "%N %.6D %.11T %c %m %G"
sinfo -p gpu -l
%z prints Slurm’s S:C:T (sockets, cores per socket, threads per core),
as in sinfo -N -o "%N %z". A node whose agent reported no CPU topology
shows N/A. In the partition view, where a row covers several nodes, the
row shows the shared value only when every node in it agrees; otherwise, or
when any node is unknown, it shows N/A.
-R / --list-reasons lists only nodes that are unavailable (down,
drain, drng, err), grouped by the admin reason set when the node was
drained or downed, one row per reason with a compressed nodelist:
REASON USER TIMESTAMP NODELIST
Memory errors alice 2026-08-27T08:03:07 node[05,08]
Not responding root 2026-08-27T09:15:44 node[01-03]
All four columns are populated from live state. USER (%u, or %U for
user(uid)) is the user who set the reason, resolved from the recorded UID;
automatic reasons (for example a heartbeat timeout) are attributed to root.
TIMESTAMP (%H) is when the reason was set. A UID that cannot be resolved
to a name, or a node whose reason predates provenance tracking, renders as
Unknown. -R -l adds a STATE column (%t).
scontrol show node appends the same provenance to the reason line as
Reason=<text> [<user>@<timestamp>] when a set-time is recorded, and the REST
node object carries reason_uid and reason_time fields.
scontrol show node ends its CPUTot= line with Slurm’s Sockets=,
CoresPerSocket= and ThreadsPerCore=, derived from the CPU topology the
node’s agent reported: cores and threads are divided evenly as slurmd does,
so uneven sockets or partly offline threads round down. A node whose agent
reported no topology (a Kubernetes-backed node, or an agent older than the
controller) leaves the three fields out. A Capabilities= line lists the
optional features the node’s agent advertised (for example signal-job).
When a node reserves cores per GPU (restricted_cores_per_gpu),
scontrol show node adds RestrictedCoresPerGPU=<R>(CPUs:<list>) naming how many
cores each GPU reserves and, across all GPUs, which CPU ids they are; a GPU job’s CPUs beyond
that count, or a CPU-only job, draw from the node’s general pool instead. Each
GPUDevice= line then ends with ReservedCPUs=<list>, that GPU’s own picks,
and AffinityCPUs=<list>, the CPUs the picks are drawn from (the GPU’s
cores=, else its NUMA node’s CPUs). Either field is left out when unknown,
and AffinityCPUs is shown even without reserved cores. The REST node object
carries the same data as sockets, cores, threads (null when
unknown), gpu_cpus (stable_id, reserved_cpus, affinity_cpus) and
capabilities. See Configuration Reference (spur.conf) for the full node-field reference.
Node states are shown as short abbreviations: idle (free), alloc (fully
allocated), mix (partly allocated), down, drain (offline, not
accepting jobs), drng (draining), err (error), unk
(unknown/unreachable), susp (suspended), and — all three only shown while
the node is currently idle — resv or maint for a node held by an admin
reservation, else plnd for a node currently held by the scheduler for a
specific pending job’s upcoming start. The idle gate is checked live, but the
job and start time shown for plnd reflect the most recent scheduling
cycle (scheduler.interval_secs), not the current instant.
Every abbreviation above, including the overlay labels resv, maint, and
plnd, can be passed to -t / --states. The long forms reserved
and planned are accepted as aliases for resv and plnd. -t resv
also matches maint nodes (a maintenance reservation is still a reservation),
while -t maint matches only maintenance nodes. -t idle returns all idle
nodes, including those shown as resv, maint, or plnd.
A trailing * on the state (idle*, mix*, down*) marks a node the
controller is not currently hearing from: its heartbeats have gone stale, so it
is given no new work until they resume, while its running jobs are untouched.
Filter on it with -t no_respond, --dead (-d, only non-responding
nodes), or --responding (-r, only responding nodes). Like the other
-t states, no_respond widens the list: sinfo -t idle,no_respond
shows idle nodes plus every non-responding node, whatever its state. -d and
-r narrow it instead: sinfo -t idle -d shows only idle nodes that are
not responding, and -d -r together match nothing.
scontrol show node shows the same condition as a +NOT_RESPONDING suffix
on State=, the REST node object carries a not_responding boolean, and
spur health stars the node and counts it as unhealthy.
The flag is the current leader’s view of its heartbeats. It is blank for the
grace window after a leadership change (see Node Liveness), so a silent
node can still take new work in that window. Once it ends, a DOWN node the
leader has not heard from shows as down*. A follower never sets it: the REST
API answers node reads from the controller it reaches, so not_responding is
false for every node when that is a follower, and the CLI shows no *
when a follower serves a read itself because it cannot forward it to the leader.
Query the leader for this flag.
Accounting History — sacct#
spur history (Slurm sacct) reports finished and running jobs from the
accounting service. With private_data = ["jobs"] it lists only your own
jobs, and -u for someone else prints nothing (see Job and usage visibility (private_data)).
spur history -u alice -S now-7days
Flags:
Long |
Short |
Description |
|---|---|---|
|
|
Filter by user. |
|
|
Filter by account. |
|
|
Earliest submit/start time to include. |
|
|
Latest time to include. |
|
|
Filter by state (comma list). Accepted values (long or short code):
|
|
|
Filter by job ID (comma list; step suffixes like |
|
|
Comma-separated field names (see below). |
|
|
Long form; adds DerivedExitCode, Start, End, TimeLimit, the three preemption fields, and Reason. |
|
|
Brief form: JobID, State, ExitCode. |
|
|
Omit the header line. |
|
|
Print fields |
|
|
As |
|
Maximum rows to print. Must be at least 1. Default |
Unlike squeue, --format here takes comma-separated field names, not
% letters. Available fields include JobID, JobName, User,
Account, Partition, State, Elapsed, NNodes, ExitCode,
DerivedExitCode, Start, End, Submit, TimeLimit, NodeList,
NCPUS, ReqMem, QOS, PreemptedBy, PreemptMode, PreemptQOS,
Borrowed, Reason, ReasonDetail, EndedBy, and ReqGPUS. Set a per-field width with
Field%N, e.g. JobName%20.
NodeList is the compressed hostlist of the nodes the job was allocated, in
the same form squeue and scontrol show job print (node[01-04]). It
is recorded when the job starts, so it stays available after the job leaves
the controller; a requeued job shows the nodes of its latest run. Jobs that
started before this release recorded no node list and show it empty.
NCPUS is the job’s total allocated CPUs, recorded when the job starts: for
a --cpus-per-gpu job this already counts the CPUs its GPUs brought in, so
it reads higher than cpus-per-task x tasks alone would suggest. Jobs that
started before this release recorded no CPU count and show
cpus-per-task x tasks.
PreemptedBy is the job ID of the higher-priority job that caused the
preemption (N/A when the job was not preempted). PreemptMode is one of
Requeue, Cancel, or Suspend. PreemptQOS is the QOS of that
preempting job — the QOS whose preempt allow-list authorized the
preemption. It is set for any job preempted by this release; jobs preempted by
an earlier release, which recorded no authorizing QOS, show N/A even though
they were preempted. All three appear in the long format (-l).
Borrowed is yes when the run took borrowed capacity under
Idle-Fill Scheduling, and no otherwise. It is the same
question squeue’s %W answers for a running job:
$ sacct --format=JobID,User,State,Borrowed
JobID User State Borrowed
19 ifbob PREEMPTED yes
21 ifalice COMPLETED no
Why a job ended#
Reason is a short code saying why the job’s run ended, and ReasonDetail
adds what Spur knew beyond the code. Use Reason to group failures, for
example sacct -n -s FAILED,NODE_FAIL --format=Reason piped through
sort | uniq -c.
$ sacct --format=JobID,State,ExitCode,Reason,ReasonDetail
JobID State ExitCode Reason ReasonDetail
41 FAILED 127:0 NonZeroExitCode exit 127: command not found (shell convention)
42 FAILED 0:11 RaisedSignal signal 11 (Segmentation_fault)
43 FAILED 1:0 CompletingTimeout no completion reported by node3
44 TIMEOUT 0:15 TimeLimit
Reason |
State |
When |
|---|---|---|
|
COMPLETED |
The job ended normally. |
|
FAILED |
The job exited with a non-zero code. For 126 and 127 the detail notes the shell convention (“not executable”, “command not found”); that is a hint, since a script may also exit 127 itself. |
|
FAILED |
A signal ended the job; the detail names it. |
|
FAILED |
The controller stopped waiting for the job’s nodes to report; the detail lists the nodes that never did. |
|
TIMEOUT |
The job reached its time limit, however the process then died. |
|
DEADLINE |
The job’s |
|
OUT_OF_MEMORY |
The kernel killed the job for exceeding its memory limit. |
|
PREEMPTED |
A higher-priority job preempted it; the detail names that job and its QOS. |
|
CANCELLED |
The job’s owner cancelled it. |
|
CANCELLED |
An operator or administrator cancelled someone else’s job. |
|
CANCELLED |
Spur cancelled it: a dependency that can never be met, an ended
reservation, a failed prolog on an interactive or |
|
CANCELLED |
Cancelled, but who did it was not recorded: the job was cancelled before this release, or by an older controller during a rolling upgrade. |
|
REQUEUED |
The job’s owner or an operator requeued it while it was running. |
|
NODE_FAIL |
A node the job ran on went down. The detail names the node and its
down reason, as |
|
NODE_FAIL |
A node the job ran on was removed from the cluster; the detail names the node and the removal reason. |
|
NODE_FAIL |
The job could not be started on every node; the detail says why. |
EndedBy is the user who cancelled the job, for UserCancel and
AdminCancel; a SystemCancel has no user and, like every other reason,
shows N/A. It is shown to operators and administrators, the same audience as
sacctmgr show txn; for anyone else it reads N/A. When [auth] runs in
disabled or permissive mode an unauthenticated caller counts as that
audience, as it does for the audit log.
Reason reads N/A for a job that is still running, for rows written
before this field existed, and for rows written through the external
RecordJobEnd accounting RPC. Accounting keeps one row per job,
so a requeued job’s next run replaces the reason with its own. During a
rolling upgrade, a job that restarts and then ends while an older controller
is leader keeps the reason of its previous run, and a run the older controller
force-finished reads NonZeroExitCode rather than CompletingTimeout.
The default columns are JobID JobName User Account Partition State Elapsed
NNodes ExitCode Reason. Scripts that read columns by position should select
them with --format.
A job cancelled or expired before it starts keeps its name, user, account,
partition, and requested resources in accounting. Its Start is Unknown
and it is charged no usage.
Time arguments accept an absolute date (YYYY-MM-DD or
YYYY-MM-DDTHH:MM:SS) or a relative offset (now-7days, now-6hours).
sacct -S 2026-07-01 -E 2026-07-25 -s FAILED --limit 500
sacct --format=JobID,JobName,State,Elapsed,ExitCode
sacct -s PREEMPTED --format=JobID,JobName,State,ExitCode,PreemptedBy,PreemptMode,PreemptQOS
sacct -s PR,CA # short codes also accepted
State names and short codes are case-insensitive. An unknown state in a list
makes the command exit with an error before querying accounting. Omit -s
or use -s all to query all states.
Running-Job Stats — sstat#
spur stat (Slurm sstat) reports live resource usage for running jobs.
--jobs/-j is required (comma list). A step suffix (30.0) matches
the base job. A non-numeric ID is an error.
spur stat -j 12345
sstat -j 12345,12346 -o JobID,NTasks,GPUAlloc,Elapsed -p
Default fields: JobID NTasks NCPUS MemAlloc GPUAlloc Elapsed Nodelist.
--format/-o takes field names; --parsable/-p produces
|-delimited output. NCPUS is the running job’s ReqTRES cpu=
count, so a --cpus-per-gpu job’s count already includes the CPUs its GPUs
brought in.
Note
The per-process Ave* and Max* fields (AveCPU, AveRSS,
MaxRSS, and similar) always report N/A — Spur does not collect
per-process metrics.
Detailed Records — scontrol show#
spur show (Slurm scontrol show) prints the full Key=Value record for
an entity: job, node, partition, reservation, step, or
assoc_mgr.
Note
The diagnostic fields below are printed on every job, with (null) for an
unset string, so a parser must key on the field name rather than on a line
being absent. Some fields still appear only when set (Comment,
Reservation, ArrayJobId, SchedNodeList, StdIn, the GPU line,
the preemption line, and the SuspendedBy/LastRequeue line), and
MinMemoryNode is spelled MinMemoryCPU for a --mem-per-cpu job.
For a live job someone acted on, scontrol show job adds a line naming who.
SuspendedBy is the user who suspended the job, only while it stays
suspended; a job paused by preemption shows the preemption line instead.
LastRequeue says why the job last went back to the queue: Failure (a
launch failure, time limit or node failure with --requeue), Preempt, or
Admin (an scontrol requeue by the owner or an operator), with whoever
requeued it in parentheses for Admin:
SuspendedBy=alice LastRequeue=Admin(ops)
The user names are shown to the same audience as EndedBy in sacct
(operators, administrators, and unauthenticated callers when [auth] is
disabled or permissive); for anyone else they are blank and
LastRequeue shows the kind alone.
scontrol show job 1024
spur show node node01
scontrol show partition gpu
Expanding node lists — scontrol show hostnames / hostlist#
scontrol show hostnames expands a hostlist expression to one host per line;
scontrol show hostlist (and hostlistsorted) does the reverse, compressing
a list back into the bracketed form. Both are pure local transforms: they never
contact the controller, so they work inside a batch script on any compute node.
With no argument, hostnames reads SLURM_JOB_NODELIST (Spur also honors
SPUR_JOB_NODELIST), which makes it the standard way to derive a rendezvous
address or an MPI hostfile from an allocation:
scontrol show hostnames 'gpu[01-03]' # -> gpu01 / gpu02 / gpu03, one per line
scontrol show hostlist gpu03,gpu01,gpu02 # -> gpu[01-03]
# Inside a job: first allocated node as the rendezvous host, all nodes to a hostfile.
export MASTER_ADDR="$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n1)"
scontrol show hostnames > "$HOME/hostfile.$SLURM_JOB_ID"
hostlist and hostlistsorted produce identical output in Spur: both remove
duplicates and sort before compressing.
Diagnosing a job that will not start#
A job pending on Reason=Resources looks the same whether the cluster is full
or the job is pinned to a busy subset of it. scontrol show job reports the
request as submitted, which distinguishes the two:
A job that no node could host even when completely idle reports
Reason=ReqNodeNotAvail, Requested node configuration is not available
instead. Waiting will not help it: either the request exceeds every node’s
capacity, or it names a resource no node declares — a typo such as
--gres 1 asks for one unit of a resource literally named 1, which is
easy to miss because it never appears in ReqTRES. Compare ReqTRES and
MinCPUsNode / MinMemoryNode below against spur show node, and
check --gres spelling. The reason is recomputed every scheduling cycle, so
the job starts on its own once matching nodes join.
Field |
Meaning |
|---|---|
|
Nodes requested with |
|
Nodes excluded with |
|
Nodes actually allocated. Empty ( |
|
Node feature constraint from |
|
Requested resource totals, e.g. |
|
|
|
|
|
Per-node minima implied by the request. Memory is in MB with an |
|
Submission flags and the requeue count for this job. |
|
Elapsed and requested wall time. |
|
First executable line of the batch script. |
|
The submit command as invoked, so any submission-flag question is answerable from one command. |
|
Earliest the job may start ( |
|
When the job began accruing age priority. |
|
While pending, the slot the scheduler is holding: when it projects the
job will start, and on which nodes. With no slot reserved, |
|
The recorded end once the job finishes, otherwise its start plus its
time limit. |
|
The last scheduling cycle that considered this job, so it advances even
on a cycle that places nothing. |
Note
StartTime, SchedNodeList, and LastSchedEval live only on the
leader and are never replicated, so a read served by a follower reports them
unset even in a healthy cluster.
So a job pinned to a busy node reads (excerpt — the full record has more fields between these lines):
JobState=PENDING Reason=Resources Dependency=(null)
...
StartTime=2026-08-27T17:09:56 EndTime=2026-08-27T17:14:56 Deadline=N/A
...
ReqNodeList=node07 ExcNodeList=(null)
NodeList=(null) SchedNodeList=node07
...
SubmitLine=sbatch -w node07 --exclusive -t 5 job.sh
The controller logs the matching scheduler-side view at info level, one line
per job when its placement outcome changes (not every cycle). A cycle with no
schedulable nodes at all is skipped before this runs, so it logs nothing. Jobs beyond the
per-cycle scheduling depth limit are not reported:
backfill did not start job job_id=1024 reason=no_suitable_nodes needed_nodes=1 candidate_nodes=0
reason is one of het_group_incomplete, no_suitable_nodes,
requested_nodes_unavailable, too_few_candidates, no_capacity_at_start,
future_slot_reserved, or refused_at_start. future_slot_reserved means
the job can be placed and the scheduler is holding a future slot for it, reported
with planned_start. refused_at_start means the job could be placed, now or
in a future slot, but on the nodes it was placed on it would be over a QOS,
partition QOS or account limit, counting the jobs started earlier in the same
cycle and the room held for preemptions in flight. It holds no slot and pends with
that limit’s reason, for example QOSGrpNodeLimit.
Note
Spur accrues age priority from submit time for every pending job, including
held and dependency-blocked ones. Slurm suspends accrual for those, so
AccrueTime can read earlier here than on an equivalent Slurm job.
Preemption provenance#
When a job has been preempted, scontrol show job includes three additional
fields:
PreemptedBy=<job_id>— the ID of the higher-priority job that triggered the preemption. Only shown once the job has been preempted; during a preemption grace period nothing has been preempted yet, so it is not printed.PreemptMode=Requeue|Cancel|Suspend— how the preemption was carried out.PreemptQOS=<name>— the QOS of the preempting job, whosepreemptallow-list authorized the preemption. Set for any job preempted by this release; jobs preempted by an earlier release showN/Aeven though they were preempted.
While a job is inside its preemption grace period, scontrol show job instead
prints a line holding only PreemptTime=<timestamp>, the moment the job was
signaled to wind down, and EndTime shows when the grace ends. PreemptTime
is omitted for a job that is not in a grace. The REST job object
(GET /api/v1/job/<id>, GET /api/v1/jobs, and the same under
/slurm/v0.0.42) reports the same as preempt_time (Unix seconds, null
outside a grace) and end_time. While the job is still running, end_time
is the projected end of its grace, not the time it finished: use job_state
to tell whether a job has ended. The same job object also reports
cpus_per_gpu and memory_per_gpu, both null when the job did not
request them.
How a job learns it is being preempted#
If the QOS your job runs under, or the partition it runs in, has a GraceTime
(ask your administrator; see preemption grace time;
the QOS value wins if it is set, otherwise the highest value among the
partitions the job runs in applies), a job selected for preemption in cancel or
requeue mode is not killed at once. It gets SIGCONT then SIGTERM, one time,
as a warning, stays RUNNING and keeps its nodes, and is preempted at the
deadline shown as EndTime (PreemptTime shows when the warning was sent).
The grace is shortened if the job’s own time limit arrives first; the deadline
is fixed when the warning is sent, so lowering the time limit later does not
shorten it. The warning is sent once: if it is lost (for example the node’s agent
is unreachable at that moment) the job is still preempted at the deadline, only
without having been warned. Without a GraceTime there is no warning period and
the job is preempted immediately.
Launch your work with
srun. The warning goes to the job’ssrunsteps, not to the batch script, so a training process started directly from the batch script gets no warning and sees only the final kill. Run it assrun python train.pyand handle SIGTERM there to checkpoint.Expect a second SIGTERM. At the deadline Spur sends SIGTERM to every process in the job, waits up to the kill wait (5 seconds by default), then sends SIGKILL. A handler that is already shutting down should ignore a repeat. The wait does not yet reach
srunsteps (kill wait note).Exiting early is fine. A job that exits during its grace is recorded as
PREEMPTED(cancelmode) or goes back toPENDING(requeuemode), and the preempting job starts as soon as the nodes are free.One node finishing early does not affect the others. The job stays
Runtil every node has finished or the deadline. A node whose work has finished is freed for the preempting job, and the other nodes keep running undisturbed, so a rank that exits first does not cut short the ranks that are still writing their checkpoint. If your batch script launches ranks withsrun, ranks exiting do not end the job: it ends when the batch script exits (the remaining nodes are then stopped), or at the deadline.You can still cancel.
scancelduring the grace ends the job as your cancel.scontrol suspendis refused.The grace is not withdrawn. The job is preempted at the deadline even if the job that triggered it is cancelled meanwhile.
Jobs on nodes managed by the Kubernetes operator get no grace and no warning.
Borrowed runs and reclaim#
When Idle-Fill Scheduling is enabled, a job that exceeded its
QOS group node quota may still run, on capacity no job with a quota claim wanted.
Such a run is borrowed, and squeue’s %W field reports it:
$ squeue -o "%i %j %u %t %N %W"
JOBID NAME USER ST NODELIST BORROWED
18 alice-legit ifalice R node-3 no
19 bob-borrow ifbob R node-4 yes
A borrowed job can be reclaimed when a job that does hold a quota claim needs its
nodes. Reclaim requeues: the job returns to the queue with its spec intact and
starts again once capacity allows, so the partial run is lost but the work is not.
It appears as an ordinary preemption, with PreemptMode=Requeue and
PreemptedBy naming the job that reclaimed it, and sacct records that the run
was borrowed.
Note that --no-requeue does not prevent reclaim, and that an over-quota job
which has not been lent capacity keeps reporting QOSGrpNodeLimit rather than a
placement reason — being over quota is the true reason it is waiting.
Quick reference — one command for any preempted job:
scontrol show job <id>
This works regardless of preemption mode (requeue, cancel, or suspend) as long as the job is still known to the controller (running, pending, suspended, or recently terminal). For cancel-preempted jobs that have been evicted from memory, fall back to:
sacct -j <id> --format=JobID,State,PreemptedBy,PreemptMode,PreemptQOS
The accounting database keeps the record permanently. In practice
scontrol show job is the right first instinct; fall back to sacct when
it reports Job <id> not found (a non-zero exit status), which means the
controller no longer holds the job. A malformed id is different:
scontrol show job abc fails with Invalid job id specified: abc, so a
script can tell a typo from a job that has left the controller.
Note
Suspend-mode preemption and accounting. When PreemptMode=Suspend,
the job receives SIGSTOP and stays running — no accounting end-record
is written. The provenance fields (PreemptedBy, PreemptMode,
PreemptQOS) are visible in scontrol show job while the job is
suspended, but are cleared when the job resumes so that a subsequent normal
completion is not miscounted as a preemption. As a result, sacct has
no record of the preemption for suspend-mode jobs: the accounting row for
that run will show the final completion state only. Requeue mode is
unaffected — it writes an accounting end-record (PREEMPTED) at the time
of preemption. Cancel mode writes the same record, but only once the run has
actually ended: the victim holds its allocation in COMPLETING until every
node reports the release, or until CompleteWait elapses.
Limits Against Usage — scontrol show assoc_mgr#
scontrol show assoc_mgr answers the question squeue cannot: how much of a
QOS or an association each user is holding right now, next to the caps that
govern them. Without it an operator has to infer per-user totals by counting
squeue rows.
scontrol show assoc_mgr
scontrol show assoc_mgr alice
scontrol show assoc_mgr users=alice
Output is two sections of Key=Value blocks — QOS records, then association
records. Each block opens with the scope itself and lists a line per user under
it:
QOS Records
QOS=highprio MaxWall=01:00:00 MaxTRESPJ=cpu=32 MaxJobsPU=2 MaxSubmitJobsPU=N MaxTRESPU=node=4 MaxSubmitJobsPA=20
GrpJobs=N(9) GrpSubmitJobs=N(11) GrpTRES=cpu=N(36),node=16(9) GrpWall=7-00:00:00(2-12:00:00)
User=alice MaxJobsPU=2(6) MaxSubmitJobsPU=N(7) MaxTRESPU=cpu=N(24),node=4(6) OverLimit=MaxJobsPU,MaxTRESPU
User=bob MaxJobsPU=2(1) MaxSubmitJobsPU=N(1) MaxTRESPU=cpu=N(4),node=4(1)
Association Records
Account=tenant-a MaxWall=N
GrpJobs=N(1) GrpSubmitJobs=N(1) GrpTRES=node=N(1)
User=alice MaxJobs=4(1) MaxSubmitJobs=N(1) MaxTRES=cpu=N(4),node=N(1) MaxTRESPJ=node=4
Reading it:
On the
Grp*line and theUser=lines every cap exceptMaxTRESPJis printed as ``Limit(Consumed)``, as in Slurm:node=4(6)is a cap of four nodes with six in use, andNmarks no cap.GrpWallfollows the same shape as a wall-clock time,budget(spent); anNin its consumed slot means the controller has not read spend yet (its usage cache holds no snapshot), which is not the same as none spent. The per-job and per-account caps (MaxWall,MaxTRESPJ,MaxSubmitJobsPA), and a QOS’s per-user caps on its scope line, print bare, with no consumption beside them, because they bound each job, account, or user rather than a total the scope accrues.For the count caps a literal
0is a real cap that blocks every job it governs. A TRES dimension is the exception:0there is treated as unset — it renders asNand is not enforced, soGrpTRES=node=0does not block a dimension the wayMaxJobs=0blocks jobs.Consumption is live, measured exactly as the scheduler measures it when it admits a job. Node counts are distinct occupied nodes, so two jobs sharing a node hold one node, not two. A TRES dimension appears when either the cap or the usage has something to say about it.
The scope line carries what belongs to the scope: its wall-clock ceiling
MaxWall, and — for a QOS, which caps every user identically — its per-job capMaxTRESPJ, the per-user caps it enforces (MaxJobsPU,MaxSubmitJobsPU,MaxTRESPU), and the per-account submit capMaxSubmitJobsPA. An association caps each(user, account)pair on its own, so its per-user caps and itsMaxTRESPJride on eachUser=line instead.MaxTRESPJbounds one job andMaxTRESPUbounds a user’s total within the QOS. An association carries no per-user TRES cap, soMaxTRESon aUser=line always readsN; a cap on one user across accounts is a partition QOS.MaxSubmitJobsPAandGrpWallare QOS-only and never appear in association records.Grp*figures are the whole scope’s, summed across every user, and stay on the scope line.GrpWallis the QOS’s wall-clock budget beside the spend measured overgrp_wall_window_days(see Accounting, Accounts, Users, and QOS). Filtering to one user narrows theUser=lines but never the group figures, since a group cap cannot be judged from one user’s share.OverLimitlists caps that are already exceeded, on the scope line for group caps — includingGrpWallonce spend reaches the budget, the state behind aQOSGrpWallLimithold — and on a user’s line for that user’s caps. It is absent when everything is within its cap. Usage over a cap is not a contradiction: caps are applied when a job is admitted and running jobs are never re-checked, so tightening a cap under running work leaves exactly this state.
Records come from the accounting definitions as well as from the queue, so a QOS
nobody is using still appears with its caps and no User= lines — an
over-tight cap on an idle QOS is exactly the kind of mistake that hides
otherwise. A QOS deleted after its jobs queued also stays reported for as long as
those jobs hold resources.
A LimitsReadable=NO line above the sections means accounting is enabled but a
cache has not loaded yet, so some caps below may be missing; the usage figures
are still current. A cluster with accounting disabled has no caps to read and so
never prints this line.
By default every caller sees every scope and may pass users=<name> to
inspect one user. When an administrator sets private_data = ["usage"] under
[auth] (see Job and usage visibility (private_data)), an unprivileged caller sees only the QOS
and accounts they take part in, and only their own User= lines; a users=
selector is replaced with their own name. sshare omits other users’
rows and leaves account totals and shares of the total blank. sreport
utilization fills in only your own usage and fairshare and leaves the other
cells blank. sreport preemptions shows only your own rows and the accounts
you coordinate when either "jobs" or "usage" is private. sprio and
sreport job sizes follow "jobs" instead. Grp* totals on the scope line
still include every user’s jobs. Operators and Administrators are exempt.
Cluster Metrics — /metrics#
Beyond the per-job CLI commands above, spurctld exports cluster-wide metrics
over HTTP in Prometheus/OpenMetrics text format. This is the surface a
monitoring stack (Prometheus, Grafana, an OpenMetrics scraper) consumes to chart
queue depth, node utilization, and resource allocation over time.
The server is controlled by the [metrics] section of spur.conf (see
Configuration Reference (spur.conf)). It listens on port 6822 and by default
binds to loopback only; set bind = "all" to expose it to a scraper on
another host. Only the Raft leader serves data — followers return
503 Service Unavailable — so point your scraper at all controllers and let it
follow the leader.
curl http://127.0.0.1:6822/metrics/jobs
curl http://127.0.0.1:6822/metrics/nodes
Endpoints#
Path |
Status |
Contents |
|---|---|---|
|
Live |
Alias for |
|
Live |
Job counts by state and aggregate allocated resources. |
|
Live |
Node counts by state, cluster resource totals, and per-node gauges. |
|
Planned |
Route exists but currently returns an empty body. |
|
Live |
Scheduler cycle statistics and preemption outcomes. |
|
Live |
Spur-managed k0s cluster lifecycle and per-node health ( |
|
Live |
Per-user/per-account job gauges and karma counters. Requires
|
Job metrics — /metrics/jobs#
All gauges. <state> expands to one metric per job state: pending,
running, completing, completed, failed, cancelled,
timeout, node_fail, preempted, suspended, deadline,
out_of_memory.
Metric |
Description |
|---|---|
|
Total number of jobs. |
|
Number of jobs in each state. |
|
CPUs allocated to running/completing jobs. |
|
Memory (bytes) allocated to running/completing jobs. |
|
GPUs allocated to running/completing jobs. |
Preemption outcomes — /metrics/scheduler#
When a pending job (the requester) preempts running jobs (the victims), Spur records whether the launch of the requester used the capacity each victim released. Counts describe victim runs, not nodes: preempting the same job on two runs counts twice. Suspension and idle-fill reclaim without an owner-directed handoff are excluded. A grace warning alone does not count; the first released slice opens one observation for that victim run.
Metric |
Type |
Meaning |
|---|---|---|
|
Counter |
Victim runs with matching released capacity in a confirmed requesting launch. |
|
Counter |
Victim runs closed without confirmed use by the requesting launch. |
|
Gauge |
Victim runs that released capacity but whose requesting launch outcome is unresolved. |
|
Gauge |
Finished jobs past retention kept in the controller until their counts reach accounting. |
|
Gauge |
Age of the oldest of those jobs, from its end time. |
For example, if a requester preempts A on n1 and B on n2, then launches only using B’s capacity, used increases by one and unused by one. Partial reports and replay do not count a victim run again. A successful launch followed by a later application failure remains used. A failed release is a launch failure, even if the controller already recorded the job as Running. Native interactive allocations count as launched once allocation registration and activation succeed.
reason has four values:
started_elsewhere: the launch succeeded without matching released capacity.launch_failed: a reserved requesting launch was rejected, held, backed off, lost its nodes, or was not released.cancelled: cancellation ended the request before use.ended_other: another end or a superseding attempt closed the request, including an outcome that could not be confirmed after controller failure.
initiator is user, admin, system, or unknown. Cancellation
uses the authenticated cancellation provenance. Scheduler placement and launch
failures use system. Missing provenance stays unknown. The label records
the observed operation, not an inferred root cause: a launch rejection caused
by user-supplied input still has a system launch-failure verdict. The
ended_other / unknown bucket is uncertainty, not proof of resource loss.
No username, job ID, or node name is a Prometheus label.
The counts are also available from:
sdiag: cluster totals, unused counts by cause, and, when finished jobs past retention are waiting for export, how many and the oldest one’s age.scontrol show job:PreemptionsUsed,PreemptionsUnused, andPreemptionsOpenon the requesting job, plusUnusedPreemptionCausesas comma-separatedreason/initiator:countpairs. Printed only when a count is nonzero.REST job responses: the same counts and causes in
preemption_usage.sreport preemptions: lifetime totals per requesting user and account (see below).
A device-backed victim counts as used only when the requester received the same node-local devices; using CPU on the same GPU node is not enough. CPU and memory have no identity, so they are matched against a capacity budget. Both are approximations: they do not prove a preemption was necessary or measure lost GPU-hours. One victim using any matching capacity counts once.
If the requesting launch fails before it reserves nodes, the observation stays
open so a retry can still use the capacity. If the requester gives up a
reservation before it starts (a backoff, a hold, or a node going down), the
observation closes as launch_failed and a later retry does not count as use.
A launch result that was not recorded before a controller restart closes as
ended_other / unknown once the new leader’s dispatch grace expires (twice
the dispatch timeout; never, if that timeout is disabled). This records
uncertainty, not a launch failure.
Cluster totals and per-job counts survive controller restart and are not
cleared by sdiag --reset.
Lifetime user/account reporting#
sreport preemptions needs PostgreSQL accounting:
sreport preemptions
sreport preemptions --user alice --parsable
sreport preemptions --account research --limit 25
sreport preemptions --account research --by-cause
sreport preemptions --account research --limit 25 --page-token '<token>'
Rows show User, Account, Used, Unused, and Open. Each row is a
requesting user/account pair; a user’s rows across accounts add to that user’s
total. A complete account report requires every page for that account. The CLI
prints the next-page token on standard error when another page exists. The
default page contains at most 100 rows; --limit accepts 1 through 1000.
--start and --end are rejected because these are lifetime totals rather
than event-time reports. Pages are read in user/account order, not from a frozen
snapshot: counts can change between pages, and a pair added before the current
position during paging is not shown.
Add --by-cause to show only nonzero unused counts, with one row per requesting
user/account, reason, and initiator. These use the same values as the scheduler
metrics and count victim runs, not nodes. For example:
$ sreport preemptions --by-cause
User Account Reason Initiator Unused
alice research cancelled user 2
alice research launch_failed system 1
With --parsable --by-cause, the columns are
User|Account|Reason|Initiator|Unused. For example:
$ sreport preemptions --parsable --noheader --by-cause --user alice
alice|research|cancelled|user|2
alice|research|launch_failed|system|1
Cause counts add up to the pair’s Unused total in the default report. Pairs
with no unused counts have no rows in the cause report. Pagination still counts
user/account pairs and keeps their causes together, so a page can have no cause
rows and still print a next-page token; continue until there is no token.
The report applies both jobs and usage privacy settings, including
coordinator visibility (see Job and usage visibility (private_data)). A scoped report is not a full
account or cluster total. Cluster totals in sdiag and the metrics are
aggregates with no per-user detail and are not scoped.
Accounting reports lag the live counts, and Open there is the last exported
value. Preemption outcome export describes how counts reach PostgreSQL
and what happens during an accounting outage.
Node metrics — /metrics/nodes#
All gauges. Cluster-wide totals carry no labels; per-node gauges carry a
node=<name> label. <state> expands to one metric per node state:
idle, alloc, mixed, down, drain, draining, error,
unknown, suspended.
Metric |
Description |
|---|---|
|
Total number of nodes. |
|
Number of nodes in each state. |
|
Number of nodes the leader is not hearing heartbeats from (the |
|
|
|
Number of nodes the leader keeps up because their agent answers its
liveness probe though their heartbeats do not reach it. Such a node can
hold finished jobs as running, so alert if this stays above |
|
Total and allocated CPUs across all nodes. |
|
Total and allocated memory (bytes) across all nodes. |
|
Total and allocated GPUs across all nodes. |
|
Total and allocated CPUs on the labeled node. |
|
Total and allocated memory (bytes) on the labeled node. |
|
Total and allocated GPUs on the labeled node. |
|
CPU load reported by the node agent. |
|
Free memory (bytes) reported by the node agent. |
k0s metrics from spurctld — /metrics/k8s#
When Spur manages a k0s cluster (spur k8s up, see
Spur-Managed Kubernetes (k0s)), spurctld exports its own view of the
cluster lifecycle and per-node health at /metrics/k8s. Every series carries
distribution="k0s" and cluster="<cluster-name>".
Metric |
Description |
|---|---|
|
Current cluster phase as a one-hot set ( |
|
1 when the cluster phase is Ready (primary alerting signal). |
|
Configured control-plane replica count. |
|
Total nodes with a k0s role, and the per-role count. |
|
Provisioning attempts and attempts that gave up before Ready. |
|
Phase transitions labeled by source and destination. |
|
Reconcile-loop iteration wall time (histogram) and error count. |
|
Nodes whose running k0s disagrees with the controller’s record
( |
|
Strays the controller stopped, counted once the node stays stopped across
two checks. Only |
|
Stops of a confirmed stray the controller declined: |
|
Stopped strays whose k0s came back. |
|
Nodes newly recorded as failing to start k0s. |
|
Whether a node’s k0s systemd unit reports active. |
|
Per-node k0s unit restarts and install time (histogram). |
Each controller counts only what it saw while it was the leader, and its
counters reset when that process restarts. A leader change also starts drift
episodes over, so a drift that is still present is counted again by the new
leader rather than staying flat. Alert on increase(...[1h]) > 0 summed
across controllers rather than on rate, which falls back to zero while the
drift is still present. The counters carry no node label, to keep series
bounded; the spurctld log line names the node.
k0s component metrics (incoming)#
Note
The endpoints in this section are incoming / in progress and are separate
from the spurctld /metrics/k8s surface above. The bundled k0s
components each expose their own upstream Kubernetes /metrics endpoint;
Spur does not yet aggregate or proxy these — the ports and paths below are the
standard Kubernetes surfaces documented here for planning a scrape
configuration.
These endpoints are served by the k0s-managed components, not by spurctld.
Control-plane endpoints live on the control-plane node; the kubelet endpoints
live on every node. Most require TLS and authentication.
Component |
Port / Path |
Status |
Contents |
|---|---|---|---|
kube-apiserver |
|
Incoming |
API request latency/counts, etcd cache, admission timings. |
kubelet |
|
Incoming |
Node agent, pod lifecycle, and volume stats. |
kubelet (cAdvisor) |
|
Incoming |
Per-container CPU/memory/network/disk usage. |
kubelet (resource) |
|
Incoming |
Node/pod CPU and memory for the metrics pipeline. |
kube-scheduler |
|
Incoming |
Scheduling attempts, latency, and queue depth. |
kube-controller-manager |
|
Incoming |
Controller work queues and reconcile timings. |
etcd |
|
Incoming |
Datastore health, latency, and DB size. |
CoreDNS |
|
Incoming |
Cluster DNS query counts, latency, and cache stats. |
Controlling Jobs#
Cancel jobs with spur cancel (Slurm scancel), by job ID or by filter:
scancel 1024 1025
spur cancel -u alice -p gpu --state PENDING
scancel --reservation maint
scancel --signal SIGTERM 2048
Filter flags include --user/-u, --partition/-p, --state/-t
(only PD or R), --name/-n, --account/-A,
--reservation/-R (drain a reservation’s jobs, mirroring squeue -R), and
--signal/-s (KILL/9, TERM/15, INT/2, and others). You must
supply job IDs or at least one of --user, --name, --partition,
--account, or --reservation; --state alone only narrows a selection.
Without --user, a filter matches every user’s jobs you can see, as in Slurm,
and each one you do not own is refused — so --reservation (or --account,
--partition) can span other users’ jobs, with authorization applied to each
cancellation. A filter given an empty value (e.g. --reservation= from an unset
variable) is rejected rather than treated as “match everything”. In filter mode,
jobs already in a terminal state are silently skipped. scancel exits 1 and reports on stderr if any cancel or signal is
refused. A job that is already finished or unknown does not fail a plain cancel
(or -s KILL) and is not reported. With another signal it fails, and
--quiet/-Q hides the report but not the exit status.
Time limits and the kill wait. At its time limit a job moves from RUNNING
to COMPLETING (CG in squeue) straight away and is sent SIGTERM. It
stays COMPLETING for up to the time-limit kill wait
(timeout_kill_wait_secs, default 30 seconds), then Spur sends SIGKILL. The
wait ends early once the job’s own process (its batch script) has exited, and
the job ends TIMEOUT with reason TimeLimit however it exits: a clean
exit on SIGTERM does not make it COMPLETED. --requeue still requeues it
on timeout. While it waits, the job still counts as running for
--dependency=singleton and for an array’s %N throttle, so a successor
or the next array task does not start until the job has ended. The SIGTERM is final, so a batch script can use the wait to
checkpoint and exit. An idle interactive allocation reaped for inactivity
(Reaping abandoned interactive allocations (inactive_limit_secs)) normally ends TIMEOUT within a moment of the
SIGTERM, without spending the wait in COMPLETING: its work runs in srun
steps and its own process exits on SIGTERM (see the note below). scancel,
scontrol requeue and the final kill of a preempted job use the general kill
wait (kill_wait_secs, default 5 seconds), as does stopping the other nodes
of a job whose batch script has exited. An srun step with no supervisor process gets the full wait that applies to it
when a stop reaches it (see the note below): 5 seconds on scancel, 30 at a
time limit, where earlier releases gave it 2 seconds.
Note
A job does not yet wait for its srun steps: it ends and frees its nodes
once its batch script (or, for an allocation, its placeholder process) exits.
On nodes where Spur runs as root, the steps are then stopped with the job
without getting the kill wait. On nodes without cgroup enforcement, they can
outlive the job and keep running beside a newly placed job. This affects every
stop that uses a kill wait: a time limit, the inactivity limit, scancel,
and the final kill of a preempted job. The preemption grace is not affected.
Change job state with spur control (Slurm scontrol):
scontrol hold 1024 # prevent from starting
scontrol release 1024 # allow a held job to start
scontrol requeue 1024 # return a job to the queue
scontrol suspend 1024 # SIGSTOP, keep the allocation
scontrol resume 1024 # SIGCONT
Suspending a job that is in its preemption grace period is refused, because the preemption that ends the grace only applies to a running job.
A running job that is held, by scontrol requeuehold (JobHeldUser) or
on reaching max_batch_requeue (JobHoldMaxRequeue), gives up its nodes
at once. If the stop sent at that moment does not reach a node, the process
keeps running there while sinfo already shows the node’s resources as free,
and a new job sent to the same GPUs is refused. An administrator can find the
job still holding them in that node’s spurd log (gpu_owners=[...]). On the
node’s next heartbeat the controller tells it to stop the held job’s process,
so the resources become usable within about 40 seconds.
Update a job or node with scontrol update and Key=Value pairs. Job
updates need JobId= and accept Priority=, TimeLimit=, Partition=,
Account=, Comment=, QOS=, and ExcNodeList= (also spelled
ExcludedNodes=). Node updates need NodeName= and accept State= and
Reason=.
ExcNodeList= takes a hostlist expression and replaces the job’s excluded
nodes, the list sbatch -x sets; an empty or blank value clears it, and an
expression that does not parse (node[07-08) is refused. It applies only
while the job is pending and not yet being started on nodes the scheduler has
already reserved for it; a job that has left that window is refused with
FAILED_PRECONDITION. Once accepted, the exclusion is recorded in the
controller’s replicated log, so it survives a controller restart or a change of
leader, and the job is never started on an excluded node. Use it to keep queued
jobs off a node you have just found to be bad. In a cluster with several
controllers, finish upgrading all of them before using it: a controller from an
earlier release cannot read that log entry.
The whole request is rejected, with a non-zero exit and nothing changed, if any
key is unknown or belongs to the other kind of update (State= on a job,
TimeLimit= beside NodeName=), any value is malformed, or a job update
names no field to change. TimeLimit=UNLIMITED, which Slurm accepts, is
refused: give an explicit limit.
scontrol update JobId=1024 TimeLimit=2:00:00 Priority=100
scontrol update JobId=1024 ExcNodeList=node[07-08]
scontrol update JobId=1024 ExcNodeList= # clear the exclusions
scontrol update NodeName=node01 State=drain Reason="maintenance"