Monitoring & Controlling Jobs#

Once jobs are submitted, you inspect the queue, the nodes, and the accounting history with a handful of commands, and you steer running jobs with cancel, hold, and update operations. This page covers viewing the queue (spur queue), nodes and partitions (spur nodes), accounting history (spur history), live job stats (spur stat), detailed records (spur show), and the controls that change a job’s state.

Note

In spur queue (squeue) and spur nodes (sinfo), -h means --noheader, not help. spur history (sacct) uses -n for the same purpose.

View the Queue — squeue#

spur queue (Slurm squeue) lists jobs currently in the system. With no arguments it shows every active job, or only your own if your site sets private_data = ["jobs"] (see Job and usage visibility (private_data)).

spur queue
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
    1       gpu train-ll    alice  R       2:14      2 node[01-02]
    2   default    hello      bob PD       0:00      1 (Priority)

The default columns are JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON). --long/-l adds the full STATE and TIME_LIMIT.

Common flags:

Long

Short

Description

--user

-u

Show only this user’s jobs.

--me

Show only your own jobs (same as -u with your username). If both --me and -u are given, the last one wins.

--partition

-p

Filter by partition.

--states

-t

Filter by state codes or names (comma list). -t all shows every state.

--jobs

-j

Show only these job IDs (comma list). A step suffix (30.0) matches the base job. A non-numeric ID is an error.

--account

-A

Filter by account.

--name

-n

Filter by job name.

--qos

-q

Filter by QOS (comma-separated list).

--reservation

-R

Filter by reservation (comma-separated list).

--format

-o

Custom column format (see below).

--long

-l

Long form; adds STATE and TIME_LIMIT.

--noheader

-h

Omit the header line.

Note

--me looks up your username on the machine you run it from. If your login token names a different user than your local account, pass -u with the token’s name instead: --me would list the local account’s jobs, or nothing when your site sets private_data = ["jobs"].

When --states is omitted the default filter is PENDING, RUNNING, SUSPENDED, COMPLETING.

--format/-o uses %[.][-][width]<letter> fields. The resolved letters:

Letter

Column

Letter

Column

%i / %A

JOBID

%C

CPUS

%j / %n

NAME

%N

NODELIST

%u

USER

%R

NODELIST(REASON)

%a

ACCOUNT

%r

REASON

%P

PARTITION

%p

PRIORITY

%q

QOS

%Z

WORK_DIR

%t

ST (short code)

%o

COMMAND

%T

STATE (full)

%v

RESERVATION

%M

TIME

%S

START_TIME

%l

TIME_LIMIT

%V

SUBMIT_TIME

%D

NODES

%e

END_TIME

%Y

SCHEDNODES

%L

TIME_LEFT

%b

GRES

%Q

PRIORITY

%W

BORROWED

%p and %Q both render the integer priority. Slurm splits these (%p is a normalized float, %Q the integer); Spur exposes the integer under both.

spur queue -u alice -t R
squeue --me -t PD
squeue -p gpu -o "%.18i %.9P %.8T %.10M %R"
squeue --states=PD,R --noheader

Verbose field names — squeue -O/--Format#

-O/--Format selects columns by field name instead of %-letter. Fields are comma-separated, each type[:[.][size][suffix]]:

  • size is a minimum width (default 20); unlike -o’s ., it never truncates.

  • A leading . right-justifies the column. The default is left-justified, the opposite of -o.

  • suffix is arbitrary text appended after the field, e.g. a separator.

Field names are case-insensitive and render the same data as their %-letter equivalents. -O and -o/--format are mutually exclusive; passing both is an error, matching Slurm.

squeue -O "JobID:10,Partition,State,QOS"
squeue -O "JobID:.18,Priority,Reason"

Supported field names: Account, ArrayJobID, Command, Comment, EndTime, GRES, JobID, Name, NodeList, NumCPUs, NumNodes, Partition, Priority, PriorityLong, QOS, Reason, ReasonList, Reservation, SchedNodes, StartTime, State, StateCompact, SubmitTime, TimeLeft, TimeLimit, TimeUsed, UserName, WorkDir.

Slurm exposes many more -O fields that Spur has no backing data for (Licenses, Dependency, Nice, Reboot, most of the tres-* family, federation fields, job-step fields, and others). Requesting an unsupported name is an error listing the supported fields, rather than the blank column Slurm prints. This is a deliberate divergence: fail fast on a typo instead of silently emitting an empty column. Priority renders Spur’s integer priority (same as PriorityLong); Spur does not compute Slurm’s normalized 0.0-1.0 float.

Projected Start Times — squeue --start#

--start answers “when will my job run, and where” for the whole queue at once, rather than one job at a time through scontrol show job:

$ squeue --start
     JOBID PARTITION     NAME     USER ST          START_TIME  NODES           SCHEDNODES NODELIST(REASON)
      1024       gpu  train.sh    alice PD 2026-08-27T17:09:56      2          node[07-08] (Resources)
      1031       gpu   eval.sh      bob PD                 N/A      1               (null) (Priority)

It restricts the view to pending jobs and orders them by projected start, soonest first; jobs with no reserved slot report N/A and sort last. Each of those defaults is overridable by passing -t, -S, or -o explicitly.

%S and %e carry the same projection in any format string, so squeue -o "%.18i %.19S %.19e" works without --start.

Note

Projections come from the backfill pass on the controller’s leader, and assume every running job uses its full time limit. A job that finishes early moves every projection behind it earlier.

Job state codes. The ST column uses these short codes:

Code

State

Code

State

PD

PENDING

CA

CANCELLED

R

RUNNING

TO

TIMEOUT

CG

COMPLETING

NF

NODE_FAIL

CD

COMPLETED

PR

PREEMPTED

F

FAILED

S

SUSPENDED

DL

DEADLINE

OOM

OUT_OF_MEMORY

For a pending job the NODELIST(REASON) column shows why it is waiting. Common reasons include Priority (waiting its turn), Resources (waiting for nodes to free up), Dependency (waiting on another job), Reservation (waiting for a reservation window), Preempted (requeue-preempted and held until its eligibility window reopens), JobArrayTaskLimit (held back by the array’s own %N concurrency cap, see Job Arrays), PartitionTresLimit (held back by a partition’s per-node resource cap; the column shows a description with the numbers, see Partition Resource Limits), and various QOS or association limit reasons (QOSMax*, AssocMax*, AssocGrp*).

View Nodes & Partitions — sinfo#

spur nodes (Slurm sinfo) shows partition and node state. The default is a partition-oriented view; -N switches to one line per node.

spur nodes
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
default*     up  infinite     4  idle node[01-04]
gpu          up  1-00:00:00   2   mix node[05-06]

Flags:

Long

Short

Description

--partition

-p

Filter by partition.

--states

-t

Filter by node state (comma-separated; a node in any listed state matches). Accepts the base states, the display-only labels resv (or reserved), maint, and plnd (or planned), and no_respond (or noresp) for nodes the controller is not hearing from. all disables the filter.

--dead

-d

Only nodes the controller is not hearing from. Combines with -t and -r; a node must match all of them.

--responding

-r

Only nodes the controller is hearing from. Combines with -t and -d; a node must match all of them.

--nodes

-n

Filter by node.

--node-oriented

-N

One line per node instead of per partition.

--list-reasons

-R

List unavailable nodes (down/drain/draining/error) grouped by reason.

--format

-o

Custom column format.

--long

-l

Long form; adds CPUS and MEMORY.

--noheader

-h

Omit the header line.

Default columns (partition view): PARTITION AVAIL TIMELIMIT NODES STATE NODELIST; -l adds CPUS and MEMORY. Node view (-N): NODELIST NODES PARTITION STATE CPUS MEMORY GRES.

sinfo -N -o "%N %.6D %.11T %c %m %G"
sinfo -p gpu -l

%z prints Slurm’s S:C:T (sockets, cores per socket, threads per core), as in sinfo -N -o "%N %z". A node whose agent reported no CPU topology shows N/A. In the partition view, where a row covers several nodes, the row shows the shared value only when every node in it agrees; otherwise, or when any node is unknown, it shows N/A.

-R / --list-reasons lists only nodes that are unavailable (down, drain, drng, err), grouped by the admin reason set when the node was drained or downed, one row per reason with a compressed nodelist:

REASON               USER      TIMESTAMP           NODELIST
Memory errors        alice     2026-08-27T08:03:07 node[05,08]
Not responding       root      2026-08-27T09:15:44 node[01-03]

All four columns are populated from live state. USER (%u, or %U for user(uid)) is the user who set the reason, resolved from the recorded UID; automatic reasons (for example a heartbeat timeout) are attributed to root. TIMESTAMP (%H) is when the reason was set. A UID that cannot be resolved to a name, or a node whose reason predates provenance tracking, renders as Unknown. -R -l adds a STATE column (%t).

scontrol show node appends the same provenance to the reason line as Reason=<text> [<user>@<timestamp>] when a set-time is recorded, and the REST node object carries reason_uid and reason_time fields.

scontrol show node ends its CPUTot= line with Slurm’s Sockets=, CoresPerSocket= and ThreadsPerCore=, derived from the CPU topology the node’s agent reported: cores and threads are divided evenly as slurmd does, so uneven sockets or partly offline threads round down. A node whose agent reported no topology (a Kubernetes-backed node, or an agent older than the controller) leaves the three fields out. A Capabilities= line lists the optional features the node’s agent advertised (for example signal-job).

When a node reserves cores per GPU (restricted_cores_per_gpu), scontrol show node adds RestrictedCoresPerGPU=<R>(CPUs:<list>) naming how many cores each GPU reserves and, across all GPUs, which CPU ids they are; a GPU job’s CPUs beyond that count, or a CPU-only job, draw from the node’s general pool instead. Each GPUDevice= line then ends with ReservedCPUs=<list>, that GPU’s own picks, and AffinityCPUs=<list>, the CPUs the picks are drawn from (the GPU’s cores=, else its NUMA node’s CPUs). Either field is left out when unknown, and AffinityCPUs is shown even without reserved cores. The REST node object carries the same data as sockets, cores, threads (null when unknown), gpu_cpus (stable_id, reserved_cpus, affinity_cpus) and capabilities. See Configuration Reference (spur.conf) for the full node-field reference.

Node states are shown as short abbreviations: idle (free), alloc (fully allocated), mix (partly allocated), down, drain (offline, not accepting jobs), drng (draining), err (error), unk (unknown/unreachable), susp (suspended), and — all three only shown while the node is currently idle — resv or maint for a node held by an admin reservation, else plnd for a node currently held by the scheduler for a specific pending job’s upcoming start. The idle gate is checked live, but the job and start time shown for plnd reflect the most recent scheduling cycle (scheduler.interval_secs), not the current instant.

Every abbreviation above, including the overlay labels resv, maint, and plnd, can be passed to -t / --states. The long forms reserved and planned are accepted as aliases for resv and plnd. -t resv also matches maint nodes (a maintenance reservation is still a reservation), while -t maint matches only maintenance nodes. -t idle returns all idle nodes, including those shown as resv, maint, or plnd.

A trailing * on the state (idle*, mix*, down*) marks a node the controller is not currently hearing from: its heartbeats have gone stale, so it is given no new work until they resume, while its running jobs are untouched. Filter on it with -t no_respond, --dead (-d, only non-responding nodes), or --responding (-r, only responding nodes). Like the other -t states, no_respond widens the list: sinfo -t idle,no_respond shows idle nodes plus every non-responding node, whatever its state. -d and -r narrow it instead: sinfo -t idle -d shows only idle nodes that are not responding, and -d -r together match nothing. scontrol show node shows the same condition as a +NOT_RESPONDING suffix on State=, the REST node object carries a not_responding boolean, and spur health stars the node and counts it as unhealthy.

The flag is the current leader’s view of its heartbeats. It is blank for the grace window after a leadership change (see Node Liveness), so a silent node can still take new work in that window. Once it ends, a DOWN node the leader has not heard from shows as down*. A follower never sets it: the REST API answers node reads from the controller it reaches, so not_responding is false for every node when that is a follower, and the CLI shows no * when a follower serves a read itself because it cannot forward it to the leader. Query the leader for this flag.

Accounting History — sacct#

spur history (Slurm sacct) reports finished and running jobs from the accounting service. With private_data = ["jobs"] it lists only your own jobs, and -u for someone else prints nothing (see Job and usage visibility (private_data)).

spur history -u alice -S now-7days

Flags:

Long

Short

Description

--user

-u

Filter by user.

--account

-A

Filter by account.

--starttime

-S

Earliest submit/start time to include.

--endtime

-E

Latest time to include.

--state

-s

Filter by state (comma list). Accepted values (long or short code): COMPLETED/CD, FAILED/F, CANCELLED/CA, TIMEOUT/TO, NODE_FAIL/NF, PREEMPTED/PR, DEADLINE/DL, RUNNING/R, PENDING/PD.

--jobs

-j

Filter by job ID (comma list; step suffixes like 30.0 match the base job). A non-numeric ID is an error. An id with no record is left out. Without --limit, every requested id that has a record is returned, up to 10000.

--format

-o

Comma-separated field names (see below).

--long

-l

Long form; adds DerivedExitCode, Start, End, TimeLimit, the three preemption fields, and Reason.

--brief

-b

Brief form: JobID, State, ExitCode.

--noheader

-n

Omit the header line.

--parsable2

-P

Print fields | delimited, without a trailing |: one header line (none with -n), no padding, no truncation.

--parsable

-p

As -P, but every line ends with a trailing |. When both are given, the last one wins.

--limit

Maximum rows to print. Must be at least 1. Default 100 when -j is omitted. sacct prints at most 10000 rows. When a page hides further matches, a warning is printed to stderr and the command still exits 0.

Unlike squeue, --format here takes comma-separated field names, not % letters. Available fields include JobID, JobName, User, Account, Partition, State, Elapsed, NNodes, ExitCode, DerivedExitCode, Start, End, Submit, TimeLimit, NodeList, NCPUS, ReqMem, QOS, PreemptedBy, PreemptMode, PreemptQOS, Borrowed, Reason, ReasonDetail, EndedBy, and ReqGPUS. Set a per-field width with Field%N, e.g. JobName%20.

NodeList is the compressed hostlist of the nodes the job was allocated, in the same form squeue and scontrol show job print (node[01-04]). It is recorded when the job starts, so it stays available after the job leaves the controller; a requeued job shows the nodes of its latest run. Jobs that started before this release recorded no node list and show it empty.

NCPUS is the job’s total allocated CPUs, recorded when the job starts: for a --cpus-per-gpu job this already counts the CPUs its GPUs brought in, so it reads higher than cpus-per-task x tasks alone would suggest. Jobs that started before this release recorded no CPU count and show cpus-per-task x tasks.

PreemptedBy is the job ID of the higher-priority job that caused the preemption (N/A when the job was not preempted). PreemptMode is one of Requeue, Cancel, or Suspend. PreemptQOS is the QOS of that preempting job — the QOS whose preempt allow-list authorized the preemption. It is set for any job preempted by this release; jobs preempted by an earlier release, which recorded no authorizing QOS, show N/A even though they were preempted. All three appear in the long format (-l).

Borrowed is yes when the run took borrowed capacity under Idle-Fill Scheduling, and no otherwise. It is the same question squeue’s %W answers for a running job:

$ sacct --format=JobID,User,State,Borrowed
JobID  User     State      Borrowed
19     ifbob    PREEMPTED  yes
21     ifalice  COMPLETED  no

Why a job ended#

Reason is a short code saying why the job’s run ended, and ReasonDetail adds what Spur knew beyond the code. Use Reason to group failures, for example sacct -n -s FAILED,NODE_FAIL --format=Reason piped through sort | uniq -c.

$ sacct --format=JobID,State,ExitCode,Reason,ReasonDetail
JobID  State      ExitCode  Reason             ReasonDetail
41     FAILED     127:0     NonZeroExitCode    exit 127: command not found (shell convention)
42     FAILED     0:11      RaisedSignal       signal 11 (Segmentation_fault)
43     FAILED     1:0       CompletingTimeout  no completion reported by node3
44     TIMEOUT    0:15      TimeLimit

Reason

State

When

None

COMPLETED

The job ended normally.

NonZeroExitCode

FAILED

The job exited with a non-zero code. For 126 and 127 the detail notes the shell convention (“not executable”, “command not found”); that is a hint, since a script may also exit 127 itself.

RaisedSignal

FAILED

A signal ended the job; the detail names it.

CompletingTimeout

FAILED

The controller stopped waiting for the job’s nodes to report; the detail lists the nodes that never did.

TimeLimit

TIMEOUT

The job reached its time limit, however the process then died.

DeadLine

DEADLINE

The job’s --deadline passed before it could run.

OutOfMemory

OUT_OF_MEMORY

The kernel killed the job for exceeding its memory limit.

Preempted

PREEMPTED

A higher-priority job preempted it; the detail names that job and its QOS.

UserCancel

CANCELLED

The job’s owner cancelled it.

AdminCancel

CANCELLED

An operator or administrator cancelled someone else’s job.

SystemCancel

CANCELLED

Spur cancelled it: a dependency that can never be met, an ended reservation, a failed prolog on an interactive or srun job, or an in-cluster daemon (including an FFI slurm_kill_job call made without credentials).

Cancelled

CANCELLED

Cancelled, but who did it was not recorded: the job was cancelled before this release, or by an older controller during a rolling upgrade.

Requeued

REQUEUED

The job’s owner or an operator requeued it while it was running.

NodeDown

NODE_FAIL

A node the job ran on went down. The detail names the node and its down reason, as sinfo -R shows it (for example Not responding after lost heartbeats), plus the uid that set it when an operator did.

NodeRemoved

NODE_FAIL

A node the job ran on was removed from the cluster; the detail names the node and the removal reason.

JobLaunchFailure

NODE_FAIL

The job could not be started on every node; the detail says why.

EndedBy is the user who cancelled the job, for UserCancel and AdminCancel; a SystemCancel has no user and, like every other reason, shows N/A. It is shown to operators and administrators, the same audience as sacctmgr show txn; for anyone else it reads N/A. When [auth] runs in disabled or permissive mode an unauthenticated caller counts as that audience, as it does for the audit log.

Reason reads N/A for a job that is still running, for rows written before this field existed, and for rows written through the external RecordJobEnd accounting RPC. Accounting keeps one row per job, so a requeued job’s next run replaces the reason with its own. During a rolling upgrade, a job that restarts and then ends while an older controller is leader keeps the reason of its previous run, and a run the older controller force-finished reads NonZeroExitCode rather than CompletingTimeout.

The default columns are JobID JobName User Account Partition State Elapsed NNodes ExitCode Reason. Scripts that read columns by position should select them with --format.

A job cancelled or expired before it starts keeps its name, user, account, partition, and requested resources in accounting. Its Start is Unknown and it is charged no usage.

Time arguments accept an absolute date (YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS) or a relative offset (now-7days, now-6hours).

sacct -S 2026-07-01 -E 2026-07-25 -s FAILED --limit 500
sacct --format=JobID,JobName,State,Elapsed,ExitCode
sacct -s PREEMPTED --format=JobID,JobName,State,ExitCode,PreemptedBy,PreemptMode,PreemptQOS
sacct -s PR,CA     # short codes also accepted

State names and short codes are case-insensitive. An unknown state in a list makes the command exit with an error before querying accounting. Omit -s or use -s all to query all states.

Running-Job Stats — sstat#

spur stat (Slurm sstat) reports live resource usage for running jobs. --jobs/-j is required (comma list). A step suffix (30.0) matches the base job. A non-numeric ID is an error.

spur stat -j 12345
sstat -j 12345,12346 -o JobID,NTasks,GPUAlloc,Elapsed -p

Default fields: JobID NTasks NCPUS MemAlloc GPUAlloc Elapsed Nodelist. --format/-o takes field names; --parsable/-p produces |-delimited output. NCPUS is the running job’s ReqTRES cpu= count, so a --cpus-per-gpu job’s count already includes the CPUs its GPUs brought in.

Note

The per-process Ave* and Max* fields (AveCPU, AveRSS, MaxRSS, and similar) always report N/A — Spur does not collect per-process metrics.

Detailed Records — scontrol show#

spur show (Slurm scontrol show) prints the full Key=Value record for an entity: job, node, partition, reservation, step, or assoc_mgr.

Note

The diagnostic fields below are printed on every job, with (null) for an unset string, so a parser must key on the field name rather than on a line being absent. Some fields still appear only when set (Comment, Reservation, ArrayJobId, SchedNodeList, StdIn, the GPU line, the preemption line, and the SuspendedBy/LastRequeue line), and MinMemoryNode is spelled MinMemoryCPU for a --mem-per-cpu job.

For a live job someone acted on, scontrol show job adds a line naming who. SuspendedBy is the user who suspended the job, only while it stays suspended; a job paused by preemption shows the preemption line instead. LastRequeue says why the job last went back to the queue: Failure (a launch failure, time limit or node failure with --requeue), Preempt, or Admin (an scontrol requeue by the owner or an operator), with whoever requeued it in parentheses for Admin:

SuspendedBy=alice LastRequeue=Admin(ops)

The user names are shown to the same audience as EndedBy in sacct (operators, administrators, and unauthenticated callers when [auth] is disabled or permissive); for anyone else they are blank and LastRequeue shows the kind alone.

scontrol show job 1024
spur show node node01
scontrol show partition gpu

Expanding node lists — scontrol show hostnames / hostlist#

scontrol show hostnames expands a hostlist expression to one host per line; scontrol show hostlist (and hostlistsorted) does the reverse, compressing a list back into the bracketed form. Both are pure local transforms: they never contact the controller, so they work inside a batch script on any compute node.

With no argument, hostnames reads SLURM_JOB_NODELIST (Spur also honors SPUR_JOB_NODELIST), which makes it the standard way to derive a rendezvous address or an MPI hostfile from an allocation:

scontrol show hostnames 'gpu[01-03]'      # -> gpu01 / gpu02 / gpu03, one per line
scontrol show hostlist gpu03,gpu01,gpu02  # -> gpu[01-03]

# Inside a job: first allocated node as the rendezvous host, all nodes to a hostfile.
export MASTER_ADDR="$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n1)"
scontrol show hostnames > "$HOME/hostfile.$SLURM_JOB_ID"

hostlist and hostlistsorted produce identical output in Spur: both remove duplicates and sort before compressing.

Diagnosing a job that will not start#

A job pending on Reason=Resources looks the same whether the cluster is full or the job is pinned to a busy subset of it. scontrol show job reports the request as submitted, which distinguishes the two:

A job that no node could host even when completely idle reports Reason=ReqNodeNotAvail, Requested node configuration is not available instead. Waiting will not help it: either the request exceeds every node’s capacity, or it names a resource no node declares — a typo such as --gres 1 asks for one unit of a resource literally named 1, which is easy to miss because it never appears in ReqTRES. Compare ReqTRES and MinCPUsNode / MinMemoryNode below against spur show node, and check --gres spelling. The reason is recomputed every scheduling cycle, so the job starts on its own once matching nodes join.

Field

Meaning

ReqNodeList

Nodes requested with -w/--nodelist. (null) when unrestricted.

ExcNodeList

Nodes excluded with -x/--exclude.

NodeList

Nodes actually allocated. Empty ((null)) while pending, except briefly while the controller reserves nodes and confirms the launch (bounded by dispatch_timeout_secs). Not the same as ReqNodeList.

Features

Node feature constraint from -C/--constraint.

ReqTRES

Requested resource totals, e.g. cpu=16,mem=32000M,node=2,gres/gpu=8.

CpusPerTres

gpu:<n> when the job set --cpus-per-gpu; omitted otherwise.

MemPerTres

gpu:<MB> when the job set --mem-per-gpu; omitted otherwise.

MinCPUsNode / MinMemoryNode

Per-node minima implied by the request. Memory is in MB with an M suffix; a bare 0 means none was requested. A --mem-per-cpu job reports MinMemoryCPU instead, as Slurm does.

Dependency, Requeue, Restarts, BatchFlag, Exclusive

Submission flags and the requeue count for this job.

RunTime / TimeLimit / TimeMin / Deadline

Elapsed and requested wall time. TimeLimit reads UNLIMITED when none was set; TimeMin and Deadline read N/A when unset.

Command

First executable line of the batch script.

SubmitLine

The submit command as invoked, so any submission-flag question is answerable from one command.

EligibleTime

Earliest the job may start (--begin, otherwise submit time).

AccrueTime

When the job began accruing age priority.

StartTime / SchedNodeList

While pending, the slot the scheduler is holding: when it projects the job will start, and on which nodes. With no slot reserved, StartTime reads N/A and SchedNodeList is omitted. StartTime becomes the real start once the job runs.

EndTime

The recorded end once the job finishes, otherwise its start plus its time limit. N/A for an unlimited job, or a pending one with no projected start.

LastSchedEval

The last scheduling cycle that considered this job, so it advances even on a cycle that places nothing. N/A before the first cycle, frozen once the job starts, and reset by a controller restart or failover. On a deep queue it also covers jobs classified but left untried past scheduler.max_jobs_per_cycle.

Note

StartTime, SchedNodeList, and LastSchedEval live only on the leader and are never replicated, so a read served by a follower reports them unset even in a healthy cluster.

So a job pinned to a busy node reads (excerpt — the full record has more fields between these lines):

JobState=PENDING Reason=Resources Dependency=(null)
...
StartTime=2026-08-27T17:09:56 EndTime=2026-08-27T17:14:56 Deadline=N/A
...
ReqNodeList=node07 ExcNodeList=(null)
NodeList=(null) SchedNodeList=node07
...
SubmitLine=sbatch -w node07 --exclusive -t 5 job.sh

The controller logs the matching scheduler-side view at info level, one line per job when its placement outcome changes (not every cycle). A cycle with no schedulable nodes at all is skipped before this runs, so it logs nothing. Jobs beyond the per-cycle scheduling depth limit are not reported:

backfill did not start job job_id=1024 reason=no_suitable_nodes needed_nodes=1 candidate_nodes=0

reason is one of het_group_incomplete, no_suitable_nodes, requested_nodes_unavailable, too_few_candidates, no_capacity_at_start, future_slot_reserved, or refused_at_start. future_slot_reserved means the job can be placed and the scheduler is holding a future slot for it, reported with planned_start. refused_at_start means the job could be placed, now or in a future slot, but on the nodes it was placed on it would be over a QOS, partition QOS or account limit, counting the jobs started earlier in the same cycle and the room held for preemptions in flight. It holds no slot and pends with that limit’s reason, for example QOSGrpNodeLimit.

Note

Spur accrues age priority from submit time for every pending job, including held and dependency-blocked ones. Slurm suspends accrual for those, so AccrueTime can read earlier here than on an equivalent Slurm job.

Preemption provenance#

When a job has been preempted, scontrol show job includes three additional fields:

  • PreemptedBy=<job_id> — the ID of the higher-priority job that triggered the preemption. Only shown once the job has been preempted; during a preemption grace period nothing has been preempted yet, so it is not printed.

  • PreemptMode=Requeue|Cancel|Suspend — how the preemption was carried out.

  • PreemptQOS=<name> — the QOS of the preempting job, whose preempt allow-list authorized the preemption. Set for any job preempted by this release; jobs preempted by an earlier release show N/A even though they were preempted.

While a job is inside its preemption grace period, scontrol show job instead prints a line holding only PreemptTime=<timestamp>, the moment the job was signaled to wind down, and EndTime shows when the grace ends. PreemptTime is omitted for a job that is not in a grace. The REST job object (GET /api/v1/job/<id>, GET /api/v1/jobs, and the same under /slurm/v0.0.42) reports the same as preempt_time (Unix seconds, null outside a grace) and end_time. While the job is still running, end_time is the projected end of its grace, not the time it finished: use job_state to tell whether a job has ended. The same job object also reports cpus_per_gpu and memory_per_gpu, both null when the job did not request them.

How a job learns it is being preempted#

If the QOS your job runs under, or the partition it runs in, has a GraceTime (ask your administrator; see preemption grace time; the QOS value wins if it is set, otherwise the highest value among the partitions the job runs in applies), a job selected for preemption in cancel or requeue mode is not killed at once. It gets SIGCONT then SIGTERM, one time, as a warning, stays RUNNING and keeps its nodes, and is preempted at the deadline shown as EndTime (PreemptTime shows when the warning was sent). The grace is shortened if the job’s own time limit arrives first; the deadline is fixed when the warning is sent, so lowering the time limit later does not shorten it. The warning is sent once: if it is lost (for example the node’s agent is unreachable at that moment) the job is still preempted at the deadline, only without having been warned. Without a GraceTime there is no warning period and the job is preempted immediately.

  • Launch your work with srun. The warning goes to the job’s srun steps, not to the batch script, so a training process started directly from the batch script gets no warning and sees only the final kill. Run it as srun python train.py and handle SIGTERM there to checkpoint.

  • Expect a second SIGTERM. At the deadline Spur sends SIGTERM to every process in the job, waits up to the kill wait (5 seconds by default), then sends SIGKILL. A handler that is already shutting down should ignore a repeat. The wait does not yet reach srun steps (kill wait note).

  • Exiting early is fine. A job that exits during its grace is recorded as PREEMPTED (cancel mode) or goes back to PENDING (requeue mode), and the preempting job starts as soon as the nodes are free.

  • One node finishing early does not affect the others. The job stays R until every node has finished or the deadline. A node whose work has finished is freed for the preempting job, and the other nodes keep running undisturbed, so a rank that exits first does not cut short the ranks that are still writing their checkpoint. If your batch script launches ranks with srun, ranks exiting do not end the job: it ends when the batch script exits (the remaining nodes are then stopped), or at the deadline.

  • You can still cancel. scancel during the grace ends the job as your cancel. scontrol suspend is refused.

  • The grace is not withdrawn. The job is preempted at the deadline even if the job that triggered it is cancelled meanwhile.

Jobs on nodes managed by the Kubernetes operator get no grace and no warning.

Borrowed runs and reclaim#

When Idle-Fill Scheduling is enabled, a job that exceeded its QOS group node quota may still run, on capacity no job with a quota claim wanted. Such a run is borrowed, and squeue’s %W field reports it:

$ squeue -o "%i %j %u %t %N %W"
JOBID NAME         USER     ST NODELIST BORROWED
18    alice-legit  ifalice  R  node-3   no
19    bob-borrow   ifbob    R  node-4   yes

A borrowed job can be reclaimed when a job that does hold a quota claim needs its nodes. Reclaim requeues: the job returns to the queue with its spec intact and starts again once capacity allows, so the partial run is lost but the work is not. It appears as an ordinary preemption, with PreemptMode=Requeue and PreemptedBy naming the job that reclaimed it, and sacct records that the run was borrowed.

Note that --no-requeue does not prevent reclaim, and that an over-quota job which has not been lent capacity keeps reporting QOSGrpNodeLimit rather than a placement reason — being over quota is the true reason it is waiting.

Quick reference — one command for any preempted job:

scontrol show job <id>

This works regardless of preemption mode (requeue, cancel, or suspend) as long as the job is still known to the controller (running, pending, suspended, or recently terminal). For cancel-preempted jobs that have been evicted from memory, fall back to:

sacct -j <id> --format=JobID,State,PreemptedBy,PreemptMode,PreemptQOS

The accounting database keeps the record permanently. In practice scontrol show job is the right first instinct; fall back to sacct when it reports Job <id> not found (a non-zero exit status), which means the controller no longer holds the job. A malformed id is different: scontrol show job abc fails with Invalid job id specified: abc, so a script can tell a typo from a job that has left the controller.

Note

Suspend-mode preemption and accounting. When PreemptMode=Suspend, the job receives SIGSTOP and stays running — no accounting end-record is written. The provenance fields (PreemptedBy, PreemptMode, PreemptQOS) are visible in scontrol show job while the job is suspended, but are cleared when the job resumes so that a subsequent normal completion is not miscounted as a preemption. As a result, sacct has no record of the preemption for suspend-mode jobs: the accounting row for that run will show the final completion state only. Requeue mode is unaffected — it writes an accounting end-record (PREEMPTED) at the time of preemption. Cancel mode writes the same record, but only once the run has actually ended: the victim holds its allocation in COMPLETING until every node reports the release, or until CompleteWait elapses.

Limits Against Usage — scontrol show assoc_mgr#

scontrol show assoc_mgr answers the question squeue cannot: how much of a QOS or an association each user is holding right now, next to the caps that govern them. Without it an operator has to infer per-user totals by counting squeue rows.

scontrol show assoc_mgr
scontrol show assoc_mgr alice
scontrol show assoc_mgr users=alice

Output is two sections of Key=Value blocks — QOS records, then association records. Each block opens with the scope itself and lists a line per user under it:

QOS Records
QOS=highprio MaxWall=01:00:00 MaxTRESPJ=cpu=32 MaxJobsPU=2 MaxSubmitJobsPU=N MaxTRESPU=node=4 MaxSubmitJobsPA=20
   GrpJobs=N(9) GrpSubmitJobs=N(11) GrpTRES=cpu=N(36),node=16(9) GrpWall=7-00:00:00(2-12:00:00)
   User=alice MaxJobsPU=2(6) MaxSubmitJobsPU=N(7) MaxTRESPU=cpu=N(24),node=4(6) OverLimit=MaxJobsPU,MaxTRESPU
   User=bob MaxJobsPU=2(1) MaxSubmitJobsPU=N(1) MaxTRESPU=cpu=N(4),node=4(1)

Association Records
Account=tenant-a MaxWall=N
   GrpJobs=N(1) GrpSubmitJobs=N(1) GrpTRES=node=N(1)
   User=alice MaxJobs=4(1) MaxSubmitJobs=N(1) MaxTRES=cpu=N(4),node=N(1) MaxTRESPJ=node=4

Reading it:

  • On the Grp* line and the User= lines every cap except MaxTRESPJ is printed as ``Limit(Consumed)``, as in Slurm: node=4(6) is a cap of four nodes with six in use, and N marks no cap. GrpWall follows the same shape as a wall-clock time, budget(spent); an N in its consumed slot means the controller has not read spend yet (its usage cache holds no snapshot), which is not the same as none spent. The per-job and per-account caps (MaxWall, MaxTRESPJ, MaxSubmitJobsPA), and a QOS’s per-user caps on its scope line, print bare, with no consumption beside them, because they bound each job, account, or user rather than a total the scope accrues.

  • For the count caps a literal 0 is a real cap that blocks every job it governs. A TRES dimension is the exception: 0 there is treated as unset — it renders as N and is not enforced, so GrpTRES=node=0 does not block a dimension the way MaxJobs=0 blocks jobs.

  • Consumption is live, measured exactly as the scheduler measures it when it admits a job. Node counts are distinct occupied nodes, so two jobs sharing a node hold one node, not two. A TRES dimension appears when either the cap or the usage has something to say about it.

  • The scope line carries what belongs to the scope: its wall-clock ceiling MaxWall, and — for a QOS, which caps every user identically — its per-job cap MaxTRESPJ, the per-user caps it enforces (MaxJobsPU, MaxSubmitJobsPU, MaxTRESPU), and the per-account submit cap MaxSubmitJobsPA. An association caps each (user, account) pair on its own, so its per-user caps and its MaxTRESPJ ride on each User= line instead. MaxTRESPJ bounds one job and MaxTRESPU bounds a user’s total within the QOS. An association carries no per-user TRES cap, so MaxTRES on a User= line always reads N; a cap on one user across accounts is a partition QOS. MaxSubmitJobsPA and GrpWall are QOS-only and never appear in association records.

  • Grp* figures are the whole scope’s, summed across every user, and stay on the scope line. GrpWall is the QOS’s wall-clock budget beside the spend measured over grp_wall_window_days (see Accounting, Accounts, Users, and QOS). Filtering to one user narrows the User= lines but never the group figures, since a group cap cannot be judged from one user’s share.

  • OverLimit lists caps that are already exceeded, on the scope line for group caps — including GrpWall once spend reaches the budget, the state behind a QOSGrpWallLimit hold — and on a user’s line for that user’s caps. It is absent when everything is within its cap. Usage over a cap is not a contradiction: caps are applied when a job is admitted and running jobs are never re-checked, so tightening a cap under running work leaves exactly this state.

Records come from the accounting definitions as well as from the queue, so a QOS nobody is using still appears with its caps and no User= lines — an over-tight cap on an idle QOS is exactly the kind of mistake that hides otherwise. A QOS deleted after its jobs queued also stays reported for as long as those jobs hold resources.

A LimitsReadable=NO line above the sections means accounting is enabled but a cache has not loaded yet, so some caps below may be missing; the usage figures are still current. A cluster with accounting disabled has no caps to read and so never prints this line.

By default every caller sees every scope and may pass users=<name> to inspect one user. When an administrator sets private_data = ["usage"] under [auth] (see Job and usage visibility (private_data)), an unprivileged caller sees only the QOS and accounts they take part in, and only their own User= lines; a users= selector is replaced with their own name. sshare omits other users’ rows and leaves account totals and shares of the total blank. sreport utilization fills in only your own usage and fairshare and leaves the other cells blank. sreport preemptions shows only your own rows and the accounts you coordinate when either "jobs" or "usage" is private. sprio and sreport job sizes follow "jobs" instead. Grp* totals on the scope line still include every user’s jobs. Operators and Administrators are exempt.

Cluster Metrics — /metrics#

Beyond the per-job CLI commands above, spurctld exports cluster-wide metrics over HTTP in Prometheus/OpenMetrics text format. This is the surface a monitoring stack (Prometheus, Grafana, an OpenMetrics scraper) consumes to chart queue depth, node utilization, and resource allocation over time.

The server is controlled by the [metrics] section of spur.conf (see Configuration Reference (spur.conf)). It listens on port 6822 and by default binds to loopback only; set bind = "all" to expose it to a scraper on another host. Only the Raft leader serves data — followers return 503 Service Unavailable — so point your scraper at all controllers and let it follow the leader.

curl http://127.0.0.1:6822/metrics/jobs
curl http://127.0.0.1:6822/metrics/nodes

Endpoints#

Path

Status

Contents

/metrics

Live

Alias for /metrics/jobs.

/metrics/jobs

Live

Job counts by state and aggregate allocated resources.

/metrics/nodes

Live

Node counts by state, cluster resource totals, and per-node gauges.

/metrics/partitions

Planned

Route exists but currently returns an empty body.

/metrics/scheduler

Live

Scheduler cycle statistics and preemption outcomes.

/metrics/k8s

Live

Spur-managed k0s cluster lifecycle and per-node health (spur_k8s_*).

/metrics/jobs-users-accts

Live

Per-user/per-account job gauges and karma counters. Requires metrics.high_cardinality = true; returns 404 otherwise. See Karma Scoring.

Job metrics — /metrics/jobs#

All gauges. <state> expands to one metric per job state: pending, running, completing, completed, failed, cancelled, timeout, node_fail, preempted, suspended, deadline, out_of_memory.

Metric

Description

spur_jobs

Total number of jobs.

spur_jobs_<state>

Number of jobs in each state.

spur_jobs_cpus_alloc

CPUs allocated to running/completing jobs.

spur_jobs_memory_alloc_bytes

Memory (bytes) allocated to running/completing jobs.

spur_jobs_gpus_alloc

GPUs allocated to running/completing jobs.

Preemption outcomes — /metrics/scheduler#

When a pending job (the requester) preempts running jobs (the victims), Spur records whether the launch of the requester used the capacity each victim released. Counts describe victim runs, not nodes: preempting the same job on two runs counts twice. Suspension and idle-fill reclaim without an owner-directed handoff are excluded. A grace warning alone does not count; the first released slice opens one observation for that victim run.

Metric

Type

Meaning

spur_scheduler_preemptions_used_total

Counter

Victim runs with matching released capacity in a confirmed requesting launch.

spur_scheduler_preemptions_unused_total{reason,initiator}

Counter

Victim runs closed without confirmed use by the requesting launch.

spur_scheduler_preemptions_open

Gauge

Victim runs that released capacity but whose requesting launch outcome is unresolved.

spur_scheduler_preemption_exports_pending

Gauge

Finished jobs past retention kept in the controller until their counts reach accounting.

spur_scheduler_preemption_export_oldest_seconds

Gauge

Age of the oldest of those jobs, from its end time.

For example, if a requester preempts A on n1 and B on n2, then launches only using B’s capacity, used increases by one and unused by one. Partial reports and replay do not count a victim run again. A successful launch followed by a later application failure remains used. A failed release is a launch failure, even if the controller already recorded the job as Running. Native interactive allocations count as launched once allocation registration and activation succeed.

reason has four values:

  • started_elsewhere: the launch succeeded without matching released capacity.

  • launch_failed: a reserved requesting launch was rejected, held, backed off, lost its nodes, or was not released.

  • cancelled: cancellation ended the request before use.

  • ended_other: another end or a superseding attempt closed the request, including an outcome that could not be confirmed after controller failure.

initiator is user, admin, system, or unknown. Cancellation uses the authenticated cancellation provenance. Scheduler placement and launch failures use system. Missing provenance stays unknown. The label records the observed operation, not an inferred root cause: a launch rejection caused by user-supplied input still has a system launch-failure verdict. The ended_other / unknown bucket is uncertainty, not proof of resource loss. No username, job ID, or node name is a Prometheus label.

The counts are also available from:

  • sdiag: cluster totals, unused counts by cause, and, when finished jobs past retention are waiting for export, how many and the oldest one’s age.

  • scontrol show job: PreemptionsUsed, PreemptionsUnused, and PreemptionsOpen on the requesting job, plus UnusedPreemptionCauses as comma-separated reason/initiator:count pairs. Printed only when a count is nonzero.

  • REST job responses: the same counts and causes in preemption_usage.

  • sreport preemptions: lifetime totals per requesting user and account (see below).

A device-backed victim counts as used only when the requester received the same node-local devices; using CPU on the same GPU node is not enough. CPU and memory have no identity, so they are matched against a capacity budget. Both are approximations: they do not prove a preemption was necessary or measure lost GPU-hours. One victim using any matching capacity counts once.

If the requesting launch fails before it reserves nodes, the observation stays open so a retry can still use the capacity. If the requester gives up a reservation before it starts (a backoff, a hold, or a node going down), the observation closes as launch_failed and a later retry does not count as use. A launch result that was not recorded before a controller restart closes as ended_other / unknown once the new leader’s dispatch grace expires (twice the dispatch timeout; never, if that timeout is disabled). This records uncertainty, not a launch failure.

Cluster totals and per-job counts survive controller restart and are not cleared by sdiag --reset.

Lifetime user/account reporting#

sreport preemptions needs PostgreSQL accounting:

sreport preemptions
sreport preemptions --user alice --parsable
sreport preemptions --account research --limit 25
sreport preemptions --account research --by-cause
sreport preemptions --account research --limit 25 --page-token '<token>'

Rows show User, Account, Used, Unused, and Open. Each row is a requesting user/account pair; a user’s rows across accounts add to that user’s total. A complete account report requires every page for that account. The CLI prints the next-page token on standard error when another page exists. The default page contains at most 100 rows; --limit accepts 1 through 1000. --start and --end are rejected because these are lifetime totals rather than event-time reports. Pages are read in user/account order, not from a frozen snapshot: counts can change between pages, and a pair added before the current position during paging is not shown.

Add --by-cause to show only nonzero unused counts, with one row per requesting user/account, reason, and initiator. These use the same values as the scheduler metrics and count victim runs, not nodes. For example:

$ sreport preemptions --by-cause
User                 Account              Reason               Initiator          Unused
alice                research             cancelled            user                    2
alice                research             launch_failed        system                  1

With --parsable --by-cause, the columns are User|Account|Reason|Initiator|Unused. For example:

$ sreport preemptions --parsable --noheader --by-cause --user alice
alice|research|cancelled|user|2
alice|research|launch_failed|system|1

Cause counts add up to the pair’s Unused total in the default report. Pairs with no unused counts have no rows in the cause report. Pagination still counts user/account pairs and keeps their causes together, so a page can have no cause rows and still print a next-page token; continue until there is no token.

The report applies both jobs and usage privacy settings, including coordinator visibility (see Job and usage visibility (private_data)). A scoped report is not a full account or cluster total. Cluster totals in sdiag and the metrics are aggregates with no per-user detail and are not scoped.

Accounting reports lag the live counts, and Open there is the last exported value. Preemption outcome export describes how counts reach PostgreSQL and what happens during an accounting outage.

Node metrics — /metrics/nodes#

All gauges. Cluster-wide totals carry no labels; per-node gauges carry a node=<name> label. <state> expands to one metric per node state: idle, alloc, mixed, down, drain, draining, error, unknown, suspended.

Metric

Description

spur_nodes

Total number of nodes.

spur_nodes_<state>

Number of nodes in each state.

spur_nodes_not_responding

Number of nodes the leader is not hearing heartbeats from (the * in sinfo), in any state, so it overlaps the per-state counts. See Node Liveness.

spur_nodes_liveness_breaker_tripped

1 while the leader’s last health pass marked no node DOWN because too many nodes went silent at once, otherwise 0. See Node Liveness.

spur_nodes_silent_answering

Number of nodes the leader keeps up because their agent answers its liveness probe though their heartbeats do not reach it. Such a node can hold finished jobs as running, so alert if this stays above 0. Set only by the leader. See Node Liveness.

spur_nodes_cpus / spur_nodes_cpus_alloc

Total and allocated CPUs across all nodes.

spur_nodes_memory_bytes / spur_nodes_memory_alloc_bytes

Total and allocated memory (bytes) across all nodes.

spur_nodes_gpus / spur_nodes_gpus_alloc

Total and allocated GPUs across all nodes.

spur_node_cpus / spur_node_cpus_alloc

Total and allocated CPUs on the labeled node.

spur_node_memory_bytes / spur_node_memory_alloc_bytes

Total and allocated memory (bytes) on the labeled node.

spur_node_gpus / spur_node_gpus_alloc

Total and allocated GPUs on the labeled node.

spur_node_cpu_load

CPU load reported by the node agent.

spur_node_free_memory_bytes

Free memory (bytes) reported by the node agent.

k0s metrics from spurctld — /metrics/k8s#

When Spur manages a k0s cluster (spur k8s up, see Spur-Managed Kubernetes (k0s)), spurctld exports its own view of the cluster lifecycle and per-node health at /metrics/k8s. Every series carries distribution="k0s" and cluster="<cluster-name>".

Metric

Description

spur_k8s_cluster_phase{phase}

Current cluster phase as a one-hot set (down, provisioning, ready, degraded); value 1 on the active phase.

spur_k8s_cluster_up

1 when the cluster phase is Ready (primary alerting signal).

spur_k8s_control_plane_replicas

Configured control-plane replica count.

spur_k8s_nodes_total / spur_k8s_nodes_by_role{role}

Total nodes with a k0s role, and the per-role count.

spur_k8s_provision_attempts_total / spur_k8s_provision_failures_total

Provisioning attempts and attempts that gave up before Ready.

spur_k8s_phase_transitions_total{from,to}

Phase transitions labeled by source and destination.

spur_k8s_reconcile_duration_seconds / spur_k8s_reconcile_errors_total

Reconcile-loop iteration wall time (histogram) and error count.

spur_k8s_drift_detected_total{kind}

Nodes whose running k0s disagrees with the controller’s record (out_of_scope, role_mismatch or not_running), counted once per episode. See Membership drift.

spur_k8s_drift_corrected_total{kind}

Strays the controller stopped, counted once the node stays stopped across two checks. Only out_of_scope is ever corrected, so it is the only kind.

spur_k8s_drift_refused_total{reason}

Stops of a confirmed stray the controller declined: disabled, control_plane and not_spurd_owned once per episode; circuit_breaker, throttled (another stray was stopped this tick), budget (the ten-minute stop budget is spent), backoff, not_leader, term_change, record_changed, stop_failed and stop_unconfirmed per attempt.

spur_k8s_drift_recurred_total

Stopped strays whose k0s came back.

spur_k8s_start_failures_total

Nodes newly recorded as failing to start k0s.

spur_k8s_node_up{node}

Whether a node’s k0s systemd unit reports active.

spur_k8s_node_restart_total{node} / spur_k8s_install_duration_seconds{node}

Per-node k0s unit restarts and install time (histogram).

Each controller counts only what it saw while it was the leader, and its counters reset when that process restarts. A leader change also starts drift episodes over, so a drift that is still present is counted again by the new leader rather than staying flat. Alert on increase(...[1h]) > 0 summed across controllers rather than on rate, which falls back to zero while the drift is still present. The counters carry no node label, to keep series bounded; the spurctld log line names the node.

k0s component metrics (incoming)#

Note

The endpoints in this section are incoming / in progress and are separate from the spurctld /metrics/k8s surface above. The bundled k0s components each expose their own upstream Kubernetes /metrics endpoint; Spur does not yet aggregate or proxy these — the ports and paths below are the standard Kubernetes surfaces documented here for planning a scrape configuration.

These endpoints are served by the k0s-managed components, not by spurctld. Control-plane endpoints live on the control-plane node; the kubelet endpoints live on every node. Most require TLS and authentication.

Component

Port / Path

Status

Contents

kube-apiserver

6443 /metrics

Incoming

API request latency/counts, etcd cache, admission timings.

kubelet

10250 /metrics

Incoming

Node agent, pod lifecycle, and volume stats.

kubelet (cAdvisor)

10250 /metrics/cadvisor

Incoming

Per-container CPU/memory/network/disk usage.

kubelet (resource)

10250 /metrics/resource

Incoming

Node/pod CPU and memory for the metrics pipeline.

kube-scheduler

10259 /metrics

Incoming

Scheduling attempts, latency, and queue depth.

kube-controller-manager

10257 /metrics

Incoming

Controller work queues and reconcile timings.

etcd

2379 /metrics

Incoming

Datastore health, latency, and DB size.

CoreDNS

9153 /metrics

Incoming

Cluster DNS query counts, latency, and cache stats.

Controlling Jobs#

Cancel jobs with spur cancel (Slurm scancel), by job ID or by filter:

scancel 1024 1025
spur cancel -u alice -p gpu --state PENDING
scancel --reservation maint
scancel --signal SIGTERM 2048

Filter flags include --user/-u, --partition/-p, --state/-t (only PD or R), --name/-n, --account/-A, --reservation/-R (drain a reservation’s jobs, mirroring squeue -R), and --signal/-s (KILL/9, TERM/15, INT/2, and others). You must supply job IDs or at least one of --user, --name, --partition, --account, or --reservation; --state alone only narrows a selection. Without --user, a filter matches every user’s jobs you can see, as in Slurm, and each one you do not own is refused — so --reservation (or --account, --partition) can span other users’ jobs, with authorization applied to each cancellation. A filter given an empty value (e.g. --reservation= from an unset variable) is rejected rather than treated as “match everything”. In filter mode, jobs already in a terminal state are silently skipped. scancel exits 1 and reports on stderr if any cancel or signal is refused. A job that is already finished or unknown does not fail a plain cancel (or -s KILL) and is not reported. With another signal it fails, and --quiet/-Q hides the report but not the exit status.

Time limits and the kill wait. At its time limit a job moves from RUNNING to COMPLETING (CG in squeue) straight away and is sent SIGTERM. It stays COMPLETING for up to the time-limit kill wait (timeout_kill_wait_secs, default 30 seconds), then Spur sends SIGKILL. The wait ends early once the job’s own process (its batch script) has exited, and the job ends TIMEOUT with reason TimeLimit however it exits: a clean exit on SIGTERM does not make it COMPLETED. --requeue still requeues it on timeout. While it waits, the job still counts as running for --dependency=singleton and for an array’s %N throttle, so a successor or the next array task does not start until the job has ended. The SIGTERM is final, so a batch script can use the wait to checkpoint and exit. An idle interactive allocation reaped for inactivity (Reaping abandoned interactive allocations (inactive_limit_secs)) normally ends TIMEOUT within a moment of the SIGTERM, without spending the wait in COMPLETING: its work runs in srun steps and its own process exits on SIGTERM (see the note below). scancel, scontrol requeue and the final kill of a preempted job use the general kill wait (kill_wait_secs, default 5 seconds), as does stopping the other nodes of a job whose batch script has exited. An srun step with no supervisor process gets the full wait that applies to it when a stop reaches it (see the note below): 5 seconds on scancel, 30 at a time limit, where earlier releases gave it 2 seconds.

Note

A job does not yet wait for its srun steps: it ends and frees its nodes once its batch script (or, for an allocation, its placeholder process) exits. On nodes where Spur runs as root, the steps are then stopped with the job without getting the kill wait. On nodes without cgroup enforcement, they can outlive the job and keep running beside a newly placed job. This affects every stop that uses a kill wait: a time limit, the inactivity limit, scancel, and the final kill of a preempted job. The preemption grace is not affected.

Change job state with spur control (Slurm scontrol):

scontrol hold 1024       # prevent from starting
scontrol release 1024    # allow a held job to start
scontrol requeue 1024    # return a job to the queue
scontrol suspend 1024    # SIGSTOP, keep the allocation
scontrol resume 1024     # SIGCONT

Suspending a job that is in its preemption grace period is refused, because the preemption that ends the grace only applies to a running job.

A running job that is held, by scontrol requeuehold (JobHeldUser) or on reaching max_batch_requeue (JobHoldMaxRequeue), gives up its nodes at once. If the stop sent at that moment does not reach a node, the process keeps running there while sinfo already shows the node’s resources as free, and a new job sent to the same GPUs is refused. An administrator can find the job still holding them in that node’s spurd log (gpu_owners=[...]). On the node’s next heartbeat the controller tells it to stop the held job’s process, so the resources become usable within about 40 seconds.

Update a job or node with scontrol update and Key=Value pairs. Job updates need JobId= and accept Priority=, TimeLimit=, Partition=, Account=, Comment=, QOS=, and ExcNodeList= (also spelled ExcludedNodes=). Node updates need NodeName= and accept State= and Reason=.

ExcNodeList= takes a hostlist expression and replaces the job’s excluded nodes, the list sbatch -x sets; an empty or blank value clears it, and an expression that does not parse (node[07-08) is refused. It applies only while the job is pending and not yet being started on nodes the scheduler has already reserved for it; a job that has left that window is refused with FAILED_PRECONDITION. Once accepted, the exclusion is recorded in the controller’s replicated log, so it survives a controller restart or a change of leader, and the job is never started on an excluded node. Use it to keep queued jobs off a node you have just found to be bad. In a cluster with several controllers, finish upgrading all of them before using it: a controller from an earlier release cannot read that log entry.

The whole request is rejected, with a non-zero exit and nothing changed, if any key is unknown or belongs to the other kind of update (State= on a job, TimeLimit= beside NodeName=), any value is malformed, or a job update names no field to change. TimeLimit=UNLIMITED, which Slurm accepts, is refused: give an explicit limit.

scontrol update JobId=1024 TimeLimit=2:00:00 Priority=100
scontrol update JobId=1024 ExcNodeList=node[07-08]
scontrol update JobId=1024 ExcNodeList=        # clear the exclusions
scontrol update NodeName=node01 State=drain Reason="maintenance"

See Also#