Architecture#
Spur runs as four binaries: a controller daemon (spurctld), a node agent
(spurd), a per-job supervisor (spurstepd), and a command-line client
(spur). Clusters that use native credentials add a fifth, the credential
mint (spurauthd). This page describes what each component does, which ports
they use, and the core scheduling concepts.
Components#
spurctld — Controller#
The controller is the scheduler and the single point of contact for clients. It
serves the gRPC API (the SlurmController and SlurmAccounting services) on
port 6817. Accounting runs in-process, backed by PostgreSQL, and the REST API
(built on Axum) is served directly by the controller — there are no separate
accounting or REST daemons.
High availability is built in through Raft log replication (openraft) and is always on: even a single-node deployment runs a one-member Raft cluster. In a multi-controller cluster, the leader handles all writes and non-leaders forward requests to it automatically, so clients can talk to any controller.
spurd — Node agent#
The node agent runs on every compute node. It registers with the controller,
sends periodic heartbeats, and receives job launch and cancel commands over gRPC
(the SlurmAgent service) on port 6818. Interactive sessions and live job
output stream directly between the client and the agent.
spurstepd — Job supervisor#
The agent does not parent the work it launches. It spawns a supervisor per job,
and a further one per numbered srun step, detached into its own session so
it is reparented to init. The supervisor owns the process tree, its cgroup and
its exit status, and the agent talks to it over a Unix socket.
That is what lets spurd be restarted or upgraded under running work: on
start it rediscovers the supervisors on disk and re-adopts them, and a step that
finished meanwhile still reports its exit status over the reconnect. Users never
invoke spurstepd directly. See Interactive & Parallel Jobs for what survives a
restart and what does not.
spur — Command-line client#
spur is a multi-call binary. It talks to the controller on port 6817 for
scheduling, admin, and accounting. Invoke it as spur <command> (for example
spur submit or spur queue), or through Slurm-compatible symlinks such as
sbatch, squeue, and sinfo. See Slurm Compatibility for the
full command map.
spurauthd — Credential mint#
Only clusters that set [auth] plugin = "spur" need the mint. It runs on
every host that makes calls — login nodes for the CLI, compute nodes for
spurd — and listens on a local Unix socket, not the network. Before each
call the CLI or agent asks it for a short-lived credential, and it takes the
caller’s identity from the kernel rather than from anything the caller sends.
See plugin = "spur" for setup.
Note
Unlike Slurm, Spur has no separate accounting or REST daemons — there is
no slurmdbd and no slurmrestd. The controller (spurctld) handles
accounting and the REST API itself. The entire distribution is five binaries:
spurctld, spurd, spurstepd, spur, and spurauthd.
Ports#
Port |
Component |
Purpose |
|---|---|---|
|
|
Controller gRPC API and accounting |
|
|
Node agent gRPC (launch, cancel, I/O streaming) |
|
|
REST API |
|
|
Raft replication (controller-to-controller) |
These are the defaults; all four are configurable in spur.conf (listen_addr,
agent_port, rest_addr, raft_listen_addr). See
Configuration Reference (spur.conf).
Concepts#
Partitions#
A partition is a job queue over a set of nodes. Partitions are defined in the
configuration file (spur.conf) and control which nodes a job can run on, along
with access limits such as permitted accounts and default time limits. See
Configuration Reference (spur.conf) for partition settings.
Jobs and job steps#
A job is a resource allocation submitted to a partition, typically from a batch
script (spur submit / sbatch) or an interactive allocation
(spur alloc / salloc). A job step is a task launched inside an existing
allocation with spur run / srun; a step shares the parent job’s resources
rather than requesting a new allocation.
Associations and QOS#
An association is the (cluster, account, user, partition) tuple that ties a
user to the resources they may use. A Quality of Service (QOS) applies limits and
priority on top of associations. Both are managed through the accounting service.
See Accounting, Accounts, Users, and QOS.
Networking and scheduling#
Spur can carry controller and agent traffic over a WireGuard mesh, giving encrypted transport between nodes without a separate VPN. The mesh is optional — deployments on a trusted LAN can run traffic directly. See Manual Deployment (systemd) for setup.
Scheduling is GPU-first: GPUs are requested with --gres=gpu:... (or the
--gpus shorthand), and the agent sets ROCR_VISIBLE_DEVICES and
CUDA_VISIBLE_DEVICES for the allocated devices at launch.