External fabric observations#
Spur can report GPU membership in a fabric partition configured by an external
manager. With [devices] auto_detect = true (the default), spurd reads
AMD UALink attributes during local device discovery and inventory refresh.
No intermediate inventory file or fabric-manager credentials are required on the agent.
Discovery performs no fabric configuration writes.
Observation is independent of job placement requirements. Jobs can request single-block placement using the discovered membership, as described below. This neither reserves a whole topology block nor verifies fabric-manager readiness or fabric isolation.
Inspecting a node#
Use scontrol show node <node> (or spur control show node <node>).
The response includes InventoryGeneration, FabricBindingVersion, BlockFileRevision, and, for GPUs with topology
metadata, full PCI BDF, node-local PCI host bridge, NUMA locality, exact VRAM
bytes, and fabric observations. For example:
GPUDevice=0 StableId=... PCI=0001:04:00.0 PCIRoot=pci0001:00 NUMA=unknown VRAMBytes=309237645312 GPUType=mi300x
FabricSource=amd-ualink Observation=observed PhysicalDomain=... FabricDomain=15 Endpoint=63 EndpointState=active Transport=UALoE
PhysicalCapacity=72 FabricCapacity=4 ActiveEndpoints=[60,61,62,63] LocalEndpoints=[60,61,62,63]
observed means the attributes were read consistently. It does not mean the
external fabric manager reports the partition as ready. Other statuses are incomplete (missing, malformed, or
unreadable fields), inconsistent (conflicting or changing observations),
unavailable (the source/device could not be observed), and unknown.
Issue kinds are missing, unreadable, malformed, too_large,
conflict, changed, unavailable, peer_conflict (another GPU on
the node reports different shared vPod facts), and unknown.
ObservationIssues identifies affected fields and stable diagnostic kinds.
Unknown NUMA locality remains unknown rather than becoming NUMA node zero.
The displayed data is the controller’s last acknowledged inventory, not a live health query. Use the node state and heartbeat alongside it. The existing refresh interval and two-observation debounce apply. Membership-only changes participate in refresh even when device IDs and GPU counts remain constant.
Identity and capacity#
PhysicalDomain identifies a physical pod; FabricDomain is a vPod (virtual
pod: the accelerator partition configured by the external fabric manager) ID
scoped to that physical pod. Equal vPod numbers on different physical pods do
not imply connectivity. The controller scopes domain
identities by the configured cluster_name administrative namespace. PCI host bridge names and GPU
stable IDs are local to the Spur node.
Capacities and membership count physical fabric endpoints. Logical GPU partitions may share one endpoint, so endpoint counts cannot be substituted for schedulable logical-GPU counts. Discovery enumerates active KFD devices; inactive DRM/XCP entries do not add schedulable GPUs. Exact VRAM comes from the KFD logical-device inventory. No GPU marketing name or performance rating is inferred from these fabric attributes.
Discovery boundary#
The AMD adapter reads only these files below each GPU’s PCI ualink directory:
ppod_id,ppod_sizevpod_id,vpod_size,vpod_active_accelslocal_accels,accel_id,accel_state,link_type
Reads are bounded and endpoint sets are normalized. Two consecutive snapshots
must agree; local GPU views of the same domain are also checked for conflicts.
This cannot provide an atomic driver snapshot or prevent an external manager
from changing the fabric after discovery. Programming interfaces such as
config, setup, and stations are not accessed. Zero bandwidth or
latency values are not used to invent scheduling costs.
On-disk CDI specs still supply device selection and injection edits. When
auto-detection is enabled, live topology is overlaid for matching AMD render
devices, with explicit PCI BDF and nonzero stable IDs checked against discovery.
A missing match is reported as unavailable. Static specs do not freeze live membership.
With auto_detect = false, CDI metadata is static and updates require
regenerating the specs. GRES-only entries do not acquire topology automatically.
Integration contract#
GpuResource.topology is optional in persisted inventory and the gRPC API.
Older agents and saved inventories omit it. New agents send typed
DeviceTopology and FabricObservation messages through registration;
node/resource responses preserve them. Unknown wire status values map to
unknown. Existing allocated device identities remain accounted for during
inventory changes, while missing devices’ old fabric observations become
unavailable. When block_file is configured, a full discovery dropout also
marks retained declaration-only GPUs unavailable. GPUs with neither a fabric
observation nor a block file are left unchanged, so a failed scan does not
alter plain GPU inventory. During
partial loss, a missing held GPU retains its accounting identity with an
unavailable marker. These markers participate in inventory refresh and disable
affected block placement; identity retention does not establish device presence.
The shared model uses vendor-neutral physical/fabric domain references,
endpoint IDs, locality, capacity, observation status, and diagnostics.
amd-ualink is its first discovery source. Internally, CDI carries the
model as JSON in the spur.amd.com/topology annotation. An invalid annotation
is reported as unavailable rather than silently disappearing. This annotation
is an inventory transport, not a validated controller static-import API.
Typed GPU predicates use these observations when explicitly requested, as described below. Per-device static import, segments, topology exclusivity, and authoritative fabric health remain unsupported.
Controller block inspection#
The controller assembles blocks from explicit GPU membership on each query. Node ordering and rack size do not create fabric membership. A block key consists of the cluster namespace, physical domain, and fabric domain. Members retain node names, stable logical GPU IDs, physical endpoint IDs, and inventory generations.
spur control topology show
scontrol show topology
scontrol topology show --json
scontrol topology export > fabric.json
scontrol topology validate fabric.json
show includes invalid blocks and GPUs whose membership cannot be resolved.
Diagnostics cover disagreement between nodes, endpoints claimed by multiple
blocks or physical devices, missing registered endpoints, unavailable nodes,
and missing or expired heartbeats. The controller’s configured heartbeat timeout
applies (90 seconds by default). A current heartbeat only establishes agent
liveness; it does not establish when the external fabric last changed.
Membership=consistent means the registered reports agree and cover the
reported active endpoints. It does not establish fabric health, available job
capacity, isolation, or a scheduling guarantee. Multiple logical GPU partitions
may share a physical endpoint when they report the same node and PCI identity.
export prints a version 1 JSON candidate membership document only when every
block is consistent and every GPU on a block’s nodes has resolved membership.
GPUs on nodes outside every block, such as plain GPU nodes, are listed by
show but do not prevent an export. The document contains
scoped block keys and exact node/stable-ID pairs. It deliberately contains no
health or availability claims. validate resolves all declared blocks and
devices against current controller inventory, rejecting duplicates, unknown
identities, namespace mismatches, changed membership, and invalid live blocks.
Unknown top-level and block fields and unsupported versions are errors; input is limited to
8 MiB. A document can describe a subset of blocks, but each declared block must
contain its complete registered membership.
Validation is read-only. It does not install the document, change membership, reserve resources, or freeze a fabric configuration. Automatic discovery remains the default; no administrator-generated intermediate file is required. Installing an optional whole-node definition file is a separate operation, described below.
Discovery in block and tree topology#
Automatic discovery supplies groups to Spur’s existing block/tree node topology. It does not introduce another topology plugin. To use discovered vPods as blocks:
[topology]
plugin = "block"
No intermediate file or block_size is needed. For example, one rack system
with vPod 15 containing nodes 0/2 and vPod 16 containing nodes 1/3 becomes two
leaves in the existing topology model. Node names and registration order do not
change those groups. The optional block file supplies leaves through the same
path, including vendor-neutral declarations without native observations.
The controller rebuilds the model each scheduling cycle, so node registration, membership refresh, and configuration reload update locality without a restart. Only consistent whole-node membership becomes a leaf. Nodes with GPUs in several blocks or with incomplete/invalid membership have no inferred leaf; they remain eligible for ordinary scheduling. Unknown nodes are not grouped together as if they shared a block. GPU-level containment still supports exact device membership independently of this node-level projection.
plugin = "tree" also accepts these flat leaves when no explicit switches are
configured. Observations do not establish links between vPods, so the controller
does not invent parent switches, distances, or bandwidth. An explicit
[[topology.switches]] hierarchy retains precedence in tree mode. Existing
block configurations without any native/file membership source retain grouping
by block_size (18 by default). Readable but incomplete or inconsistent fabric
observations do not fall back to groups inferred from node count. Unavailable
markers alone, such as those left by a failed discovery scan, carry no membership
evidence: if no block or file remains, block_size grouping applies.
plugin = "none" disables the locality model.
The existing submission options request locality using this model:
spur submit -p gpu -N 2 -n 2 --gpus 6 --topology block --wrap 'hostname'
spur submit -p gpu -N 2 -n 2 --gpus 6 --topology tree --wrap 'hostname'
Both options prefer nearby nodes and permit crossing leaves when needed; neither
is a hard single-vPod requirement. --topology none skips the locality hint.
Omitting the option retains ordinary resource ordering. To require containment,
add the job constraint below. Locality hints cannot relax that requirement.
Job placement#
Spur observes membership; it does not create, resize, or delete fabric groups, and no partition setting imposes a single-block requirement on every GPU job. Request hard containment with an existing constraint option:
spur submit -p gpu -N 2 -n 2 --gpus 6 --constraint='same(gpu.fabric_domain_id)' --wrap 'hostname'
spur alloc -p gpu -N 2 -n 2 --gpus 6 --constraint='same(gpu.vpod_id)'
spur run -p gpu -N 2 -n 2 --gpus 6 --constraint='same(gpu.fabric_domain_id)' hostname
same(gpu.vpod_id) is an alias for the vendor-neutral
same(gpu.fabric_domain_id). Both require all selected GPUs to belong to
one scoped, observed or declared block. Equal local IDs in different physical
domains do not satisfy the constraint. The same block inventory supplies the
existing block/tree topology and exact-GPU placement. No new topology plugin
or partition-name list is needed.
Per-GPU property constraints#
The existing --constraint option also filters the actual allocated GPU
identities by model, VRAM, physical domain, fabric domain, NUMA node, and PCI
root. A matching node is insufficient: every selected GPU must match.
Field |
Operators |
Example |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
VRAM accepts bytes or binary/decimal units such as GiB and GB; it is a
per-device capacity filter, not a request to reserve a portion of GPU memory.
NUMA IDs are integers greater than or equal to zero. Unknown values never satisfy a predicate,
including !=. String comparisons are exact and case-sensitive. Domain ID
filters use the ordinary partition and node eligibility rules; there is no
separate administrator-only domain-pinning policy in this version.
Join predicates and required node features with commas or &. Expressions
are limited to 4096 bytes and 32 terms. OR and grouping parentheses, apart from
same(field), are rejected. Legacy feature-only constraints retain their
comma-separated syntax: fast,large means both features; fast&large alone
retains its previous meaning as one literal feature name.
same(field) requires one value across all selected GPUs for any field in
the table. NUMA and PCI-root identities are scoped to a node, so identical
local labels on different nodes do not satisfy a same-NUMA or same-root request;
these two groupings are rejected at submit unless the job requests -N 1.
The constraint selects GPUs; it does not pin job CPUs or memory to that NUMA
node. Equality on a fabric-domain ID also requires one scoped block, avoiding
accidental combinations of repeated local IDs in different racks. A file block
name does not manufacture a GPU observation: direct domain-ID filters still
require reported values, while same(gpu.fabric_domain_id) can use a declared
block when native membership is absent.
Malformed expressions are rejected at submit, including an unknown field such as
same(gpu.vpodid) and near-misses such as GPU.vram>=192GiB,
same[gpu.vpod_id], or gpu.type:mi300x, which would otherwise be read as
node feature names. A job_submit hook that rewrites the constraint is
validated again after the hook runs.
spur submit -N 1 -n 1 --gpus 2 --constraint='gpu.type=mi300x&gpu.vram>=192GiB&same(gpu.pcie_root_id)' --wrap 'hostname'
spur submit -N 2 -n 2 --gpus-per-node 2 --topology block --constraint='gpu.vram>=192GiB&same(gpu.vpod_id)' --wrap 'hostname'
spur run -N 1 -n 1 --gpus 2 --constraint='same(gpu.numa_node_id)' hostname
These expressions require a GPU request or whole-node exclusivity. Upgrade
controllers and clients together before using them. Agents from this release
report FabricBindingVersion=3; older agents report 0 and are never selected
for constrained placement.
The controller persists the exact selected GPU identities and property
snapshots with their inventory generation. Admission and the start gate require
that generation to be current. Steps inside a running allocation, including after
supervisor recovery, instead re-attest the bound GPUs’ own facts, so an agent
restart, a change to another GPU, or another GPU’s peer_conflict does not
block them. Before launch, the agent re-reads the selected devices’ UALink
attributes; other properties, such as VRAM and NUMA locality, are checked
against the inventory the agent last reported. Inspection with spur control show job
includes GpuPropertyNode, Generation, and StableIds. A change to a
bound GPU’s own properties rejects subsequent steps of that allocation. This does
not terminate GPU processes already executing when the hardware changes.
Plain GPU jobs use ordinary scheduling and can span blocks. With a same-fabric constraint,
a job waits if no single block can fit it; there is no automatic fallback.
--topology none cannot disable that requirement. Omitting -p selects the
configured default partition, or the first partition if no default is marked;
the explicit job constraint keeps the same meaning for either choice and for
comma-separated partition lists. Partition names have no built-in rack meaning.
Bare --exclusive retains whole-node allocation semantics. Combined with the
same-fabric constraint, every GPU reserved on an exclusive node must belong to
the selected block. ExclusiveTopo=YES is rejected for partitions; reserving a
node does not reserve all nodes in its block. ExclusiveTopo=NO is accepted as
Slurm’s default. Steps inherit their parent allocation’s
binding. Adding a GPU property constraint to an existing step is rejected; specify
it when creating the allocation.
For a same-fabric job, the scheduler considers complete, consistent blocks whose candidate nodes advertise binding enforcement. It filters each node’s inventory to that block’s exact GPU IDs, applies any additional GPU predicates and grouping, and runs the existing resource and backfill solver. A property-only job applies the same filters without requiring containment in one fabric block. It chooses the earliest feasible start, preferring smaller blocks at the same start and then the scoped block key. Node lists, GPU types and quantities, CPU/memory requirements, and reservations still apply. Selected GPU IDs must remain free throughout the proposed interval. Future shadow reservations hold those same concrete resources so shorter jobs can fill gaps without consuming them at the reserved start.
The controller validates the selected block again before recording the placement.
Each per-node allocation persists its block key, active endpoints, exact logical
GPU membership, binding version, and inventory generation. scontrol show job
displays the resulting per-node fabric binding. A failed same-fabric search
reports FabricUnavailable in the pending reason description. A property-only
request that cannot fit reports GpuPropertiesUnavailable; inspect
spur control show node for its reported GPU facts.
The agent validates bindings before admission, then rechecks the inventory generation under the allocation lock before reserving resources. Inventory publication uses the same lock; file and device reads run outside it. AMD UALink bindings reread the allowlisted attributes of the bound PCI devices. Missing observations, changed domains/endpoints, a changed inventory generation, or an unsupported binding version reject the launch. Other discovery sources use their current typed inventory for binding checks. No device or fabric configuration is written. Capable agents start with a nonzero random inventory epoch, fencing dispatches made against an earlier agent instance.
A supervisor retains the parent binding across an agent restart, and numbered/interactive steps validate it before executing; they cannot acquire devices outside the parent allocation. Steps re-attest only the bound devices: each must still report the bound domain and endpoint, and its own endpoint must remain active. An agent restart, a generation change from other devices, a drained or stale node elsewhere in the block, or another vPod member becoming inactive does not block new steps. A change to a bound device’s own membership does, until the allocation is requeued; for file bindings, so does removing any of the allocation’s nodes from the bound block.
This is a snapshot-based placement contract. An external manager can still change a fabric after preflight or while a job is running; operators must coordinate such changes with running allocations. Observed membership is not an assertion of fabric readiness, bandwidth, security isolation, or an atomic fabric lease.
Heterogeneous job components with GPU property or containment constraints are rejected at submit until group-wide binding is supported. Constrained jobs do not trigger automatic preemption, idle-fill reclamation, or federation forwarding, whose current selection paths do not carry fabric requirements. Ordinary backfill, including future resource reservations, remains available. Segments, whole-block exclusivity, and per-device static import remain future extensions.
Optional administrator or operator block file#
For a rack system whose placement groups are maintained by an administrator or
external operator and not available through native fabric discovery, set
block_file as a fallback membership source:
[topology.external]
block_file = "/etc/spur/topology.conf"
Native discovery remains authoritative when block_file is configured.
The file adds whole-node groups for nodes without native fabric observations;
discovered vPods need not be listed. The filename has no special meaning; an
absolute path is required. Install the same setting and file on the controller
and agents participating in file-defined groups. Native-only agents do not require the
file and use native bindings. All nodes can remain in one partition. The job
constraint selects hard containment; plain GPU jobs retain ordinary scheduling.
The file is text, with one record per block:
# Two groups in one rack system
BlockName=group-a Nodes=node[01-02]
BlockName=group-b Nodes=node[03-04]
Each record requires BlockName and Nodes. Field names are case
insensitive; field order is arbitrary. Node lists support bracket ranges and
comma-separated names. # begins a comment and a trailing backslash joins
lines. Block names use letters, digits, underscores, dots, and hyphens, with a
maximum of 128 bytes. Node names use the same characters, with a maximum of
255 bytes. Files are limited to 1 MiB and 65,536 declared nodes. Duplicate
fields, block names, or node memberships are errors.
This is a deliberately small subset of the block record format. BlockSizes
and other fields are rejected: higher-level aggregation is not implemented,
and each declared block is a boundary for jobs requesting containment. A node can belong to
only one declared block. The record covers all currently registered GPUs on
each named node; it cannot describe separate groups within a node. GPU counts,
types, identities, free resources, and health are never taken from this file.
Discovered nodes omitted from the file keep their native block membership.
Nodes with neither native membership nor a file declaration cannot supply GPUs
to same-fabric jobs, but remain available to ordinary scheduling.
If any named node has native fabric observations, the entire file record is
ignored. An exact match with one complete native block produces an informational
notice. A conflicting grouping, subdivision, mixed record, or unavailable native
membership produces a warning. A file cannot merge or subdivide discovered
vPods. A record mixing discovered and non-discovered nodes is ignored in full;
write a separate record for the non-discovered nodes. Notices appear in
spur control topology show and topology validate --format block;
topology show --json includes structured notices. Reconfigure also logs
them. These notices do not make healthy native blocks unavailable.
For example, if nodes 01/02 form discovered vPod 15 and nodes 03/04 have no
native fabric observations, only BlockName=group-b Nodes=node[03-04] is
needed. Both groups participate in the existing block/tree topology and
same-fabric scheduling. A conflicting BlockName=group-a Nodes=node[01-04]
would instead be ignored, leaving vPod 15 intact and nodes 03/04 undeclared.
Missing or inconsistent native observations remain unavailable for affected placement; the file cannot turn failed discovery into usable membership. During an agent’s lifetime, losing a previously observed native source retains an unavailable marker until observations recover. This history is in memory; a fresh agent cannot distinguish an absent source from hardware that never exposed it. After a restart, the administrator remains responsible for the correctness of any fallback declaration. Native endpoint coverage, heartbeat, and exact-device binding checks still apply.
If none of a declared block’s GPUs reports native membership, the file is an explicit administrator declaration. This path is vendor neutral. Spur enforces node membership and exact live GPU identities without asserting connectivity, bandwidth, fabric health, or external-manager readiness. An external operator is responsible for ensuring its declaration reflects the configured system. This does not add fabric programming or manager credentials to Spur.
Creating and validating a candidate#
An administrator can write the records directly. Export is useful for inspection or as a starting point for declarations on non-discovered nodes:
spur control topology export --format block > topology.conf.candidate
spur control topology validate --format block topology.conf.candidate
export --format block prints deterministic BlockName/Nodes records.
It preserves names for file-defined groups and generates names such as
block0 for automatic groups. It rejects invalid groups, a node shared by
multiple groups, or unresolved GPUs on an included node. Review and validate
the output before installation. It contains no inferred topology hierarchy.
validate --format block checks syntax locally, then resolves membership
against the controller’s current live inventory through a read-only RPC. The
original bounded text is sent unchanged, so expansion of a compact hostlist
does not inflate the validation request beyond the file-size limit. Validation
does not require the candidate revision to be deployed to agents, so it can
be used before installation. Missing nodes, stale agents, missing GPUs, or
invalid file-only membership are reported as errors. Records superseded by
native discovery produce notices instead; unrelated unhealthy native blocks
do not reject a valid file-only candidate. The result reports applied and
ignored record counts. Exported native groups are ignored if installed while
native observations are present. Success does not test fabric connectivity or
install the file. Plain export and validate
retain the existing JSON format; JSON manifests are inspection/validation
artifacts and are not accepted as block_file input.
Deployment and updates#
For first activation, upgrade all controllers and participating agents, install
the validated file on controllers and file-only agents, and restart those
agents to load the new block_file setting. Reconfigure or restart the controller to enable the
setting. A controller can start before nodes register; affected blocks remain
unavailable until complete live inventory and matching revisions arrive.
Reconfigure rejects a file-only block that names an unregistered node, a node
without GPUs, or GPUs without stable identities, so wait for registration before
using it for activation. A drained, down, or stale node only leaves its block
unavailable and is logged; it does not block unrelated configuration changes. Records
superseded by native discovery do not require matching files on native agents.
For later membership changes:
Generate a complete candidate and run
topology validate --format block.Coordinate with running allocations. Moving a node to another block rejects new steps of allocations bound to the old block; requeue them as appropriate.
Atomically replace the file on participating agents and controllers. Use a temporary file in the same directory followed by rename; keep the directory readable by the daemon and writable only by the administrator/operator.
Run
spur control reconfigureon the controller. It parses and validates the complete candidate before changing partitions or the active config.Inspect
spur control topology showandspur control show node NODE. Resume constrained submissions once the desired blocks have no issues and the participating agents report the expected revision.
There is no automatic controller filesystem watch. Agents reread the file through the existing inventory refresh/debounce loop and directly before admission and new steps. The controller retains a parsed immutable snapshot until reconfigure/restart. A malformed or missing candidate causes reconfigure to fail while preserving the previous configuration. A missing, malformed, or different agent file fences that agent’s declared block; another valid block can still schedule. Native agents ignore file revisions, so file changes do not change their inventory generation or invalidate their native bindings. Restore the matching file and wait for inventory refresh to make the affected block available again.
Revisions hash normalized block names and expanded node memberships. Comments, whitespace, field order, and record order do not change the revision. Membership or block-name changes do. During rollout, revision differences deliberately make affected file-only placement unavailable. Native blocks remain independent of file revisions. The revision covers the whole file, so an edit to any record fences new placements and dispatches made against an older revision. Steps of a running file-bound allocation instead re-read the file and require every node of the allocation to remain in the bound block. Plan updates as a coordinated operation. Step dispatch is not an atomic multi-node barrier: during partial rollout, a step may start on nodes that still validate the old binding while another node rejects it. Quiesce affected allocations before changing membership. Existing running processes are not moved or killed by a file update.
High-availability deployments follow the existing controller configuration
model: reconfigure applies on the active leader, and does not replicate
file contents or local configuration to followers. Deploy matching files and
configuration to every controller and restart followers with that configuration
before relying on failover. This feature adds no distributed file rollout or
atomic fabric lease.
Scheduling and inspection#
File blocks use the same backfill algorithm and exact-device admission checks
as automatic blocks. Earliest feasible start wins, then smaller blocks, then
block-key order.
A smaller job may share a block when its CPUs, memory, and GPUs are free. A constrained job
cannot combine two blocks even if their summed free GPUs meet its request.
--exclusive still reserves whole nodes, not a whole block.
For four nodes without native fabric observations, four GPUs per node, and the two records above:
spur submit -p gpu -N 2 -n 2 --gpus 6 --constraint='same(gpu.fabric_domain_id)' -t 00:10:00 --wrap 'sleep 300'
spur submit -p gpu -N 2 -n 2 --gpus 6 --constraint='same(gpu.fabric_domain_id)' -t 00:10:00 --wrap 'sleep 300'
spur submit -p gpu -N 4 -n 4 --gpus 12 --constraint='same(gpu.fabric_domain_id)' -t 00:10:00 --wrap 'sleep 300'
spur control topology show
spur control show job JOB_ID
The first two jobs can use different eight-GPU blocks. The twelve-GPU job stays
pending because it cannot fit within either block. Omitting -p uses the
configured default partition; the constraint is independent of that default.
topology show includes Source=file and Revision for file-only blocks.
Native blocks retain their native domain IDs. Job inspection includes BlockFileRevision and
BindingVersion=2 for file bindings, or BindingVersion=1 for native
fabric bindings, alongside exact per-node GPU bindings. The constraint uses
the existing submission flag; no job ID environment variable
is required when submitting a new job.