External fabric observations#

Spur can report GPU membership in a fabric partition configured by an external manager. With [devices] auto_detect = true (the default), spurd reads AMD UALink attributes during local device discovery and inventory refresh. No intermediate inventory file or fabric-manager credentials are required on the agent. Discovery performs no fabric configuration writes.

Observation is independent of job placement requirements. Jobs can request single-block placement using the discovered membership, as described below. This neither reserves a whole topology block nor verifies fabric-manager readiness or fabric isolation.

Inspecting a node#

Use scontrol show node <node> (or spur control show node <node>). The response includes InventoryGeneration, FabricBindingVersion, BlockFileRevision, and, for GPUs with topology metadata, full PCI BDF, node-local PCI host bridge, NUMA locality, exact VRAM bytes, and fabric observations. For example:

GPUDevice=0 StableId=... PCI=0001:04:00.0 PCIRoot=pci0001:00 NUMA=unknown VRAMBytes=309237645312 GPUType=mi300x
   FabricSource=amd-ualink Observation=observed PhysicalDomain=... FabricDomain=15 Endpoint=63 EndpointState=active Transport=UALoE
   PhysicalCapacity=72 FabricCapacity=4 ActiveEndpoints=[60,61,62,63] LocalEndpoints=[60,61,62,63]

observed means the attributes were read consistently. It does not mean the external fabric manager reports the partition as ready. Other statuses are incomplete (missing, malformed, or unreadable fields), inconsistent (conflicting or changing observations), unavailable (the source/device could not be observed), and unknown. Issue kinds are missing, unreadable, malformed, too_large, conflict, changed, unavailable, peer_conflict (another GPU on the node reports different shared vPod facts), and unknown. ObservationIssues identifies affected fields and stable diagnostic kinds. Unknown NUMA locality remains unknown rather than becoming NUMA node zero.

The displayed data is the controller’s last acknowledged inventory, not a live health query. Use the node state and heartbeat alongside it. The existing refresh interval and two-observation debounce apply. Membership-only changes participate in refresh even when device IDs and GPU counts remain constant.

Identity and capacity#

PhysicalDomain identifies a physical pod; FabricDomain is a vPod (virtual pod: the accelerator partition configured by the external fabric manager) ID scoped to that physical pod. Equal vPod numbers on different physical pods do not imply connectivity. The controller scopes domain identities by the configured cluster_name administrative namespace. PCI host bridge names and GPU stable IDs are local to the Spur node.

Capacities and membership count physical fabric endpoints. Logical GPU partitions may share one endpoint, so endpoint counts cannot be substituted for schedulable logical-GPU counts. Discovery enumerates active KFD devices; inactive DRM/XCP entries do not add schedulable GPUs. Exact VRAM comes from the KFD logical-device inventory. No GPU marketing name or performance rating is inferred from these fabric attributes.

Discovery boundary#

The AMD adapter reads only these files below each GPU’s PCI ualink directory:

  • ppod_id, ppod_size

  • vpod_id, vpod_size, vpod_active_accels

  • local_accels, accel_id, accel_state, link_type

Reads are bounded and endpoint sets are normalized. Two consecutive snapshots must agree; local GPU views of the same domain are also checked for conflicts. This cannot provide an atomic driver snapshot or prevent an external manager from changing the fabric after discovery. Programming interfaces such as config, setup, and stations are not accessed. Zero bandwidth or latency values are not used to invent scheduling costs.

On-disk CDI specs still supply device selection and injection edits. When auto-detection is enabled, live topology is overlaid for matching AMD render devices, with explicit PCI BDF and nonzero stable IDs checked against discovery. A missing match is reported as unavailable. Static specs do not freeze live membership. With auto_detect = false, CDI metadata is static and updates require regenerating the specs. GRES-only entries do not acquire topology automatically.

Integration contract#

GpuResource.topology is optional in persisted inventory and the gRPC API. Older agents and saved inventories omit it. New agents send typed DeviceTopology and FabricObservation messages through registration; node/resource responses preserve them. Unknown wire status values map to unknown. Existing allocated device identities remain accounted for during inventory changes, while missing devices’ old fabric observations become unavailable. When block_file is configured, a full discovery dropout also marks retained declaration-only GPUs unavailable. GPUs with neither a fabric observation nor a block file are left unchanged, so a failed scan does not alter plain GPU inventory. During partial loss, a missing held GPU retains its accounting identity with an unavailable marker. These markers participate in inventory refresh and disable affected block placement; identity retention does not establish device presence.

The shared model uses vendor-neutral physical/fabric domain references, endpoint IDs, locality, capacity, observation status, and diagnostics. amd-ualink is its first discovery source. Internally, CDI carries the model as JSON in the spur.amd.com/topology annotation. An invalid annotation is reported as unavailable rather than silently disappearing. This annotation is an inventory transport, not a validated controller static-import API.

Typed GPU predicates use these observations when explicitly requested, as described below. Per-device static import, segments, topology exclusivity, and authoritative fabric health remain unsupported.

Controller block inspection#

The controller assembles blocks from explicit GPU membership on each query. Node ordering and rack size do not create fabric membership. A block key consists of the cluster namespace, physical domain, and fabric domain. Members retain node names, stable logical GPU IDs, physical endpoint IDs, and inventory generations.

spur control topology show
scontrol show topology
scontrol topology show --json
scontrol topology export > fabric.json
scontrol topology validate fabric.json

show includes invalid blocks and GPUs whose membership cannot be resolved. Diagnostics cover disagreement between nodes, endpoints claimed by multiple blocks or physical devices, missing registered endpoints, unavailable nodes, and missing or expired heartbeats. The controller’s configured heartbeat timeout applies (90 seconds by default). A current heartbeat only establishes agent liveness; it does not establish when the external fabric last changed.

Membership=consistent means the registered reports agree and cover the reported active endpoints. It does not establish fabric health, available job capacity, isolation, or a scheduling guarantee. Multiple logical GPU partitions may share a physical endpoint when they report the same node and PCI identity.

export prints a version 1 JSON candidate membership document only when every block is consistent and every GPU on a block’s nodes has resolved membership. GPUs on nodes outside every block, such as plain GPU nodes, are listed by show but do not prevent an export. The document contains scoped block keys and exact node/stable-ID pairs. It deliberately contains no health or availability claims. validate resolves all declared blocks and devices against current controller inventory, rejecting duplicates, unknown identities, namespace mismatches, changed membership, and invalid live blocks. Unknown top-level and block fields and unsupported versions are errors; input is limited to 8 MiB. A document can describe a subset of blocks, but each declared block must contain its complete registered membership.

Validation is read-only. It does not install the document, change membership, reserve resources, or freeze a fabric configuration. Automatic discovery remains the default; no administrator-generated intermediate file is required. Installing an optional whole-node definition file is a separate operation, described below.

Discovery in block and tree topology#

Automatic discovery supplies groups to Spur’s existing block/tree node topology. It does not introduce another topology plugin. To use discovered vPods as blocks:

[topology]
plugin = "block"

No intermediate file or block_size is needed. For example, one rack system with vPod 15 containing nodes 0/2 and vPod 16 containing nodes 1/3 becomes two leaves in the existing topology model. Node names and registration order do not change those groups. The optional block file supplies leaves through the same path, including vendor-neutral declarations without native observations.

The controller rebuilds the model each scheduling cycle, so node registration, membership refresh, and configuration reload update locality without a restart. Only consistent whole-node membership becomes a leaf. Nodes with GPUs in several blocks or with incomplete/invalid membership have no inferred leaf; they remain eligible for ordinary scheduling. Unknown nodes are not grouped together as if they shared a block. GPU-level containment still supports exact device membership independently of this node-level projection.

plugin = "tree" also accepts these flat leaves when no explicit switches are configured. Observations do not establish links between vPods, so the controller does not invent parent switches, distances, or bandwidth. An explicit [[topology.switches]] hierarchy retains precedence in tree mode. Existing block configurations without any native/file membership source retain grouping by block_size (18 by default). Readable but incomplete or inconsistent fabric observations do not fall back to groups inferred from node count. Unavailable markers alone, such as those left by a failed discovery scan, carry no membership evidence: if no block or file remains, block_size grouping applies. plugin = "none" disables the locality model.

The existing submission options request locality using this model:

spur submit -p gpu -N 2 -n 2 --gpus 6 --topology block --wrap 'hostname'
spur submit -p gpu -N 2 -n 2 --gpus 6 --topology tree --wrap 'hostname'

Both options prefer nearby nodes and permit crossing leaves when needed; neither is a hard single-vPod requirement. --topology none skips the locality hint. Omitting the option retains ordinary resource ordering. To require containment, add the job constraint below. Locality hints cannot relax that requirement.

Job placement#

Spur observes membership; it does not create, resize, or delete fabric groups, and no partition setting imposes a single-block requirement on every GPU job. Request hard containment with an existing constraint option:

spur submit -p gpu -N 2 -n 2 --gpus 6 --constraint='same(gpu.fabric_domain_id)' --wrap 'hostname'
spur alloc -p gpu -N 2 -n 2 --gpus 6 --constraint='same(gpu.vpod_id)'
spur run -p gpu -N 2 -n 2 --gpus 6 --constraint='same(gpu.fabric_domain_id)' hostname

same(gpu.vpod_id) is an alias for the vendor-neutral same(gpu.fabric_domain_id). Both require all selected GPUs to belong to one scoped, observed or declared block. Equal local IDs in different physical domains do not satisfy the constraint. The same block inventory supplies the existing block/tree topology and exact-GPU placement. No new topology plugin or partition-name list is needed.

Per-GPU property constraints#

The existing --constraint option also filters the actual allocated GPU identities by model, VRAM, physical domain, fabric domain, NUMA node, and PCI root. A matching node is insufficient: every selected GPU must match.

GPU property syntax#

Field

Operators

Example

gpu.type (alias gpu.gpu_type)

= !=

gpu.type=mi300x

gpu.vram (alias gpu.vram_bytes)

= != > >= < <=

gpu.vram>=192GiB

gpu.numa_node_id

= != > >= < <=

gpu.numa_node_id=0

gpu.pcie_root_id

= !=

gpu.pcie_root_id=pci0001:00

gpu.physical_domain_id (alias gpu.physical_pod_id)

= !=

gpu.physical_domain_id=rack-a

gpu.fabric_domain_id (alias gpu.vpod_id)

= !=

gpu.fabric_domain_id=15

VRAM accepts bytes or binary/decimal units such as GiB and GB; it is a per-device capacity filter, not a request to reserve a portion of GPU memory. NUMA IDs are integers greater than or equal to zero. Unknown values never satisfy a predicate, including !=. String comparisons are exact and case-sensitive. Domain ID filters use the ordinary partition and node eligibility rules; there is no separate administrator-only domain-pinning policy in this version.

Join predicates and required node features with commas or &. Expressions are limited to 4096 bytes and 32 terms. OR and grouping parentheses, apart from same(field), are rejected. Legacy feature-only constraints retain their comma-separated syntax: fast,large means both features; fast&large alone retains its previous meaning as one literal feature name.

same(field) requires one value across all selected GPUs for any field in the table. NUMA and PCI-root identities are scoped to a node, so identical local labels on different nodes do not satisfy a same-NUMA or same-root request; these two groupings are rejected at submit unless the job requests -N 1. The constraint selects GPUs; it does not pin job CPUs or memory to that NUMA node. Equality on a fabric-domain ID also requires one scoped block, avoiding accidental combinations of repeated local IDs in different racks. A file block name does not manufacture a GPU observation: direct domain-ID filters still require reported values, while same(gpu.fabric_domain_id) can use a declared block when native membership is absent.

Malformed expressions are rejected at submit, including an unknown field such as same(gpu.vpodid) and near-misses such as GPU.vram>=192GiB, same[gpu.vpod_id], or gpu.type:mi300x, which would otherwise be read as node feature names. A job_submit hook that rewrites the constraint is validated again after the hook runs.

spur submit -N 1 -n 1 --gpus 2 --constraint='gpu.type=mi300x&gpu.vram>=192GiB&same(gpu.pcie_root_id)' --wrap 'hostname'
spur submit -N 2 -n 2 --gpus-per-node 2 --topology block --constraint='gpu.vram>=192GiB&same(gpu.vpod_id)' --wrap 'hostname'
spur run -N 1 -n 1 --gpus 2 --constraint='same(gpu.numa_node_id)' hostname

These expressions require a GPU request or whole-node exclusivity. Upgrade controllers and clients together before using them. Agents from this release report FabricBindingVersion=3; older agents report 0 and are never selected for constrained placement.

The controller persists the exact selected GPU identities and property snapshots with their inventory generation. Admission and the start gate require that generation to be current. Steps inside a running allocation, including after supervisor recovery, instead re-attest the bound GPUs’ own facts, so an agent restart, a change to another GPU, or another GPU’s peer_conflict does not block them. Before launch, the agent re-reads the selected devices’ UALink attributes; other properties, such as VRAM and NUMA locality, are checked against the inventory the agent last reported. Inspection with spur control show job includes GpuPropertyNode, Generation, and StableIds. A change to a bound GPU’s own properties rejects subsequent steps of that allocation. This does not terminate GPU processes already executing when the hardware changes.

Plain GPU jobs use ordinary scheduling and can span blocks. With a same-fabric constraint, a job waits if no single block can fit it; there is no automatic fallback. --topology none cannot disable that requirement. Omitting -p selects the configured default partition, or the first partition if no default is marked; the explicit job constraint keeps the same meaning for either choice and for comma-separated partition lists. Partition names have no built-in rack meaning.

Bare --exclusive retains whole-node allocation semantics. Combined with the same-fabric constraint, every GPU reserved on an exclusive node must belong to the selected block. ExclusiveTopo=YES is rejected for partitions; reserving a node does not reserve all nodes in its block. ExclusiveTopo=NO is accepted as Slurm’s default. Steps inherit their parent allocation’s binding. Adding a GPU property constraint to an existing step is rejected; specify it when creating the allocation.

For a same-fabric job, the scheduler considers complete, consistent blocks whose candidate nodes advertise binding enforcement. It filters each node’s inventory to that block’s exact GPU IDs, applies any additional GPU predicates and grouping, and runs the existing resource and backfill solver. A property-only job applies the same filters without requiring containment in one fabric block. It chooses the earliest feasible start, preferring smaller blocks at the same start and then the scoped block key. Node lists, GPU types and quantities, CPU/memory requirements, and reservations still apply. Selected GPU IDs must remain free throughout the proposed interval. Future shadow reservations hold those same concrete resources so shorter jobs can fill gaps without consuming them at the reserved start.

The controller validates the selected block again before recording the placement. Each per-node allocation persists its block key, active endpoints, exact logical GPU membership, binding version, and inventory generation. scontrol show job displays the resulting per-node fabric binding. A failed same-fabric search reports FabricUnavailable in the pending reason description. A property-only request that cannot fit reports GpuPropertiesUnavailable; inspect spur control show node for its reported GPU facts.

The agent validates bindings before admission, then rechecks the inventory generation under the allocation lock before reserving resources. Inventory publication uses the same lock; file and device reads run outside it. AMD UALink bindings reread the allowlisted attributes of the bound PCI devices. Missing observations, changed domains/endpoints, a changed inventory generation, or an unsupported binding version reject the launch. Other discovery sources use their current typed inventory for binding checks. No device or fabric configuration is written. Capable agents start with a nonzero random inventory epoch, fencing dispatches made against an earlier agent instance.

A supervisor retains the parent binding across an agent restart, and numbered/interactive steps validate it before executing; they cannot acquire devices outside the parent allocation. Steps re-attest only the bound devices: each must still report the bound domain and endpoint, and its own endpoint must remain active. An agent restart, a generation change from other devices, a drained or stale node elsewhere in the block, or another vPod member becoming inactive does not block new steps. A change to a bound device’s own membership does, until the allocation is requeued; for file bindings, so does removing any of the allocation’s nodes from the bound block.

This is a snapshot-based placement contract. An external manager can still change a fabric after preflight or while a job is running; operators must coordinate such changes with running allocations. Observed membership is not an assertion of fabric readiness, bandwidth, security isolation, or an atomic fabric lease.

Heterogeneous job components with GPU property or containment constraints are rejected at submit until group-wide binding is supported. Constrained jobs do not trigger automatic preemption, idle-fill reclamation, or federation forwarding, whose current selection paths do not carry fabric requirements. Ordinary backfill, including future resource reservations, remains available. Segments, whole-block exclusivity, and per-device static import remain future extensions.

Optional administrator or operator block file#

For a rack system whose placement groups are maintained by an administrator or external operator and not available through native fabric discovery, set block_file as a fallback membership source:

[topology.external]
block_file = "/etc/spur/topology.conf"

Native discovery remains authoritative when block_file is configured. The file adds whole-node groups for nodes without native fabric observations; discovered vPods need not be listed. The filename has no special meaning; an absolute path is required. Install the same setting and file on the controller and agents participating in file-defined groups. Native-only agents do not require the file and use native bindings. All nodes can remain in one partition. The job constraint selects hard containment; plain GPU jobs retain ordinary scheduling.

The file is text, with one record per block:

# Two groups in one rack system
BlockName=group-a Nodes=node[01-02]
BlockName=group-b Nodes=node[03-04]

Each record requires BlockName and Nodes. Field names are case insensitive; field order is arbitrary. Node lists support bracket ranges and comma-separated names. # begins a comment and a trailing backslash joins lines. Block names use letters, digits, underscores, dots, and hyphens, with a maximum of 128 bytes. Node names use the same characters, with a maximum of 255 bytes. Files are limited to 1 MiB and 65,536 declared nodes. Duplicate fields, block names, or node memberships are errors.

This is a deliberately small subset of the block record format. BlockSizes and other fields are rejected: higher-level aggregation is not implemented, and each declared block is a boundary for jobs requesting containment. A node can belong to only one declared block. The record covers all currently registered GPUs on each named node; it cannot describe separate groups within a node. GPU counts, types, identities, free resources, and health are never taken from this file. Discovered nodes omitted from the file keep their native block membership. Nodes with neither native membership nor a file declaration cannot supply GPUs to same-fabric jobs, but remain available to ordinary scheduling.

If any named node has native fabric observations, the entire file record is ignored. An exact match with one complete native block produces an informational notice. A conflicting grouping, subdivision, mixed record, or unavailable native membership produces a warning. A file cannot merge or subdivide discovered vPods. A record mixing discovered and non-discovered nodes is ignored in full; write a separate record for the non-discovered nodes. Notices appear in spur control topology show and topology validate --format block; topology show --json includes structured notices. Reconfigure also logs them. These notices do not make healthy native blocks unavailable.

For example, if nodes 01/02 form discovered vPod 15 and nodes 03/04 have no native fabric observations, only BlockName=group-b Nodes=node[03-04] is needed. Both groups participate in the existing block/tree topology and same-fabric scheduling. A conflicting BlockName=group-a Nodes=node[01-04] would instead be ignored, leaving vPod 15 intact and nodes 03/04 undeclared.

Missing or inconsistent native observations remain unavailable for affected placement; the file cannot turn failed discovery into usable membership. During an agent’s lifetime, losing a previously observed native source retains an unavailable marker until observations recover. This history is in memory; a fresh agent cannot distinguish an absent source from hardware that never exposed it. After a restart, the administrator remains responsible for the correctness of any fallback declaration. Native endpoint coverage, heartbeat, and exact-device binding checks still apply.

If none of a declared block’s GPUs reports native membership, the file is an explicit administrator declaration. This path is vendor neutral. Spur enforces node membership and exact live GPU identities without asserting connectivity, bandwidth, fabric health, or external-manager readiness. An external operator is responsible for ensuring its declaration reflects the configured system. This does not add fabric programming or manager credentials to Spur.

Creating and validating a candidate#

An administrator can write the records directly. Export is useful for inspection or as a starting point for declarations on non-discovered nodes:

spur control topology export --format block > topology.conf.candidate
spur control topology validate --format block topology.conf.candidate

export --format block prints deterministic BlockName/Nodes records. It preserves names for file-defined groups and generates names such as block0 for automatic groups. It rejects invalid groups, a node shared by multiple groups, or unresolved GPUs on an included node. Review and validate the output before installation. It contains no inferred topology hierarchy.

validate --format block checks syntax locally, then resolves membership against the controller’s current live inventory through a read-only RPC. The original bounded text is sent unchanged, so expansion of a compact hostlist does not inflate the validation request beyond the file-size limit. Validation does not require the candidate revision to be deployed to agents, so it can be used before installation. Missing nodes, stale agents, missing GPUs, or invalid file-only membership are reported as errors. Records superseded by native discovery produce notices instead; unrelated unhealthy native blocks do not reject a valid file-only candidate. The result reports applied and ignored record counts. Exported native groups are ignored if installed while native observations are present. Success does not test fabric connectivity or install the file. Plain export and validate retain the existing JSON format; JSON manifests are inspection/validation artifacts and are not accepted as block_file input.

Deployment and updates#

For first activation, upgrade all controllers and participating agents, install the validated file on controllers and file-only agents, and restart those agents to load the new block_file setting. Reconfigure or restart the controller to enable the setting. A controller can start before nodes register; affected blocks remain unavailable until complete live inventory and matching revisions arrive. Reconfigure rejects a file-only block that names an unregistered node, a node without GPUs, or GPUs without stable identities, so wait for registration before using it for activation. A drained, down, or stale node only leaves its block unavailable and is logged; it does not block unrelated configuration changes. Records superseded by native discovery do not require matching files on native agents.

For later membership changes:

  1. Generate a complete candidate and run topology validate --format block.

  2. Coordinate with running allocations. Moving a node to another block rejects new steps of allocations bound to the old block; requeue them as appropriate.

  3. Atomically replace the file on participating agents and controllers. Use a temporary file in the same directory followed by rename; keep the directory readable by the daemon and writable only by the administrator/operator.

  4. Run spur control reconfigure on the controller. It parses and validates the complete candidate before changing partitions or the active config.

  5. Inspect spur control topology show and spur control show node NODE. Resume constrained submissions once the desired blocks have no issues and the participating agents report the expected revision.

There is no automatic controller filesystem watch. Agents reread the file through the existing inventory refresh/debounce loop and directly before admission and new steps. The controller retains a parsed immutable snapshot until reconfigure/restart. A malformed or missing candidate causes reconfigure to fail while preserving the previous configuration. A missing, malformed, or different agent file fences that agent’s declared block; another valid block can still schedule. Native agents ignore file revisions, so file changes do not change their inventory generation or invalidate their native bindings. Restore the matching file and wait for inventory refresh to make the affected block available again.

Revisions hash normalized block names and expanded node memberships. Comments, whitespace, field order, and record order do not change the revision. Membership or block-name changes do. During rollout, revision differences deliberately make affected file-only placement unavailable. Native blocks remain independent of file revisions. The revision covers the whole file, so an edit to any record fences new placements and dispatches made against an older revision. Steps of a running file-bound allocation instead re-read the file and require every node of the allocation to remain in the bound block. Plan updates as a coordinated operation. Step dispatch is not an atomic multi-node barrier: during partial rollout, a step may start on nodes that still validate the old binding while another node rejects it. Quiesce affected allocations before changing membership. Existing running processes are not moved or killed by a file update.

High-availability deployments follow the existing controller configuration model: reconfigure applies on the active leader, and does not replicate file contents or local configuration to followers. Deploy matching files and configuration to every controller and restart followers with that configuration before relying on failover. This feature adds no distributed file rollout or atomic fabric lease.

Scheduling and inspection#

File blocks use the same backfill algorithm and exact-device admission checks as automatic blocks. Earliest feasible start wins, then smaller blocks, then block-key order. A smaller job may share a block when its CPUs, memory, and GPUs are free. A constrained job cannot combine two blocks even if their summed free GPUs meet its request. --exclusive still reserves whole nodes, not a whole block.

For four nodes without native fabric observations, four GPUs per node, and the two records above:

spur submit -p gpu -N 2 -n 2 --gpus 6 --constraint='same(gpu.fabric_domain_id)' -t 00:10:00 --wrap 'sleep 300'
spur submit -p gpu -N 2 -n 2 --gpus 6 --constraint='same(gpu.fabric_domain_id)' -t 00:10:00 --wrap 'sleep 300'
spur submit -p gpu -N 4 -n 4 --gpus 12 --constraint='same(gpu.fabric_domain_id)' -t 00:10:00 --wrap 'sleep 300'
spur control topology show
spur control show job JOB_ID

The first two jobs can use different eight-GPU blocks. The twelve-GPU job stays pending because it cannot fit within either block. Omitting -p uses the configured default partition; the constraint is independent of that default. topology show includes Source=file and Revision for file-only blocks. Native blocks retain their native domain IDs. Job inspection includes BlockFileRevision and BindingVersion=2 for file bindings, or BindingVersion=1 for native fabric bindings, alongside exact per-node GPU bindings. The constraint uses the existing submission flag; no job ID environment variable is required when submitting a new job.