Overview

VPP-native boot

How a node brings up its management network entirely inside VPP — the seeded, seedless (DHCP every NIC), and factory tiers, how they're selected, and how discovery finds a fresh node.

A SiHA node has no kernel networking for management: no dhclient on an interface, no cloud-init writing /etc/network, no SSH. The supervisor (PID 1) brings the management plane up inside VPP, before the control plane starts — so the very socket you reach the node on rides the dataplane.

The shape of it

At boot, PID 1 resolves the management NIC(s), hands each to VPP as a DPDK port, and builds an L3 path per NIC to a kernel tap that apid listens on:

A dedicated management VRF — table 4096. Every management address lives in its own routing table, isolated from the data interfaces and untouched by the control-plane’s interface controller.
One LCP tap per NIC — mgmt0, mgmt1, … VPP’s Linux Control Plane mirrors each VPP interface to a kernel netdev and punts traffic destined to its address up to it. These are the only management netdevs the kernel ever sees.
apid on :6443. The gRPC front door binds the taps; every client reaches the node over mTLS through any of them.

A node may declare more than one management NIC — static, DHCP, or a mix. Each is brought up independently into VRF 4096 with its own tap, and every address it produces goes into apid’s certificate SANs, so the node answers on any of them. If one NIC fails to come up, the others still do (best-effort).

Where the address comes from is the only thing that varies — so there are really two DHCP-capable tiers, seeded (the seed names the NICs; static, DHCP, or a mix) and seedless (no seed at all → DHCP every NIC), plus a factory fallback for when nothing leases.

The seeded tier — static, DHCP, or a mix

When the platform seed names one or more management NICs, PID 1 brings up each of them independently: adopt the port as a DPDK PMD, bind it to VRF 4096, give it an address, and tap it (mgmt0, mgmt1, …). How a NIC gets its address is a per-NIC choice — a fixed static address or a DHCP lease — so a single node can run some NICs static and others DHCP at once. Every resulting address lands in apid’s certificate SANs, so the node answers on any of them. The data NICs are left alone — the control plane adopts them later, from the first DataplaneConfig.

adopt PMD — VPP binds the port as a DPDK interfacelcp punt — Linux-CP mirrors the VPP address to the kernel tapone static, one DHCP — each NIC is independent, same VRF

Static — the seed carries the address (the network-config delivered over the vsock/9p reader); PID 1 assigns it directly to the VPP interface.

DHCP — the seed names the link but not the address. The catch: LCP is push-only, VPP → kernel, so a kernel dhclient lease on the tap would never flow back into VPP — VPP wouldn’t own the address nor answer ARP for it. So the DHCP client runs inside VPP, one per NIC: it acquires the lease straight onto the VPP interface, in VRF 4096, and LCP mirrors it out to the tap.

The seedless tier — DHCP every NIC

Sometimes there is no seed at all: no config_drive, no metadata, no vsock user-data — a bare factory node, or a cloud whose metadata service never answers. Rather than pick one NIC and hope, PID 1 synthesizes a DHCP request on every PCI NIC and runs the exact same fan-out as a DHCP seed: one VPP DHCP client per port, into VRF 4096, each tapped.

The rule is simple — an interface is management if and only if it has an IP. Every NIC that wins a lease becomes a management interface (its address in the cert SANs); a NIC that gets none within the timeout is left for the dataplane. A two-NIC node on two DHCP segments therefore comes up reachable on both — the same end state as a seed that had named both, discovered rather than declared.

adopt PMD — VPP binds the port as a DPDK interfacelcp punt — Linux-CP mirrors the VPP address to the kernel tapleases — a DHCP client per NIC, inside VPP

One default route. N leased NICs would otherwise each install a 0.0.0.0/0 in the shared management VRF — ECMP that some cloud fabrics’ per-port anti-spoofing silently drops. So only the lowest-PCI leased NIC keeps its default route; every other management NIC is connected-route only. Deterministic, and it keeps “adopt on the primary” working.

Timeout. Each NIC’s DHCP wait is bounded (30 s) so a slow relay never mis-classifies a real management NIC as data. The NICs are tried in turn, so a node with many DHCP-less ports simply takes longer to settle.

If no NIC leases once every port has been tried, the node drops to the factory fallback below — the only posture where all NICs become one surface.

Fallback — factory / plug-anywhere

A brand-new node with no seed and nothing leased on any NIC still shouldn’t sit dark. Rather than that, PID 1 bridges every NIC into one L2 domain and gives it a loopback BVI holding a break-glass 192.168.1.1/24 in VRF 4096. Plug a cable into any port and you reach .1 — the operator adopts from there.

bridge — every port joined into one L2 domainlcp punt — Linux-CP mirrors the VPP address to the kernel tapbreak-glass gateway

This tier is strictly the factory posture. An adopted node that somehow has no management address is treated as a fault (a “degraded” state), never silently bridged — bridging every port on a live node would be a rogue gateway.

How a tier is chosen

The tier is selected from the seed and the node’s adoption state; the seedless→fallback split is then decided at runtime, after the DHCP fan-out has drained. Highest precedence first:

Condition	Tier
seed names one or more management networks	seeded — each NIC static (it has an address) or DHCP
no seed, node not adopted, at least one NIC leases	seedless — DHCP every NIC; the leased ones become management
no seed, node not adopted, no NIC leases	fallback (`.1`)
else (adopted, but no management address)	degraded — error, do not bridge

The chosen tier logs plainly on the serial console (tier=seeded / tier=fallback, each NIC logging mode=static or mode=dhcp); a seedless boot additionally logs one mgmt-probe … result=leased|timeout line per NIC, so which cable came up is readable straight from the console. Every management IP the tier produces feeds the server certificate’s SANs before the PKI is minted — so the cert is valid on every address you’ll actually reach the node on.

Cloud or metal — the image says which

The ladder itself is selected by a platform stamp baked into the image: siha.platform=cloud|metal on the kernel command line, inside the signed UKI — so the boot posture is part of what SecureBoot attests, not a runtime guess.

cloud (the default) runs the full ladder above: look for a platform seed first (config share, config drive, metadata service), then DHCP, then fallback. This is the posture for any environment that can hand the node its identity at first boot — a virtualized platform, a cloud tenant, a lab hypervisor.
metal skips the seed machinery entirely — on bare hardware there is nothing to probe, so the node never wastes boot time looking. DHCP on every NIC is the nominal path (logged plainly, never as an anomaly), the factory fallback (192.168.1.1 on every port) is the safety net, and enrollment is always discover → adopt: rack it, cable it, adopt it — pure zero-touch provisioning, no per-node preparation of any kind.

Why a boot-time stamp rather than runtime detection? Because the two postures disagree about what silence means. On cloud, “no seed found” is a warning-worthy anomaly (the platform should have provided one); on metal it is Tuesday. Baking the intent into the signed image makes the node’s behavior deterministic and auditable instead of heuristic.

Both stamps converge on the same end state; metal just gets there without ever probing for a seed. A node that A/B-boots across stamps keeps its identity — adoption and PKI survive — while its network-shape state (management set, fabric MTU) is reset for re-detection.

⚠️ What is supported today. cloud is the validated, supported posture: it is exercised end-to-end on virtualized and cloud deployments (seed, DHCP and fallback tiers, adoption, config apply, upgrades). metal is experimental for now: the code path is complete and continuously tested on virtual machines, but it has not yet been validated on physical appliances (real NICs, hardware IOMMU, enforced SecureBoot on real firmware). Treat metal images as preview material until that validation lands — and expect this note to disappear when it does.

Surviving a dataplane restart

Pushing a DataplaneConfig re-renders VPP’s startup.conf and restarts VPP with a health-checked, last-known-good two-phase apply. A VPP restart wipes the entire dataplane — including the management PMDs, VRF 4096 and the mgmt taps. So PID 1 re-runs the same bring-up after the restart, restoring the management plane in place; the node stays reachable across a config apply instead of vanishing until the next reboot.

Day 2 — trimming the management surface

At boot a node comes up reachable on every management NIC the seed named — deliberately: during adoption you may not yet know which cable you’ll patch. Once adopted and sure of your topology, you scope management down to the NIC(s) you actually want with a ManagementInterfaces resource — the list of ports that stay management. Every NIC you drop is released back to a plain data port: it lands in table 0 with no address (a harmless orphan), free for you to declare as an Interface{kind: pmd} and use for traffic. The re-wire is live — no VPP restart, no reboot, no lost session on the port you keep.

# ManagementInterfaces — singleton (id: default), reconciled live
interfaces:
  - pci: "0000:05:00.0"        # keep management here (or match by `mac:`)

A NIC is named by pci (its BDF) or mac — never a kernel name (udev-unstable). Each must be a currently-bound PMD; an unbound BDF or a set that resolves to zero reachable NICs is rejected before anything is re-wired. Access control on the surviving interface is your ordinary ACL / ACLBinding (the management interface is published as a COSI Interface so a binding can target its sw_if_index) — defense-in-depth on top of mTLS and the management VRF.

Recovery — recreate, or break-glass. A management-set change reconciles live, after the health-gated A/B upgrade already committed, so a mistake here is outside auto-rollback’s reach. There’s no in-place auto-revert: siha nodes are cattle, so the recovery model is to re-provision — the node comes back on its seed’s wide management set — not to nurse a stuck one. Two guards keep a mistake cheap: a set resolving to zero reachable NICs (or an unbound BDF) is rejected before any re-wire, and if you still lock yourself out the break-glass path (serial console + writable-partition fallback IP) fixes it in place.

Discovery — finding a fresh node

You can’t reach a factory node by name yet, and its management plane lives entirely in VPP — there’s no kernel interface to probe. sihactl discover solves this with an all-hosts multicast solicit:

The solicit goes to 224.0.0.1:15353. VPP auto-installs a has-local multicast route for 224.0.0.1 (the management tap is an implicit member), so the packet rides ip4-mfib → ip4-punt → punt-redirect down to mgmt0, where the supervisor’s responder answers with the node’s identity — its management IP, CA fingerprint, and whether it’s still un-adopted.

Why not a subnet broadcast? VPP’s Linux Control Plane only punts local-unicast and multicast to the tap — a subnet-directed broadcast is dropped at the FIB, before any local delivery. A broadcast solicit simply never reaches the responder on a VPP-native node; multicast is the path that does.

Because discovery is multicast, several un-adopted nodes on the same segment all answer — you see every node waiting to be adopted, which is exactly the point.

Next: adopt the node and apply your first resources.