VPP: a graph of packet nodes

Forwarding packets fast is less about clever algorithms than about respecting the machine. Modern CPUs hate surprises: a cold instruction cache, an unpredictable branch, a memory access the prefetcher didn’t see coming — each one stalls the pipeline for tens or hundreds of cycles. At 10 GbE line rate, that’s 14.88 Mpps for 64-byte frames, roughly 67 nanoseconds per packet. You do not have cycles to waste on cache misses.

VPP (Vector Packet Processing) is the dataplane at the heart of SiHA, and its entire design is built around keeping the CPU happy. Two ideas do the heavy lifting: vectors and the graph.

Batches, not packets

A naive router processes one packet at a time: pull a packet off the NIC, run it through the whole forwarding path — parse Ethernet, look up the route, apply ACLs, rewrite headers, transmit — then start over. Every packet re-walks the same code, and every packet pays the cost of pulling that code back into the instruction cache.

VPP inverts this. It takes packets off DPDK in bursts and processes them as a vector — a batch of up to 256 packets that travels together through the forwarding path. Instead of running the entire path for one packet, VPP runs one stage of the path for the whole vector, then hands the vector to the next stage.

The graph

That forwarding path is a directed graph of nodes. Each node is a small, focused piece of code: ethernet-input parses L2 headers, ip4-lookup does the FIB lookup, ip4-rewrite writes the new headers, and feature nodes like acl and nat44 do exactly what they say. A packet’s journey is a walk through this graph, node to node, and the vector flows through it as a unit.

The node is the unit of work, and the vector is the unit of scheduling. A node’s inner loop looks like “for each packet in this vector, do my one job.”

Why vectors are fast

This is the core insight, and it isn’t magic — it’s cache efficiency.

• Hot instruction cache. A node’s code is loaded into the I-cache once per vector, then executed over all N packets. The naive one-packet-at-a-time loop reloads the entire forwarding path’s code for every single packet. Amortizing the code fetch across a batch is where most of the win comes from.
• Working prefetcher. Because the node knows it has N packets queued, it can prefetch the data for packet i+1 while working on packet i. Memory latency gets hidden instead of stalling the core.
• Amortized fixed costs. Function-call overhead, loop setup, per-node bookkeeping — all divided across the vector instead of paid per packet.

The beautiful part is that this gets better under load. A busier interface delivers larger bursts, which means bigger vectors, which means better amortization. VPP’s efficiency rises exactly when you need it most.

Plugins are graph nodes

Features in VPP are just nodes you splice into the graph. This is what makes it a platform rather than a fixed pipeline. dpdk_plugin feeds packets in from the NIC; acl_plugin, nat44, and lb_plugin add filtering, translation, and load-balancing nodes; linux_cp bridges to the host kernel. Enabling a feature on an interface is, mechanically, wiring a node into the path those packets take.

How SiHA drives it

VPP runs entirely in userspace as a single process — no kernel modules in the fast path. On the SiHA appliance it’s supervised as a child of the PID-1 supervisor, cycled and health-gated like any other daemon.

The SiHA control plane programs VPP over its binary API using govpp. Each feature has a thin Go wrapper — interfaces, ACLs, NAT, LB — that translates declarative COSI resources into the corresponding binary-API calls. Those wrappers are also where we absorb a few upstream VPP quirks so the rest of the control plane doesn’t have to know about them; the wrapper files document each workaround with a Why: note.

The dataplane’s own configuration — VPP’s startup.conf — is owned by the control plane, not hand-edited on the box. When an operator pushes a DataplaneConfig, SiHA re-renders the file and restarts VPP with a health-checked, two-phase apply: snapshot the current config as last-known-good, write the new one, restart, then probe. If VPP doesn’t come back healthy, SiHA restores the last-known-good config and refuses to retry the failing revision until the operator edits it. The dataplane never gets stuck in a crash loop because someone typo’d a config.

The whole design is a bet that forwarding speed comes from cache discipline, and that safety comes from a declarative control plane that treats the dataplane config as code — not from clever tricks in the hot path.

Want to see how the control plane and dataplane fit together end to end? Read the architecture overview.