DPDK: escaping the kernel

The Linux network stack is a marvel of generality. It speaks every protocol, handles every corner case, and is exactly the wrong tool for an appliance whose entire job is to move packets from one wire to another as fast as physically possible.

The cost of asking the kernel to forward

Follow a single packet through a conventional kernel path and count the overhead:

• The NIC raises an interrupt on arrival. At line rate that interrupt fires millions of times a second, and each one is a context switch out of whatever the CPU was doing.
• The frame is DMA’d into a kernel socket buffer, then copied across the kernel/userspace boundary when your application reads it.
• Reading and writing means syscalls — more context switches, more cache pollution.
• The generic stack walks every layer (netfilter hooks, routing lookup, qdiscs) for every packet, whether you need it or not.

None of this matters at a few thousand packets per second. But 10 GbE line rate is 14.88 Mpps for 64-byte frames — that leaves roughly 67 nanoseconds per packet. A single cache miss can blow that budget. At 25 and 100 GbE the per-packet overhead of the generic stack simply dominates; the CPU spends all its time on bookkeeping and none on forwarding.

DPDK: give the NIC to userspace

DPDK — the Data Plane Development Kit — inverts the model. Instead of the kernel owning the NIC and lending you packets, the application owns the NIC directly.

The device is unbound from its kernel driver and rebound to a userspace-capable one — vfio-pci, or uio_pci_generic when the kernel isn’t locked down. From there a poll-mode driver (PMD) maps the NIC’s descriptor rings straight into the application’s address space and reads them in a tight loop. No interrupts. No socket buffers. No trip through the generic stack.

The techniques that make this fast:

• Poll-mode instead of interrupts. A pinned core spins reading the RX ring. There is no interrupt latency because there are no interrupts.
• Hugepages. Packet buffers and rings live in 2 MB or 1 GB pages, so the TLB covers gigabytes of memory with a handful of entries — far fewer misses on the hot path.
• Zero-copy DMA. The NIC DMAs frames directly into userspace mbuf structures. The application processes the same bytes the hardware wrote; nothing is copied to hand it over.
• Run-to-completion on pinned cores. A packet is received, classified, transformed, and transmitted on one core without ever being handed to a scheduler.
• Burst RX. The driver pulls a batch of packets per poll, amortizing per-call cost and keeping the instruction cache hot.

The tradeoff: a core that never sleeps

There is no free lunch. A poll-mode core sits at 100% CPU whether one packet or fourteen million are arriving — it is always spinning, asking “anything yet?”

You trade an entire CPU core for deterministic, sub-microsecond latency and line-rate throughput. On a general-purpose server that would be wasteful. On a firewall / router / load-balancer whose whole reason to exist is forwarding, it is exactly the right trade.

How SiHA wires it up

On a SiHA appliance, data NICs bind to DPDK by default — kernel bypass isn’t an opt-in tuning knob, it’s the baseline. A few specifics fall out of running on an immutable, secure-boot appliance OS:

• Hugepages are reserved at boot via the kernel command line, before anything else claims memory, so the pool is always available and never fragmented.
• The bind driver auto-selects. Under enforced SecureBoot the kernel is locked down, which blocks uio_pci_generic’s PCI BAR mapping — so SiHA binds vfio-pci and goes through the IOMMU. When the kernel isn’t locked down (the debug build), it falls back to uio_pci_generic. The appliance figures this out itself by reading the lockdown state; the operator declares an interface, not a driver.
• An af-packet fallback stays available for debugging. When you need the kernel to see the frames — for a quick tcpdump, or to isolate whether a problem is in the dataplane or the wire — you can flip an interface to af-packet mode and give it back to the stack. It’s slow, and that’s fine, because it only exists for when you’re staring at something that shouldn’t be happening.

What runs on top

DPDK gets packets into userspace at line rate — but a raw poll loop isn’t a firewall. Something has to classify, route, NAT, and load-balance those mbufs once they arrive. That something is VPP, the vectorized forwarding engine SiHA drives as its dataplane, and it’s where the story gets interesting.

More on how VPP turns a stream of raw frames into a declaratively-configured router over on the blog.