SONiC-VPP, Commissioned
Volume II of the SONiC-VPP series

SONiC-VPP, commissioned.

Volume I took the router apart. This volume puts it into service: how the image is built, what runs between power-on and the first forwarded packet, where the Linux kernel sits on the packet path, how the VM and container builds differ, what is actually supported, how to tune it, and how it behaves on virtual NICs and on Compute Engine.

SONiC-VPP PLATFORM=vpp  x86_64
Dataplane
FD.io VPP, packaged 2606-0.10 (upstream 12154be, 16 Sep 2026)
Packet I/O
DPDK 26.07, bundled by VPP
Base OS
Debian 13 trixie
SAI library
libsaivs, switch type VPP (SwitchVpp)
SAI→VPP code
21.7k lines C++ + 8.4k lines C
VPP patches
19 carried on upstream
Custom plugins
sonic_ext, tunterm_acl, ip_validate
Release branches
202511, 202605
Build targets
KVM image, ONIE, raw, single container
Test topologies
t0-vpp, t1-vpp, t1-lag-vpp, dualtor-vpp, dualtor-aa-vpp
Read from sonic-platform-vpp main @ 21bb1c5 and sonic-sairedis vslib/vpp, 9 Oct 2026
Previously in this series

Volume I: SONiC, disassembled.

Twelve atoms from NOS to memif, the five Redis databases and seven containers, a BGP route traced from fpmsyncd to the VPP graph, the three packaging form factors, the LCP interface mapping, a five-platform comparison, and the fidelity ladder.

This volume

Systems and strategy. The atoms below are only the ones Volume I did not cover. Everything here was checked against source code, not just documentation, because the docs in this project trail the code by a year in places.

Vocabulary carried forward from Volume I
SAI.
Switch Abstraction Interface, the vendor-neutral C API SONiC uses to program any dataplane.
APPL_DB / ASIC_DB.
Redis databases holding desired state in SONiC terms, and the same state translated into SAI objects.
orchagent.
The daemon that turns APPL_DB entries into SAI objects in ASIC_DB.
syncd.
The process that applies ASIC_DB to a real dataplane through a vendor SAI library.
fpmsyncd.
Receives routes from FRR's zebra and writes them to APPL_DB.
DPDK, PMD.
Data Plane Development Kit; its poll-mode drivers read NIC rings from userspace without interrupts.
VPP, graph node, vector.
Vector Packet Processing moves batches (vectors) of up to 256 packets through a directed graph of small processing nodes.
TAP, veth.
Kernel virtual interfaces: a TAP connects the kernel to a userspace program; a veth pair is a virtual cable between namespaces.
LCP.
VPP's Linux Control Plane plugin. It pairs each VPP interface with a kernel TAP so Linux daemons see a normal interface.
SwitchVpp.
The SAI implementation for VPP. Since SONiC 2025.05 it lives in sonic-sairedis vslib/vpp as a subclass of the virtual switch, not as a separate libsaivpp.
§ 01Atoms

The new parts bin

These are the parts you meet as soon as you build, boot or tune SONiC-VPP. Tap one for its definition. Later sections use these words without stopping to define them again.

§ 02Components

Bill of materials

Five repositories produce SONiC-VPP. Only two contain code written for it: sonic-platform-vpp, which holds the build rules, the VPP patches and plugins, the containers and the boot scripts, and sonic-sairedis/vslib/vpp, which translates SAI calls into VPP API messages. Everything else is stock SONiC or stock VPP.

FD.io VPP upstream pinned SHA 12154be DPDK 26.07 inside sonic-platform-vpp patches, plugins, rules, dockers, boot scripts VPP .debs (10) 2606-0.10+b1sonic1 Buildkite or local build sonic-sairedis vslib/vpp: SwitchVpp* vppxlate (C) libsaivs + syncd-vs switch type VPP selected by sai.profile docker-syncd-vpp vpp + syncd, privileged /dev/hugepages mounted stock SONiC dockers swss, bgp, teamd, database, lldp, snmp… sonic-buildimage, PLATFORM=vpp sonic-vpp.img.gz · .bin · .raw · docker-sonic-vpp.gz sonic-mgmt nightly: t0-vpp, t1-lag-vpp on KVM apply series 19 patches, 3 plugins compiles into vpp-dev headers vpp debs libsaivs
The only hard coupling is the dashed arrow. libsaivs is compiled against the exact VPP API headers in the VPP packages, so a VPP package and a libsaivs built from different VPP sources disagree about message layouts at runtime. The package version string exists largely to prevent that.

What runs, and where

vpp
The dataplane process, inside the syncd container. Started by vpp_init.sh under supervisord. Owns the data NICs, the forwarding tables and the LCP taps.
syncd + SwitchVpp
Same container. Reads ASIC_DB and calls SwitchVpp, which keeps its own object state and turns SAI create, set and remove calls into VPP API requests.
vppxlate
The C layer under SwitchVpp. Builds about 70 kinds of VPP binary-API messages, holds the request socket and a separate event socket, and reads counters from the stats segment.
vpp-cfg-init
A factory-default hook on the host, shipped in the platform-modules-vpp package. Runs on first boot, discovers the data ports and writes the port plan.
vpp_ports_setup.sh
Run by vpp-port-config.service before syncd starts. Moves each data NIC from its kernel driver to a userspace I/O driver, and back on stop.
vppcfgd
Optional. Renders VPP's startup.conf from a PLATFORM|vpp entry in CONFIG_DB when VPP_CONF_DB=y. Packaged in the single container.
docker-sonic-vpp
The single-container build: Redis, the swss daemons, syncd with VPP, FRR, teamd and vppcfgd under one supervisord.
docker-gbsyncd-vpp
The gearbox (external PHY) syncd container, built for parity with other platforms. It has no physical PHY to drive here.
The patch and plugin ledger (19 patches, 3 plugins)

Every VPP change SONiC carries, from vppbld/patches/series and vppbld/plugins/src. Gaps in numbering are retired patches.

Build plumbing

0001Use the libnl SONiC already ships; drop tshark from VPP's build dependencies.
0002Make xdp-tools build against the libbpf VPP pulls in.
0005Allow a custom version string, so packages carry +b1sonic1.
0006Fix a curl invocation that passed conflicting output flags.

Making VPP behave like an ASIC with a CPU port

0004linux-cp: remove the ethertype registration conflict check, so the host path can claim ethertypes other plugins also register.
0010Startup option to stop installing the class E (240/4) drop route.
0021Send TTL-expired transit packets to a sonic_ext punt node instead of having VPP generate the ICMP reply, when SONiC installed a TTL_ERROR trap.
0023Startup option to skip VPP's local source-address check.

Features

0003Export vxlan_main so other plugins can reach VXLAN state.
0009Multipoint-to-point IP-in-IP tunnels, for dual-ToR decapsulation.
0012-14sFlow per-port sample rate, per-port direction, and the APIs for both.
0017VXLAN VNET decap that matches on local address and VNI regardless of outer source, for secondary-VTEP designs.
0018ACL rules that match on ingress interface, so SAI's IN_PORTS qualifier works instead of being dropped.
0019An ACL permit-and-mirror action for Everflow.
0020Run ethernet-output features on L2 adjacencies, so ERSPAN GRE headers can be fixed up without a custom GRE API.

Load balancing

0011Opt-in hashing on the inner headers of IP-in-IP, GRE and NVGRE, for ECMP and LAG.
0015Hash the inner 5-tuple when encapsulating VXLAN, so tunnelled flows spread across underlay paths.

Plugins

sonic_extThe SONiC behaviour plugin. Punts via the original LAG or VLAN member, cross-connects host traffic straight to the wire, sends IPv6 ND to the kernel, polices punted traffic per CoPP class, redirects unresolved-next-hop packets to the kernel, filters foreign VLAN tags, and fixes up mirror encapsulation. Each feature can be turned off with SONIC_EXT_CONFIG.
tunterm_aclApplies an ACL after VXLAN decapsulation, matching inner destination and redirecting.
ip_validateDrops unicast packets with impossible addresses (loopback, multicast or class E sources, link-local, unspecified) on every interface.
What the ledger says

Only a third of the carried code adds features. The rest changes VPP defaults that a switch ASIC does not have: VPP answers IPv6 neighbor discovery itself, sends its own ICMP errors, resolves next hops itself and drops some addresses on principle. SONiC expects the hardware to trap those packets to the CPU and let Linux decide. Much of SONiC-VPP's engineering goes into making a software router act like silicon with a CPU port.

§ 03Components

From source to image

SONiC-VPP is a platform inside SONiC's normal build, the way a switch vendor is. You configure the build for PLATFORM=vpp, and the build pulls in platform/vpp, a git submodule pinned to a commit of sonic-platform-vpp.

  1. Fetch and pinmake init checks out all submodules, including platform/vpp at the commit sonic-buildimage pins.
  2. Build the build containersmake configure PLATFORM=vpp builds the Debian trixie (and bookworm) sonic-slave images. This is most of the first-build time; later builds reuse them.
  3. Get VPP packagesvppbld tries to download ten .deb files named by VPP_VERSION_SONIC from the sonic-vpp Buildkite registry. On a miss it clones gerrit.fd.io, checks out the SHA in vppbld/vpp_version, copies in the three plugins, applies the 19 patches and runs VPP's own make pkg-deb with the Mellanox PMDs turned off.
  4. Compile the SAI librarysonic-sairedis builds libsaivs, including vslib/vpp, against the VPP development headers. On trixie the VPP packages are declared build dependencies of libsairedis for this reason.
  5. Assemble containersdocker-syncd-vpp comes from SONiC's trixie syncd template plus libsaivs and syncd-vs. Every other container is the standard SONiC one.
  6. Assemble the installerThe image adds the platform-modules-vpp package (first-boot hook, port service, loaders) and loads psample for sFlow and mpls_router for MPLS at boot.
KVM image (the one CI tests)
git clone --recurse-submodules https://github.com/sonic-net/sonic-buildimage.git
cd sonic-buildimage
make init
NOBULLSEYE=1 NOBUSTER=1 make configure PLATFORM=vpp
NOBULLSEYE=1 NOBUSTER=1 make SONIC_BUILD_JOBS=4 target/sonic-vpp.img.gz
# output: target/sonic-vpp.img.gz   (CI also builds INSTALL_DEBUG_TOOLS=y)
Single container
sudo modprobe ip_tables iptable_nat iptable_filter   # needed by docker-in-slave
NOBULLSEYE=1 NOBUSTER=1 BUILD_SKIP_TEST=y INCLUDE_FIPS=n \
  make SONIC_BUILD_JOBS=4 target/docker-sonic-vpp.gz
sonic-vpp.img.gz
KVM disk image

Full multi-container SONiC, installed through an ONIE recovery ISO inside QEMU during the build. This is what sonic-mgmt tests every night.

sonic-vpp.bin
ONIE installer

The same image for x86 servers that boot ONIE. Not exercised by the public nightly runs.

sonic-vpp.raw
raw disk image

The same image as a raw disk, for platforms that import raw disks rather than qcow2.

docker-sonic-vpp.gz
single container

A reduced SONiC in one container: no SNMP, LLDP, telemetry or platform monitoring. It needs INCLUDE_FIPS=n and has no sanity tests in CI.

The version string is a cache key

VPP_VERSION in rules/vpp.mk (today 2606-0.10) names the packages, and the build fetches prebuilt packages by that name. Change a patch or a plugin without bumping the minor number and every downstream build silently downloads the old packages, while libsaivs compiles against your new headers. The result is a VPP and a SAI library that disagree about message CRCs. The rule in vpp.mk is to bump on any change under vppbld/.

Also: the packages are labelled 2606, but the pinned commit is from the 26.10 development cycle on VPP master. The label tracks the base release; the SHA is what actually ships.

§ 04Systems

Power-on to first packet

On the KVM image, eight things happen in order between the hypervisor handing over NICs and VPP forwarding. Most problems people hit, such as ports missing, VPP not starting, or the second port being named Ethernet4, trace back to one of these steps.

Boot sequence
Step through it. Each step shows the file or state it produces.


    
The port plan is written once

syncd_vpp_env, the ifmap and the initial config_db.json come from a factory-default hook. They are not regenerated on reboot. If you add, remove or reorder data NICs after first boot, VPP will be started with a stale PCI list. Regenerate by re-running the hook (or config-setup factory) rather than editing the environment of the syncd container, which reads the file through its /etc/sonic mount.

§ 05Systems

The kernel boundary

VPP forwards transit traffic. The Linux kernel never does. But the kernel still owns everything SONiC's daemons need it to own: the neighbor tables, the sockets for BGP, LACP through teamd, ICMP errors, and the routes FRR installs. Every packet that needs one of those crosses a TAP. Pick a packet and watch where it goes.

Packet path tracer
Node names are the real VPP graph nodes in the pinned build.

    Why the kernel FIB doesn't matter

    zebra still installs every route into the kernel, so ip route inside SONiC looks complete. Nothing on the transit path reads it. VPP's linux-nl plugin, which would mirror kernel changes into VPP, is turned off (NO_LINUX_NL=y). The only way state reaches VPP is fpmsyncd → APPL_DB → orchagent → ASIC_DB → SwitchVpp. The kernel FIB is used only for packets the router itself originates, which leave through a TAP and are cross-connected to the wire.

    How each interface type is wired

    Front-panel port
    VPP bobm0, created from startup.conf, paired with TAP Ethernet0. Direct pair.
    PortChannel
    VPP BondEthernet10 paired with helper TAP be10. A tc filter redirects be10's ingress to the kernel's PortChannel10, where teamd runs LACP. sonic_ext sends LACP frames up the member's own TAP so teamd knows the arrival port.
    Loopback
    VPP loop0 paired with tap_Loopback0, tc-redirected to Loopback0.
    Sub-interface
    bobm0.10 and Ethernet0.10 are created automatically under the parent pair (lcp-auto-subint).
    VLAN interface
    A VPP bridge domain with a BVI. Punted traffic goes up the TAP of the member port it arrived on.

    Other kernel touchpoints

    TAP transport
    VPP's TAP interfaces are virtio devices backed by the kernel's vhost-net, so punts and host-originated packets cost a vhost round trip.
    NIC drivers
    uio_pci_generic or vfio-pci hold the data NICs. Kernel networking never sees them again until syncd stops.
    sFlow
    VPP samples in its own graph and sends samples to hsflowd over the kernel's psample netlink channel, which the image loads at boot.
    MPLS
    The image loads mpls_router so intfmgrd can enable MPLS on the TAPs; the labels themselves are programmed into VPP.
    Namespaces
    On the VM image SONiC's containers share the host network namespace, so TAPs created inside syncd are visible to swss, bgp and teamd. In the single container, one pause container holds the namespace for everything.
    § 06Systems

    VM or container

    Volume I compared the packaging options at a glance. The difference that matters in practice is where the kernel is and how ports get in. A VM has its own kernel and discovers its NICs at boot. The container borrows the host's kernel and is handed its ports by a launch script.

    KVM image guest: Debian 13, its own kernel bgp swss database syncd+ vpp guest kernel: TAPs Ethernet0, Ethernet4 … eth0 mgmtkernel driver eth1…ethN (PCI)uio_pci_generic → DPDK bobmN Single container host kernel (shared with everything else on the host) netns held by pause container "<name>-nss" docker-sonic-vpp (privileged, supervisord) redis, swss, syncd+vpp, FRR, teamd… veth (moved into netns)af_packet: host-vethXDPDK off if all veth PCI NICvfio-pci bound on hostDPDK: bobmN
    Same SONiC, same VPP. What changes is who owns the kernel (the guest or the host), who binds the NICs (a systemd service in the guest or a launch script on the host), and whether ports are PCI devices or veths read through AF_PACKET.
    § 07Systems

    What's wired

    Status here is graded on evidence in code and tests, not roadmap. "Supported" means the SAI translation exists and sonic-mgmt or plugin tests exercise it. "Partial" means code exists but coverage is thin or explicitly skipped. The project README's own feature list dates from the first phase and is years behind the code.

    Where SONiC-VPP sits among SONiC flavors

    § 08Systems under load

    Tuning for throughput

    VPP's speed comes from three things: forwarding threads pinned to dedicated cores, each NIC queue polled by exactly one thread, and packet buffers in hugepages. SONiC-VPP ships tuned for small lab VMs, not for throughput. Its defaults are one worker, one receive queue per port, 2,048 buffers per port and a couple dozen megabytes of 2 MB hugepages. Every one of those has to change before the numbers look like VPP's published ones.

    The cost of polling

    A worker spins at 100% of a core whether or not traffic flows. In September 2026 the project cut the VM default from two workers to one after measuring a two-DUT KVM testbed: VPP CPU per DUT fell from 282% to 178%, host CPU for both DUTs fell by about a third, and a flaky BGP test began passing because the control plane got its cores back. The virtio PMD used in KVM testbeds cannot run in interrupt or adaptive mode under VPP, so worker count is the only lever.

    Core and buffer planner
    Lay out a VM's cores, then check the buffer pool and hugepages against what SONiC-VPP's launch script provides.
    OS, IRQs VPP main worker, polling worker, no queue SONiC control plane
    Generated configuration
    
        

    Reference points, not promises: on Compute Engine one VPP worker forwarded about 18 Mpps of 64-byte IPv4 on a c3-highcpu-4, and 19 workers reached about 108 Mpps on an h3-standard-88 (roughly 5.7 Mpps per worker once scaled). A single flow always lands on one queue, so one worker is the ceiling for any one 5-tuple.

    Knobs, in order of impact

    How to tell what's limiting you

    docker exec -it syncd vppctl show runtime            # vectors/call near 256 = thread saturated; near 1 = idle
    docker exec -it syncd vppctl show hardware-interfaces # rx missed / no-buf = rings or buffers starved
    docker exec -it syncd vppctl show interface rx-placement  # which thread polls which queue
    docker exec -it syncd vppctl show buffers             # pool size and free count
    docker exec -it syncd vppctl show errors              # per-node drop reasons, CoPP policer drops included
    docker exec -it syncd vppctl show threads             # main and worker lcores actually used
    § 09Systems under load

    What the guest actually sees

    The SONiC-VPP scripts treat every port as one of two things: a veth, which VPP reads through AF_PACKET, or a PCI device, which DPDK drives. What sits behind that PCI device decides everything about performance. Pick a NIC type and compare the path a packet takes from a VPP worker to the wire.

    NIC ladder
    § 10Systems under load

    On Google's fabric

    Compute Engine's third-generation machines (C3, C3D, H3, C4 and later) offer only gVNIC, implemented on the Titanium offload hardware rather than in host software. VPP's bundled DPDK includes the gve driver, so VPP can drive a gVNIC directly. The network around the VM is the harder part: a VPC is a routed L3 network with no shared Ethernet segment, so anything SONiC assumes about wires has to be rethought.

    VPC mgmt SSH via IAP VPC left client 10.10.1.40 VPC right server 10.20.2.40 SONiC-VPP VM can-ip-forward nic0 → eth0 mgmt nic1 → Ethernet0 nic2 → Ethernet4 gve PMDvfio, no-IOMMU route in left: 10.20.0.0/16 → next hop nic1 ARP for anything → fabric gateway MAC 42:01:… no broadcast, no LACP/LLDP/VLAN trunks between VMs
    Each vNIC sits in its own VPC. Traffic reaches the router only because a VPC route points at it, and leaves only because can-ip-forward allows it to send packets that are not its own. There is no shared L2 segment anywhere in the picture.

    Rules of the fabric

    One VPC per vNIC
    A multi-NIC VM has traditionally needed each vNIC in a different VPC network, and vNIC count is capped by machine size (typically up to 8). Plan on every SONiC port costing a VPC.
    L3 only
    No broadcast or multicast. The fabric answers ARP with Google's gateway MAC, and the published VPP-on-GCE setup pinned static neighbors to those MACs. LACP, LLDP, STP and 802.1Q trunks do not cross a VPC; emulate links with VXLAN or GENEVE between VMs if you need them.
    Transit
    Set --can-ip-forward. Point VPC routes at the VM (next hop instance or IP), or peer BGP with Cloud Router through an NCC Router Appliance.
    MTU
    VPC default is 1460, configurable up to 8896. DPDK's gve driver does not support jumbo frames, and SONiC ports default to MTU 9100. Set each port's MTU to the VPC MTU or lower.
    Bandwidth
    Counted per VM, not per vNIC: about 2 Gbps per vCPU, capped at 100 Gbps on C3 and C4, and 200 Gbps with Tier_1 networking.
    Queues
    Each gVNIC's queue count is fixed when the VM is created and bounded by vCPUs. VPP can't poll more RX queues than the vNIC exposes.
    Driver binding
    No virtual IOMMU, so DPDK binds with vfio-pci in no-IOMMU mode. The VM image's default, uio_pci_generic, relies on legacy interrupt support a modern virtual NIC may not offer; verify before relying on it. Keep nic0 on the kernel for management.
    Three ways to run it on GCE

    
        
    § 11Strategic

    How it changes

    SONiC-VPP is developed in the open under the SONiC Virtual Dataplane Workgroup, chaired by Cisco engineers. In the twelve months to October 2026, sonic-platform-vpp took 79 commits, most from Microsoft and Cisco, with contributions from PANTHEON.tech and Nexthop. A dataplane feature usually touches three repositories in this order.

    1. DesignAn HLD in docs/HLD of sonic-platform-vpp: mirror and Everflow, IP-in-IP, route flow counters and LAG all started this way.
    2. Change VPP, minimallyPrefer a node in a custom plugin to a patch. Where a patch is needed, recent ones resolve hooks with vlib_get_plugin_symbol, so stock VPP behaves exactly as before when the plugin is absent.
    3. Test inside VPPmake ci-build; make ci-test runs VPP's own unit tests plus the plugins' tests. A preflight fails the job if network namespaces can't be created, because those suites would otherwise skip silently and pass.
    4. Bump and publishBump the VPP_VERSION minor number, build, and upload the packages to Buildkite. Base VPP refreshes roughly monthly.
    5. TranslateImplement the SAI side in sonic-sairedis/vslib/vpp, with unit tests there.
    6. IntegrateOpen a draft sonic-buildimage PR that moves the platform/vpp submodule, so the KVM image goes through sonic-mgmt sanity before the platform PR merges.
    7. Prove it nightlyElastictest runs the t0-vpp and t1-lag-vpp suites every night. Progress is measured by deleting entries from tests_mark_conditions_sonic_vpp.yaml, which currently lists 61 skips, most marked "to be included".

    Recent history

    § 12Strategic

    What it's for, and what you give up

    It is becoming SONiC's reference software DUT

    Read the commit log and a pattern shows. Recent work was driven by sonic-mgmt test cases: drop counters on t0, CoPP on t1-lag, BGP on dual-ToR, Everflow. The KVM image is tested nightly; the single container is not tested at all; and a commit keeping the syncd-vpp container privileged carries the note that it is not used in production. Cisco's NANOG 95 talk also reports cloud providers using it as a gateway, but the engineering center of gravity is test fidelity.

    The hard problem is semantics, not speed

    VPP has been fast for a decade. What SONiC-VPP spends its effort on is making VPP's behavior match what SONiC's tests expect from an ASIC: who answers ND, who sends ICMP errors, which ingress port a punted LACP frame reports, how a CoPP policer counts. Each shim is a place where VPP and real silicon can disagree. If you rely on SONiC-VPP to predict hardware behavior, the sonic_ext feature list is the list of things to double-check.

    Failure domains

    Shared fate
    VPP and syncd live in one container. If either crashes, packet I/O stops and there is no state reconciliation; restart the container or VM. Warm and fast reboot are not designed yet.
    Programming rate
    SAI bulk route calls become one VPP API request per route. Large convergence events are bounded by API round trips through one main thread.
    Table sizing
    The template sizes VPP's heap (2 GB) and FIB pools (256K entries) for about 100,000 routes. Bigger tables need a bigger template.
    Cores
    Polling threads cost a core each, always. On a shared host, cores per DUT is the scaling limit, not memory.
    Elephant flows
    RSS puts each flow on one queue, so no single flow goes faster than one worker.

    Choosing a shape

    Not the right tool for

    Queueing, buffering, PFC or WRED behavior; warm-reboot testing; anything that depends on a specific ASIC's pipeline quirks; and multi-terabit throughput. Use hardware for those.

    Next in this series

    Volume III could open the translation layer itself: SwitchVpp's object model, how it fakes single-attribute SAI updates on an API that has none, what bulk calls cost, and how CRC drift between VPP and libsaivs shows up at runtime.