higebu.com

Why was Linux SRv6 H.Encaps.L2 still slow with tunsrc set?

Thu, 01 Oct 2026 00:00:00 +0900

In the previous post, setting tunsrc made both H.Encaps.V6 and H.Encaps.L2 faster. Even so, H.Encaps.L2 stayed at 660.7 kpps against 976.4 kpps for H.Encaps.V6 (both PDR, with tunsrc set). For the same input packets, H.Encaps.L2 additionally carries the 14-byte inner Ethernet header, which should not cost a third of the throughput. It turned out to be an unnecessary memory allocation on every packet, and the fix is now in net-next.

Test setup

Everything below was measured on net-next, just before and at the fix, built from the same configuration (based on Ubuntu’s, with CONFIG_INIT_ON_ALLOC_DEFAULT_ON=y). The SUT (the machine under test) has a Xeon E5-2650 v3 at 2.30 GHz and forwards 64-byte frames on a single core, connected back to back to a T-Rex tester with Intel 82599ES (ixgbe) NICs, with iommu=pt and tunsrc set. Throughput is measured with my fork of SRPerf (updated to run on Ubuntu 24.04 and Python 3.12) as in the previous post: PDR (Partial Drop Rate, the highest offered rate with at most 0.5% packet loss) and MRR (Maximum Receive Rate, the receive rate when sending at line rate).

What the flame graph showed

I took a flame graph of the forwarding core while sending H.Encaps.L2 traffic at a fixed 500 kpps, which the kernel before the fix forwards without loss.

Including their callees, pskb_expand_head() takes 21.1% of the sampled cycles, kmalloc_reserve() 18.7% and memset_orig() 14.1%. Every memset_orig() sample is on the path seg6_do_srh() → pskb_expand_head() → kmalloc_reserve(), so the zeroing comes from reallocating the skb head. The code shows that this happens on every packet.

Why the skb head was reallocated on every packet

seg6_do_srh() in net/ipv6/seg6_iptunnel.c pushes the Ethernet header back in front of the packet before building the outer IPv6 header, and in the L2 encapsulation modes it called pskb_expand_head() unconditionally to make room for it. pskb_expand_head() always allocates a new head and copies the headroom and the linear data into it, even when the existing headroom is already large enough. The IPv6 encapsulation modes use skb_cow_head() instead, which reallocates only when the headroom is too small or the head is shared with a clone.

In this setup, the skb reaches seg6_do_srh() with 206 bytes of headroom, while a single-segment L2 encapsulation needs 94 bytes (14 for the Ethernet header, 40 for the IPv6 header, 24 for the SRH and 16 reserved for the outgoing link layer). The skbs were not cloned either, so the reallocation was not needed.

The cost is made larger by CONFIG_INIT_ON_ALLOC_DEFAULT_ON. This is a hardening option, added in Linux 5.3, that zero-fills page and slab allocations to reduce the risk of exposing stale data. Ubuntu and Debian enable it by default, and it was enabled in the kernels here, so every reallocated head was also cleared with memset(). It can be turned off with init_on_alloc=0, but turning off a security feature to win back performance is the wrong way around; not reallocating is the better fix.

The unconditional reallocation itself dates back to when H.Encaps.L2 was added in 2017, so the kernel used in the SRPerf paper (net-next during the 5.2 cycle) had it as well. That kernel predates init_on_alloc, which was merged in 5.3, though, and in the paper H.Encaps.L2 was much closer to H.Encaps.V6 (about 828 against 978 kpps, 0.85). To see how much of the gap came from the zeroing, I booted the kernel before the fix with init_on_alloc=0:

init_on_alloc	H.Encaps.V6	H.Encaps.L2	L2/V6
1 (default)	937.7	644.0	0.69
0	938.0	773.9	0.83

(MRR in kpps, mean of 10 runs, before the fix)

Without the zeroing, L2/V6 goes from 0.69 to 0.83, close to the paper.

The fix

The fix asks skb_cow_head() for the whole encapsulation up front:

- if (pskb_expand_head(skb, skb->mac_len, 0, GFP_ATOMIC) < 0)
- return -ENOMEM;
+ headroom = skb->mac_len + sizeof(struct ipv6hdr) +
+ ipv6_optlen(tinfo->srh) +
+ dst_dev_overhead(cache_dst, skb);
+
+ err = skb_cow_head(skb, headroom);
+ if (unlikely(err))
+ return err;

When the headroom is large enough and the skb is not cloned, nothing is reallocated. When the headroom really is too small, the head is reallocated once, and the later skb_cow_head() in the IPv6 encapsulation code then finds the room it needs instead of reallocating a second time.

After the fix, the allocation no longer appears in the flame graph under the same load:

pskb_expand_head(), kmalloc_reserve() and memset_orig() have no samples left, and seg6_do_srh() falls from 28.2% to 5.6% of the cycles. Dividing the cycles of each 30-second capture by the 15 million packets forwarded in it, the core spends 2602 cycles per packet instead of 3862.

Measurement

	H.Encaps.L2 PDR
before the fix	654.6 kpps
after the fix	965.7 kpps (+47.5%)

(mean of 10 PDR searches, each using 10-second trials)

H.Encaps.V6 does not take this code path, and its MRR stays at about 938 kpps on both kernels. With init_on_alloc=0, the kernel with the fix gives an L2/V6 ratio of 1.02 (956.8 against 939.1 kpps MRR): H.Encaps.L2 is as fast as H.Encaps.V6, so the gap that was left after removing the zeroing was the reallocation.

Getting it upstream

The first version only asked skb_cow_head() for skb->mac_len. Eric Dumazet pointed out that a header-cloned skb could then still be reallocated twice, once to unclone it and again for the outer header, so the second version asked for the whole encapsulation as above.

While reviewing the second version, Sashiko, the AI review bot on netdev, found a different bug. dst_dev_overhead() reserves room for the egress device’s link-layer header, but the code rebuilds the ingress MAC header in front of the packet. When the ingress MAC header is longer, for example a VLAN-tagged frame received with reorder_hdr off, the offset of the rebuilt header wraps around and the copy lands about 64 KB past the buffer. This was not specific to the L2 modes, so I fixed it separately in dst_dev_overhead() and sent it to the net tree. That fix was merged first, and the third version of this patch went into net-next, which currently tracks 7.3-rc, so it should be in Linux 7.4.

References

seg6: reallocate the skb head on L2 encapsulation only when needed (net-next, bf515c9fae87)
net: ipv6: keep room for the mac header in dst_dev_overhead() (net, 87cd6b717e40)
Patch threads: v1, v3
A. Abdelsalam, P. L. Ventre, C. Scarpitta, A. Mayer, S. Salsano, P. Camarillo, F. Clad, C. Filsfils, “SRPerf: A Performance Evaluation Framework for IPv6 Segment Routing”, IEEE Transactions on Network and Service Management, vol. 18, no. 2, pp. 2320-2333, 2021
The kernel’s command-line parameters: init_on_alloc
SRPerf (GitHub) and my fork
How much slower is Linux SRv6 H.Encaps without tunsrc?
Brendan Gregg’s FlameGraph

How much slower is Linux SRv6 H.Encaps without tunsrc?

Fri, 25 Sep 2026 00:00:00 +0900

I have been implementing the SRv6 Mobile User Plane (RFC 9433) behaviors in the Linux kernel, and I am measuring them with SRPerf to see whether they keep up with the existing SRv6 behaviors. SRPerf is the measurement framework from the paper SRPerf: A Performance Evaluation Framework for IPv6 Segment Routing, which benchmarked the SRv6 behaviors of Linux and VPP. The published version targets an older environment, so I use my fork, updated to run on Ubuntu 24.04 and Python 3.12.

The paper says that H.Encaps gets slower if the tunnel source address (tunsrc) is not configured. It does not say how much slower, so I measured it.

What the SRPerf paper says about TUNSRC

The description of the experimental setup in the paper reads:

We configured the TUNSRC for the SRv6 policy headend behaviors doing encapsulation. The latter allows to configure the IPv6 source address of the IPv6 outer header. The TUNSRC has to be configured otherwise the Linux kernel will try to get the address from the interface which will cause a performance drop in the performance of the encaps behavior.

The authors measured with tunsrc configured, but the paper gives no number for the drop. Note that the published SRPerf forwarding-behaviour.cfg does not configure tunsrc, so you need to add it yourself to measure under the same conditions as the paper.

What happens without tunsrc

The source address of the outer IPv6 header is chosen by set_tun_src() in net/ipv6/seg6_iptunnel.c (v7.1 source).

 if (route_tunsrc && !ipv6_addr_any(route_tunsrc)) {
 memcpy(saddr, route_tunsrc, sizeof(struct in6_addr));
 } else {
 rcu_read_lock();
 tun_src = rcu_dereference(sdata->tun_src);

 if (!ipv6_addr_any(tun_src)) {
 memcpy(saddr, tun_src, sizeof(struct in6_addr));
 } else {
 ipv6_dev_get_saddr(net, dev, daddr,
 IPV6_PREFER_SRC_PUBLIC, saddr);
 }

The kernel looks at the per-route tunsrc (since Linux 7.1) and then the per-network-namespace tunsrc (the one set with ip sr tunsrc set). If both are ::, it calls ipv6_dev_get_saddr(). tunsrc defaults to ::, so unless you configure it, the kernel compares the addresses on the host against the RFC 6724 rules for every packet. This is what the paper means by “the Linux kernel will try to get the address from the interface”.

Measuring with and without tunsrc

I followed the paper’s method: the SUT (the machine under test) forwards on a single core, and I measured PDR (Partial Drop Rate, the highest offered rate with at most 0.5% packet loss) and MRR (Maximum Receive Rate, the receive rate when sending at line rate), 10 runs of 10 seconds each. The SUT has a Xeon E5-2650 v3 (2.30 GHz) and is directly connected to the tester running T-Rex with Intel 82599ES (10GbE) NICs. The IOMMU is in passthrough mode (iommu=pt).

With the same kernel and the same configuration, I only switched tunsrc between :: and 12:2::2. 12:2::2 is the address the kernel picks when tunsrc is ::, so the only difference between the two runs is how the source address is chosen. For comparison, I also measured plain IPv6 and H.Insert (inline), which do not build an outer IPv6 header and therefore never reach set_tun_src().

	without tunsrc	with tunsrc	change
plain IPv6	1196.0	1192.2	-0.3%
H.Insert	998.3	1001.3	+0.3%
H.Encaps.V6	569.8	967.2	+69.7%
H.Encaps.L2	445.2	656.2	+47.4%

(MRR, kpps)

plain IPv6 and H.Insert barely moved, and only the two encap behaviors improved. Without tunsrc, H.Encaps.V6 was 41% slower and H.Encaps.L2 was 32% slower. H.Encaps.L2 also has a separate slowdown unrelated to tunsrc, and these numbers were taken on a kernel without that fix (I wrote about it in the next post).

Converting PDR to cycles per packet (2.30 GHz / PDR), the difference is 1660 to 1670 cycles for both H.Encaps.V6 and H.Encaps.L2, which is the cost of the source address selection alone. This cost may depend on how many addresses the host has, which I did not measure.

With tunsrc configured, H.Encaps.V6 reached a PDR of 976 kpps, almost the same as the 978 kpps in the paper.

How to configure tunsrc

With iproute2:

# per network namespace
ip sr tunsrc set 2001:db8::1
ip sr tunsrc show

# per route (Linux 7.1 and iproute2 7.1 or later)
ip -6 route add 2001:db8:100::/64 encap seg6 mode encap tunsrc 2001:db8::1 segs fc00::1 dev eth0

With FRR, put the following in the configuration (frr.conf):

segment-routing
 srv6
 encapsulation
 source-address 2001:db8::1

FRR does not set tunsrc unless source-address is configured. I checked this by starting FRR: with only an SRv6 locator configured, ip sr tunsrc show stayed at ::, and after adding source-address it showed that address. bgpd’s SRv6 L3VPN does not set a per-route source address either, so without source-address you get the slower path.

References

A. Abdelsalam, P. L. Ventre, C. Scarpitta, A. Mayer, S. Salsano, P. Camarillo, F. Clad, C. Filsfils, “SRPerf: A Performance Evaluation Framework for IPv6 Segment Routing”, IEEE Transactions on Network and Service Management, vol. 18, no. 2, pp. 2320-2333, 2021
SRPerf (GitHub) and my fork
RFC 8986: Segment Routing over IPv6 (SRv6) Network Programming
RFC 9433: Segment Routing over IPv6 for the Mobile User Plane
RFC 6724: Default Address Selection for Internet Protocol Version 6 (IPv6)
Linux v7.1 net/ipv6/seg6_iptunnel.c
FRR User Guide: Zebra, SRv6 source-address

VMFork using mob

Wed, 27 Apr 2016 23:57:00 +0000

We cannot do VMFork without PowerCLI Extension for now. But that means there are VMFork related APIs in vSphere API.

I explored those APIs using the MOB(The Managed Object Browser). Then I found it and I understand how to use it.

Disclaimer: VMFork is not officially supported by VMware, please use at your own risk.

There are following methods in VirtualMachine object.

CreateForkChild_Task
EnableForkParent_Task
DisableForkParent_Task
RetrieveForkChildren_Task
RetrieveForkParent_Task

Also, there are config.forkconfiginfo and runtime.quiescedforkparent in VirtualMachine properties.

These methods and properties are found in the HTML source of VM page on the MOB.

This time, we want to run the VMFork, so we use EnableForkParent_Task and CreateForkChild_Task.

How to run the VMFork using the MOB.

1. Open the MOB.

https://<vcenter_ip_or_hostname>/mob/

2. Open the VirtualMachine you want to use as parent VM.

Open the parent VM with using rootFolder or searchIndex. The URL would be like here.

https://<vcenter_ip_or_hostname>/mob/?moid=<your_vm_moid>

3. Run EnableForkParent_Task

Open following URL.

https://<vcenter_ip_or_hostname>/mob/?moid=<your_vm_moid>&method=enableForkParent

This method has no parameters, you have only to click Invoke Method.

4. Quiesce the parent VM

Run this command on the parent VM via console or SSH.

vmtoolsd --cmd 'vmfork-begin -1 -1'

If you want to automate it, you can use GuestOperationsManager.ProcessManager.

PowerCLI’s Enable-InstantCloneVM upload PreQuiesceScript and PostCloneScript with operations up to this point.

You can upload these scripts with GuestOperationsManager instead of PowerCLI.

Also, It seems that PowerCLI’s Enable-InstantCloneVM runs PreQuiesceScript and PostCloneScript with GuestOperationsManager. The command is like here.

$prescript_path;vmtoolsd --cmd 'vmfork-begin -1 -1';$postscript_path

5. Run VMFork

Use CreateForkChild_Task. It is same as PowerCLI’s New-InstantCloneVM. The URL is following.

https://<vcenter_ip_or_hostname>/mob/?moid=<your_vm_moid>&method=createForkChild

CreateForkChild_Task has parameters, name and spec. name is child VM name. And I guess spec is VirtualMachineCreateChildSpec. Minimun spec is here.

<spec>
 <persistent>false</persistent>
</spec>

Click Invoke Method. The child VM will be created in an instant.

References

vSphere Client 5.5 on Ubuntu 14.04

Sun, 15 Mar 2015 12:30:00 +0000

Versions

Ubuntu 14.04.2 amd64
wine 1.7.38
mono 4.5.4

Install Wine and .NET

Run following commands:

sudo add-apt-repository ppa:ubuntu-wine/ppa
sudo apt-get update
sudo apt-get install -y wine1.7 wine-mono4.5.4 wine-gecko2.34

Confirm Wine version

$ wine --version
wine-1.7.38

Set up 32bit Wine

WINEARCH=win32 WINEPREFIX=~/.wine32 winecfg

Install .NET

WINEARCH=win32 WINEPREFIX=~/.wine32 winetricks dotnet20
WINEARCH=win32 WINEPREFIX=~/.wine32 winetricks dotnet30
WINEARCH=win32 WINEPREFIX=~/.wine32 winetricks dotnet20sp1
WINEARCH=win32 WINEPREFIX=~/.wine32 winetricks dotnet30sp1
WINEARCH=win32 WINEPREFIX=~/.wine32 winetricks dotnet35
WINEARCH=win32 WINEPREFIX=~/.wine32 winetricks dotnet35sp1

If the above commands exits with error, and shows a download url. Download the file and put into specified directory.

Install vSphere Client

Download vSphere Client Installer, and run following command:

WINEARCH=win32 WINEPREFIX=~/.wine32 wine Downloads/VMware-viclient-all-5.5.0-1993072.exe

Run vSphere Client

Run following command:

WINEARCH=win32 WINEPREFIX=$HOME/.wine32 wine $HOME/.wine32/drive_c/Program\ Files/VMware/Infrastructure/Virtual\ Infrastructure\ Client/Launcher/VpxClient.exe

Or double click the shortcut on your desktop.

Console doesn’t work now. If you want to use console, you should use VMware Workstation or VNC.