How ZeroTier Eliminated Kernel Extensions on MacOS

How ZeroTier Eliminated Kernel Extensions on MacOS

… version 10.13 or newer at least …

Since we first released ZeroTier for MacOS we’ve used a kernel extension based on a stripped down and rebranded version of tuntaposx. In our version we increased the maximum MTU, removed tun support (we only use tap), and changed the interface names to start with "zt".

To release this kernel extension we had to get a special kernel extension signing key from Apple, which was about as easy as getting security clearance to see alien bodies at Area 51. We built it, signed it, and haven’t rebuilt it for years.

We learned a while ago that Apple wants to phase out conventional kernel extensions for obvious reasons: they’re a massive security and stability "hole". This process is likely going to start with the upcoming MacOS Catalina release.

Back when the first ZeroTier release for Mac was being developed a search was performed to determine if a way existed to avoid the kernel extension requirement. Back then we came up empty-handed. When we learned of Apple’s impending plans to sunset kernel extensions we repeated our search and this time we delved deep, going so far as to actually read the Darwin kernel’s source code at opensource.apple.com.

We found something! It appeared silently in MacOS 10.13, seemed to do what we needed, but had absolutely no documentation whatsoever.

Starting in 10.13 Darwin contains something called a "fake Ethernet" device. They start with "feth", can be created with the "ifconfig" command, and appear to behave very much like "veth pairs" on Linux. Create two "feth" interfaces, peer them, and now packets injected in one come out of the other.

Our Apple Stack Overflow question and now the ZeroTier source code itself remain just about the only result you find when you search on this topic. Since there was no documentation we started experimenting and developed a technique for using these interfaces that seems to work very well and perform decently enough for ordinary desktop use. Performance seems *slightly *worse than the old tap interface kext but not enough to be particularly noticeable.

The method we found is pure black magic. A BPF (Berkeley Packet Filter) socket seems to be needed to receive packets from the pair, while an AF_NDRV socket (yet another bit of almost undocumented internal magic) seems to be the best way to inject them. Injection can also be done via BPF but the AF_NDRV method seemed to yield superior performance.

If you’re a coder and want the dirty details check out MacEthernetTapAgent.c.

If you are using ZeroTier 1.4.2+ and are running MacOS 10.13 or newer, try typing "ifconfig" at the command line. You should see a series of “feth” pairs. Our code computes 4-digit numeric IDs based on network IDs (so they’re always the same) and uses IDs below 5000 for the member of the pair that gets IPs assigned and a corresponding ID above 5000 for the member of the pair used for I/O. (Like Linux "veth" devices they must exist in pairs. A single "feth" device doesn’t appear to want to communicate even if injection and packet sampling is local.)

We don’t think the "feth" technique will live forever. Starting in MacOS Catalina Apple is introducing DriverKit. DriverKit replaces old installed-as-root kernel extensions with a more microkernel-like way to run drivers in user-space (somehow, we’re not clear on the details yet).

It looks very much like DriverKit will let us develop a new Ethernet tap device driver for ZeroTier that runs in user space and works at least as well if not better than the old tuntaposx driver. Once Catalina is out and this API is out of beta we intend to explore this option as this is going to be the "right way" to do it. It should deliver superior performance too. We might even be able to finally put ZeroTier in the Mac App Store for easier installation and updates on MacOS!

Of course we’ll have to keep "feth" around in the code base until versions prior to 10.15 (Catalina) die off, and keep the kernel extension around until versions prior to 10.13 die off. Such is life if you’re trying to support a substantial user base.

Edit:

People might ask why we don’t use the NetworkExtension framework or the “utun” device type. It’s because NetworkExtension and "utun" only support layer 3 "tun" type interfaces, not layer 2 "tap" interfaces.

We do have code to glue a layer 3 tunnel to a layer 2 virtual network by implementing our own IPv4 ARP and IPv6 NDP. This is how we work on phones (iOS and Android). It’s not ideal though. Many desktop users want real “tap” devices for various reasons including bridging to VMs, doing real multicast, and running exotic protocols.

Apple also seems to have hard-coded an exclusion for "utun" devices into Bonjour and other system services. The fact that these work over ZeroTier is a major draw for MacOS users, so that means we can’t use "utun" or NetworkExtension on desktop (unless Apple broadens these interfaces) without losing quite a lot of capability.