Manual

1. Introduction

ZeroTier is a smart Ethernet switch for planet Earth.

It’s a distributed network hypervisor built atop a cryptographically secure global peer to peer network. It provides advanced network virtualization and management capabilities on par with an enterprise SDN switch, but across both local and wide area networks and connecting almost any kind of app or device.

This manual describes the design and operation of ZeroTier and its associated services, apps, and libraries. Its intended audience includes IT professionals, network administrators, information security experts, and developers.

The first section (2) of this guide explains ZeroTier’s design and operation at a high level and is written for those with at least an intermediate knowledge of topics like TCP/IP and Ethernet networking. It’s not required reading for most users, but understanding how things work in detail helps clarify everything else and helps tremendously with troubleshooting should anything go wrong.

The remaining sections deal more concretely with deployment and administration.

Table of Contents

  1. Introduction
  2. Network Hypervisor Overview
    1. VL1: The ZeroTier Peer to Peer Network
      1. Network Topology and Peer Discovery
      2. Addressing
      3. Cryptography
      4. Trusted Paths for Fast Local SDN
      5. Multipath
    2. VL2: The Ethernet Virtualization Layer
      1. Network Identifiers and Controllers
      2. Certificates and Other Credentials
      3. Multicast, ARP, NDP, and Special Addressing Modes
      4. Ethernet Bridging
      5. Public Networks
      6. Ad-Hoc Networks
      7. Quality of Service (QoS)
  3. The Network Rules Engine
    1. Rule Sets and Rule Evaluation
      1. Actions and Match Conditions
    2. Capabilities
    3. Tags
    4. Rule Definition Language
      1. An Introductory Example
      2. Rule Definition Language Syntax
      3. Actions, Matches, Operators, and Constants
    5. Useful Design Patterns
      1. TCP Whitelisting
      2. Locking Down UDP
      3. Traffic Observation and Interception
      4. The Classified System Pattern
  4. ZeroTier One
    1. JSON API
    2. Local Configuration Options
    3. Network Controller Configuration
    4. Creating Your Own Roots (a.k.a. “Moons”)
  5. SDK
    1. Socket API (libzt)
      1. Starting the Service
      2. Joining a Network
      3. Communicating with peers
      4. Handling events
        1. Node Events
        2. Network Events
        3. Peer Events
        4. Path Events
        5. Route Events
        6. Address Events
        7. Network Stack Events (debugging)
        8. Netif Events (debugging)
      5. Errors
      6. Thread Model
      7. Statistics
      8. Network Controller Mode
    2. Frame-based API (libztcore)
    3. Downloads
    4. Examples
      1. C Example
      2. Java Example

2. Network Hypervisor Overview

The ZeroTier network hypervisor (currently found in the node/ subfolder of the ZeroTierOne git repository) is a self-contained network virtualization engine that implements an Ethernet virtualization layer similar to VXLAN on top of a global encrypted peer to peer network.

The ZeroTier protocol is original, though aspects of it are similar to VXLAN and IPSec. It has two conceptually separate but closely coupled layers in the OSI model sense: VL1 and VL2. VL1 is the underlying peer to peer transport layer, the “virtual wire,” while VL2 is an emulated Ethernet layer that provides operating systems and apps with a familiar communication medium.

2.1. VL1: The ZeroTier Peer to Peer Network

A global data center requires a global wire closet.

In conventional networks L1 (OSI layer 1) refers to the actual CAT5/CAT6 cables or wireless radio channels over which data is carried and the physical transceiver chips that modulate and demodulate it. VL1 is a peer to peer network that does the same thing by using encryption, authentication, and a lot of networking tricks to create virtual wires on a dynamic as-needed basis.

2.1.1. Network Topology and Peer Discovery

VL1 is designed to be zero-configuration. A user can start a new ZeroTier node without having to write configuration files or provide the IP addresses of other nodes. It’s also designed to be fast. Any two devices in the world should be able to locate each other and communicate almost instantly.

To achieve this VL1 is organized like DNS. At the base of the network is a collection of always-present root servers whose role is similar to that of DNS root name servers. Roots run the same software as regular endpoints but reside at fast stable locations on the network and are designated as such by a world definition. World definitions come in two forms: the planet and one or more moons. The protocol includes a secure mechanism allowing world definitions to be updated in-band if root servers’ IP addresses or ZeroTier addresses change.

There is only one planet. Earth’s root servers are operated by ZeroTier, Inc. as a free service. There are currently twelve root servers organized into two six-member clusters distributed across every major continent and multiple network providers. Almost everyone in the world has one within less than 100ms network latency from their location.

A node can “orbit” any number of moons. A moon is just a convenient way to add user-defined root servers to the pool. Users can create moons to reduce dependency on ZeroTier, Inc. infrastructure or to locate root servers closer for better performance. For on-premise SDN use a cluster of root servers can be located inside a building or data center so that ZeroTier can continue to operate normally if Internet connectivity is lost.

Nodes start with no direct links to one another, only upstream to roots (planet and moons). Every peer on VL1 possesses a globally unique 40-bit (10 hex digit) ZeroTier address, but unlike IP addresses these are opaque cryptographic identifiers that encode no routing information. To communicate peers first send packets “up” the tree, and as these packets traverse the network they trigger the opportunistic creation of direct links along the way. The tree is constantly trying to “collapse itself” to optimize itself to the pattern of traffic it is carrying.

Peer to peer connection setup goes like this:

  1. A wants to send a packet to B, but since it has no direct path it sends it upstream to R (a root).
  2. If R has a direct link to B, it forwards the packet there. Otherwise it sends the packet upstream until planetary roots are reached. Planetary roots know about all nodes, so eventually the packet will reach B if B is online.
  3. R also sends a message called rendezvous to A containing hints about how it might reach B. Meanwhile the root that forwards the packet to B sends rendezvous informing B how it might reach A.
  4. A and B get their rendezvous messages and attempt to send test messages to each other, possibly accomplishing hole punching of any NATs or stateful firewalls that happen to be in the way. If this works a direct link is established and packets no longer need to take the scenic route.

Since roots forward packets, A and B can reach each other instantly. A and B then begin attempting to make a direct peer to peer connection. If this succeeds it results in a faster lower latency link. We call this transport triggered link provisioning since it’s the forwarding of the packet itself that triggers the peer to peer network to attempt direct connection.

VL1 never gives up. If a direct path can’t be established, communication can continue through (slower) relaying. Direct connection attempts continue forever on a periodic basis. VL1 also has other features for establishing direct connectivity including LAN peer discovery, port prediction for traversal of symmetric IPv4 NATs, and explicit port mapping using uPnP and/or NAT-PMP if these are available on the local physical LAN.

A blog post from 2014 by ZeroTier’s original author explains some of the reasoning behind VL1’s design.

2.1.2. Addressing

Every node is uniquely identified on VL1 by a 40-bit (10 hex digit) ZeroTier address. This address is computed from the public portion of a public/private key pair. A node’s address, public key, and private key together form its identity.

On devices running ZeroTier One the node identity is stored in identity.public and identity.secret in the service’s home directory.

When ZeroTier starts for the first time it generates a new identity. It then attempts to advertise it upstream to the network. In the very unlikely event that the identity’s 40-bit unique address is taken, it discards it and generates another.

Identities are claimed on a first come first serve basis and currently expire from planetary roots after 60 days of inactivity. If a long-dormant device returns it may re-claim its identity unless its address has been taken in the meantime (again, highly unlikely).

The address derivation algorithm used to compute addresses from public keys imposes a computational cost barrier against the intentional generation of a collision. Currently it would take approximately 10,000 CPU-years to do so (assuming e.g. a 3ghz Intel core). This is expensive but not impossible, but it’s only the first line of defense. After generating a collision an attacker would then have to compromise all upstream nodes, network controllers, and anything else that has recently communicated with the target node and replace their cached identities.

ZeroTier addresses are, once advertised and claimed, a very secure method of unique identification.

When a node attempts to send a message to another node whose identity is not cached, it sends a whois query upstream to a root. Roots provide an authoritative identity cache.

2.1.3. Cryptography

If you don’t know much about cryptography you can safely skip this section. TL;DR: packets are end-to-end encrypted and can’t be read by roots or anyone else, and we use modern 256-bit crypto in ways recommended by the professional cryptographers that created it.

Asymmetric public key encryption is Curve25519/Ed25519, a 256-bit elliptic curve variant.

Every VL1 packet is encrypted end to end using (as of the current version) 256-bit Salsa20 and authenticated using the Poly1305 message authentication (MAC) algorithm. MAC is computed after encryption (encrypt-then-MAC) and the cipher/MAC composition used is identical to the NaCl reference implementation.

As of today we do not implement forward secrecy or other stateful cryptographic features in VL1. We don’t do this for the sake of simplicity, reliability, and code footprint, and because frequently changing state makes features like clustering and fail-over much harder to implement. See our discussion on GitHub.

We may implement forward secrecy in the future. For those who want this level of security today, we recommend using other cryptographic protocols such as SSL or SSH over ZeroTier. These protocols typically implement forward secrecy, but using them over ZeroTier also provides the secondary benefit of defense in depth. Most cryptography is compromised not by a flaw in encryption but through bugs in the implementation. If you’re using two secure transports, the odds of a critical bug being discovered in both at the same time is very low. The CPU overhead of double-encryption is not significant for most work loads.

2.1.4. Trusted Paths for Fast Local SDN

To support the use of ZeroTier as a high performance SDN/NFV protocol over physically secure networks the protocol supports a feature called trusted paths. It is possible to configure all ZeroTier devices on a given network to skip encryption and authentication for traffic over a designated physical path. This can cut CPU use noticeably in high traffic scenarios but at the cost of losing virtually all transport security.

Trusted paths do not prevent communication with devices elsewhere, since traffic over other paths will be encrypted and authenticated normally.

We don’t recommend the use of this feature unless you really need the performance and you know what you’re doing. We also recommend thinking carefully before disabling transport security on a cloud private network. Larger cloud providers such as Amazon and Azure tend to provide good network segregation but many less costly providers offer private networks that are “party lines” and are not much more secure than the open Internet.

2.1.5. Multipath

As of 1.4.0 (coming soon) ZeroTier will allow peers to communicate over multiple physical paths simultaneously and will automatically load balance according to path strength. A peer supporting multipath logic will fall back to classical non-multipath behavior when communicating with peers which do not support it or that do not have it enabled. This feature is accomplished via passive and active measurement techniques and all additional traffic overhead will cease when user traffic ceases, there is no ambient traffic overhead when not in use.

Modes

Currently two modes are supported: random and proportionally balanced. The former will send traffic over any paths that are detected and while no balancing takes place it will stop sending traffic over a path if it expires. The latter will continuously measure the quality of each path and allocate traffic across them in proportion to their observed stability and performance.

Notion of link quality

Quality (specifically quality relative to other paths) is the value we use to determine how to allocate traffic across paths. It is composed of two sub-quantities: stability and performance. These separate quantities are important for future quality of service (QoS) efforts. For instance we may wish to map certain types of traffic to paths with high stability when it might not require high performance (or vice versa). These quantities are defined as follows:

  • Stability: This is essentially a measure of how well behaved a path is. Namely how its latency varies over time, how throughput varies over time, packet loss ratio, and packet error ratio.
  • Performance: This is simply a measure of how much traffic this link is known to be able to carry.

NOTE: The above quantities are the sum of a series of weighted contributions by more fundamental metrics:

  • Latency milliseconds, [0, 2^16]: How long it takes a packet to travel from one node to another.
  • Packet Delay Variance (PDV) milliseconds, [0, 2^16]: How the latency between two nodes varies over time. This is very similar to the concept of jitter but not exactly.
  • Throughput Disturbance Coefficient (TDC) unitless, [0, 1]: How the observed throughput of a path varies with time. It could be the result of traffic shaping or congestion.
  • Packet Loss Ratio (PLR) unitless, [0, 1]: The percentage of packets which are lost.
  • Packet Error Ratio (PER) unitless, [0, 1]: The percentage of packets which failed authentication/CRC checks.
  • Max Throughput bytes per second, [0, 2^64]: Maximum number of bytes observed to have been sent over a path within one second. This value will slowly decrease over time so that it acts like a rubber band around a tube.

New protocol verbs

While understanding the following isn’t important for one to use this feature, it’s here in case anyone is curious:

  • VERB_ACK: An acknowledgment of receipt of a series of recent packets from another peer. This is used to calculate relative throughput values and to detect packet loss. Every packet type with the exception of VERB_ACK and VERB_QOS_MEASUREMENT are counted toward this value.
       <[4] 32-bit number of bytes received since last ACK>
    

    Upon receipt of this packet, the local peer will verify that the correct number of bytes were received by the remote peer. If these values do not agree it could be an indicator of packet loss. Additionally, the local peer knows the interval of time that has elapsed since the last received ACK. With this information it can compute a rough estimate of the current throughput. This is sent at a rate of once every 250 ms.

  • VERBQOSMEASUREMENT: A packet containing timing measurements useful for estimating path quality. Composed of a list of pairs for an arbitrary set of recent packets. This is used to sample for latency and packet delay variance.
     <[8] 64-bit packet ID of previously-received packet>
     <[1] 8-bit packet sojourn time>
     <...repeat until end of max 1400 byte packet...>
    

    The number of possible records per QoS packet is: (1400 * 8) / 72 = 155. This packet should be sent very rarely as it can be somewhat large if the connection is saturated. Future versions might use a bloom table to probabilistically determine these values in a vastly more space-efficient manner.

    Note: The ‘internal packet sojourn time’ is a slight misnomer as it is a measure of the amount of time between when a packet was received and the egress time of its tracking QoS packet. This is sent at a rate of once every second.

Enabling Multipath

Communication over multiple paths is enabled by setting multipathMode in local.conf. Restart ZeroTier and it will automatically detect available physical interfaces and begin allocating across all paths.

{
    "settings": {
        "multipathMode": 2
    }
}

Where:

  • 0 = ZT_MULTIPATH_NONE: No active multipath. Traffic is merely sent over the strongest path. This mode will automatically failover to the next-strongest path in the event that a path goes down.
  • 1 = ZT_MULTIPATH_RANDOM: Traffic is randomly distributed among all active paths.
  • 2 = ZT_MULTIPATH_PROPORTIONALLY_BALANCED: Traffic is allocated across all active paths in proportion to their observed stability and performance (quality). Will cease sending traffic over links that appear to be stale.

Using the raw JSON output format of the status command zerotier-cli -j status will yield something like the following which contains an entry for each peer, aggregate link properties, and individual path properties. This example shows traffic split roughly evenly across the paths on separate physical interfaces.

"settings": {
    "multipathMode": 2
}

...

"multipath": {
  "9f47c03a5b": {
   "aggregateLinkLatency": 25,
   "aggregateLinkPacketDelayVariance": 6,
   "paths": [
    {
     "address": "172.116.130.93/43032",
     "allocation": 0.512172758579254,
     "ifname": "phy0",
     "latency": 13.0,
     "stability": 0.820895612239838
    },
    {
     "address": "2605:e000:1600:c315:0000:ff0b:eca2:452c/9993",
     "allocation": 0.487827271223068,
     "ifname": "wlan0",
     "latency": 30.0,
     "stability": 0.772978186607361
    }
   ]
  }
}

If one builds with the ZT_DEBUG=1 flag, a series of traces will be emitted which detail the current state of aggregate links to each peer, for instance:

link to peer 9f47c03a5b is composed of (2) physical paths (en0, ipv4, 0.512), (en4, ipv4, 0.487), has packet delay variance (6 ms), mean latency (25 ms)
link to peer 9f47c03a5b is fully redundant
link to peer 9f47c03a5b is no longer redundant

Performance and security considerations

To minimize performance overhead the following decisions were made:

  • To reduce the speed at which ACK/QoS counters and tables are populated, only packets with IDs divisible by 16 are included in measurements, this amounts to about 6.25% of all packets.
  • To reduce computational overhead all non-trivial measurements or those measurements resulting from statistical analysis are cached and their update frequency is not a function of user traffic.
  • Rate limits are imposed on the number of VERB_ACK and VERB_QOS_MEASUREMENT packets that will be accepted for processing by a peer.

2.2. VL2: The Ethernet Virtualization Layer

VL2 is a VXLAN-like network virtualization protocol with SDN management features. It implements secure VLAN boundaries, multicast, rules, capability based security, and certificate based access control.

VL2 is built atop and carried by VL1, and in so doing it inherits VL1’s encryption and endpoint authentication and can use VL1 asymmetric keys to sign and verify credentials. VL1 also allows us to implement VL2 entirely free of concern for underlying physical network topology. Connectivity and routing efficiency issues are VL1 concerns. It’s important to understand that there is no relationship between VL2 virtual networks and VL1 paths. Much like VLAN multiplexing on a wired LAN, two nodes that share multiple network memberships in common will still only have one VL1 path (virtual wire) between them.

2.2.1. Network Identifiers and Controllers

Each VL2 network (VLAN) is identified by a 64-bit (16 hex digit) ZeroTier network ID that contains the 40-bit ZeroTier address of the network’s controller and a 24-bit number identifying the network on the controller.

Network ID: 8056c2e21c123456
            |         |
            |         Network number on controller
            |
            ZeroTier address of controller

When a node joins a network or requests a network configuration update, it sends a network config query message (via VL1) to the network’s controller. The controller can then use the node’s VL1 address to look it up on the network and send it the appropriate certificates, credentials, and configuration information. From the perspective of VL2 virtual networks, VL1 ZeroTier addresses can be thought of as port numbers on an enormous global-scale virtual switch.

A common misunderstanding is to conflate network controllers with root servers (planet and moons). Root servers are connection facilitators that operate at the VL1 level. Network controllers are configuration managers and certificate authorities that belong to VL2. Generally root servers don’t join or control virtual networks and network controllers are not root servers, though it is possible to have a node do both.

Controller Security Considerations

Network controllers serve as certificate authorities for ZeroTier virtual networks. As such, their identity.secret files should be guarded closely and backed up securely. Compromise of a controller’s secret key would allow an attacker to issue fraudulent network configurations or admit unauthorized members, while loss of the secret key results in loss of ability to control the network in any way or issue configuration updates and effectively renders the network unusable.

It is important that controllers’ system clocks remain relatively accurate (to within 30-60 seconds) and that they are secure against remote tampering. Many cloud providers provide secure time sources either directly via the hypervisor or via NTP servers within their networks.

2.2.2. Certificates and Other Credentials

All credentials issued by network controllers to member nodes in a given network are signed by the controller’s secret key to allow all network members to verify them. Credentials have timestamp fields populated by the controller, allowing relative comparison without the need to trust the node’s local system clock.

Credentials are issued only to their owners and are then pushed peer to peer by nodes that wish to communicate with other nodes on the network. This allows networks to grow to enormous sizes without requiring nodes to cache large numbers of credentials or to constantly consult the controller.

Credential Types

  • Certificates of Membership: a certificate that a node presents to obtain the right to communicate on a given network. Certificates of membership are accepted if they agree, meaning that the submitting member’s certificate’s timestamp differs from the recipient’s certificate’s timestamp by no more than the recipient certificate’s maximum timestamp delta value. This creates a decentralized moving-window scheme for certificate expiration without requiring node clock synchronization or constant checking with the controller.
  • Revocations: a revocation instantaneously revokes a given credential by setting a hard timestamp limit before which it will not be accepted. Revocations are rapidly propagated peer to peer among members of a network using a rumor mill algorithm, allowing a controller to revoke a member credential across the entire network even if its connection to some members is unreliable.
  • Capabilities: a capability is a bundle of network rules that is signed by the controller and can be presented to other members of a network to grant the presenter elevated privileges within the framework of the network’s base rule set. More on this in the section on rules.
  • Tags: a tag is a key/value pair signed by the controller that is automatically presented by members to one another and can be matched on in base or capability network rules. Tags can be used to categorize members by role, department, classification, etc.
  • Certificates of Ownership: these certify that a given network member owns something, such as an IP address. These are currently only used to lock down networks against IP address spoofing but could be used in the future to certify ownership of other network-level entities that can be matched in a filter.

2.2.3. Multicast, ARP, NDP, and Special Addressing Modes

ZeroTier networks support multicast via a simple publish/subscribe system.

When a node wishes to receive multicasts for a given multicast group, it advertises membership in this group to other members of the network with which it is communicating and to the network controller. When a node wishes to send a multicast it both consults its cache of recent advertisements and periodically solicits additional advertisements.

Broadcast (Ethernet ff:ff:ff:ff:ff:ff) is treated as a multicast group to which all members subscribe. It can be disabled at the network level to reduce traffic if it is not needed. IPv4 ARP receives special handling (see below) and will still work if normal broadcast is disabled.

Multicasts are propagated using simple sender-side replication. This places the full outbound bandwidth load for multicast on the sender and minimizes multicast latency. Network configurations contain a network-wide multicast limit configurable at the network controller. This specifies the maximum number of other nodes to which any node will send a multicast. If the number of known recipients in a given multicast group exceeds the multicast limit, the sender chooses a random subset.

There is no global limit on multicast recipients, but setting the multicast limit very high on very large networks could result in significant bandwidth overhead.

Special Handling of IPv4 ARP Broadcasts

IPv4 ARP is built on simple Ethernet broadcast and scales poorly on large or distributed networks. To improve ARP’s scalability ZeroTier generates a unique multicast group for each IPv4 address detected on its system and then transparently intercepts ARP queries and sends them only to the correct group. This converts ARP into effectively a unicast or narrow multicast protocol (like IPv6 NDP) and allows IPv4 ARP to work reliably across wide area networks without excess bandwidth consumption. A similar strategy is implemented under the hood by a number of enterprise switches and WiFi routers designed for deployment on extremely large LANs. This ARP emulation mode is transparent to the OS and application layers, but it does mean that packet sniffers will not see all ARP queries on a virtual network the way they typically can on smaller wired LANs.

Multicast-Free IPv6 Addressing Modes

IPv6 uses a protocol called NDP in place of ARP. It is similar in role and design but uses narrow multicast in place of broadcast for superior scalability on large networks. This protocol nevertheless still imposes the latency of an additional multicast lookup whenever a new address is contacted. This can add hundreds of milliseconds over a wide area network, or more if latencies associated with pub/sub recipient lookup are significant.

IPv6 addresses are large enough to easily encode ZeroTier addresses. For faster operation and better scaling we’ve implemented several special IPv6 addressing modes that allow the local node to emulate NDP. These are ZeroTier’s rfc4193 and 6plane IPv6 address assignment schemes. If these addressing schemes are enabled on a network, nodes locally intercept outbound NDP queries for matching addresses and then locally generate spoofed NDP replies.

Both modes dramatically reduce initial connection latency between network members. 6plane additionally exploits NDP emulation to transparently assign an entire IPv6 /80 prefix to every node without requiring any node to possess additional routing table entries. This is designed for virtual machine and container hosts that wish to auto-assign IPv6 addresses to guests and is very useful on microservice architecture backplane networks.

Finally there is a security benefit to NDP emulation. ZeroTier addresses are cryptographically authenticated, and since Ethernet MAC addresses on networks are computed from ZeroTier addresses these are also secure. NDP emulated IPv6 addressing modes are therefore not vulnerable to NDP reply spoofing.

Normal non-NDP-emulated IPv6 addresses (including link-local addresses) can coexist with NDP-emulated addressing schemes. Any NDP queries that do not match NDP-emulated addresses are sent via normal multicast.

2.2.4. Ethernet Bridging

ZeroTier emulates a true Ethernet switch. This includes the ability to L2 bridge other Ethernet networks (wired LAN, WiFi, virtual backplanes, etc.) to virtual networks using conventional Ethernet bridging.

To act as a bridge a network member must be designated as such by the controller. This is for security reasons as normal network members are not permitted to send traffic from any origin other than their MAC address. Designated bridges also receive special treatment from the multicast algorithm, which more aggressively and directly queries them for group subscriptions and replicates all broadcast traffic and ARP requests to them. As a result bridge nodes experience a slightly higher amount of multicast bandwidth overhead.

Bridging has been tested extensively on Linux using the Linux kernel native bridge, which cleanly handles network MTU mismatch. There are third party reports of bridging working on other platforms. The details of setting up bridging, including how to selectively block traffic like DHCP that may not be wanted across the bridge, are beyond the scope of this manual.

2.2.5. Public Networks

It is possible to disable access control on a ZeroTier network. A public network’s members do not check certificates of membership, and new members to a public network are automatically marked as authorized by their host controller. It is not possible to de-authorize a member from a public network.

Rules on the other hand are enforced, so it’s possible to implement a special purpose public network that only allows access to a few things or that only allows a restricted subset of traffic.

Public networks are useful for testing and for peer to peer “party lines” for gaming, chat, and other applications. Participants in public networks are warned to pay special attention to security. If joining a public network be careful not to expose vulnerable services or accidentally share private files via open network shares or HTTP servers. Make sure your operating system, applications, and services are fully up to date.

ZeroTier, Inc. operates a public network called Earth (no relation to the root server planet definition of the same name) with the network ID 8056c2e21c000001. Earth issues IPv4 addresses in the unused IPv4 space 28.0.0.0/7 and rfc4193 IPv6 addresses and allows multicast for service discovery. It’s essentially a global LAN party.

2.2.6. Ad-Hoc Networks

A special kind of public network called an ad-hoc network may be accessed by joining a network ID with the format:

ffSSSSEEEE000000
| |   |   |
| |   |   Reserved for future use, must be 0
| |   End of port range (hex)
| Start of port range (hex)
Reserved ZeroTier address prefix indicating a controller-less network

Ad-hoc networks are public (no access control) networks that have no network controller. Instead their configuration and other credentials are generated locally. Ad-hoc networks permit only IPv6 UDP and TCP unicast traffic (no multicast or broadcast) using 6plane format NDP-emulated IPv6 addresses. In addition an ad-hoc network ID encodes an IP port range. UDP packets and TCP SYN (connection open) packets are only allowed to desintation ports within the encoded range.

For example ff00160016000000 is an ad-hoc network allowing only SSH, while ff0000ffff000000 is an ad-hoc network allowing any UDP or TCP port.

Keep in mind that these networks are public and anyone in the entire world can join them. Care must be taken to avoid exposing vulnerable services or sharing unwanted files or other resources.

2.2.7. Quality of Service (QoS)

As of 1.4.1 (planned near future) ZeroTier will support new traffic prioritization rules that build on the link measuring features described in Section 2.1.5. Nine buckets will be available for traffic classification: [0, 1, 2, 3, (4), 5, 6, 7, 8], each geometrically-increasing in priority where bucket 4 is the default in the absence of a classification rule for a given type of traffic.

For instance, say you’d like to prioritize your VoIP traffic over standard web traffic:

priority 6
  ipprotocol udp
  and dport 5060-5065
  and sport 5060-5065
;

priority 3
   ipprotocol tcp
   and dport 80
   and sport 80
;

This would place VoIP traffic on ports 5060 to 5065 at a higher priority 6 than the standard port 80 web traffic in bucket 3.

3. The Network Rules Engine

Traffic on ZeroTier networks can be observed and controlled with a system of globally applied network rules. These are enforced in a distributed fashion by both the senders and the receivers of packets. To escape the rules engine a malicious attacker would need to fully compromise both sides of any conversation.

The ZeroTier VL2 rules engine differs from most other firewalls and SDN rules engines in several ways. The most immediately relevant of these is that the ZeroTier rules engine is stateless, meaning it lacks connection tracking. This means that bidirectional whitelisting can’t be accomplished by simply whitelisting reply packets to established connections. Instead some thought must be put into how to allow both sides of a desired flow. Rule patterns to achieve the most common desired objectives are included in this manual.

The decision to make our rules engine stateless was a design trade-off driven by several concerns. First we wanted to keep complexity, code footprint, and memory use very low to support small embedded devices. The second and more fundamental reason is that distributed stateful filtering requires distributed state synchronization. This would have added a large volume of additional sync traffic as well as introducing inescapablenew sources of instability and failure and a lot of surface area for security vulnerabilities.

While ZeroTier lacks state tracking, its rules engine includes something not found anywhere else in the enterprise networking space: capability-based security and device tagging. Capabilities and tags allow extremely complex micro-segmented network rule schemes to be implemented in a sane, conceptual way that is both easier for human beings to understand and more efficient for machines to handle.

This section assumes some level of familiarity with network rules as they’re commonly used on firewalls and routers, etc. While the rules engine is part of VL2, it’s been given its own section in this manual due to the depth and cross-cutting nature of the topic.

3.1. Rule Sets and Rule Evaluation

Rule sets are ordered lists of one or more rules, with each rule consisting of one or more match conditions followed by one action. As a rule set is evaluated, each match is tested in order and is then ANDed or ORed with the previous match result state. When an action is encountered it is taken if the result of the preceding matches is true. An action with no preceding matches is always taken. If no permissive actions are taken by any rule set the packet is discarded.

Here is a simple rule set that constrains Ethernet traffic on a network to only IPv4, ARP, or IPv6 as it would appear in the raw JSON format used by ZeroTier One’s built-in network controller implementation. (Don’t worry if this seems verbose and difficult. We have a more human-friendly way of writing rule sets, but before we introduce it it’s important to understand what is really happening.)

[
  {
    "etherType": 2048,
    "not": true,
    "or": false,
    "type": "MATCH_ETHERTYPE"
  },
  {
    "etherType": 2054,
    "not": true,
    "or": false,
    "type": "MATCH_ETHERTYPE"
  },
  {
    "etherType": 34525,
    "not": true,
    "or": false,
    "type": "MATCH_ETHERTYPE"
  },
  {
    "type": "ACTION_DROP"
  },
  {
    "type": "ACTION_ACCEPT"
  }
]

This checks whether an Ethernet level packet is not IPv4 (ethertype 2048) and not IPv4 ARP (ethertype 2054) and not IPv6 (ethertype 34525). If all three matches evaluate to true (meaning the ethertype is none of these) then the drop action is taken. Otherwise the accept action is taken.

Networks have one base rule set that is applied to all traffic. Its size is constrained to 1024 entries (each match or action is an entry). It should be used to set the overall policies for all members of the network, and for most common use cases it’s all you’ll need. For more complex scenarios, both capabilities and tags provide methods of both managing complexity and scaling the overall size of a network’s rule system.

3.1.1. Actions and Match Conditions

These are the available matches and actions in raw form. The rule definition language outlined in section 3.4 provides a friendlier way for human beings to specify rules.

Action Argument(s) Description
ACTION_DROP Drop packet and terminate all rule evaluation, including capabilities.
ACTION_BREAK Terminate evaluation of this rule set but continue evaluating capabilities.
ACTION_ACCEPT Accept packet and terminate further evaluation.
ACTION_TEE length,address Send a copy of up to the first length bytes (-1 for all) to a ZeroTier address.
ACTION_REDIRECT address Transparently redirect this packet to a ZeroTier address without changing its headers.
Match Condition Argument(s) Description
MATCH_SOURCE_ZEROTIER_ADDRESS zt VL1 source address.
MATCH_DEST_ZEROTIER_ADDRESS zt VL1 destination address.
MATCH_MAC_SOURCE mac Ethernet source address.
MATCH_MAC_DEST mac Ethernet destination address.
MATCH_IPV4_SOURCE ip IPv4 source address.
MATCH_IPV4_DEST ip IPv4 destination address.
MATCH_IPV6_SOURCE ip IPv6 source address.
MATCH_IPV6_DEST ip IPv6 destination address.
MATCH_IP_TOS mask,start,end IP TOS field bitwise-ANDed with mask is within range.
MATCH_IP_PROTOCOL ipProtocol IP protocol number.
MATCH_ETHERTYPE etherType Ethernet frame type.
MATCH_ICMP icmpType,icmpCode ICMP type and code, if applicable. (V4 or V6)
MATCH_IP_SOURCE_PORT_RANGE start,end IP (V4 or V6) source port range (inclusive).
MATCH_IP_DEST_PORT_RANGE start,end IP (V4 or V6) destination port range (inclusive).
MATCH_CHARACTERISTICS mask Bitwise AND characteristic bits with mask, true if result is nonzero.
MATCH_FRAME_SIZE_RANGE start,end Ethernet frame size range (inclusive).
MATCH_RANDOM probability Match if a random 32-bit number is less than or equal to the probability.
MATCH_TAGS_DIFFERENCE id,value Difference between tags with this ID is less than or equal to the value.
MATCH_TAGS_BITWISE_AND id,value Tags ANDed together equal value.
MATCH_TAGS_BITWISE_OR id,value Tags ORed together equal value.
MATCH_TAGS_BITWISE_XOR id,value Tags XORed together equal value.
MATCH_TAGS_EQUAL id,value Both tags equal the same value.
MATCH_TAG_SENDER id,value Sending side’s tag equals this value.
MATCH_TAG_RECEIVER id,value Receiving side’s tag equals this value.

3.2. Capabilities

A capability is a small rule set that is bundled into a credential object, signed by the network controller, and issued to only those member(s) permitted to exercise it. When a member detects that outgoing traffic does not match the base rule set but is allowed by one of its capabilities, it periodically pushes the matching capability credential to the recipient ahead of the packet(s) in question. Peer to peer capability distribution is automatic and is triggered by capability match.

When the recipient receives the capability it authenticates it by checking its signature and timestamp and, provided the capability is valid, adds it to the set of capabilities to apply to incoming traffic from the capability’s owner. The sender has effectively told the recipient “I can too send this packet! Teacher says so!”

Capabilities allow large systems of rules to be broken down into functional aspects and then distributed intelligently only to those members with a need to know. This avoids the bandwidth and storage overhead of distributing huge monolithic rule sets and organizes rules conceptually to make them easier for administrators to understand.

There are three terminating actions that can be taken in a rule set: accept, break, and drop. The accept action terminates rule evaluation and accepts the packet. The break action terminates the evaluation of the current rule set but permits the further evaluation of capabilities. The drop action terminates rule evaluation and drops the packet without checking capabilities in the base rule set, but is equivalent to break in capability rule sets. In most cases break should be used unless certain traffic must be absolutely prohibited under any circumstance.

In the simple base rule set example in section 3.1 the drop action is taken in the unapproved case. This means that ethernet whitelisting cannot be overridden by a capability. If we change ACTION_DROP in our example to ACTION_BREAK, then it becomes possible to issue the following capability:

[
  {
    "etherType": 2114,
    "not": false,
    "or": false,
    "type": "MATCH_ETHERTYPE"
  },
  {
    "type": "ACTION_ACCEPT"
  }
]

Ethertype 2114 is wake-on-LAN, a special packet that can cause some systems to wake from sleep mode. If we place the above tiny rule set into a capability and issue it to a device, this device but no others will now be permitted to send wake-on-LAN magic packets. (Wake-on-LAN requires hardware support so it would only work to target devices plugged into a physical network bridged to a ZeroTier network, but don’t worry about that here. It’s just an example of special traffic.)

Capability rule sets are limited to only 64 entries. The idea is to keep them small and simple. A capability should grant one thing or one small set of conceptually related things.

3.3. Tags

ZeroTier provides a second mechanism to control rule set complexity. Tags are 32-bit numeric key-value pair credentials that are issued to network members and signed by the controller. They are then distributed peer to peer on a need to know basis in a similar manner to capabilities.

Tags provide a way to conditionally drop or allow traffic between members by member classification. They allow very detailed network micro-segmentation by member role, permission, function, etc. without resulting in a combinatorial explosion in rules table size.

Let’s say we want to permit traffic on TCP ports 139 and 445 (netbios/CIFS file sharing) only between systems that belong to the same department. Our company has 12,000 devices and 10 departments. Without tags this would require 144,000,000 rules, but with tags it can be accomplished by only a few.

First a tag is created to represent the department. Let’s give it tag ID 100. Each member system receives the tag with a value from 1 to 10 indicating which department it belongs to. We can then add the following rules to our network’s base rule set (or to a capability if so desired):

[
  {
    "type": "MATCH_IP_DEST_PORT_RANGE",
    "not": false,
    "or": false,
    "start": 139,
    "end": 139
  },
  {
    "type": "MATCH_IP_DEST_PORT_RANGE",
    "not": false,
    "or": true,
    "start": 445,
    "end": 445
  },
  {
    "type": "MATCH_IP_PROTOCOL",
    "not": false,
    "or": false,
    "ipProtocol": 6
  },
  {
    "type": "MATCH_TAGS_DIFFERENCE",
    "not": false,
    "or": false,
    "id": 10,
    "value": 0
  },
  {
    "type": "ACTION_ACCEPT"
  }
]

This tells members in our network to accept TCP packets on ports 139 or 445 if the difference between tags with tag ID 10 is zero, meaning they match. (If a member does not have a value for this tag, it does not match.) Now all members of the same department can access CIFS file shares, but CIFS sharing between departments could still be prohibited. (TCP whitelisting requires some additional rules due to the stateless nature of our rules engine. See the section below on rule design patterns.)

Tags can be compared on numeric value or as bit fields via several different bit mask operations allowing many different systems of member classification to be implemented.

3.4. Rule Definition Language

Raw rule sets are verbose and difficult to write, so we created a minimal rule definition langugage that’s easier for human beings.

The parser for this language can be found in the rule-compiler subfolder of the ZeroTierOne project and as zerotier-rule-compiler on NPM. It’s written in JavaScript and is the same code that powers the in-browser editor in ZeroTier Central.

3.4.1. An Introductory Example

# Whitelist only IPv4 (/ARP) and IPv6 traffic and allow only ZeroTier-assigned IP addresses
drop                      # drop cannot be overridden by capabilities
  not ethertype ipv4      # frame is not ipv4
  and not ethertype arp   # AND is not ARP
  and not ethertype ipv6  # AND is not ipv6
  or not chr ipauth       # OR IP addresses are not authenticated (1.2.0+ only!)
;

# Allow SSH, HTTP, and HTTPS by allowing all TCP packets (including SYN/!ACK) to these ports
accept
  ipprotocol tcp
  and dport 22 or dport 80 or dport 443
;

# Create a tag for which department someone is in
tag department
  id 1000                 # arbitrary, but must be unique
  enum 100 sales          # has no meaning to filter, but used in UI to offer a selection
  enum 200 engineering
  enum 300 support
  enum 400 manufacturing
;

# Allow Windows CIFS and netbios between computers in the same department using a tag
accept
  ipprotocol tcp
  and tdiff department 0  # difference between department tags is 0, meaning they match
  and dport 139 or dport 445
;

# Drop TCP SYN,!ACK packets (new connections) not explicitly whitelisted above
break                     # break can be overridden by a capability
  chr tcp_syn             # TCP SYN (TCP flags will never match non-TCP packets)
  and not chr tcp_ack     # AND not TCP ACK
;

# Create a capability called "superuser" that lets its holders override all but the initial "drop"
cap superuser
  id 1000 # arbitrary, but must be unique
  accept; # allow with no match conditions means allow anything and everything
;

# Accept other packets
accept;

This creates a network that can pass IPv4 (and ARP) and IPv6 traffic but no other Ethernet frame types. In addition the not chr ipauth condition drops traffic between IP addresses that have not been assigned by ZeroTier to their respective sources or destinations, blocking all IP spoofing. These are enforced with a hard drop, preventing them from being overridden by any capability.

All UDP is allowed, but all non-whitelisted new TCP connections (SYN/!ACK packets) are blocked. This is done with a soft break, allowing it to be overridden by capabilities. New TCP connections are allowed on ports 22, 80, and 443 for everyone. Windows file sharing is allowed only between computers in the same department by way of a tag. A super-user capability that can be assigned to administrative nodes allows the sender to initiate any kind of connection.

See section 3.5.1 for a discussion of how we accomplish TCP whitelisting here.

3.4.2. Rule Definition Language Syntax

# The remainder of this line is a comment.

action [ ... args ... ]
  [and|or] [not] match [... args ...]
  [ ... additional matches ... ]
;

tag <name>
  id <id>                 # arbitrary unique 32-bit ID
  [default <value>]       # default tag value to assign to members
  [enum <value> <name>]   # value can be any 32-bit unsigned integer
  [flag <bit> <name>]     # bit can be 0 to 31
  [ ... additional enums or flags ... ]
;

cap <name>
  id <id>                 # arbitrary unique 32-bit ID
  action [ ... args ... ]
    [ ... ]
  ;
  [ ... additional action blocks ... ]
;

macro <name[($var1,...)]>
  action [ ... args ... ]
    [ ... ]
  ;
;

include <name(value,...)>

Each action, tag, capability, or macro block ends with a semicolon. White space separates things. Indentation is not significant. Hash symbols indicate that the remainder of a line is a comment.

As described in sections 3.1 through 3.3, a rule set is composed of one or more sequences of match,[match],…,action in which the action is taken if the chain of matches evaluates to true. Capabilities are small bundles of rules that can be assigned to nodes to give them special abilities, and tags are key/value pairs that can be assigned to nodes to allow rules to be selectively applied.

Macros are just a convenient way of defining more complex actions that can then be applied in multiple places. An example of a macro might be:

macro allowtcp($port)
  accept
    ipprotocol tcp
    and dport $port
  ;
;

You could then allow TCP ports in a standard TCP whitelisting scheme by just saying:

include allowtcp(80)
include allowtcp(443)
include allowtcp(8080)

Tag enum and flag directives have no meaning to the actual ZeroTier rules engine, but they can be used by user interfaces like ZeroTier Central to make it easier to assign and search tags.

3.4.3. Actions, Matches, Operators, and Constants

Action Argument(s) Description
drop Drop packet and terminate all rule evaluation, including capabilities.
break Terminate evaluation of this rule set but continue evaluating capabilities.
accept Accept packet and terminate further evaluation.
tee <length> <address> Send a copy of up to the first length bytes (-1 for all) to a ZeroTier address.
redirect <address> Transparently redirect this packet to a ZeroTier address without changing its headers.
Match Condition Argument(s) Description
ztsrc <address> VL1 source address.
ztdest <address> VL1 destination address.
macsrc <address> Ethernet source address.
macdest <address> Ethernet destination address.
ipsrc <address/prefix> Source IP address or network (V4 or V6 auto-detected).
ipdest <address/prefix> Destination IP address or network (V4 or V6 auto-detected).
iptos <mask> <start[-end]> IP TOS field bitwise-ANDed with mask is within range.
ipprotocol <protocol> IP protocol number.
ethertype <type> Ethernet frame type.
icmp <type> <code> ICMP type (V4 or V6) and code. Use -1 for code if not applicable.
sport <start[-end]> IP (V4 or V6) source port range (inclusive).
dport <start[-end]> IP (V4 or V6) destination port range (inclusive).
chr <mask/name> Bitwise AND characteristic bits with mask, true if result is nonzero.
framesize <start[-end]> Ethernet frame size range (inclusive).
random <probability> Match randomly with probability 0.0 (0%) to 1.0 (100%).
tdiff <id> <value> Difference between tags with this ID is less than or equal to the value.
tand <id> <value> Tags ANDed together equal value.
tor <id> <value> Tags ORed together equal value.
txor <id> <value> Tags XORed together equal value.
teq <id> <value> Both tags equal the same value.
tseq <id> <value> Sending side’s tag equals this value.
treq <id> <value> Receiving side’s tag equals this value.

Match conditions may be joined by and (default if none specified) or or and may be modified by not.

For convenience the following symbols can be used when matching on certain packet attributes:

  • IP protocols: icmp (for IPv4), igmp, ipip, tcp, egp, igp, udp, rdp, esp, ah, icmp6 (for IPv6), l2tp, sctp, and udplite.
  • Ethernet frame types: ipv4, arp, ipv6, wol (wake on LAN), rarp, atalk, aarp, ipxa, ipxb.
  • Packet characteristics (bit masks for chr):
    • inbound: packet is being filtered on the receiving side (use not inbound for sending side)
    • multicast: destination is a multicast or broadcast MAC
    • broadcast: destination is the broadcast MAC
    • ipauth: sender IP is assigned by ZeroTier to the sending node
    • tcp_fin: packet is TCP with FIN flag set
    • tcp_syn: packet is TCP with SYN flag set
    • tcp_rst: packet is TCP with RST flag set
    • tcp_psh: packet is TCP with PSH flag set
    • tcp_ack: packet is TCP with ACK flag set
    • tcp_urg: packet is TCP with URG flag set
    • tcp_ece: packet is TCP with ECE flag set
    • tcp_cwr: packet is TCP with CWR flag set
    • tcp_ns: packet is TCP with NS flag set
    • tcp_rs2: packet is TCP with RS2 (reserved bit 2) flag set
    • tcp_rs1: packet is TCP with RS1 (reserved bit 1) flag set
    • tcp_rs0: packet is TCP with RS0 (reserved bit 0) flag set

3.5. Useful Design Patterns

3.5.1. TCP Whitelisting

First, add this at or near the bottom of your rules:

# Block TCP SYN,!ACK to prevent new non-whitelisted TCP connections from being initiated
# unless previously whitelisted or allowed by a capability.
break chr tcp_syn and not tcp_ack;

Then above the SYN,!ACK break (or in a capability) add rules to allow TCP packets with permitted destination ports:

# Allow TCP port 80 (HTTP)
accept ipprotocol tcp and dport 80;

Section 3.4.1 shows a complete example.

ZeroTier’s filter is stateless. If we block all TCP packets except those with the correct destination port, this will prevent reply packets from returning to their senders.

Allowing reply packets with the correct (inverse) source port seems like a simple fix, but this is insecure as it allows anyone to initiate any TCP connection if they bind to whitelisted source ports.

The best solution is to block all non-whitelisted TCP packets with flags SYN,!ACK (SYN and not ACK) and and then whitelist desired destination ports. This way TCP replies and control traffic are allowed, but new connections cannot be opened unless they are permitted.

3.5.2. Locking Down UDP

UDP is tougher to deal with in a stateless paradigm. It’s connectionless so there is no way to specifically select a new session vs. an existing session. The best way to lock down UDP on a network is to use tags to allow it to and from things like DNS servers that need to speak it.

tag udpserver
  id 1000
  default 0
;

accept
  ipprotocol udp
  and tor udpserver 1
  or chr multicast
;

break ipprotocol udp;

First we define a tag called udpserver with a default value of 0. We don’t set any enums or flags for this tag since it will be used as a boolean. For servers that need to respond to DNS queries, set the udpserver to 1.

Then we accept UDP traffic if the value of the udpserver tag is 1 when both sender and receiver tags are ORed together, or if UDP traffic is multicast. This allows multicast mDNS and Netbios announcements and allows UDP traffic to and from UDP servers, but prohibits other horizontal UDP traffic.

3.5.3. Traffic Observation and Interception

Here’s a simple rule to monitor everything:

# Send a copy of EVERY packet on both sender and receiver side to ZeroTier address "deadbeef11".
tee -1 deadbeef11;

That’s going to flood deadbeef11 with two full copies of every single packet, since it will match on both the sender and the recipient side. A less bandwidth-intensive security monitor setup might look like this:

tee 128 deadbeef11
  chr inbound
  and chr tcp_syn or chr tcp_rst or chr tcp_fin
  or random 0.1
;

tee 128 deadbeef22
  not chr inbound
  and chr tcp_syn or chr tcp_rst or chr tcp_fin
  or random 0.1
;

This is a bit more clever. It sends the first 128 bytes of every TCP SYN, RST, or FIN packet (TCP connection open and close) to one observer on the inbound side and another observer on the outbound side. It also sends the first 128 bytes of other packets with a probability of one in ten. The first 128 bytes of a packet will be enough to see the Ethernet and IP headers as well as layer 4+ information about many protocols.

This would allow observers to watch every new TCP connection on the network and also passively monitor other traffic in a “fuzzy” probabilistic fashion without using very much bandwidth. We split sending and receiving observers to prevent duplicate packets and also to allow us to detect cases where one side is failing to tee traffic more frequently than would be easily explained by packet loss.

There are many, many variations on the above that are possible but hopefully these will be enough to get you started.

Passive out-of-line monitoring with tee is not perfect. If bandwidth between two parties exceeds the bandwidth of one or both parties to the observer, packets will get dropped. It also provides no mechanism to filter traffic in depth since the observer cannot directly intervene (though it could de-authorize a member via the controller API).

Active in-line monitoring can be achieved with redirect:

redirect deadbeef11
  ipprotocol tcp
  and dport 80 or sport 80
;

This will pipe all HTTP traffic through deadbeef11. The target node will receive each Ethernet frame intact including its original Ethernet source and destination MAC address. To forward the packet on to its final destination, simply re-send it unmodified via the same interface on which it was received. The intermediate node could scan, modify, and interrupt HTTP traffic across the network. To regular network participants this is completely invisible as it occurs at layer 2 (Ethernet), though some performance degradation may be noticed especially if the link is running across WAN.

3.5.4. The Classified System Pattern

One useful pattern for access control resembles the way military organizations classify data. Information is deemed classified, and only those who have the required level of classification are allowed to access it.

A model resembling this can be implemented in ZeroTier network rules as follows:

# Is this member classified?
tag classified
  id 2
  enum 0 no
  enum 1 secret
  enum 2 top
  default no
;

# Clearance flags (a bit like groups)
tag clearance
  id 1
  default 0
  flag 0 staging
  flag 1 production
  flag 2 financial
  flag 3 security
  flag 4 executive
;

# If one party is classified, require at least one overlapping clearance bit
break
  not tor classified 0
  and tand clearance 0
;

Initially members will be assigned a default classification of 0 (“no”). These can freely communicate since the bitwise OR of their classified tags will be zero. Neither member possesses a classification requirement.

To restrict access to a member, set its classified tag to secret or top. (In this example there is no difference, but two levels are included in case you want to implement some kind of more detailed segmentation based on these.) Now the first match (not tor classified 0) will be true and the packet will be dropped unless the two communicating members have at least once clearance bit in common (tand clearance 0).

4. ZeroTier One

ZeroTier One is a service that can run on laptops, desktops, servers, virtual machines, and containers to provide virtual network connectivity through a virtual network port much like a VPN client. It can also act as a network controller and as a federated root server.

Binary packages are available on the ZeroTier site and source code is found on GitHub.

After the service is installed and started, networks can be joined using their 16-digit network IDs. Each network appears as a virtual “tap” network port on your system that behaves just like an ordinary Ethernet port.

The ZeroTier One service keeps its configuration and state information in its working directory. It’s found by default at the following locations:

  • Windows: C:\ProgramData\ZeroTier\One
  • Macintosh: /Library/Application Support/ZeroTier/One
  • Linux: /var/lib/zerotier-one
  • FreeBSD/OpenBSD: /var/db/zerotier-one

4.1. JSON API

Macintosh and Windows versions of ZeroTier One come with a graphical UI that runs as a tray / task bar app, and all three versions include a command line interface called zerotier-cli. These interfaces control the service using a local JSON API available via http on ZeroTier’s primary port (9993 by default) and authenticated using a token stored as authtoken.secret in the service’s working directory.

The API may be accessed directly via HTTP GET and POST requests that include the X-ZT1-Auth header whose value must be the contents of authtoken.secret. Most objects and fields are read-only. Networks and moons can be added by POSTing and removed by DELETEing. Bold fields are read/write (rw) or write-only (ro) and can be modified by POSTing a JSON object. Types are strict in POSTs, e.g. an integer field must be 1 and not "1".

GET /status
address string 10-digit (40 bit) ZeroTier address of this node
clock integer Current system clock at time of this request
cluster object|null Cluster status if clustering is enabled (usually null)
config object Contents of local.conf configuration file (see section 4.2)
online boolean True if node can communicate with at least one root
planetWorldId integer World ID of current planet (always 149604618 except in testing scenarios)
planetWorldTimestamp integer Timestamp of current planet
publicIdentity string Public identity of this node (address and public key)
tcpFallbackActive boolean If true, node is tunneling through a ZeroTier TCP relay (slow)
version string ZeroTier One version
versionBuild integer Build version
versionMajor integer Major version number
versionMinor integer Minor version number
versionRev integer Revision portion of version number
GET /peer
[ … array of peers … ]
GET /peer/##########
address string 10-digit ZeroTier address of peer
latency integer Last measured latency in milliseconds
role string LEAF for normal nodes, MOON or PLANET for roots
version string Version of node, if known (learned via direct paths)
versionMajor integer Node major version
versionMinor integer Node minor version
versionRev integer Node revision
paths [object] Array of path objects
   address string Physical address of other end of path
   active boolean Is path currently considered active?
   expired boolean Is path expired?
   preferred boolean Is path preferred?
   lastReceive integer Timestamp of last packet received
   lastSend integer Timestamp of last packet sent
   linkQuality float Link quality, 0.0 to 1.0
   trustedPathId integer Trusted path ID or 0 if not a trusted path
GET /network
[ … array of networks … ]
GET,POST,DELETE /network/################
id string 16-digit network ID
nwid string 16-digit network ID (deprecated)
mac string Ethernet MAC address of this node on this network
mtu integer Port MTU
portDeviceName string Operating system device name or null if not applicable
portError integer OS-specific error code if a port error occurred or 0 if no error
broadcastEnabled boolean True if Ethernet broadcast (ff:ff:ff:ff:ff:ff) is enabled
bridge boolean True if this port can be bridged (L2 bridging)
dhcp boolean Should DHCP be used? (currently unused, always false)
name string Short name of network as configured at controller
status string OK, NOT_FOUND, ACCESS_DENIED, or PORT_ERROR
type string Network authentication mode: PUBLIC or PRIVATE
allowManaged (rw) boolean If true, ZT-managed IPs and routes are assigned
allowGlobal (rw) boolean If true, ZT-managed IPs and routes can overlap public IP space
allowDefault (rw) boolean If true, network can override system default route (full tunnel)
assignedAddresses [string] Array of ZT-managed IP assignments (IP/bits)
routes [object] Array of ZT-managed routes
   target string Network (IP/bits) of target for this route
   via string|null Next hop or null if this is a local (same LAN) network
   flags integer Currently unused, always 0
   metric integer Route metric, currently unused, always 0
GET /moon
[ … array of moons … ]
GET,POST,DELETE /moon/################
id string 16-digit hex moon ID
timestamp integer Most recent timestamp of moon’s world definition
signature string Hex signature
updatesMustBeSignedBy string Public key that must be used to sign the next world update
waiting boolean If true, we are waiting for the seed to respond
seed (ro) string 10-digit ZT address of any member of moon, sent in initial POST
roots [object] Array of root definitions for this moon’s world
   identity string Identity of root (address and public key)
   stableEndpoints [string] Array of root’s physical addresses

4.2. Local Configuration Options

A file called local.conf in the ZeroTier home folder contains configuration options that apply to the local node. It can be used to set up trusted paths, blacklist physical paths, set up physical path hints for certain nodes, and define trusted upstream devices (federated roots). In a large deployment it can be deployed using a tool like Puppet, Chef, SaltStack, etc. to set a uniform configuration across systems. It’s a JSON format file that can also be edited and rewritten by ZeroTier One itself, so ensure that proper JSON formatting is used.

Settings available in local.conf (this is not valid JSON, and JSON does not allow comments):

{
    "physical": { /* Settings that apply to physical L2/L3 network paths. */
        "NETWORK/bits": { /* Network e.g. 10.0.0.0/24 or fd00::/32 */
            "blacklist": true|false, /* If true, blacklist this path for all ZeroTier traffic */
            "trustedPathId": 0|!0 /* If present and nonzero, define this as a trusted path (see below) */
        } /* ,... additional networks */
    },
    "virtual": { /* Settings applied to ZeroTier virtual network devices (VL1) */
        "##########": { /* 10-digit ZeroTier address */
            "try": [ "IP/port"/*,...*/ ], /* Hints on where to reach this peer if no upstreams/roots are online */
            "blacklist": [ "NETWORK/bits"/*,...*/ ] /* Blacklist a physical path for only this peer. */
        }
    },
    "settings": { /* Other global settings */
        "primaryPort": 0-65535, /* If set, override default port of 9993 and any command line port */
        "portMappingEnabled": true|false, /* If true (the default), try to use uPnP or NAT-PMP to map ports */
        "softwareUpdate": "apply"|"download"|"disable", /* Automatically apply updates, just download, or disable built-in software updates */
        "softwareUpdateChannel": "release"|"beta", /* Software update channel */
        "softwareUpdateDist": true|false, /* If true, distribute software updates (only really useful to ZeroTier, Inc. itself, default is false) */
        "interfacePrefixBlacklist": [ "XXX",... ], /* Array of interface name prefixes (e.g. eth for eth#) to blacklist for ZT traffic */
        "allowManagementFrom": "NETWORK/bits"|null, /* If non-NULL, allow JSON/HTTP management from this IP network. Default is 127.0.0.1 only. */
        "allowTcpFallbackRelay": true|false /* Allow or disallow establishment of TCP relay connections (true by default) */
    }
}
  • trustedPathId: A trusted path is a physical network over which encryption and authentication are not required. This provides a performance boost but sacrifices all ZeroTier’s security features when communicating over this path. Only use this feature if you know what you are doing and really need the performance! To set up a trusted path, all devices on the same trusted physical network must have the same trusted path ID. Trusted path IDs are arbitrary unsigned 64-bit integers. These are not secrets. The security of a trusted path depends on its physical configuration. Take special care that any firewalls at its boundaries do not allow traffic in our out with IPs overlapping the trusted network range.

An example local.conf:

{
    "physical": {
        "10.0.0.0/24": {
            "blacklist": true
        },
        "10.10.10.0/24": {
            "trustedPathId": 101010024
        },
    },
    "virtual": {
        "feedbeef12": {
            "role": "UPSTREAM",
            "try": [ "10.10.20.1/9993" ],
            "blacklist": [ "192.168.0.0/24" ]
        }
    },
    "settings": {
        "softwareUpdate": "apply",
        "softwraeUpdateChannel": "release"
    }
}

4.3. Network Controller Configuration

See the network controller microservice docs on GitHub. By default the controller microservice is included in all ZeroTier One builds for Windows, Macintosh, Linux, and BSD (as of version 1.2.0).

4.4. Creating Your Own Roots (a.k.a. Moons)

As of version 1.2.2 this feature is still considered somewhat experimental. Please let us know if you experience problems. We’ll remove this notice when it’s been in the wild for a while.

As described in section 2.1.1 all ZeroTier nodes on a planet effectively inhabit a single data center. This makes it easy to directly connect devices anywhere, but it has the disadvantage of not working without an Internet connection. Network connections are far from perfectly reliable, and sometimes for security reasons a user may wish to “air gap” a set of nodes from the rest of the Internet entirely.

In 1.2.0 we introduced the ability to add your own user-defined roots. Since the data center we inhabit is the planet, a user-defined set of roots is called a moon. When a node “orbits” a moon, it adds the moon’s roots to its root server set. Nodes orbiting moons will still use planetary roots, but they’ll use the moon’s roots if they look faster or if nothing else is available.

The first step in creating a moon is to deploy a set of root servers. In most cases we recommend two. These are regular ZeroTier nodes, but ones that are always on and have static (physical) IP addresses. These static IPs could be global Internet IPs or physical intranet IPs that are only reachable internally. In the latter case your moon’s roots won’t work outside your office, but that doesn’t matter. Roaming nodes will just use planetary roots instead.

We recommend that root servers do not act as network controllers, join networks, or perform any other overlapping functions. They need good reliable network connections but otherwise require very little RAM, storage, or CPU. A root could be a small VM, VPS, or cloud instance, or a small device like a Raspberry Pi. If you provision your roots as VMs, take care that they do not all reside on the same physical hardware. This would defeat the purpose of having two.

The next step is to create a world definition using zerotier-idtool. You will need the identity.public files from each of your root servers. Pick one root (doesn’t matter which) and run zerotier-idtool initmoon <identity.public of one root> >>moon.json. The zerotier-idtoolcommand will output a JSON version of your world definition to stdout, so we redirect it to moon.json.

Examine this file. It will contain something like:

{
  "id": "deadbeef00",
  "objtype": "world",
  "roots": [
    {
      "identity": "deadbeef00:0:34031483094...",
      "stableEndpoints": []
    }
  ],
  "signingKey": "b324d84cec708d1b51d5ac03e75afba501a12e2124705ec34a614bf8f9b2c800f44d9824ad3ab2e3da1ac52ecb39ac052ce3f54e58d8944b52632eb6d671d0e0",
  "signingKey_SECRET": "ffc5dd0b2baf1c9b220d1c9cb39633f9e2151cf350a6d0e67c913f8952bafaf3671d2226388e1406e7670dc645851bf7d3643da701fd4599fedb9914c3918db3",
  "updatesMustBeSignedBy": "b324d84cec708d1b51d5ac03e75afba501a12e2124705ec34a614bf8f9b2c800f44d9824ad3ab2e3da1ac52ecb39ac052ce3f54e58d8944b52632eb6d671d0e0",
  "worldType": "moon"
}

The world ID is technically arbitrary and could be any random 64-bit value. By convention we use the address of one of the roots.

The world definition JSON file contains secrets, so it’s important to keep it in a safe place. The signingKey_SECRET is what will allow you to update your world definition automatically in the future.

Now add your other root(s) and define their stable endpoints. You’ll end up with something that looks like:

{
  "id": "deadbeef00",
  "objtype": "world",
  "roots": [
    {
      "identity": "deadbeef00:0:34031483094...",
      "stableEndpoints": [ "10.0.0.2/9993","2001:abcd:abcd::1/9993" ]
    },
    {
      "identity": "feedbeef11:0:83588158384...",
      "stableEndpoints": [ "10.0.0.3/9993","2001:abcd:abcd::3/9993" ]
    }
  ],
  "signingKey": "b324d84cec708d1b51d5ac03e75afba501a12e2124705ec34a614bf8f9b2c800f44d9824ad3ab2e3da1ac52ecb39ac052ce3f54e58d8944b52632eb6d671d0e0",
  "signingKey_SECRET": "ffc5dd0b2baf1c9b220d1c9cb39633f9e2151cf350a6d0e67c913f8952bafaf3671d2226388e1406e7670dc645851bf7d3643da701fd4599fedb9914c3918db3",
  "updatesMustBeSignedBy": "b324d84cec708d1b51d5ac03e75afba501a12e2124705ec34a614bf8f9b2c800f44d9824ad3ab2e3da1ac52ecb39ac052ce3f54e58d8944b52632eb6d671d0e0",
  "worldType": "moon"
}

The static IP addresses you use must be reachable from all the places you want these roots to serve. If you’re deploying these for use at a physical location, use internal IPs. If you want them to be usable off-site, use public IPs from your DMZ or host them at a cloud provider with a data center presence close to you. Low-cost cloud hosts that provide simple static direct IP addressing and dual-stack IPv4/IPv6 support like Digital Ocean, Vultr, and Linode make ideal places to host roots. The lowest priced instances at these providers are more than sufficient in most cases.

The third step is to generate the actual signed world with zerotier-idtool genmoon moon.json. In this case this will generate a file called 000000deadbeef00.moon. This does not contain secret keys but is signed by the secret from the JSON file.

Now go to your roots, create (if it does not exist) a subdirectory of their working directories (usually /var/lib/zerotier-one on Linux) called moons.d, and copy the signed moon file there. Now restart the roots and they should be ready.

You can add these roots to regular nodes in one of two ways: by placing the same world definition file in their moons.d directories or by using the zerotier-cli orbit command: zerotier-cli orbit deadbeef00 deadbeef00. The first argument is the world ID (which we can shorten by removing the two leading zeroes) and the second is the address of any of its roots. This will contact the root and obtain the full world definition from it if it’s online and reachable.

Once you’ve “orbited” your moon, try zerotier-cli listpeers. You should see the roots you’ve created listed as MOON instead of LEAF. They will now be used as alternative root servers.

5. SDK – Embedding ZeroTier Into an Application

The ZeroTier SDK can be used in two very different ways: libztcore which is the platform-agnostic network virtualization engine, and libzt which is the engine paired with a lightweight userspace network stack. libzt is a superset of libztcore and its usage is distinguished by the fact that it exposes a standard socket API and simple network/service control API. The ZeroTier source code is open source and is licensed under the GNU GPL v3 (not LGPL). If you’d like to embed it in a closed-source commercial product or appliance, please e-mail contact@zerotier.com to discuss commercial licensing. Otherwise it can be used for free. Before we dive into the technicals, the first thing to understand are the differences between the two APIs as each is intended for a very different purpose:

  • libzt is intended for convenience and simplicity:
    • socket API: zts_socket(), zts_connect(), zts_bind(), ...
    • control API: zts_start(), zts_join(), zts_leave(), ....
  • libztcore is intended for raw performance. If your goal is simply moving frames as quickly as possible and you’re willing to put in some extra work, this is what you’re looking for. The API is described in include/ZeroTierOne.h. For an example of how this API is used, see the living documentation that is service/OneService.cpp.
    • core API: ZT_VirtualNetworkFrameFunction(), ZT_WirePacketSendFunction(), ...

5.1 Socket API (libzt)

The libzt project combines our network virtualization engine with a lightweight user-space TCP/IP stack and a Posix-compliant network API. The result can be built into an application as a library, allowing it to access virtual networks without elevated permissions or special OS access to create tun/tap ports.

5.1.1 Starting the Service

The next few sections explain how to use the control API. These functions are non-blocking and will return an error code specified in include/ZeroTierConstants.h and will result in the generation of callback events. It is your responsibility to handle these events.

To start the service, simply call:

zts_start(char *path, void (*userCallbackFunc)(struct zts_callback_msg*), int port)

At this stage, if a cryptographic identity for this node does not already exist, it will generate a new one and store it on disk, the node’s address (commonly referred to as nodeId) will be derived from this identity and will be presented to you upon receiving the ZTS_EVENT_NODE_ONLINE shown below.

NOTE: The first argument path is a path where you will allow ZeroTier to store its automatically-generated cryptographic identity files (identity.public and identity.secret), these files are your keys to communicating on the network. Keep them safe and keep them unique. If any two nodes are online using the same identities you will have a bad time. The second argument userCallbackFunc is a function that you specify to handle all generated events for the life of your program (see below):

void myZeroTierEventCallback(struct zts_callback_msg *msg)
{
    if (msg->eventCode == ZTS_EVENT_NODE_ONLINE) {
        printf("ZTS_EVENT_NODE_ONLINE, nodeId=%llx\n", msg->node->address);
        // You can join networks now!
    }
    // ...
}

After calling zts_start() you will receive one or more of the following events:

ZTS_EVENT_NODE_OFFLINE
ZTS_EVENT_NODE_ONLINE
ZTS_EVENT_NODE_DOWN
ZTS_EVENT_NODE_IDENTITY_COLLISION
ZTS_EVENT_NODE_UNRECOVERABLE_ERROR
ZTS_EVENT_NODE_NORMAL_TERMINATION

After receiving ZTS_EVENT_NODE_ONLINE you will be allowed to join or leave networks.

At the end of your program or when no more network activity is anticipated, the user application can shut down the service with zts_stop(). However, it is safe to leave the service running in the background indefinitely as it doesn’t consume much memory or CPU while at idle. zts_stop()is a non-blocking call and will itself issue a series of events indicating that various aspects of the ZeroTier service have successfully shut down.

It is worth noting that while zts_stop() will stop the service, but the user-space network stack will continue operating in a headless hibernation mode. This is intended behavior due to the fact that the network stack we’ve chosen doesn’t currently support the notion of shutdown since it was initially designed for embedded applications that are simply switched off. If you do need a way to shut everything down and free all resources you can call zts_free(), but please note that calling this function will prevent all subsequent zts_start() calls from succeeding and will require a full application restart if you want to run the service again. The events ZTS_EVENT_NODE_ONLINE and ZTS_EVENT_NODE_OFFLINE can be seen periodically throughout the lifetime of your application depending on the reliability of your underlying network link, these events are lagging indicators and are typically only triggered every thirty (30) seconds.

Lastly, the function zts_restart() is provided as a way to restart the ZeroTier service along with all of its virtual interfaces. The network stack will remain online and undisturbed during this call. Note that this call will temporarily block until the service has fully shut down, then will return and you may then watch for the appropriate startup callbacks mentioned above.

5.1.2 Joining a Network

Joining a ZeroTier virtual network is as easy as calling zts_join(uint64_t networkId). Similarly there is a zts_leave(uint64_t networkId). Note that zts_start() must be called and a ZTS_EVENT_NODE_ONLINE event must be received before these calls will succeed. After calling zts_join()any one of the following events may be generated:

ZTS_EVENT_NETWORK_NOT_FOUND
ZTS_EVENT_NETWORK_CLIENT_TOO_OLD
ZTS_EVENT_NETWORK_REQUESTING_CONFIG
ZTS_EVENT_NETWORK_OK
ZTS_EVENT_NETWORK_ACCESS_DENIED
ZTS_EVENT_NETWORK_READY_IP4
ZTS_EVENT_NETWORK_READY_IP6
ZTS_EVENT_NETWORK_DOWN

ZTS_EVENT_NETWORK_READY_IP4 and ZTS_EVENT_NETWORK_READY_IP6 are combinations of a few different events. They signal that the network was found, joined successfully, an IP address was assigned and the network stack’s interface is ready to process traffic of the indicated type. After this point you should be able to communicate with peers on the network.

5.1.3 Communicating with peers

After successfully starting the service and joining a network, communicating with other nodes (peers) on that network is as easy as it would ordinarily be without ZeroTier. However, one thing to be aware of is the difference between relay and P2P modes. In the event that a direct connection cannot be established between your nodes, ZeroTier offers a free relaying service, this means that your nodes are reachable almost instantaneously but at a temporary performance cost. One should wait to send large amounts of traffic until a ZTS_EVENT_PEER_P2P is received for the node that you’re interested in talking to. This event usually only takes a few seconds to appear after data has initially been sent. Similarly if after some time ZeroTier determines that a previously known path to one of your nodes is no longer available you will see a ZTS_EVENT_PEER_RELAYevent.

One can use zts_get_peer_status(uint64_t peerId) to query the current reachability state of another node. This function will actually return the previously mentioned event values, plus an additional one called ZTS_EVENT_PEER_UNREACHABLE if no known direct path exists between the calling node and the remote node.

5.1.4 Handling events

As mentioned in previous sections, the control API works by use of non-blocking calls and the generation of a few dozen different event types. Depending on the type of event there may be additional contextual information attached to the zts_callback_msg object that you can use. This contextual information will be housed in one of the following structures which are defined in include/ZeroTier.h:

struct zts_callback_msg
{
    int eventCode;
    struct zts_node_details *node;
    struct zts_network_details *network;
    struct zts_netif_details *netif;
    struct zts_virtual_network_route *route;
    struct zts_physical_path *path;
    struct zts_peer_details *peer;
    struct zts_addr_details *addr;
};

Here’s an example of a callback function:

void myZeroTierEventCallback(struct zts_callback_msg *msg)
{
    if (msg->eventCode == ZTS_EVENT_NODE_ONLINE) {
        printf("ZTS_EVENT_NODE_ONLINE, node=%llx\n", msg->node->address);
        // You can join networks now!
    }
}

In this callback function you can perform additional non-blocking API calls or other work. While not returning control to the service isn’t forbidden (the event messages are generated by a separate thread) it is recommended that you return control as soon as possible as not returning will prevent the user application from receiving additional callback event messages which may be time-sensitive.

A typical ordering of messages may look like the following:

ZTS_EVENT_NETIF_UP --- network=a09acf023be465c1, mac=73b7abcfc207, mtu=10000
ZTS_EVENT_ADDR_NEW_IP4 --- addr=11.7.7.184 (on network=a09acf023be465c1)
ZTS_EVENT_ADDR_NEW_IP6 --- addr=fda0:9acf:233:e4b0:7099:9309:4c9b:c3c7 (on network=a09acf023be465c1)
ZTS_EVENT_NODE_ONLINE, node=c4c7ba3cf
ZTS_EVENT_NETWORK_READY_IP4 --- network=a09acf023be465c1
ZTS_EVENT_NETWORK_READY_IP6 --- network=a09acf023be465c1
ZTS_EVENT_PEER_P2P --- node=74d0f5e89d
ZTS_EVENT_PEER_P2P --- node=9d219039f3
ZTS_EVENT_PEER_P2P --- node=a09acf0233

5.1.4.1 Node Events

These events pertain to the state of the current node. This message type will arrive with a zts_node_details object accessible via msg->node. Additionally, one can query the status of the node with zts_get_node_status(), this will return the status as an integer value only.

ZTS_EVENT_NODE_OFFLINE
ZTS_EVENT_NODE_ONLINE
ZTS_EVENT_NODE_DOWN
ZTS_EVENT_NODE_IDENTITY_COLLISION
ZTS_EVENT_NODE_UNRECOVERABLE_ERROR
ZTS_EVENT_NODE_NORMAL_TERMINATION

5.1.4.2 Network Events

These events pertain to the state of the indicated network. This event type will arrive with a zts_network_details object accessible via msg->network. If for example you want to know the number of assigned routes for your network you can use msg->network->num_routes. Similarly for the MTU, use msg->network->mtu. Additionally, one can query the status of the network with zts_get_network_status(uint64_t networkId), this will return the status as an integer value only.

ZTS_EVENT_NETWORK_NOT_FOUND
ZTS_EVENT_NETWORK_CLIENT_TOO_OLD
ZTS_EVENT_NETWORK_REQUESTING_CONFIG
ZTS_EVENT_NETWORK_OK
ZTS_EVENT_NETWORK_ACCESS_DENIED
ZTS_EVENT_NETWORK_READY_IP4
ZTS_EVENT_NETWORK_READY_IP6
ZTS_EVENT_NETWORK_DOWN

5.1.4.3 Peer Events

These events are triggered when the reachability status of a peer has changed, this can happen at any time. This event type will arrive with a zts_peer_details object for additional context. Additionally, one can query the status of the network with zts_get_peer_status(uint64_t peerId), this will return the status as an integer value only.

ZTS_EVENT_PEER_P2P
ZTS_EVENT_PEER_RELAY
ZTS_EVENT_PEER_UNREACHABLE

5.1.4.4 Path Events

These events are triggered when a direct path to a peer has been discovered or is now considered too old to be used. You will see these in conjunction with peer events. This event type will arrive with a zts_physical_path object for additional context.

ZTS_EVENT_PATH_DISCOVERED
ZTS_EVENT_PATH_ALIVE
ZTS_EVENT_PATH_DEAD

5.1.4.5 Route Events

This event type will arrive with a zts_virtual_network_route object for additional context.

ZTS_EVENT_ROUTE_ADDED
ZTS_EVENT_ROUTE_REMOVED

5.1.4.6 Address Events

These events are triggered when new addresses are assigned to the node on a particular virtual network. This event type will arrive with a zts_addr_details object for additional context.

ZTS_EVENT_ADDR_ADDED_IP4
ZTS_EVENT_ADDR_REMOVED_IP4
ZTS_EVENT_ADDR_ADDED_IP6
ZTS_EVENT_ADDR_REMOVED_IP6

5.1.4.7 Network Stack Events (debugging)

These events aren’t very important to the application developer but are important for debugging. These signal whether the userspace networking stack was brought up successfully. You can ignore these in most cases. This event type will arrive with no additional contextual information.

ZTS_EVENT_STACK_UP
ZTS_EVENT_STACK_DOWN

5.1.4.8 Netif Events (debugging)

These events aren’t very important to the application developer but are important for debugging. These signal whether the userspace networking stack was brought up successfully. You can ignore these in most cases. This event type will arrive with a zts_netif_details object for additional context.

ZTS_EVENT_NETIF_UP
ZTS_EVENT_NETIF_DOWN
ZTS_EVENT_NETIF_REMOVED
ZTS_EVENT_NETIF_LINK_UP
ZTS_EVENT_NETIF_LINK_DOWN

5.1.5 Errors

Just as there are two APIs (socket and control), there are two sets of error codes. The control API (zts_start(), zts_join(), etc) errors defined in include/ZeroTierConstants.h are:

  • ZTS_ERR_OK: Everything is ok
  • ZTS_ERR_INVALID_ARG: An argument provided by the user application is invalid (e.g. out of range, NULL, etc)
  • ZTS_ERR_SERVICE: The service isn’t initialized or is for some reason currently unavailable. Try again.
  • ZTS_ERR_INVALID_OP: For some reason this API operation is not permitted or doesn’t make sense at this time.
  • ZTS_ERR_NO_RESULT: The call succeeded, but no object or relevant result was available
  • ZTS_ERR_GENERAL: General internal failure (memory allocation, null reference, etc)

The socket API error codes are defined in doc/errno.h

NOTE: For Android/Java (or similar) which use JNI, the socket API’s error codes are negative values

NOTE: For protocol-level errors such as dropped packets or internal network stack errors, see the section Statistics

5.1.6 Thread Model

The control API for libzt is thread safe and can be called at any time from any thread. There is a single internal lock guarding access to this API. The socket API is similar in this regard. Callback events are generated by a separate thread and are independent from the rest of the API’s internal locking mechanism. Not returning from a callback event won’t impact the rest of the API but it will prevent your application from receiving future events so it is in your application’s best interest to perform as little work as possible in the callback function and promptly return control back to ZeroTier.

Note: Internally, libzt will spawn a number of threads for various purposes: a thread for the core service, a thread for the network stack, a low priority thread to process callback events, and a thread for each network joined. However, the vast majority of work is performed by the core service and stack threads.

5.1.7 Statistics

Protocol and service statistics are available in debug builds of libzt. These statistics are detailed fully in the section of include/ZeroTier.h that is guarded by LWIP_STATS. The protocol constants are defined in include/ZeroTierConstants.h. An example usage is as follows:

C++ example:

struct zts_stats_proto stats;

// Get count of received pings
if (zts_get_protocol_stats(ZTS_STATS_PROTOCOL_ICMP, &stats) == ZTS_ERR_OK) {
    printf("icmp.recv=%d\n", stats.recv);
}

// Get count of dropped TCP packets
if (zts_get_protocol_stats(ZTS_STATS_PROTOCOL_TCP, &stats) == ZTS_ERR_OK) {
    printf("tcp.drop=%d\n", stats.drop);
}

5.1.8 Network Controller Mode

The library form of ZeroTier can act as a network controller and in libzt this is controlled via the zts_controller_* API calls specified in include/ZeroTier.h. Currently controller mode is not available in the iOS and macOS framework builds.

5.2 Frame-based API (libztcore)

For more information on the libztcore API see include/ZeroTierOne.h

5.3 Downloads

SDK bundles for common architectures and platforms can be found here: http://zerotier.com/download.shtml

5.4 Examples

5.4.1 C Example

#include <arpa/inet.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

#include "ZeroTier.h"

bool node_ready = false;
bool network_ready = false;

void myZeroTierEventCallback(struct zts_callback_msg *msg)
{
    switch (msg->eventCode)
    {
        case ZTS_EVENT_NODE_ONLINE:
            printf("ZTS_EVENT_NODE_ONLINE, nodeId=%llx\n", msg->node->address);
            node_ready = true;
            break;
        case ZTS_EVENT_NODE_OFFLINE:
            printf("ZTS_EVENT_NODE_OFFLINE\n");
            node_ready = false;
            break;
        case ZTS_EVENT_NETWORK_READY_IP4:
            printf("ZTS_EVENT_NETWORK_READY_IP4, networkId=%llx\n", msg->network->nwid);
            network_ready = true;
            break;
        case ZTS_EVENT_PEER_P2P:
            printf("ZTS_EVENT_PEER_P2P, nodeId=%llx\n", msg->peer->address);
            break;
        case ZTS_EVENT_PEER_RELAY:
            printf("ZTS_EVENT_PEER_RELAY, nodeId=%llx\n", msg->peer->address);
            break;
        default:
            break;
    }
}

int main()
{
    char *str = "welcome to the machine";
    char *remoteIp = "11.7.7.223";
    int remotePort = 8082;
    int fd, err = 0;
    struct zts_sockaddr_in addr;
    addr.sin_family = ZTS_AF_INET;
    addr.sin_addr.s_addr = inet_addr(remoteIp);
    addr.sin_port = htons(remotePort);

    // Set up ZeroTier service and wait for callbacks
    int port = 9994;
    int nwid = 0x0123456789abcdef;
    zts_start("test/path", &myZeroTierEventCallback, port);
    printf("Waiting for node to come online...\n");
    while (!node_ready) { sleep(1); }
    zts_join(nwid);
    printf("Joined virtual network. Requesting configuration...\n");
    while (!network_ready) { sleep(1); }

    // Socket API example
    if ((fd = zts_socket(AF_INET, SOCK_STREAM, 0)) < 0) {
        printf("error creating socket\n");
    }
    if ((err = zts_connect(fd, (const struct sockaddr *)&addr, sizeof(addr))) < 0) {
        printf("error connecting to remote host\n");
    }
    if ((err = zts_write(fd, str, strlen(str))) < 0) {
        printf("error writing to socket\n");
    }
    zts_close(fd);
    zts_stop();
    return 0;
}

5.3.2 Java Example

Starting ZeroTier:

MyZeroTierEventListener listener = new MyZeroTierEventListener();
ZeroTier.start(getApplicationContext().getFilesDir() + "/zerotier", listener, myPort);
// Wait for EVENT_NODE_ONLINE
while (listener.isOnline == false) {
    try {
        Thread.sleep(interval);
    } catch (Exception e) { }
}
ZeroTier.join(myNetworkId);
// Wait for EVENT_NETWORK_READY_IP4/6
while (listener.isNetworkReady == false) {
    try {
        Thread.sleep(interval);
    } catch (Exception e) { }
}
// Now you can use the socket API!

An example event listener:

import com.zerotier.libzt.ZeroTier;
import com.zerotier.libzt.ZeroTierEventListener;
import com.zerotier.libzt.ZeroTierPeerDetails;

public class MyZeroTierEventListener implements ZeroTierEventListener
{
    public void onZeroTierEvent(long id, int eventCode)
    {
        if (eventCode == ZeroTier.EVENT_NODE_ONLINE) {
            System.out.println("EVENT_NODE_ONLINE: nodeId=" + Long.toHexString(ZeroTier.get_node_id()));
            isOnline = true;
        }
        if (eventCode == ZeroTier.EVENT_NODE_OFFLINE) {
            System.out.println("EVENT_NODE_OFFLINE");
        }
        if (eventCode == ZeroTier.EVENT_NETWORK_READY_IP4) {
            System.out.println("ZTS_EVENT_NETWORK_READY_IP4: nwid=" + Long.toHexString(id));
            if (id == myNetworkId) {
                isNetworkReady = true;
            }
        }
        if (eventCode == ZeroTier.EVENT_PEER_P2P) {
            System.out.println("EVENT_PEER_P2P: id=" + Long.toHexString(id));
        }
        if (eventCode == ZeroTier.EVENT_PEER_RELAY) {
            System.out.println("EVENT_PEER_RELAY: id=" + Long.toHexString(id));
        }
        // ...
    }
}