ZeroTier Manual

1. Introduction

ZeroTier is a smart Ethernet switch for planet Earth.

It's a distributed network hypervisor built atop a cryptographically secure global peer to peer network. It provides advanced network virtualization and management capabilities on par with an enterprise SDN switch, but across both local and wide area networks and connecting almost any kind of app or device.

This manual describes the design and operation of ZeroTier and its associated services, apps, and libraries. Its intended audience includes IT professionals, network administrators, information security experts, and developers.

The first section (2) of this guide explains ZeroTier's design and operation at a high level and is written for those with at least an intermediate knowledge of topics like TCP/IP and Ethernet networking. It's not required reading for most users, but understanding how things work in detail helps clarify everything else and helps tremendously with troubleshooting should anything go wrong.

The remaining sections deal more concretely with deployment and administration.

Table of Contents

  1. Introduction
  2. Network Hypervisor Overview
    1. VL1: The ZeroTier Peer to Peer Network
      1. Network Topology and Peer Discovery
      2. Addressing
      3. Cryptography
      4. Trusted Paths for Fast Local SDN
    2. VL2: The Ethernet Virtualization Layer
      1. Network Identifiers and Controllers
      2. Certificates and Other Credentials
      3. Multicast, ARP, NDP, and Special Addressing Modes
      4. Ethernet Bridging
      5. Public Networks
      6. Ad-Hoc Networks
  3. The Network Rules Engine
    1. Rule Sets and Rule Evaluation
      1. Actions and Match Conditions
    2. Capabilities
    3. Tags
    4. Rule Definition Language
      1. An Introductory Example
      2. Rule Definition Language Syntax
      3. Actions, Matches, Operators, and Constants
    5. Useful Design Patterns
      1. TCP Whitelisting
      2. Locking Down UDP
      3. Traffic Observation and Interception
      4. The Classified System Pattern
  4. ZeroTier One
    1. JSON API
    2. Local Configuration Options
    3. Network Controller Configuration
    4. Creating Your Own Roots (a.k.a. "Moons")

2. Network Hypervisor Overview

The ZeroTier network hypervisor (currently found in the node/ subfolder of the ZeroTierOne git repository) is a self-contained network virtualization engine that implements an Ethernet virtualization layer similar to VXLAN on top of a global encrypted peer to peer network.

The ZeroTier protocol is original, though aspects of it are similar to VXLAN and IPSec. It has two conceptually separate but closely coupled layers in the OSI model sense: VL1 and VL2. VL1 is the underlying peer to peer transport layer, the "virtual wire," while VL2 is an emulated Ethernet layer that provides operating systems and apps with a familiar communication medium.

2.1. VL1: The ZeroTier Peer to Peer Network

A global data center requires a global wire closet.

In conventional networks L1 (OSI layer 1) refers to the actual CAT5/CAT6 cables or wireless radio channels over which data is carried and the physical transciever chips that modulate and demodulate it. VL1 is a peer to peer network that does the same thing by using encryption, authentication, and a lot of networking tricks to create virtual wires on a dyniamic as-needed basis.

2.1.1. Network Topology and Peer Discovery

VL1 is designed to be zero-configuration. A user can start a new ZeroTier node without having to write configuration files or provide the IP addresses of other nodes. It's also designed to be fast. Any two devices in the world should be able to locate each other and communicate almost instantly.

To achieve this VL1 is organized like DNS. At the base of the network is a collection of always-present root servers whose role is similar to that of DNS root name servers. Roots run the same software as regular endpoints but reside at fast stable locations on the network and are designated as such by a world definition. World definitions come in two forms: the planet and one or more moons. The protocol includes a secure mechanism allowing world definitions to be updated in-band if root servers' IP addresses or ZeroTier addresses change.

There is only one planet. Earth's root servers are operated by ZeroTier, Inc. as a free service. There are currently twelve root servers organized into two six-member clusters distributed across every major continent and multiple network providers. Almost everyone in the world has one within less than 100ms network latency from their location.

A node can "orbit" any number of moons. A moon is just a convenient way to add user-defined root servers to the pool. Users can create moons to reduce dependency on ZeroTier, Inc. infrastructure or to locate root servers closer for better performance. For on-premise SDN use a cluster of root servers can be located inside a building or data center so that ZeroTier can continue to operate normally if Internet connectivity is lost.

Nodes start with no direct links to one another, only upstream to roots (planet and moons). Every peer on VL1 possesses a globally unique 40-bit (10 hex digit) ZeroTier address, but unlike IP addresses these are opaque cryptographic identifiers that encode no routing information. To communicate peers first send packets "up" the tree, and as these packets traverse the network they trigger the opportunistic creation of direct links along the way. The tree is constantly trying to "collapse itself" to optimize itself to the pattern of traffic it is carrying.

Peer to peer connection setup goes like this:

  1. A wants to send a packet to B, but since it has no direct path it sends it upstream to R (a root).
  2. If R has a direct link to B, it forwards the packet there. Otherwise it sends the packet upstream until planetary roots are reached. Planetary roots know about all nodes, so eventually the packet will reach B if B is online.
  3. R also sends a message called rendezvous to A containing hints about how it might reach B. Meanwhile the root that forwards the packet to B sends rendezvous informing B how it might reach A.
  4. A and B get their rendezvous messages and attempt to send test messages to each other, possibly accomplishing hole punching of any NATs or stateful firewalls that happen to be in the way. If this works a direct link is established and packets no longer need to take the scenic route.

Since roots forward packets, A and B can reach each other instantly. A and B then begin attempting to make a direct peer to peer connection. If this succeeds it results in a faster lower latency link. We call this transport triggered link provisioning since it's the forwarding of the packet itself that triggers the peer to peer network to attempt direct connection.

VL1 never gives up. If a direct path can't be established, communication can continue through (slower) relaying. Direct connection attempts continue forever on a periodic basis. VL1 also has other features for establishing direct connectivity including LAN peer discovery, port prediction for traversal of symmetric IPv4 NATs, and explicit port mapping using uPnP and/or NAT-PMP if these are available on the local physical LAN.

A blog post from 2014 by ZeroTier's original author explains some of the reasoning behind VL1's design.

2.1.2. Addressing

Every node is uniquely identified on VL1 by a 40-bit (10 hex digit) ZeroTier address. This address is computed from the public portion of a public/private key pair. A node's address, public key, and private key together form its identity.

On devices running ZeroTier One the node identity is stored in identity.public and identity.secret in the service's home directory.

When ZeroTier starts for the first time it generates a new identity. It then attempts to advertise it upstream to the network. In the very unlikely event that the identity's 40-bit unique address is taken, it discards it and generates another.

Identities are claimed on a first come first serve basis and currently expire from planetary roots after 60 days of inactivity. If a long-dormant device returns it may re-claim its identity unless its address has been taken in the meantime (again, highly unlikely).

The address derivation algorithm used to compute addresses from public keys imposes a computational cost barrier against the intentional generation of a collision. Currently it would take approximately 10,000 CPU-years to do so (assuming e.g. a 3ghz Intel core). This is expensive but not impossible, but it's only the first line of defense. After generating a collision an attacker would then have to compromise all upstream nodes, network controllers, and anything else that has recently communicated with the target node and replace their cached identities.

ZeroTier addresses are, once advertised and claimed, a very secure method of unique identification.

When a node attempts to send a message to another node whose identity is not cached, it sends a whois query upstream to a root. Roots provide an authoritative identity cache.

2.1.3. Cryptography

If you don't know much about cryptography you can safely skip this section. TL;DR: packets are end-to-end encrypted and can't be read by roots or anyone else, and we use modern 256-bit crypto in ways recommended by the professional cryptographers that created it.

Asymmetric public key encryption is Curve25519/Ed25519, a 256-bit elliptic curve variant.

Every VL1 packet is encrypted end to end using (as of the current version) 256-bit Salsa20 and authenticated using the Poly1305 message authentication (MAC) algorithm. MAC is computed after encryption (encrypt-then-MAC) and the cipher/MAC composition used is identical to the NaCl reference implementation.

As of today we do not implement forward secrecy or other stateful cryptographic features in VL1. We don't do this for the sake of simplicity, reliability, and code footprint, and because frequently changing state makes features like clustering and fail-over much harder to implement. See our discussion on GitHub.

We may implement forward secrecy in the future. For those who want this level of security today, we recommend using other cryptographic protocols such as SSL or SSH over ZeroTier. These protocols typically implement forward secrecy, but using them over ZeroTier also provides the secondary benefit of defense in depth. Most cryptography is compromised not by a flaw in encryption but through bugs in the implementation. If you're using two secure transports, the odds of a critical bug being discovered in both at the same time is very low. The CPU overhead of double-encryption is not significant for most work loads.

2.1.4. Trusted Paths for Fast Local SDN

To support the use of ZeroTier as a high performance SDN/NFV protocol over physically secure networks the protocol supports a feature called trusted paths. It is possible to configure all ZeroTier devices on a given network to skip encryption and authentication for traffic over a designated physical path. This can cut CPU use noticably in high traffic scenarios but at the cost of losing virtually all transport security.

Trusted paths do not prevent communication with devices elsewhere, since traffic over other paths will be encrypted and authenticated normally.

We don't recommend the use of this feature unless you really need the performance and you know what you're doing. We also recommend thinking carefully before disabling transport security on a cloud private network. Larger cloud providers such as Amazon and Azure tend to provide good network segregation but many less costly providers offer private networks that are "party lines" and are not much more secure than the open Internet.

2.2. VL2: The Ethernet Virtualization Layer

VL2 is a VXLAN-like network virtualization protocol with SDN management features. It implements secure VLAN boundaries, multicast, rules, capability based security, and certificate based access control.

VL2 is built atop and carried by VL1, and in so doing it inherits VL1's encryption and endpoint authentication and can use VL1 asymmetric keys to sign and verify credentials. VL1 also allows us to implement VL2 entirely free of concern for underlying physical network topology. Connectivity and routing efficiency issues are VL1 concerns. It's important to understand that there is no relationship between VL2 virtual networks and VL1 paths. Much like VLAN multiplexing on a wired LAN, two nodes that share multiple network memberships in common will still only have one VL1 path (virtual wire) between them.

2.2.1. Network Identifiers and Controllers

Each VL2 network (VLAN) is identified by a 64-bit (16 hex digit) ZeroTier network ID that contains the 40-bit ZeroTier address of the network's controller and a 24-bit number identifying the network on the controller.

Network ID: 8056c2e21c123456
            |         |
            |         Network number on controller
            |
            ZeroTier address of controller

When a node joins a network or requests a network configuration update, it sends a network config query message (via VL1) to the network's controller. The controller can then use the node's VL1 address to look it up on the network and send it the appropriate certificates, credentials, and configuration information. From the perpsective of VL2 virtual networks, VL1 ZeroTier addresses can be thought of as port numbers on an enormous global-scale virtual switch.

A common misunderstanding is to conflate network controllers with root servers (planet and moons). Root servers are connection facilitators that operate at the VL1 level. Network controllers are configuration managers and certificate authorities that belong to VL2. Generally root servers don't join or control virtual networks and network controllers are not root servers, though it is possible to have a node do both.

Controller Security Considerations

Network controllers serve as certificate authorities for ZeroTier virtual networks. As such, their identity.secret files should be guarded closely and backed up securely. Compromise of a controller's secret key would allow an attacker to issue fraudulent network configurations or admit unauthorized members, while loss of the secret key results in loss of ability to control the network in any way or issue configuration updates and effectively renders the network unusable.

It is important that controllers' system clocks remain relatively accurate (to within 30-60 seconds) and that they are secure against remote tampering. Many cloud providers provide secure time sources either directly via the hypervisor or via NTP servers within their networks.

2.2.2. Certificates and Other Credentials

All credentials issued by network controllers to member nodes in a given network are signed by the controller's secret key to allow all network members to verify them. Credentials have timestamp fields populated by the controller, allowing relative comparison without the need to trust the node's local system clock.

Credentials are issued only to their owners and are then pushed peer to peer by nodes that wish to communicate with other nodes on the network. This allows networks to grow to enormous sizes without requiring nodes to cache large numbers of credentials or to constantly consult the controller.

Credential Types

  • Certificates of Membership: a certificate that a node presents to obtain the right to communicate on a given network. Certificates of membership are accepted if they agree, meaning that the submitting member's certificate's timestamp differs from the recipient's certificate's timestamp by no more than the recipient certificate's maximum timestamp delta value. This creates a decentralized moving-window scheme for certificate expiration without requiring node clock synchronization or constant checking with the controller.

  • Revocations: a revocation instantaneously revokes a given credential by setting a hard timestamp limit before which it will not be accepted. Revocations are rapidly propagated peer to peer among members of a network using a rumor mill algorithm, allowing a controller to revoke a member credential across the entire network even if its connection to some members is unreliable.

  • Capabilities: a capability is a bundle of network rules that is signed by the controller and can be presented to other members of a network to grant the presenter elevated privileges within the framework of the network's base rule set. More on this in the section on rules.

  • Tags: a tag is a key/value pair signed by the controller that is automatically presented by members to one another and can be matched on in base or capability network rules. Tags can be used to categorize members by role, department, classification, etc.

  • Certificates of Ownership: these certify that a given network member owns something, such as an IP address. These are currently only used to lock down networks against IP address spoofing but could be used in the future to certify ownership of other network-level entities that can be matched in a filter.

2.2.3. Multicast, ARP, NDP, and Special Addressing Modes

ZeroTier networks support multicast via a simple publish/subscribe system.

When a node wishes to receive multicasts for a given multicast group, it advertises membership in this group to other members of the network with which it is communicating and to the network controller. When a node wishes to send a multicast it both consults its cache of recent advertisements and periodically solicits additional advertisements.

Broadcast (Ethernet ff:ff:ff:ff:ff:ff) is treated as a multicast group to which all members subscribe. It can be disabled at the network level to reduce traffic if it is not needed. IPv4 ARP receives special handling (see below) and will still work if normal broadcast is disabled.

Multicasts are propagated using simple sender-side replication. This places the full outbound bandwidth load for multicast on the sender and minimizes multicast latency. Network configurations contain a network-wide multicast limit configurable at the network controller. This specifies the maximum number of other nodes to which any node will send a multicast. If the number of known recipients in a given multicast group exceeds the multicast limit, the sender chooses a random subset.

There is no global limit on multicast recipients, but setting the multicast limit very high on very large networks could result in significant bandwidth overhead.

Special Handling of IPv4 ARP Broadcasts

IPv4 ARP is built on simple Ethernet broadcast and scales poorly on large or distributed networks. To improve ARP's scalability ZeroTier generates a unique multicast group for each IPv4 address detected on its system and then transparently intercepts ARP queries and sends them only to the correct group. This converts ARP into effectively a unicast or narrow multicast protocol (like IPv6 NDP) and allows IPv4 ARP to work reliably across wide area networks without excess bandwidth consumption. A similar strategy is implemented under the hood by a number of enterprise switches and WiFi routers designed for deployment on extremely large LANs. This ARP emulation mode is transparent to the OS and application layers, but it does mean that packet sniffers will not see all ARP queries on a virtual network the way they typically can on smaller wired LANs.

Multicast-Free IPv6 Addressing Modes

IPv6 uses a protocol called NDP in place of ARP. It is similar in role and design but uses narrow multicast in place of broadcast for superior scalability on large networks. This protocol nevertheless still imposes the latency of an additional multicast lookup whenever a new address is contacted. This can add hundreds of milliseconds over a wide area network, or more if latencies associated with pub/sub recipient lookup are significant.

IPv6 addresses are large enough to easily encode ZeroTier addresses. For faster operation and better scaling we've implemented several special IPv6 addressing modes that allow the local node to emulate NDP. These are ZeroTier's rfc4193 and 6plane IPv6 address assignment schemes. If these addressing schemes are enabled on a network, nodes locally intercept outbound NDP queries for matching addresses and then locally generate spoofed NDP replies.

Both modes dramatically reduce initial connection latency between network members. 6plane additionally exploits NDP emulation to transparently assign an entire IPv6 /80 prefix to every node without requiring any node to possess additional routing table entries. This is designed for virtual machine and container hosts that wish to auto-assign IPv6 addresses to guests and is very useful on microservice architecture backplane networks.

Finally there is a security benefit to NDP emulation. ZeroTier addresses are cryptographically authenticated, and since Ethernet MAC addresses on networks are computed from ZeroTier addresses these are also secure. NDP emulated IPv6 addressing modes are therefore not vulnerable to NDP reply spoofing.

Normal non-NDP-emulated IPv6 addresses (including link-local addresses) can coexist with NDP-emulated addressing schemes. Any NDP queries that do not match NDP-emulated addresses are sent via normal multicast.

2.2.4. Ethernet Bridging

ZeroTier emulates a true Ethernet switch. This includes the ability to L2 bridge other Ethernet networks (wired LAN, WiFi, virtual backplanes, etc.) to virtual networks using conventional Ethernet bridging.

To act as a bridge a network member must be designated as such by the controller. This is for security reasons as normal network members are not permitted to send traffic from any origin other than their MAC address. Designated bridges also receive special treatment from the multicast algorithm, which more aggressively and directly queries them for group subscriptions and replicates all broadcast traffic and ARP requests to them. As a result bridge nodes experience a slightly higher amount of multicast bandwidth overhead.

Bridging has been tested extensively on Linux using the Linux kernel native bridge, which cleanly handles network MTU mismatch. There are third party reports of bridging working on other platforms. The details of setting up bridging, including how to selectively block traffic like DHCP that may not be wanted across the bridge, are beyond the scope of this manual.

2.2.5. Public Networks

It is possible to disable access control on a ZeroTier network. A public network's members do not check certificates of membership, and new members to a public network are automatically marked as authorized by their host controller. It is not possible to de-authorize a member from a public network.

Rules on the other hand are enforced, so it's possible to implement a special purpose public network that only allows access to a few things or that only allows a restricted subset of traffic.

Public networks are useful for testing and for peer to peer "party lines" for gaming, chat, and other applications. Participants in public networks are warned to pay special attention to security. If joining a public network be careful not to expose vulnerable services or accidentally share private files via open network shares or HTTP servers. Make sure your operating system, applications, and services are fully up to date.

ZeroTier, Inc. operates a public network called Earth (no relation to the root server planet definition of the same name) with the network ID 8056c2e21c000001. Earth issues IPv4 addresses in the unused IPv4 space 28.0.0.0/7 and rfc4193 IPv6 addresses and allows multicast for service discovery. It's essentially a global LAN party. After joining Earth visit http://earth.zerotier.net/ to get a page showing your Earth virtual IP address and Ethernet MAC address.

2.2.6. Ad-Hoc Networks

A special kind of public network called an ad-hoc network may be accessed by joining a network ID with the format:

ffSSSSEEEE000000
| |   |   |
| |   |   Reserved for future use, must be 0
| |   End of port range (hex)
| Start of port range (hex)
Reserved ZeroTier address prefix indicating a controller-less network

Ad-hoc networks are public (no access control) networks that have no network controller. Instead their configuration and other credentials are generated locally. Ad-hoc networks permit only IPv6 UDP and TCP unicast traffic (no multicast or broadcast) using 6plane format NDP-emulated IPv6 addresses. In addition an ad-hoc network ID encodes an IP port range. UDP packets and TCP SYN (connection open) packets are only allowed to desintation ports within the encoded range.

For example ff00160016000000 is an ad-hoc network allowing only SSH, while ff0000ffff000000 is an ad-hoc network allowing any UDP or TCP port.

Keep in mind that these networks are public and anyone in the entire world can join them. Care must be taken to avoid exposing vulnerable services or sharing unwanted files or other resources.

3. The Network Rules Engine

Traffic on ZeroTier networks can be observed and controlled with a system of globally applied network rules. These are enforced in a distributed fashion by both the senders and the receivers of packets. To escape the rules engine a malicious attacker would need to fully compromise both sides of any conversation.

The ZeroTier VL2 rules engine differs from most other firewalls and SDN rules engines in several ways. The most immediately relevant of these is that the ZeroTier rules engine is stateless, meaning it lacks connection tracking. This means that bidirectional whitelisting can't be accomplished by simply whitelisting reply packets to established connections. Instead some thought must be put into how to allow both sides of a desired flow. Rule patterns to achieve the most common desired objectives are included in this manual.

The decision to make our rules engine stateless was a design trade-off driven by several concerns. First we wanted to keep complexity, code footprint, and memory use very low to support small embedded devices. The second and more fundamental reason is that distributed stateful filtering requires distributed state synchronization. This would have added a large volume of additional sync traffic as well as introducing inescapable new sources of instablity and failure and a lot of surface area for security vulnerabilities.

While ZeroTier lacks state tracking, its rules engine includes something not found anywhere else in the enterprise networking space: capability-based security and device tagging. Capabilities and tags allow extremely complex micro-segmented network rule schemes to be implemented in a sane, conceptual way that is both easier for human beings to understand and more efficient for machines to handle.

This section assumes some level of familiarity with network rules as they're commonly used on firewalls and routers, etc. While the rules engine is part of VL2, it's been given its own section in this manual due to the depth and cross-cutting nature of the topic.

3.1. Rule Sets and Rule Evaluation

Rule sets are ordered lists of one or more rules, with each rule consisting of one or more match conditions followed by one action. As a rule set is evaluated, each match is tested in order and is then ANDed or ORed with the previous match result state. When an action is encountered it is taken if the result of the preceding matches is true. An action with no preceding matches is always taken. If no permissive actions are taken by any rule set the packet is discarded.

Here is a simple rule set that constrains Ethernet traffic on a network to only IPv4, ARP, or IPv6 as it would appear in the raw JSON format used by ZeroTier One's built-in network controller implementation. (Don't worry if this seems verbose and difficult. We have a more human-friendly way of writing rule sets, but before we introduce it it's important to understand what is really happening.)

[
  {
    "etherType": 2048,
    "not": true,
    "or": false,
    "type": "MATCH_ETHERTYPE"
  },
  {
    "etherType": 2054,
    "not": true,
    "or": false,
    "type": "MATCH_ETHERTYPE"
  },
  {
    "etherType": 34525,
    "not": true,
    "or": false,
    "type": "MATCH_ETHERTYPE"
  },
  {
    "type": "ACTION_DROP"
  },
  {
    "type": "ACTION_ACCEPT"
  }
]

This checks whether an Ethernet level packet is not IPv4 (ethertype 2048) and not IPv4 ARP (ethertype 2054) and not IPv6 (ethertype 34525). If all three matches evaluate to true (meaning the ethertype is none of these) then the drop action is taken. Otherwise the accept action is taken.

Networks have one base rule set that is applied to all traffic. Its size is constrained to 1024 entries (each match or action is an entry). It should be used to set the overall policies for all members of the network, and for most common use cases it's all you'll need. For more complex scenarios, both capabilities and tags provide methods of both managing complexity and scaling the overall size of a network's rule system.

3.1.1. Actions and Match Conditions

These are the available matches and actions in raw form. The rule definition language outlined in section 3.4 provides a friendlier way for human beings to specify rules.

Action Argument(s) Description
ACTION_DROPDrop packet and terminate all rule evaluation, including capabilities.
ACTION_BREAKTerminate evaluation of this rule set but continue evaluating capabilities.
ACTION_ACCEPTAccept packet and terminate further evaluation.
ACTION_TEElength,address Send a copy of up to the first length bytes (-1 for all) to a ZeroTier address.
ACTION_REDIRECTaddressTransparently redirect this packet to a ZeroTier address without changing its headers.

Match Condition

Argument(s)

Description
MATCH_SOURCE_ZEROTIER_ADDRESS ztVL1 source address.
MATCH_DEST_ZEROTIER_ADDRESSztVL1 destination address.
MATCH_MAC_SOURCEmacEthernet source address.
MATCH_MAC_DESTmacEthernet destination address.
MATCH_IPV4_SOURCEipIPv4 source address.
MATCH_IPV4_DESTipIPv4 destination address.
MATCH_IPV6_SOURCEipIPv6 source address.
MATCH_IPV6_DESTipIPv6 destination address.
MATCH_IP_TOSmask,start,end IP TOS field bitwise-ANDed with mask is within range.
MATCH_IP_PROTOCOLipProtocolIP protocol number.
MATCH_ETHERTYPEetherTypeEthernet frame type.
MATCH_ICMPicmpType,icmpCode ICMP type and code, if applicable. (V4 or V6)
MATCH_IP_SOURCE_PORT_RANGEstart,endIP (V4 or V6) source port range (inclusive).
MATCH_IP_DEST_PORT_RANGEstart,endIP (V4 or V6) destination port range (inclusive).
MATCH_CHARACTERISTICSmaskBitwise AND characteristic bits with mask, true if result is nonzero.
MATCH_FRAME_SIZE_RANGEstart,endEthernet frame size range (inclusive).
MATCH_RANDOMprobabilityMatch if a random 32-bit number is less than or equal to the probability.
MATCH_TAGS_DIFFERENCEid,valueDifference between tags with this ID is less than or equal to the value.
MATCH_TAGS_BITWISE_ANDid,valueTags ANDed together equal value.
MATCH_TAGS_BITWISE_ORid,valueTags ORed together equal value.
MATCH_TAGS_BITWISE_XORid,valueTags XORed together equal value.
MATCH_TAGS_EQUALid,valueBoth tags equal the same value.
MATCH_TAG_SENDERid,valueSending side's tag equals this value.
MATCH_TAG_RECEIVERid,valueReceiving side's tag equals this value.

3.2. Capabilities

A capability is a small rule set that is bundled into a credential object, signed by the network controller, and issued to only those member(s) permitted to exercise it. When a member detects that outgoing traffic does not match the base rule set but is allowed by one of its capabilities, it periodically pushes the matching capability credential to the recipient ahead of the packet(s) in question. Peer to peer capability distribution is automatic and is triggered by capability match.

When the recipient receives the capability it authenticates it by checking its signature and timestamp and, provided the capability is valid, adds it to the set of capabilities to apply to incoming traffic from the capability's owner. The sender has effectively told the recipient "I can too send this packet! Teacher says so!"

Capabilities allow large systems of rules to be broken down into functional aspects and then distributed intelligently only to those members with a need to know. This avoids the bandwidth and storage overhead of distributing huge monolithic rule sets and organizes rules conceptually to make them easier for administrators to understand.

There are three terminating actions that can be taken in a rule set: accept, break, and drop. The accept action terminates rule evaluation and accepts the packet. The break action terminates the evaluation of the current rule set but permits the further evaluation of capabilities. The drop action terminates rule evaluation and drops the packet without checking capabilities in the base rule set, but is equivalent to break in capability rule sets. In most cases break should be used unless certain traffic must be absolutely prohibited under any circumstance.

In the simple base rule set example in section 3.1 the drop action is taken in the unapproved case. This means that ethernet whitelisting cannot be overridden by a capability. If we change ACTION_DROP in our example to ACTION_BREAK, then it becomes possible to issue the following capability:

[
  {
    "etherType": 2114,
    "not": false,
    "or": false,
    "type": "MATCH_ETHERTYPE"
  },
  {
    "type": "ACTION_ACCEPT"
  }
]

Ethertype 2114 is wake-on-LAN, a special packet that can cause some systems to wake from sleep mode. If we place the above tiny rule set into a capability and issue it to a device, this device but no others will now be permitted to send wake-on-LAN magic packets. (Wake-on-LAN requires hardware support so it would only work to target devices plugged into a physical network bridged to a ZeroTier network, but don't worry about that here. It's just an example of special traffic.)

Capability rule sets are limited to only 64 entries. The idea is to keep them small and simple. A capability should grant one thing or one small set of conceptually related things.

3.3. Tags

ZeroTier provides a second mechanism to control rule set complexity. Tags are 32-bit numeric key-value pair credentials that are issued to network members and signed by the controller. They are then distributed peer to peer on a need to know basis in a similar manner to capabilities.

Tags provide a way to conditionally drop or allow traffic between members by member classification. They allow very detailed network micro-segmentation by member role, permission, function, etc. without resulting in a combinatorial explosion in rules table size.

Let's say we want to permit traffic on TCP ports 139 and 445 (netbios/CIFS file sharing) only between systems that belong to the same department. Our company has 12,000 devices and 10 departments. Without tags this would require 144,000,000 rules, but with tags it can be accomplished by only a few.

First a tag is created to represent the department. Let's give it tag ID 100. Each member system receives the tag with a value from 1 to 10 indicating which department it belongs to. We can then add the following rules to our network's base rule set (or to a capability if so desired):

[
  {
    "type": "MATCH_IP_DEST_PORT_RANGE",
    "not": false,
    "or": false,
    "start": 139,
    "end": 139
  },
  {
    "type": "MATCH_IP_DEST_PORT_RANGE",
    "not": false,
    "or": true,
    "start": 445,
    "end": 445
  },
  {
    "type": "MATCH_IP_PROTOCOL",
    "not": false,
    "or": false,
    "ipProtocol": 6
  },
  {
    "type": "MATCH_TAGS_DIFFERENCE",
    "not": false,
    "or": false,
    "id": 10,
    "value": 0
  },
  {
    "type": "ACTION_ACCEPT"
  }
]

This tells members in our network to accept TCP packets on ports 139 or 445 if the difference between tags with tag ID 10 is zero, meaning they match. (If a member does not have a value for this tag, it does not match.) Now all members of the same department can access CIFS file shares, but CIFS sharing between departments could still be prohibited. (TCP whitelisting requires some additional rules due to the stateless nature of our rules engine. See the section below on rule design patterns.)

Tags can be compared on numeric value or as bit fields via several different bit mask operations allowing many different systems of member classification to be implemented.

3.4. Rule Definition Language

Raw rule sets are verbose and difficult to write, so we created a minimal rule definition langugage that's easier for human beings.

The parser for this language can be found in the rule-compiler subfolder of the ZeroTierOne project and as zerotier-rule-compiler on NPM. It's written in JavaScript and is the same code that powers the in-browser editor in ZeroTier Central.

3.4.1. An Introductory Example

# Whitelist only IPv4 (/ARP) and IPv6 traffic and allow only ZeroTier-assigned IP addresses
drop                      # drop cannot be overridden by capabilities
  not ethertype ipv4      # frame is not ipv4
  and not ethertype arp   # AND is not ARP
  and not ethertype ipv6  # AND is not ipv6
  or not chr ipauth       # OR IP addresses are not authenticated (1.2.0+ only!)
;

# Allow SSH, HTTP, and HTTPS by allowing all TCP packets (including SYN/!ACK) to these ports
accept
  ipprotocol tcp
  and dport 22 or dport 80 or dport 443
;

# Create a tag for which department someone is in
tag department
  id 1000                 # arbitrary, but must be unique
  enum 100 sales          # has no meaning to filter, but used in UI to offer a selection
  enum 200 engineering
  enum 300 support
  enum 400 manufacturing
;

# Allow Windows CIFS and netbios between computers in the same department using a tag
accept
  ipprotocol tcp
  and tdiff department 0  # difference between department tags is 0, meaning they match
  and dport 139 or dport 445
;

# Drop TCP SYN,!ACK packets (new connections) not explicitly whitelisted above
break                     # break can be overridden by a capability
  chr tcp_syn             # TCP SYN (TCP flags will never match non-TCP packets)
  and not chr tcp_ack     # AND not TCP ACK
;

# Create a capability called "superuser" that lets its holders override all but the initial "drop"
cap superuser
  id 1000 # arbitrary, but must be unique
  accept; # allow with no match conditions means allow anything and everything
;

# Accept other packets
accept;

This creates a network that can pass IPv4 (and ARP) and IPv6 traffic but no other Ethernet frame types. In addition the not chr ipauth condition drops traffic between IP addresses that have not been assigned by ZeroTier to their respective sources or destinations, blocking all IP spoofing. These are enforced with a hard drop, preventing them from being overridden by any capability.

All UDP is allowed, but all non-whitelisted new TCP connections (SYN/!ACK packets) are blocked. This is done with a soft break, allowing it to be overridden by capabilities. New TCP connections are allowed on ports 22, 80, and 443 for everyone. Windows file sharing is allowed only between computers in the same department by way of a tag. A super-user capability that can be assigned to administrative nodes allows the sender to initiate any kind of connection.

See section 3.5.1 for a discussion of how we accomplish TCP whitelisting here.

3.4.2. Rule Definition Language Syntax

# The remainder of this line is a comment.

action [ ... args ... ]
  [and|or] [not] match [... args ...]
  [ ... additional matches ... ]
;

tag <name>
  id <id>                 # arbitrary unique 32-bit ID
  [default <value>]       # default tag value to assign to members
  [enum <value> <name>]   # value can be any 32-bit unsigned integer
  [flag <bit> <name>]     # bit can be 0 to 31
  [ ... additional enums or flags ... ]
;

cap <name>
  id <id>                 # arbitrary unique 32-bit ID
  action [ ... args ... ]
    [ ... ]
  ;
  [ ... additional action blocks ... ]
;

macro <name[($var1,...)]>
  action [ ... args ... ]
    [ ... ]
  ;
;

include <name(value,...)>

Each action, tag, capability, or macro block ends with a semicolon. White space separates things. Indentation is not significant. Hash symbols indicate that the remainder of a line is a comment.

As described in sections 3.1 through 3.3, a rule set is composed of one or more sequences of match,[match],...,action in which the action is taken if the chain of matches evaluates to true. Capabilities are small bundles of rules that can be assigned to nodes to give them special abilities, and tags are key/value pairs that can be assigned to nodes to allow rules to be selectively applied.

Macros are just a convenient way of defining more complex actions that can then be applied in multiple places. An example of a macro might be:

macro allowtcp($port)
  accept
    ipprotocol tcp
    and dport $port
  ;
;

You could then allow TCP ports in a standard TCP whitelisting scheme by just saying:

include allowtcp(80)
include allowtcp(443)
include allowtcp(8080)

Tag enum and flag directives have no meaning to the actual ZeroTier rules engine, but they can be used by user interfaces like ZeroTier Central to make it easier to assign and search tags.

3.4.3. Actions, Matches, Operators, and Constants

Action Argument(s) Description
dropDrop packet and terminate all rule evaluation, including capabilities.
breakTerminate evaluation of this rule set but continue evaluating capabilities.
acceptAccept packet and terminate further evaluation.
tee<length> <address> Send a copy of up to the first length bytes (-1 for all) to a ZeroTier address.
redirect<address>Transparently redirect this packet to a ZeroTier address without changing its headers.

Match Condition

Argument(s)

Description
ztsrc <address>VL1 source address.
ztdest<address>VL1 destination address.
macsrc<address>Ethernet source address.
macdest<address>Ethernet destination address.
ipsrc<address/prefix>Source IP address or network (V4 or V6 auto-detected).
ipdest<address/prefix>Destination IP address or network (V4 or V6 auto-detected).
iptos<mask> <start[-end]> IP TOS field bitwise-ANDed with mask is within range.
ipprotocol<protocol>IP protocol number.
ethertype<type>Ethernet frame type.
icmp<type> <code> ICMP type (V4 or V6) and code. Use -1 for code if not applicable.
sport<start[-end]>IP (V4 or V6) source port range (inclusive).
dport<start[-end]>IP (V4 or V6) destination port range (inclusive).
chr<mask/name>Bitwise AND characteristic bits with mask, true if result is nonzero.
framesize<start[-end]>Ethernet frame size range (inclusive).
random<probability>Match randomly with probability 0.0 (0%) to 1.0 (100%).
tdiff<id> <value>Difference between tags with this ID is less than or equal to the value.
tand<id> <value>Tags ANDed together equal value.
tor<id> <value>Tags ORed together equal value.
txor<id> <value>Tags XORed together equal value.
teq<id> <value>Both tags equal the same value.
tseq<id> <value>Sending side's tag equals this value.
treq<id> <value>Receiving side's tag equals this value.

Match conditions may be joined by and (default if none specified) or or and may be modified by not.

For convenience the following symbols can be used when matching on certain packet attributes:

  • IP protocols: icmp (for IPv4), igmp, ipip, tcp, egp, igp, udp, rdp, esp, ah, icmp6 (for IPv6), l2tp, sctp, and udplite.
  • Ethernet frame types: ipv4, arp, ipv6, wol (wake on LAN), rarp, atalk, aarp, ipxa, ipxb.
  • Packet characteristics (bit masks for chr):
    • inbound: packet is being filtered on the receiving side (use not inbound for sending side)
    • multicast: destination is a multicast or broadcast MAC
    • broadcast: destination is the broadcast MAC
    • ipauth: sender IP is assigned by ZeroTier to the sending node
    • tcp_fin: packet is TCP with FIN flag set
    • tcp_syn: packet is TCP with SYN flag set
    • tcp_rst: packet is TCP with RST flag set
    • tcp_psh: packet is TCP with PSH flag set
    • tcp_ack: packet is TCP with ACK flag set
    • tcp_urg: packet is TCP with URG flag set
    • tcp_ece: packet is TCP with ECE flag set
    • tcp_cwr: packet is TCP with CWR flag set
    • tcp_ns: packet is TCP with NS flag set
    • tcp_rs2: packet is TCP with RS2 (reserved bit 2) flag set
    • tcp_rs1: packet is TCP with RS1 (reserved bit 1) flag set
    • tcp_rs0: packet is TCP with RS0 (reserved bit 0) flag set

3.5. Useful Design Patterns

3.5.1. TCP Whitelisting

First, add this at or near the bottom of your rules:

# Block TCP SYN,!ACK to prevent new non-whitelisted TCP connections from being initiated
# unless previously whitelisted or allowed by a capability.
break chr tcp_syn and not tcp_ack;

Then above the SYN,!ACK break (or in a capability) add rules to allow TCP packets with permitted destination ports:

# Allow TCP port 80 (HTTP)
accept ipprotocol tcp and dport 80;

Section 3.4.1 shows a complete example.

ZeroTier's filter is stateless. If we block all TCP packets except those with the correct destination port, this will prevent reply packets from returning to their senders.

Allowing reply packets with the correct (inverse) source port seems like a simple fix, but this is insecure as it allows anyone to initiate any TCP connection if they bind to whitelisted source ports.

The best solution is to block all non-whitelisted TCP packets with flags SYN,!ACK (SYN and not ACK) and and then whitelist desired destination ports. This way TCP replies and control traffic are allowed, but new connections cannot be opened unless they are permitted.

3.5.2. Locking Down UDP

UDP is tougher to deal with in a stateless paradigm. It's connectionless so there is no way to specifically select a new session vs. an existing session. The best way to lock down UDP on a network is to use tags to allow it to and from things like DNS servers that need to speak it.

tag udpserver
  id 1000
  default 0
;

accept
  ipprotocol udp
  and tor udpserver 1
  or chr multicast
;

break ipprotocol udp;

First we define a tag called udpserver with a default value of 0. We don't set any enums or flags for this tag since it will be used as a boolean. For servers that need to respond to DNS queries, set the udpserver to 1.

Then we accept UDP traffic if the value of the udpserver tag is 1 when both sender and receiver tags are ORed together, or if UDP traffic is multicast. This allows multicast mDNS and Netbios announcements and allows UDP traffic to and from UDP servers, but prohibits other horizontal UDP traffic.

3.5.3. Traffic Observation and Interception

Here's a simple rule to monitor everything:

# Send a copy of EVERY packet on both sender and receiver side to ZeroTier address "deadbeef11".
tee -1 deadbeef11;

That's going to flood deadbeef11 with two full copies of every single packet, since it will match on both the sender and the recipient side. A less bandwidth-intensive security monitor setup might look like this:

tee 128 deadbeef11
  chr inbound
  and chr tcp_syn or chr tcp_rst or chr tcp_fin
  or random 0.1
;

tee 128 deadbeef22
  not chr inbound
  and chr tcp_syn or chr tcp_rst or chr tcp_fin
  or random 0.1
;

This is a bit more clever. It sends the first 128 bytes of every TCP SYN, RST, or FIN packet (TCP connection open and close) to one observer on the inbound side and another observer on the outbound side. It also sends the first 128 bytes of other packets with a probability of one in ten. The first 128 bytes of a packet will be enough to see the Ethernet and IP headers as well as layer 4+ information about many protocols.

This would allow observers to watch every new TCP connection on the network and also passively monitor other traffic in a "fuzzy" probabilistic fashion without using very much bandwidth. We split sending and receiving observers to prevent duplicate packets and also to allow us to detect cases where one side is failing to tee traffic more frequently than would be easily explained by packet loss.

There are many, many variations on the above that are possible but hopefully these will be enough to get you started.

Passive out-of-line monitoring with tee is not perfect. If bandwidth between two parties exceeds the bandwidth of one or both parties to the observer, packets will get dropped. It also provides no mechanism to filter traffic in depth since the observer cannot directly intervene (though it could de-authorize a member via the controller API).

Active in-line monitoring can be achieved with redirect:

redirect deadbeef11
  ipprotocol tcp
  and dport 80 or sport 80
;

This will pipe all HTTP traffic through deadbeef11. The target node will receive each Ethernet frame intact including its original Ethernet source and destination MAC address. To forward the packet on to its final destination, simply re-send it unmodified via the same interface on which it was received. The intermediate node could scan, modify, and interrupt HTTP traffic across the network. To regular network participants this is completely invisible as it occurs at layer 2 (Ethernet), though some performance degradation may be noticed especially if the link is running across WAN.

3.5.4. The Classified System Pattern

One useful pattern for access control resembles the way military organizations classify data. Information is deemed classified, and only those who have the required level of classification are allowed to access it.

A model resembling this can be implemented in ZeroTier network rules as follows:

# Is this member classified?
tag classified
  id 2
  enum 0 no
  enum 1 secret
  enum 2 top
  default no
;

# Clearance flags (a bit like groups)
tag clearance
  id 1
  default 0
  flag 0 staging
  flag 1 production
  flag 2 financial
  flag 3 security
  flag 4 executive
;

# If one party is classified, require at least one overlapping clearance bit
break
  not tor classified 0
  and tand clearance 0
;

Initially members will be assigned a default classification of 0 ("no"). These can freely communicate since the bitwise OR of their classified tags will be zero. Neither member possesses a classification requirement.

To restrict access to a member, set its classified tag to secret or top. (In this example there is no difference, but two levels are included in case you want to implement some kind of more detailed segmentation based on these.) Now the first match (not tor classified 0) will be true and the packet will be dropped unless the two communicating members have at least once clearance bit in common (tand clearance 0).

4. ZeroTier One

ZeroTier One is a service that can run on laptops, desktops, servers, virtual machines, and containers to provide virtual network connectivity through a virtual network port much like a VPN client. It can also act as a network controller and as a federated root server.

Binary packages are available on the ZeroTier site and source code is found on GitHub.

After the service is installed and started, networks can be joined using their 16-digit network IDs. Each network appears as a virtual "tap" network port on your system that behaves just like an ordinary Ethernet port.

The ZeroTier One service keeps its configuration and state information in its working directory. It's found by default at the following locations:

  • Windows: C:\ProgramData\ZeroTier\One
  • Macintosh: /Library/Application Support/ZeroTier/One
  • Linux: /var/lib/zerotier-one
  • FreeBSD/OpenBSD: /var/db/zerotier-one

4.1. JSON API

Macintosh and Windows versions of ZeroTier One come with a graphical UI that runs as a tray / task bar app, and all three versions include a command line interface called zerotier-cli. These interfaces control the service using a local JSON API available via http on ZeroTier's primary port (9993 by default) and authenticated using a token stored as authtoken.secret in the service's working directory.

The API may be accessed directly via HTTP GET and POST requests that include the X-ZT1-Auth header whose value must be the contents of authtoken.secret. Most objects and fields are read-only. Networks and moons can be added by POSTing and removed by DELETEing. Bold fields are read/write (rw) or write-only (ro) and can be modified by POSTing a JSON object. Types are strict in POSTs, e.g. an integer field must be 1 and not "1".


GET/status
addressstring10-digit (40 bit) ZeroTier address of this node
clockintegerCurrent system clock at time of this request
clusterobject|nullCluster status if clustering is enabled (usually null)
configobjectContents of local.conf configuration file (see section 4.2)
onlinebooleanTrue if node can communicate with at least one root
planetWorldIdintegerWorld ID of current planet (always 149604618 except in testing scenarios)
planetWorldTimestampintegerTimestamp of current planet
publicIdentitystringPublic identity of this node (address and public key)
tcpFallbackActivebooleanIf true, node is tunneling through a ZeroTier TCP relay (slow)
versionstringZeroTier One version
versionBuildintegerBuild version
versionMajorintegerMajor version number
versionMinorintegerMinor version number
versionRevintegerRevision portion of version number
GET/peer
[ ... array of peers ... ]
GET/peer/##########
addressstring10-digit ZeroTier address of peer
latencyintegerLast measured latency in milliseconds
rolestringLEAF for normal nodes, MOON or PLANET for roots
versionstringVersion of node, if known (learned via direct paths)
versionMajorintegerNode major version
versionMinorintegerNode minor version
versionRevintegerNode revision
paths[object]Array of path objects
   addressstringPhysical address of other end of path
   activebooleanIs path currently considered active?
   expiredbooleanIs path expired?
   preferredbooleanIs path preferred?
   lastReceiveintegerTimestamp of last packet received
   lastSendintegerTimestamp of last packet sent
   linkQualityfloatLink quality, 0.0 to 1.0
   trustedPathIdintegerTrusted path ID or 0 if not a trusted path
GET/network
[ ... array of networks ... ]
GET,POST,DELETE/network/################
idstring16-digit network ID
nwidstring16-digit network ID (deprecated)
macstringEthernet MAC address of this node on this network
mtuintegerPort MTU
portDeviceNamestringOperating system device name or null if not applicable
portErrorintegerOS-specific error code if a port error occurred or 0 if no error
broadcastEnabledbooleanTrue if Ethernet broadcast (ff:ff:ff:ff:ff:ff) is enabled
bridgebooleanTrue if this port can be bridged (L2 bridging)
dhcpbooleanShould DHCP be used? (currently unused, always false)
namestringShort name of network as configured at controller
statusstringOK, NOT_FOUND, ACCESS_DENIED, or PORT_ERROR
typestringNetwork authentication mode: PUBLIC or PRIVATE
allowManaged (rw)booleanIf true, ZT-managed IPs and routes are assigned
allowGlobal (rw)booleanIf true, ZT-managed IPs and routes can overlap public IP space
allowDefault (rw)booleanIf true, network can override system default route (full tunnel)
assignedAddresses[string]Array of ZT-managed IP assignments (IP/bits)
routes[object]Array of ZT-managed routes
   targetstringNetwork (IP/bits) of target for this route
   viastring|nullNext hop or null if this is a local (same LAN) network
   flagsintegerCurrently unused, always 0
   metricintegerRoute metric, currently unused, always 0
GET/moon
[ ... array of moons ... ]
GET,POST,DELETE/moon/################
idstring16-digit hex moon ID
timestampintegerMost recent timestamp of moon's world definition
signaturestringHex signature
updatesMustBeSignedBystringPublic key that must be used to sign the next world update
waitingbooleanIf true, we are waiting for the seed to respond
seed (ro)string10-digit ZT address of any member of moon, sent in initial POST
roots[object]Array of root definitions for this moon's world
   identitystringIdentity of root (address and public key)
   stableEndpoints[string]Array of root's physical addresses

4.2. Local Configuration Options

A file called local.conf in the ZeroTier home folder contains configuration options that apply to the local node. It can be used to set up trusted paths, blacklist physical paths, set up physical path hints for certain nodes, and define trusted upstream devices (federated roots). In a large deployment it can be deployed using a tool like Puppet, Chef, SaltStack, etc. to set a uniform configuration across systems. It's a JSON format file that can also be edited and rewritten by ZeroTier One itself, so ensure that proper JSON formatting is used.

Settings available in local.conf (this is not valid JSON, and JSON does not allow comments):

{
    "physical": { /* Settings that apply to physical L2/L3 network paths. */
        "NETWORK/bits": { /* Network e.g. 10.0.0.0/24 or fd00::/32 */
            "blacklist": true|false, /* If true, blacklist this path for all ZeroTier traffic */
            "trustedPathId": 0|!0 /* If present and nonzero, define this as a trusted path (see below) */
        } /* ,... additional networks */
    },
    "virtual": { /* Settings applied to ZeroTier virtual network devices (VL1) */
        "##########": { /* 10-digit ZeroTier address */
            "try": [ "IP/port"/*,...*/ ], /* Hints on where to reach this peer if no upstreams/roots are online */
            "blacklist": [ "NETWORK/bits"/*,...*/ ] /* Blacklist a physical path for only this peer. */
        }
    },
    "settings": { /* Other global settings */
        "primaryPort": 0-65535, /* If set, override default port of 9993 and any command line port */
        "portMappingEnabled": true|false, /* If true (the default), try to use uPnP or NAT-PMP to map ports */
        "softwareUpdate": "apply"|"download"|"disable", /* Automatically apply updates, just download, or disable built-in software updates */
        "softwareUpdateChannel": "release"|"beta", /* Software update channel */
        "softwareUpdateDist": true|false, /* If true, distribute software updates (only really useful to ZeroTier, Inc. itself, default is false) */
        "interfacePrefixBlacklist": [ "XXX",... ], /* Array of interface name prefixes (e.g. eth for eth#) to blacklist for ZT traffic */
        "allowManagementFrom": "NETWORK/bits"|null /* If non-NULL, allow JSON/HTTP management from this IP network. Default is 127.0.0.1 only. */
    }
}
  • trustedPathId: A trusted path is a physical network over which encryption and authentication are not required. This provides a performance boost but sacrifices all ZeroTier's security features when communicating over this path. Only use this feature if you know what you are doing and really need the performance! To set up a trusted path, all devices on the same trusted physical network must have the same trusted path ID. Trusted path IDs are arbitrary unsigned 64-bit integers. These are not secrets. The security of a trusted path depends on its physical configuration. Take special care that any firewalls at its boundaries do not allow traffic in our out with IPs overlapping the trusted network range.

An example local.conf:

{
    "physical": {
        "10.0.0.0/24": {
            "blacklist": true
        },
        "10.10.10.0/24": {
            "trustedPathId": 101010024
        },
    },
    "virtual": {
        "feedbeef12": {
            "role": "UPSTREAM",
            "try": [ "10.10.20.1/9993" ],
            "blacklist": [ "192.168.0.0/24" ]
        }
    },
    "settings": {
        "softwareUpdate": "apply",
        "softwraeUpdateChannel": "release"
    }
}

4.3. Network Controller Configuration

See the network controller microservice docs on GitHub. By default the controller microservice is included in all ZeroTier One builds for Windows, Macintosh, Linux, and BSD (as of version 1.2.0).

4.4. Creating Your Own Roots (a.k.a. Moons)

As of version 1.2.2 this feature is still considered somewhat experimental. Please let us know if you experience problems. We'll remove this notice when it's been in the wild for a while.

As described in section 2.1.1 all ZeroTier nodes on a planet effectively inhabit a single data center. This makes it easy to directly connect devices anywhere, but it has the disadvantage of not working without an Internet connection. Network connections are far from perfectly reliable, and sometimes for security reasons a user may wish to "air gap" a set of nodes from the rest of the Internet entirely.

In 1.2.0 we introduced the ability to add your own user-defined roots. Since the data center we inhabit is the planet, a user-defined set of roots is called a moon. When a node "orbits" a moon, it adds the moon's roots to its root server set. Nodes orbiting moons will still use planetary roots, but they'll use the moon's roots if they look faster or if nothing else is available.

The first step in creating a moon is to deploy a set of root servers. In most cases we recommend two. These are regular ZeroTier nodes, but ones that are always on and have static (physical) IP addresses. These static IPs could be global Internet IPs or physical intranet IPs that are only reachable internally. In the latter case your moon's roots won't work outside your office, but that doesn't matter. Roaming nodes will just use planetary roots instead.

We recommend that root servers do not act as network controllers, join networks, or perform any other overlapping functions. They need good reliable network connections but otherwise require very little RAM, storage, or CPU. A root could be a small VM, VPS, or cloud instance, or a small device like a Raspberry Pi. If you provision your roots as VMs, take care that they do not all reside on the same physical hardware. This would defeat the purpose of having two.

The next step is to create a world definition using zerotier-idtool. You will need the identity.public files from each of your root servers. Pick one root (doesn't matter which) and run zerotier-idtool initmoon <identity.public of one root> >>moon.json. The zerotier-idtool command will output a JSON version of your world definition to stdout, so we redirect it to moon.json.

Examine this file. It will contain something like:

{
  "id": "deadbeef00",
  "objtype": "world",
  "roots": [
    {
      "identity": "deadbeef00:0:34031483094...",
      "stableEndpoints": []
    }
  ],
  "signingKey": "b324d84cec708d1b51d5ac03e75afba501a12e2124705ec34a614bf8f9b2c800f44d9824ad3ab2e3da1ac52ecb39ac052ce3f54e58d8944b52632eb6d671d0e0",
  "signingKey_SECRET": "ffc5dd0b2baf1c9b220d1c9cb39633f9e2151cf350a6d0e67c913f8952bafaf3671d2226388e1406e7670dc645851bf7d3643da701fd4599fedb9914c3918db3",
  "updatesMustBeSignedBy": "b324d84cec708d1b51d5ac03e75afba501a12e2124705ec34a614bf8f9b2c800f44d9824ad3ab2e3da1ac52ecb39ac052ce3f54e58d8944b52632eb6d671d0e0",
  "worldType": "moon"
}

The world ID is technically arbitrary and could be any random 64-bit value. By convention we use the address of one of the roots.

The world definition JSON file contains secrets, so it's important to keep it in a safe place. The signingKey_SECRET is what will allow you to update your world definition automatically in the future.

Now add your other root(s) and define their stable endpoints. You'll end up with something that looks like:

{
  "id": "deadbeef00",
  "objtype": "world",
  "roots": [
    {
      "identity": "deadbeef00:0:34031483094...",
      "stableEndpoints": [ "10.0.0.2/9993","2001:abcd:abcd::1/9993" ]
    },
    {
      "identity": "feedbeef11:0:83588158384...",
      "stableEndpoints": [ "10.0.0.3/9993","2001:abcd:abcd::3/9993" ]
    }
  ],
  "signingKey": "b324d84cec708d1b51d5ac03e75afba501a12e2124705ec34a614bf8f9b2c800f44d9824ad3ab2e3da1ac52ecb39ac052ce3f54e58d8944b52632eb6d671d0e0",
  "signingKey_SECRET": "ffc5dd0b2baf1c9b220d1c9cb39633f9e2151cf350a6d0e67c913f8952bafaf3671d2226388e1406e7670dc645851bf7d3643da701fd4599fedb9914c3918db3",
  "updatesMustBeSignedBy": "b324d84cec708d1b51d5ac03e75afba501a12e2124705ec34a614bf8f9b2c800f44d9824ad3ab2e3da1ac52ecb39ac052ce3f54e58d8944b52632eb6d671d0e0",
  "worldType": "moon"
}

The static IP addresses you use must be reachable from all the places you want these roots to serve. If you're deploying these for use at a physical location, use internal IPs. If you want them to be usable off-site, use public IPs from your DMZ or host them at a cloud provider with a data center presence close to you. Low-cost cloud hosts that provide simple static direct IP addressing and dual-stack IPv4/IPv6 support like Digital Ocean, Vultr, and Linode make ideal places to host roots. The lowest priced instances at these providers are more than sufficient in most cases.

The third step is to generate the actual signed world with zerotier-idtool genmoon moon.json. In this case this will generate a file called 000000deadbeef00.moon. This does not contain secret keys but is signed by the secret from the JSON file.

Now go to your roots, create (if it does not exist) a subdirectory of their working directories (usually /var/lib/zerotier-one on Linux) called moons.d, and copy the signed moon file there. Now restart the roots and they should be ready.

You can add these roots to regular nodes in one of two ways: by placing the same world definition file in their moons.d directories or by using the zerotier-cli orbit command: zerotier-cli orbit deadbeef00 deadbeef00. The first argument is the world ID (which we can shorten by removing the two leading zeroes) and the second is the address of any of its roots. This will contact the root and obtain the full world definition from it if it's online and reachable.

Once you've "orbited" your moon, try zerotier-cli listpeers. You should see the roots you've created listed as MOON instead of LEAF. They will now be used as alternative root servers.