Using Flow Rules To Direct Users to Services

Erik Hollensbe
May 18, 2022

The setup

I have a network with several services on it:

And I want to make sure a number of things are possible:

  • All users can reach the above 3 services on their appropriate ports

  • All users can resolve DNS.

  • Users cannot see each other or interact with each other.

This is actually quite simple with flow rules in a way that is not affected by name or IP changes.

Identity

ZeroTier has the notion of universal identity which is a key pair that identifies your client, stored with the client is the private key, and the public key is used for interactions with external entities. With this identity you can authenticate yourself to networks; if you’ve used the product you’re probably familiar with what an identity is at its basics.

Flow Rules can be used to manage networks by identity by providing an authorization layer. The ztsrc and ztdest flow rule directives can be used to allow traffic to and from identities; something we’ll see used later on in this article.

Rule Evaluation

In ZeroTier Flow Rules, rules are evaluated to the end of the rule set unless a break, drop, or accept statement is received. This means:

break not ethertype ipv4; accept;

Will accept packets for ipv4 but will not accept ipv6.

Grouping

Flow Rules have a unique method of grouping themselves: simply left to right. What this means is that this will have a different effect than you think:

accept ztdest 1234567890 and dport 22 or dport 80; drop;

Parentheses aren’t allowed in the flow rules syntax, but if they were this would look something like:

accept ((ztdest 1234567890 and dport 22) or dport 80); drop;

The appropriate rule is:

accept ztdest 1234567890 and dport 22; accept ztdest 1234567890 and dport 80; drop;

Our Rule Set

So, to get what we want assuming we want to have SSH access as well, we need to:

  • Allow 22 and 53 over TCP and UDP on ZeroNSD

  • Allow ports 22, 80, and 32400 over TCP on Plex. Also support multicast for DLNA operation.

  • Allow ports 22 and 80 on TCP for Gitea.

So let’s go over what that would look like. Let’s assume these identities for the different hosts:

  • deadbeef01 is ZeroNSD, our DNS server.

  • cafebead01 is Plex, our media server.

  • feedcafe01 is Gitea, our code repository.

To get these identities, one can zerotier-cli info from the CLI on the host, or check inside ZeroTier Central:

Installing the Rule Set

Look for this dialog:

Then install this rule set in the control panel:

drop not ethertype ipv4 and not ethertype ipv6 and not ethertype arp;

drop not chr ipauth;

accept ztdest deadbeef01 and dport 22 and ipprotocol tcp; accept ztdest deadbeef01 and dport 53; accept ztsrc deadbeef01;

accept ztdest cafebead01 and dport 22 and ipprotocol tcp; accept ztdest cafebead01 and dport 80 and ipprotocol tcp; accept ztdest cafebead01 and dport 32400 and ipprotocol tcp; accept ztdest cafebead01 and chr multicast; accept ztsrc cafebead01;

accept ztdest feedcafe01 and dport 80 and ipprotocol tcp; accept ztdest feedcafe01 and dport 22 and ipprotocol tcp; accept ztsrc feedcafe01;

accept ethertype arp;

drop;

Let’s go through these lines group by group to cover what they do.

drop not ethertype ipv4 and not ethertype ipv6 and not ethertype arp;

This rule just drops all results that are not basic IP traffic; since ZeroTier is layer 2, it can be used for more than internet traffic.

drop not chr ipauth;

This makes sure that all IPs that go through this filter rule set are authenticated via an identity.

accept ztdest deadbeef01 and dport 22 and ipprotocol tcp; accept ztdest deadbeef01 and dport 53; accept ztsrc deadbeef01;

This rule, and the others like it, allow ports into the destination (in this case, ZeroNSD’s host) and allow traffic to exit that destination to the hosts that request responses from it. The others are similar, just for different ports.

NOTE: there is a rule set feature for macros which could theoretically make this easier; these are being overhauled at this time due to a few issues with them. It is suggested you just inline all your rules for now.

drop;

Finally, drop all remaining traffic. This will disallow all other nodes from talking to each other.

Applying Rule Sets

Rule sets have to propagate to a large p2p network, so sometimes this can take some time — over a minute in some cases. Measure twice, cut once!

Getting More Advanced

Here’s a more advanced rule set that uses capabilities to expose dynamic configuration to ZeroTier Central so you can write one set of rules, and apply them to different hosts through the GUI.

cap dns id 10 accept ipprotocol udp; accept ipprotocol tcp; accept dport 53; ;

cap http id 11 accept dport 80 or dport 443 and ipprotocol tcp; ;

cap htpc id 12 accept dport 80 or dport 32400 and ipprotocol tcp; ;

cap admin id 42 accept;

drop not chr ipauth;

Let arp, ping to servers work.

accept ethertype arp or ipprotocol icmp4;

accept dport 53;

accept;

break; # not drop, so we can use capabilities.

After saving these rules and reloading the network’s page, you will see this near the bottom of the page:

Which allows you to enable the companion rule sets for the different capabilities. With this rule:

cap dns id 10 accept ipprotocol udp; accept ipprotocol tcp; accept dport 53; ;

Checking the dns capability now enables the rules:

accept ipprotocol udp; accept ipprotocol tcp; accept dport 53;

Which are what we need to serve DNS. http and htpc reveal ports over TCP.

If you have no capabilities, you can reach the servers that have their traffic exposed; otherwise you cannot reach anything. Just like a coffee shop! However, if you are admin you can go anywhere.

Feedback on rule sets

We’re actively seeking feedback for rule set improvements; if you think you have some ideas, hit us up on our Tracker and we can start the conversation!

...

Author’s note: Many thanks to ZeroTier Engineer Travis LaDuke for his help co-authoring this article.