In a previous post, we shared our affinity for ZeroTier:

...a self-contained network virtualization engine that implements an Ethernet virtualization layer similar to VXLAN on top of a global encrypted peer to peer network

This post doesn't go into all of the many reasons we love ZeroTier, but it's certainly one of our favorite network security tools for little reasons like this:

...packets are end-to-end encrypted and can't be read by roots or anyone else, and we use modern 256-bit crypto in ways recommended by the professional cryptographers that created it....

In this long overdue post, we'll share a few tricks that we use to heavily secure our own ZeroTier networks to not only restrict communication to authorized hosts on a directional basis, but even capture and inspect packets!

Mesh P2P VPN network

CONTROL ALL OF THE TRAFFIC!

One of the most powerful features of ZeroTier is its "flow rules." Example network flow rules can be found in ZeroTier's documentation.

Flow rules are how we strictly whitelist systems that are allowed to talk on certain networks, such as the OpenSOC.io Blue Team CTF @ DEF CON 26, where we had nearly 300 participants threat hunting in our environment from anywhere with internet access.

The follow snippet is similar to the actual flow rules used at DEF CON last year to keep ~300 participants from "vuln scanning" one another :) You will notice that the rules support comments which is very helpful for annotating the purpose of a particular block of rules.

# Allow only IPv4, IPv4 ARP, and IPv6 Ethernet frames.
drop
	not ethertype ipv4
	and not ethertype arp
	#and not ethertype ipv6
;

# This prevents IP spoofing but also 
# blocks manual IP management at the OS level and
# bridging unless special rules to exempt 
# certain hosts or traffic are added before
# this rule.
drop
	not chr ipauth
;

# Rules to protect participants from one another.
#
# Only allow traffic destined for the DFIR systems that 
# participants need to access.
#
# This can also be accomplished using a system's
# ZeroTier "node ID", a unique 40-bit (10 hex digit) address 
# instead of ZT-assigned IP address
accept
  ipprotocol tcp
	and ipdest 172.23.28.39/32 # opensoc elk
	or ipdest 172.23.96.133/32 # opensoc moloch
	or ipdest 172.23.82.175/32 # opensoc kolide
	or ipdest 172.23.209.200/32 # opensoc graylog
	or ipdest 172.23.62.148/32 # opensoc moloch
	or ipdest 172.23.70.65/32 # opensoc wazuh
	or ipdest 172.23.69.133/32 # opensoc scoreboard
  and dport 80 or dport 443
;
# Drop TCP SYN,!ACK packets (new connections) 
# not explicitly whitelisted above
break
  chr tcp_syn             # TCP SYN (TCP flags will never match non-TCP packets)
  and not chr tcp_ack     # AND not TCP ACK
;

# Accept anything else. This is required since default is 'drop'.
accept;

You might notice that we have explicitly whitelisted a list of destinations which means that by default, traffic between clients on the network is prohibited as it does not match the flow rules defined prior to the break rule.

TAP ALL OF THE TRAFFIC!

Another incredibly powerful feature of ZeroTier is the ability to tap the entire network regardless of how widely distributed its nodes are. Using the tee ability within a flow rule essentially copies every frame sent/received by nodes on the network and sends it to a node of your choice such as an IDS or full packet capture solution such as Moloch.

The tee action in ZeroTier is similar to the tee command in Linux.

Send a copy of up to the first length bytes (-1 for all) to a ZeroTier address.

This is what tee looks like inside of a set of flow rules...

# <snip>

drop
	not chr ipauth
;

tee -1 8e30f50a25; # copy all traffic to node 8e30f50a25

accept
	ztsrc a4cd9aef46 # The Gibson
;

# <snip>

The snippet above uses tee to copy full-length packets (hence the -1 length argument) to ZeroTier node 8e30f50a25 which is a Suricata IDS in our instance.

Sure beats a span port!

In addition to the "tapping" of all traffic in this example, we've explicitly whitelisted "The Gibson" by its ZeroTier node ID a4cd9aef46 as the only system allowed to initiate traffic as the ztsrc on this network. This moves us closer to a zero-trust model where even if one of the other nodes became compromised, they are disallowed to communicate with other nodes.

VERSION CONTROL ALL OF THE RULES!

With rules that are as simple as a bash script, it's incredibly easy to store them in a version control system like git, which is exactly what we do.