...a self-contained network virtualization engine that implements an Ethernet virtualization layer similar to VXLAN on top of a global encrypted peer to peer network
This post doesn't go into all of the many reasons we love ZeroTier, but it's certainly one of our favorite network security tools for little reasons like this:
...packets are end-to-end encrypted and can't be read by roots or anyone else, and we use modern 256-bit crypto in ways recommended by the professional cryptographers that created it....
In this long overdue post, we'll share a few tricks that we use to heavily secure our own ZeroTier networks to not only restrict communication to authorized hosts on a directional basis, but even capture and inspect packets!
Mesh P2P VPN network
CONTROL ALL OF THE TRAFFIC!
Flow rules are how we strictly whitelist systems that are allowed to talk on certain networks, such as the OpenSOC.io Blue Team CTF @ DEF CON 26, where we had nearly 300 participants threat hunting in our environment from anywhere with internet access.
The follow snippet is similar to the actual flow rules used at DEF CON last year to keep ~300 participants from "vuln scanning" one another :) You will notice that the rules support comments which is very helpful for annotating the purpose of a particular block of rules.
# Allow only IPv4, IPv4 ARP, and IPv6 Ethernet frames. drop not ethertype ipv4 and not ethertype arp #and not ethertype ipv6 ; # This prevents IP spoofing but also # blocks manual IP management at the OS level and # bridging unless special rules to exempt # certain hosts or traffic are added before # this rule. drop not chr ipauth ; # Rules to protect participants from one another. # # Only allow traffic destined for the DFIR systems that # participants need to access. # # This can also be accomplished using a system's # ZeroTier "node ID", a unique 40-bit (10 hex digit) address # instead of ZT-assigned IP address accept ipprotocol tcp and ipdest 172.23.28.39/32 # opensoc elk or ipdest 172.23.96.133/32 # opensoc moloch or ipdest 172.23.82.175/32 # opensoc kolide or ipdest 172.23.209.200/32 # opensoc graylog or ipdest 172.23.62.148/32 # opensoc moloch or ipdest 172.23.70.65/32 # opensoc wazuh or ipdest 172.23.69.133/32 # opensoc scoreboard and dport 80 or dport 443 ; # Drop TCP SYN,!ACK packets (new connections) # not explicitly whitelisted above break chr tcp_syn # TCP SYN (TCP flags will never match non-TCP packets) and not chr tcp_ack # AND not TCP ACK ; # Accept anything else. This is required since default is 'drop'. accept;
You might notice that we have explicitly whitelisted a list of destinations which means that by default, traffic between clients on the network is prohibited as it does not match the flow rules defined prior to the
TAP ALL OF THE TRAFFIC!
Another incredibly powerful feature of ZeroTier is the ability to tap the entire network regardless of how widely distributed its nodes are. Using the
tee ability within a flow rule essentially copies every frame sent/received by nodes on the network and sends it to a node of your choice such as an IDS or full packet capture solution such as Moloch.
Send a copy of up to the first length bytes (-1 for all) to a ZeroTier address.
This is what
tee looks like inside of a set of flow rules...
# <snip> drop not chr ipauth ; tee -1 8e30f50a25; # copy all traffic to node 8e30f50a25 accept ztsrc a4cd9aef46 # The Gibson ; # <snip>
The snippet above uses
tee to copy full-length packets (hence the
-1 length argument) to ZeroTier node
8e30f50a25 which is a Suricata IDS in our instance.
Sure beats a span port!
In addition to the "tapping" of all traffic in this example, we've explicitly whitelisted "The Gibson" by its ZeroTier node ID
a4cd9aef46 as the only system allowed to initiate traffic as the
ztsrc on this network. This moves us closer to a zero-trust model where even if one of the other nodes became compromised, they are disallowed to communicate with other nodes.
VERSION CONTROL ALL OF THE RULES!
With rules that are as simple as a bash script, it's incredibly easy to store them in a version control system like git, which is exactly what we do.