Over time and for various reasons, I've amassed quite the catalog of cloud-hosted servers. This has caused much anxiety in the form of rapidly expanding attack surface which I've met painstakingly with manually managed firewall rules and nginx ACLs... Not anymore!

OpenVPN hub-and-spoke setup (old school)

OUT WITH OPENVPN

My old infrastructure was moderately secure by means of only being accessible from a jumpbox OpenVPN server. While the public-facing services were exposed to the world, all management services (ssh, netdata, ELK, admin login pages, etc) were locked down to only being accessible from the jumpbox. This setup got the job done, but did not come without its flaws. Managing OpenVPN for many clients is less than fun... Not to mention, the remote systems were not members of the VPN network, they were simply destinations that whitelisted the OpenVPN source IP. This means that if any of the distributed systems needed to talk to one another over insecure protocols, ie mongodb replicas, etc, then additional means needed to be implemented for server-to-server communication.

All of these problems could be solved with the hub-and-spoke OpenVPN method I used previously, but again, the same problems exist.

1. The "hub", or OpenVPN server, becomes a single point of failure. Like any client-server architecture, if the server is down, the clients are SOL.
2. The management of a CA and client configurations/certificates of OpenVPN infrastructure is no fun task unless you throw down some $$ on OpenVPN's web gui (meh..)
3. Unless every system is a client, doesn't provide secure server-to-server comms where a private LAN does not exist.

ENTER ZEROTIER

ZeroTier is an awesome service that facilitates the incredibly easy implementation of mesh VPN networks. It borrows a lot of concepts from SD-WAN and P2P. If you are familiar with these technologies, you will have a much easier time understanding the magic behind ZeroTier.

Borrowed directly from their site:

ZeroTier delivers the capabilities of VPNs, SDN, and SD-WAN with a single system. Manage all your connected resources across both local and wide area networks as if the whole world is a single data center.

Why is this better than OpenVPN?

Many reasons.

Mesh P2P VPN network
also a potential method of summoning cyber demons?

1. Gone are the days of managing a CA and all of the client certificates and configurations. Now, I fully manage all of my networks and clients from a very sophisticated web UI.
   
   Once a client "joins" a network, I can fully manage its network membership from the "Members" section of the web UI. Notice how each client must be authorized onto the network.

2. No more single point of failure! Every system that is joined to the network is now a peer within a mesh. It's a giant software-defined-network. No one system going down can affect all other systems, or your ability to get to them.

3. Every system on the mesh network can securely communicate with every other system! That means insecure apps can chatter within the network without risk of interception. This concept is similar to (the once free) Hamachi for those of you that remember it before LogMeIn acquired and slapped a price tag on it.

4. ZT supports ethernet bridging which effectively extends a switched network between physical locations. You could have a DHCP service at DigitalOcean handing out LAN IP addresses to a cluster of VMs at Vultr, because why not?

5. You can even write network flow rules to restrict communication using attributes from layers 2-4 of the OSI model.

    drop
        not ethertype ipv4
        and not ethertype arp
        and not ethertype ipv6
    ;
    allow
        ipsrc 10.0.0.0/8
        and ipdest 10.0.0.0/8
        and dport 443
    ;
   

Hah! I trust nobody with my secret traffic!

That's cool, because neither does ZeroTier. Not only is the project fully open source for your curious code reviewing adventures, but they have published a very in-depth manual on the technology and the many layers of cryptographic awesomeness that is protecting your precious packets. No need for me to rewrite all of the material they've already published in that guide, so if you have security concerns, I'd recommend starting here. The section starts off perfectly with:

TL;DR: packets are end-to-end encrypted and can't be read by roots or anyone else, and we use modern 256-bit crypto in ways recommended by the professional cryptographers that created it.

If after reading this MIT-level writeup you still can't shake off the tinfoil hat, roll your own version, because open source!

Sounds expensive!

One of the best parts about this service is that it's FREE for unlimited networks and a total of 100 devices. That's far beyond the needs of most average VPS tinkerer types. However, if you have more than 100 devices, fear not because an upgrade to unlimited devices across unlimited networks is a very reasonable $29/month.

This looks complicated...

If you are new to VPNs, SDN, SD-WAN or P2P, this could look a little daunting at first. I will say, once you see how simple it really is, you'll think you did it wrong.

I will follow up this post with a few how-to guides to pave a road for folks looking to get started with ZeroTier.

Also read: Locking Down ZeroTier peer-to-peer networks