I tweeted this the other day, and had a lot of folks reach out asking for more details/a diagram of this setup.
places like digitalocean and vultr have beefy systems for less
— Whitney Champion (@shortxstack) November 20, 2019
SO, we stand up a big system there, and a small proxy in AWS behind an ALB/cognito/okta
lock them down, join them to the same @ZeroTier network, and 💥
less AWS$, secure, reliable
As I mentioned in the tweet, sometimes we need to spin up ad-hoc systems on the fly. Often times, it's for an incident response engagement. We're huge fans of Skadi and Timesketch--these things need a LOT of horsepower. That comes at a price, as many of you know, especially in some of the bigger cloud providers.
We're also huge fans of Vultr and Digital Ocean. They provide that horsepower for a lot less money, BUT, they don't have all the other bells and whistles that AWS provides us.
Problem is, some of those bells and whistles we absolutely need. We use Okta for just about everything in our organization. Single sign on / multifactor is critical. For a lot of our infrastructure, we can do this very easily by integrating it with AWS Cognito and our ALB's. We also have other mechanisms in place in various Lambda functions that handle other pieces of the auth flow.
We wanted the best of both of these worlds. Cheaper systems, with all the same security as the rest of our infrastructure. Who doesn't, right?
This is basically what that looks like.
We spin up a small instance (Nginx, Apache, HAProxy, it can be whatever you want) in AWS.
We have our Big Giant Server in Vultr (or the provider of your choice).
We are heavy users of ZeroTier, a mesh VPN service (you should go read more about it in Eric's blog posts here and here). We install ZeroTier on each system, and join them to the same network. This allows them to talk to each other and literally takes seconds to set up.
Cognito (handling our Okta integration) is in front of the ALB, which is in front of the proxy, which sends us straight to our Big Giant Server.
SSH is via bastion and VPN only, and absolutely nothing is open to the public except for UDP 9993 (necessary for ZeroTier), and 80/443 for the ALB.
There are likely other ways to tackle this problem--this is the route we took, and it works quite well :)