Dealing with hostnames, domains and environments

Unless you've completely jumped on the container band wagon you probably still need to maintain a number of stateful servers, which typically live in a number of environments, like dev, stg and prd. You probably also have a number of servers in a cluster spread out on different availability zones.

To keep things simple and predictable we have to use a consistent naming scheme. What pattern you end up choosing doesn't matter that much, as long as you're doing it consistently. The way described is how I like to do it.

Hostnames

Say you have a server named foobar in all environments, and you ssh into the one in prd. How would you tell? There's nothing in the default shell prompt that would suggest which environment you're currently working on. However, if you prefix the hostname with current environment your local prompt will read like vegardx@prd-foobar:~$.

And foobar is probably not a single server, more like a cluster of machines. So we add n+1 to the hostname, which helps us tell exactly what machine we're talking to. And now your prompt looks like vegardx@prd-foobar01:~$, where server number 01 in cluster foobar on environment prd.

Since we're dealing with multiple availability zones it also makes sense to add information about that to the hostname, so we postfix that to name as well, which makes the prompt look quite descriptive: vegardx@prd-foobar01a, where a defines the availability zone within the region.

flowchart LR
    A["Env<br/>prd"] --> B["Name<br/>foobar"] --> C["Count<br/>01"] --> D["AZ<br/>a"]

Service domain

Since we now have fairly predictable and unique hostnames we should extend this with a domain name as well. When working with multiple environments it makes sense to do sub-zone delegation, so that each environment has its own zone, and is completely isolated from each other.

flowchart TD
    root["netwerk.io"] --> dev["dev.netwerk.io"]
    root --> stg["stg.netwerk.io"]
    root --> prd["prd.netwerk.io"]

So now the fully qualified domain name for a machine in prd would be something like prd-foobar07c.prd.netwerk.io. The environment gets duplicated since we do sub-zone delegation, as mentioned earlier.

Clusters are named in the same predictable way, we just remove the count and availability zone in the name, like so prd-foobar.prd.netwerk.io. This could point to a load balancer in front of machines or machines directly. This is very useful as it can be used by other domains with CNAME.

SSH and ProxyJump

We now have a completely predictable and consistent naming scheme that also resolves on every level we can add a few lines to our local ssh config to ssh directly into a host, via bastion / jump hosts.

In this scenario I don't need to expose my ssh-agent, and if I need to ssh into the provisioning use (-l admin for Debian on AWS) the key is already loaded up. With ssh prd-foobar01a it will automatically ProxyJump via prd-bastion01a.prd.netwerk.io and into prd-foobar01a.prd.netwerk.io.

Host dev stg prd
  HostName %h-bastion01a.%h.netwerk.io
  IdentityFile ~/.ssh/id_rsa
  UseKeychain yes

Host dev-*
  HostName %h.dev.netwerk.io
  ProxyJump dev
  IdentityFile ~/.ssh/id_rsa
  IdentityFile ~/.ssh/dev-internal.pem
  UseKeychain yes

Host stg-*
  HostName %h.stg.netwerk.io
  ProxyJump stg
  IdentityFile ~/.ssh/id_rsa
  IdentityFile ~/.ssh/stg-internal.pem
  UseKeychain yes

Host prd-*
  HostName %h.prd.netwerk.io
  ProxyJump prd
  IdentityFile ~/.ssh/id_rsa
  IdentityFile ~/.ssh/prd-internal.pem
  UseKeychain yes