Situation-specific shell config - the final 10%

As you may know, I store my shell configuration in git, so I can easily share the same shell config across multiple machines. One of the downsides to having a single shared set of configuration, however, is that you don’t want to store sensitive data in there. Another downside is that you are very likely to have small differences in configuration between different systems. You might have a Golang development environment on your work system but not on your private system. You might want to use different AWS credentials on your work system, or you might have to regularly switch between different credential sets.

I quickly got tired of maintaining things in .bash_profile.local (which was ignored by git). Not only because it was cumbersome, but also because it started to contain settings that I didn’t need on all my systems, but I did need them on some. So I started to work on a better way to deal with this.

Step one - encrypting sensitive data

This one turned out to be a fairly easy fix. The key component is git-crypt, which does the heavy lifting. Installing git-crypt on my Mac was a breeze, as Homebrew (once again) had me covered:

$ brew install git-crypt

Git-crypt also comes with clear documentation. Setting up my repo was as easy as entering the working directory and typing:

$ git-crypt init

Now, at this point, git-crypt is not actually encrypting anything yet. In order to start encrypting files, we need to tell git which files to encrypt. This can be done by creating a .gitattributes file in your repo, and add some filters in that file (similar to .gitignore). The result looks a bit like this:

dotdir/.env/** filter=git-crypt diff=git-crypt
dotdir/.slackcat filter=git-crypt diff=git-crypt

In my case, all files created in the dotdir/.env/ subdirectory (or child subdirs) will be encrypted, as well as one specific other file containing my credentials for Slackcat.

To use the encrypted data on other machines, you can either set up your repo to work with GPG keys, or take the somewhat easier route of exporting a static key and storing it in your private Keybase folder:

$ git-crypt export-key /keybase/private/<mykeybaseusername>/mykey

Now, on any other system I added to my Keybase account, I can decrypt the sensitive data in my dotfiles repo:

$ cd ~/git/dotfiles
$ git-crypt unlock /keybase/private/<mykeybaseusername>/mykey

Step two - situation-specific config

After sifting through a bunch of different .bash_profile.local files from different machines, I found that I had a few different categories of situation-specific config:

• Project Specific: for a specific project (or client), I might use specific SSH keys, specific shell aliases, etcetera.
• Machine Specific: some machine has Golang or RVM, some other machines don’t. Some machines use Virtualbox for Docker Machine, some others use Vmware Fusion. Some machines are primarily used for work projects, others are primarily for a private project.
• Credentials: Within a project, I found that the only thing that really changes from time to time, is credential sets for things like AWS accounts. You might use different accounts for production/non-production setups within a single project, for instance.

_(* sidenote: I **absolutely recommend** using separate AWS accounts to separate production and non-production environments)_

To address the different kinds of config, I went for a fairly simple, low-tech solution. I created a directory structure:

.env/
├── credentials
│   └── work
│       └── aws-infralab.env
├── global.env
├── machines
│   ├── ananas.env
│   ├── banana.env
│   ├── kiwi.env
│   └── pear.env
└── projects
    ├── privateprojectA.env
    ├── workprojectA.env
    ├── workprojectB.env
    └── zzz-secretproject.env

As you can see, I created subdirectories under .env for the 3 categories of situation-specific config, and all files env with a .env extension. This is just a convention I chose, and you can just as easily choose to do something else.

You may wonder what that global.env file is doing there. At this point, it holds a single line, to load my personal SSH key. I could’ve just as easily stored that in .bash_profile, but I actually am considering thinning out that file a little, and moving specific settings to global.env for readability.

As part of my dotfiles setup, the .env directory is symlinked in ${HOME}/.env.

Loading the right config

First of all, I added some config to my (shared) .bash_profile to trigger this process:

## Source the functions for situation-specific shell config
. situation-specific.sh

## Load the global env
load_env global

## Load the machine-specific env for this machine
load_env machines/$(hostname -s)

If you want to try this setup, you can find the functions for situation-specific shell config in this Gist

UPDATE: I just published my envtools functions as a Git repo. Use at your own risk, of course.

A note on MacOS and hostnames

At work I initially had problems loading my machine-specific config. As it turns out, my problems were caused by our company DHCP server assigning a dynamic hostname to my machine. Fortunately, we can fix this.

A Mac has 3 distinct hostnames, and the value it gets from the DHCP server:

• ComputerName: you can set this via the Sharing panel in System Preferences
• LocalHostName: the sanitized version of ComputerName
• HostName: unset by default

Normally, hostname -s would return the value of LocalHostName, except when it gets overridden by DHCP. The solution is to manually set HostName:

$ scutil --set HostName banana

You will be prompted for your admin credentials, but after that you’ll be fine. The output of hostname -s will consistently be banana, even if your DHCP server tries to mess with your hostname.

Chaining things

So far, we’ve loaded a global env, and a machine-specific env. What about the rest? Well, it’s mostly about chaining things. For instance, let’s consider this machine ‘banana’ which is the MacBook Pro I use for work. So by default, I would like it to get the settings that are specific to work, or the current project I’m working on. Let’s look at the machine-specific config in .env/machines/banana.env:

load_env projects/workprojectA

export C_DOCKER_MACHINE="docker-vm"
if [ $(docker-machine status ${C_DOCKER_MACHINE}) == 'Running' ]; then
  dmshell &> /dev/null
fi

As part of the machine-specific config for banana, the project-specific config for ‘workprojectA’ gets loaded. And we set some machine-specific things for my docker-machine.

If we then look at the config for that project, we see something like this:

ssh_load_key benny_work &> /dev/null

load_env credentials/work/aws-infralab

export PATH="${HOME}/git/work/ops-tools/bin:${PATH}"

A project-specific SSH key gets loaded into my agent, I load the credentials for my work’s AWS lab environment, and I add a specific directory to my $PATH to quickly access some scripts.

If I now open a new terminal on my machine, I see something like this:

Last login: Thu Jul 27 16:07:33 on ttys003
Loading ENV: global
Loading ENV: credentials/work/aws-infralab
Loading ENV: projects/workprojectA
Loading ENV: machines/banana
[benny:~] $

Step 3 - switching situations

So far we’ve established that on a specific machine, it will load the proper relevant config. But what if I quicly want to switch to my private project settings on my work machine? I can just load it:

$ load_env projects/privateprojectA
Loading ENV: projects/privateprojectA

If I now check my currently loaded envs, I’ll see something like this:

$ get_env
projects/privateprojectA machines/banana projects/workprojectA credentials/work/aws-infralab global

Any environment variable in a loaded env will take precedence over the same environment variable set in a loaded env further on the right. So if we were to set FOO=bar in the global env, FOO=bat in workprojectA, and FOO=baz in privateprojectA, my shell would now show:

$ echo $FOO
baz

Note however, that this only works for environment variables. We did not unload functions or aliases. So any functions or aliases from the workprojectA env will still work after loading the privateprojectA env.

Step 4 - switching credential sets

The final bit is about credential sets. These credential files are the same as all the other env files, but with one very important difference: credential files should only contain environment variables!

For example, this is the credential file for the AWS lab environment:

# AWS Stuff
export AWS_ACCESS_KEY='<redacted>'
export AWS_SECRET_KEY='<redacted>'
export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY}
export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_KEY}
export AWS_DEFAULT_REGION="eu-west-1"

When switching credential sets, we want to make sure any previous credentials are unloaded first. We can do this with the unload_credentials command that is included in the functions library:

$ unload_credentials credentials/work/aws-infralab
Unloading ENV: credentials/work/aws-infralab

Let’s check if it really worked:

$ get_env
machines/banana projects/workprojectA global
$ env | grep AWS
<no output>

Loading a different set of credentials works the same as loading any other set of situation-specific config:

$ load_env credentials/private/myaws
Loading ENV: credentials/private/myaws

What’s next?

At this point the implementation is a little rough around the edges still. There is some tab-completion, but it’s not perfect, and some functions have no completion at all. Also, I might want to add some functionality that automatically unloads a credential set when you try to load a new credential set that tries to set the same type of credentials. This way you can have separate credential files for AWS and GCP, for instance, but as soon as you load a second credential file containing AWS credentials, it would unload the existing one.