Over the past few years, Kubernetes has grown from something that ‘seemed very interesting, but also very complex’ to something that you can get as a service from pretty much everyone. Cloud vendors like Amazon, Google, Microsoft, Oracle and DigitalOcean are offering their own flavor of Kubernetes-as-a-Service. Big players in infrastructure like RedHat, Mirantis, Rancher, and Vmware have built Kubernetes offerings for companies that can’t (or don’t want to) use those ‘as a Service’ solutions. Docker Desktop has even shipped with Kubernetes built-in for quite some time now. So basically, Kubernetes is everywhere now and as a result more and more companies are adopting it. (Whether or not they really need to, is something for another blog post)

In this blog post, however, I want to dive into a very common workflow issue: juggling multiple Kubernetes clusters. Or actually, their configs. Over the past few years I’ve worked in environments where I regularly interacted with multiple Kubernetes clusters. A different cluster for staging or production; or even multiple different clusters per environment. And of course my local Docker Desktop K8s, that is used during local development. The issue at hand here is simple: I need to be able to easily run kubectl commands against different clusters.

Configs and Contexts

Before we dive into juggling strategies, let’s set some definitions. In order for kubectl (and other tools) to communicate to a Kubernetes cluster, they need configuration. The hostname of the cluster, some form of authentication, and which namespace to use. We call this a ‘context’. You can store this configuration in a kubeconfig file, and each kubeconfig can hold multiple contexts.

Approach 1: a single kubeconfig

The first approach, that many will take, is adding that second cluster to the existing default kubeconfig file. Just edit that ~/.kube/config file and add the bit of YAML for your second cluster, and off you go. You now have 2 contexts. The kubectl command offers the --context flag to specify which context to use for your get pods command, if you don’t specify a context, the default will be used.

## default context
$ kubectl get pods

## any other context
$ kubectl get pods --context foobar

Easy stuff. But you’ll get tired of typing --context foobar pretty quickly. And occasionally, you’ll forget to add it. So, can we make this easier? Yes! We can switch the default context, like this:

$ kubectl config use-context foobar

Now all subsequent kubectl commands will use the context foobar. And you can switch back and forth. To make switching even easier, you can use kubectx, which offers kubectx for switching contexts, and kubens for switching namespaces in a cluster:

$ kubectx foobar

$ kubens kube-system

These tools also integrate with the highly recommended fzf, which you should really add to your shell config anyway.

Downsides

So that’s a pretty straightforward approach already, but I can already hear you thinking ‘if this is Approach 1, there’s more coming. What are the downsides?’ Are there downsides? Unfortunately, yes there are. The first downside is that this approach makes it hard to use 2 contexts at the same time. Say that you’re running two terminal windows, and you want to compare your deployments on staging and production, then it would be pretty nice to just run kubectl describe mydeployment in both, and get the relevant results. Well, with this approach you can’t, because there can only be a single current-context in your kubeconfig. So, if staging is your current-context you’d need to add --context production to the commands for that context. This is also true when you’re using kubectx to switch contexts. If you run kubectx staging in your left terminal, it changes current-context for the kubeconfig file, so your right terminal will also be on staging now.

The second downside is that it’s possible to run kubectx production and forget about it. So the next time you run kubectl apply -f myhax.yaml, it’ll go to production and not Docker Desktop… whoops! You can somewhat prevent this error by installing something like kube-ps1 which will show the current context in your prompt:

(⎈ |docker-desktop:default) [benny:~] $  

Finally there’s the issue of ‘YAML merging’. Increasingly, managed Kubernetes offerings allow you to download your kubeconfig from a nice UI. But if you actually want to have a single kubeconfig, you need to merge that downloaded YAML file into your kubeconfig file. It’s not impossible, but as your YAML file gets bigger, chances of breaking things increase as well. You won’t be the first or the last to end up with a completely broken kubeconfig because they forgot an - or :, or because a config block was misaligned by a single space. Good times!

Approach 2: multiple kubeconfigs

A logical next step is to divide your list of contexts over multiple kubeconfig files. Initially you may be tempted to create different kubeconfig files for different projects, like ‘work’, ‘clientA’, or ‘sideprojectB’, and each of those kubeconfig files will then hold the contexts relevant to that project. To switch to a different kubeconfig, you can either add --kubeconfig <path to kubeconfig file> to a kubectl command, or set the KUBECONFIG environment variable. Within a kubeconfig file, you can still use --context or kubectx as you would. However, the only advantage to this approach is that the list of contexts gets shorter. The other disadvantages still remain.

Approach 3: multiple kubeconfigs, each with a single context

This third approach is how I’ve currently set up my workflow. Every context has its own kubeconfig file. It addressess all downsides I’ve mentioned earlier:

• Using different contexts at the same time: simply set the KUBECONFIG environment variable to different kubeconfigs in your different terminals
• Accidentally applying something to the wrong cluster: my ~/.kube/config now only holds Docker Desktop, so if I accidentally apply something, I’m applying it to my local cluster which is completely fine
• YAML merging: no longer an issue

Downsides?

Obviously, there are some downsides here. Otherwise everyone would be doing this, right? The biggest downside is that you now have to deal with possibly a lot of kubeconfig files. Eventually you’ll grow tired of running export KUBECONFIG=<now where did I put that file again?>. Fortunately we can fix that.

I’ve been using 2 approaches to make switching kubeconfigs easy:

Direnv for kubeconfigs

By adding direnv to your shell you can simply drop .envrc files wherever you like to have things set/unset in your shell. So, if I want to use a certain Kubernetes context for a certain software project, I could add the .envrc to the git repo of that software project. Then, whenever I enter that directory, I’m automatically using the right context.

$ cd ~/git/myproject
$ cat .envrc
export KUBECONFIG=~/.env/config/myproject/kubeconfig/staging

Also, when I leave the directory ~/git/myproject, direnv will revert $KUBECONFIG to what it was previously (or unset it if it wasn’t set to begin with). In practice, this is how that works:

## current-context: Docker Desktop, since that is what's in ~/.kube/config
$ cd ~/git/myproject

## current-context: myproject/staging, since I set KUBECONFIG using direnv
$ cd ../

## current-context: Docker Desktop, since KUBECONFIG got unset by direnv 
## after leaving the directory
$ export KUBECONFIG=~/.env/config/personal/kubeconfig/gke

## current-context: manually set to my personal GKE cluster
$ cd ~/git/myproject

## current-context: myproject/staging because of direnv
$ cd ~

## current-context: back to my personal GKE since that was what was in
## KUBECONFIG previously
$ unset KUBECONFIG

## current-context: back to Docker Desktop, since KUBECONFIG is unset

Shell Functions for Kube config

While I mostly enjoy the automation of direnv, I’m currently at a project where the switching between clusters isn’t really tied to specific Git repos or directories. I need to explicitly switch, but I still dislike long export commands. So I’ve been extending my shell library for Situation-specific shell config for config files. You could already see a little sneak peek of that in the kubeconfig paths in the previous section.

The shell functions I’ve added assume that kubeconfigs live in my ~/.env/ tree, and are namespaced under ~/.env/config/<project>/kubeconfig. The primary function is ks which stands for kubeconfig-select (but I like short commands). You can find the exact function below:

# Select KUBECONFIG file -- allows to use multiple clusters in different terminals at the same time
ks() {
  case $# in
    1)
      local config_namespace=${ENVTOOLS_CONFIG_NAMESPACE?ERROR: Cannot determine Config Namespace}
      local kubeconfig_id=${1}
      ;;
    2)
      local config_namespace=${1}
      local kubeconfig_id=${2}
      ;;
    *)
      echo "USAGE: $0 <config namespace> <Kubeconfig ID>"
      return 1
      ;;
  esac

  local config_dir="${HOME}/.env/config/${config_namespace}"

  if [[ ! -d ${config_dir} ]]; then
    echo "ERROR: Config Namespace does not exist"
    return 1
  fi

  local kubeconfig="${config_dir}/kubeconfig/${kubeconfig_id}"

  if [[ ! -r ${kubeconfig} ]]; then
    echo "ERROR: Kubeconfig ${kubeconfig_id} not found"
    return 1
  fi

  export KUBECONFIG=${kubeconfig}
  kubeon
}

I can use ks in two different ways: I can either specify the ‘config namespace’ on the command line, or set an environment variable (this is where direnv comes in again). For my current assignment I’ve basically dropped an .envrc in ~/git/mycustomer that sets ENVTOOLS_CONFIG_NAMESPACE to mycustomer, so whenever I’m in that directory tree, I can quickly specify a Kubernetes context to use:

$ cd ~/git/mycustomer
$ ks staging-aws

Boom! That’s it. The kubeon on the final line of the ks function also enables kube-ps1 so I can immediately see which context is active.

Conclusion

In my opinion, the cleanest and safest way to work with different Kubernetes clusters is to use multiple kubeconfig files, where every kubeconfig only holds a single context. To make it usable, add direnv or a shell function like ks and make sure to organize those kubeconfig files in a sensible way. It ensures that you have to be specific about which cluster to interact with, with your local one as a ‘safe default’, but minimizes hassle involved with being so explicit.