Back to 2b site

Kubernetes Federated Clusters on AWS

TL;DR

This blog will discuss federated Kubernetes installations. Why and when we should use them and provide a working example of such a setup on AWS’s EKS.
This blog post includes working code examples.

As engineers with a never ending task list – which only grows as we strive for the next best thing we can add to our system – context is vital. We must be able to understand the needs and solutions of both our Production Systems and Development teams as quickly as possible, and be able to tailor the best solutions.

In the world of Kubernetes, some concepts are similar, and may share some features, but are also distinct deployment models. 

The most common of all is a single cluster that is used by one development team where no partitioning of any kind is needed.

 Multi-Cluster management is the ability to gain visibility and control over many of these separate custers. 

While single-cluster is the easiest deployment path, it is not the best use of compute resources.  It requires having all of the overhead associated with a cluster for each and every team/use. This is amplified when security and quality assurance teams also want their own place to test and deploy the same applications.

Kubernetes Multi-Tenancy is the ability to have multiple distinct areas (usually namespaces) within a single cluster. Here, quotas and security policies can be applied at the boundaries to improve overall resource utilization and simplify the overall infrastructure required to support an environment.

You may still need to have multiple separate clusters in play for the reasons above. And these scenarios can still have multi-tenancy in some or all of the clusters, but it has become a multi-cluster deployment. 

A multi-cluster management allows for a single interface which can see and control aspects of every cluster that is connected to the multi-cluster management plane.

But what if this is too much?

Kubernetes Federation extends some attributes Multi-cluster, but does not cover all the multi-cluster scenarios. 

When multiple clusters are federated, they share pieces of their configuration which is managed by what is called the ‘host cluster’ and sent to the member clusters of the federation.  The benefit of having the high order parts of a configuration shared is that any resources configured to take advantage of the federation will treat all member clusters as a single distributed cluster with oversight by the host cluster.

Think of it as a sort of a ‘cluster-in-cluster’, if you will.

When do we need it?

Kubernetes Federation can deploy multiple cluster applications in different regions and design for disaster recovery. The host cluster is the fundamental abstraction for supporting the Kubernetes federation, and it carries any configuration that will generate the members of the cluster. 

It is also a member which can run the real workloads, but generally, any organization will have the host cluster as a standalone cluster for clarity. Consider the option of giving each team an ecosystem of their own, while those are all orchestrated in a single interface for all scale, expansion and operation needs.

Application Deployment and More

Federation helps to deploy applications that need high availability, resilient, and automatic scaling functionality between clusters. It’s capable of automatically detecting failures of clusters and compensating for the number of lost replicas in federated applications across all clusters, including those in different regions.

One API can manage the entire configuration, including the ‘clusters-in-cluster’ configurations and tasks to be performed. In our example, we’ll use Terraform code to deploy infrastructure in AWS.

A note: At the date of this solution’s publication terraform does not support Kubefed configurations so we will need to configure it manually as detailed below.

Before we get into the process, here are a few key concept to aid understanding:

Terraform: Terraform is a tool for managing infrastructure as code, cloud platform management, and ability to create modules for self-service infrastructure.

Helm: Helm is a Kubernetes deployment tool used in the automation of the creation, packaging, configuration, and deployment of applications and services to Kubernetes clusters.

Set up a working example

Let’s begin our step-by-step walk-through of the process.

Note: This example can be used on either Linux, Mac or WSL on Windows.

To begin with our deployment, we’ll need to install Terraform, Kubectl and helm. We’ll need to clone the example repository, and will be able to do so using the git clone command.

git clone https://github.com/2bcloud-io/kub-fed-blog
Cd terraform

Once we’ve successfully cloned our repository, we’ll need to execute three Terraform commands to provision infrastructure.
terraform init 
terraform plan 
terraform apply

With the infrastructure provisioned, we need to confederate the two clusters. Next, export all necessary environment variables to be used in future commands. Here, we’re going to export the account IDs, names, regions of both clusters, which appear as follows without a unique variable name.

unset AWS_ACCOUNT_ID
unset EKS_1_NAME
unset EKS_1_REGION
unset EKS_2_NAME
unset EKS_2_REGION

When you export all necessary environment variables to be used in future commands, be sure to change all environment variables if you would like to use different names, as seen below.

export AWS_ACCOUNT_ID=xxxxxxxxxxx
export EKS_1_NAME=eks-federated-1
export EKS_1_REGION=us-east-1
export EKS_2_NAME=eks-federated-2
export EKS_2_REGION=us-west-2

Kubernetes Configuration

Next, we’ll get the config from the cloud using the following commands that identify the name and region variable:

aws eks update-kubeconfig –name ${EKS_1_NAME} –region ${EKS_1_REGION}
aws eks update-kubeconfig –name ${EKS_2_NAME} –region ${EKS_2_REGION}

get_k8s_context

Once pulled from the cloud, we’ll rename the kubernetes configuration using these commands:

kubectl config rename-context $(kubectl config get-contexts –no-headers=true -o name | grep ${EKS_1_REGION}:${AWS_ACCOUNT_ID}) ${EKS_1_NAME}.${EKS_1_REGION}
kubectl config rename-context $(kubectl config get-contexts –no-headers=true -o name | grep ${EKS_2_REGION}:${AWS_ACCOUNT_ID}) ${EKS_2_NAME}.${EKS_2_REGION}

get_k8s_context_after_rename

Next, choose which cluster you want to use from the control plane and switch context to that cluster. For our example, we are going to install KubeFed control plane in eks-federated-1 cluster.

Here’s the code and an example:
Once renamed, we’ll need to get and view the clusters using this command:

kubectl config get-contexts

After getting context, we’ll have two clusters, and will need to select one to complete the installation of the application to be deployed.

kubectl config use-context eks-federated-1

Add the helm repository:

helm repo add kubefed-charts
https://raw.githubusercontent.com/kubernetes-sigs/kubefed/master/charts
helm repo update

Complete the installation of the application:

helm –namespace kube-federation-system upgrade –install kubefed kubefed-charts/kubefed –create-namespace

kubefed_check_deployment

Check your installation to see applications inside the cluster.

Federating the clusters is next. To begin, join the first cluster to kubefed, then do the same with the second cluster. We’ll use these commands:

kubefedctl join ${EKS_1_NAME}.${EKS_1_REGION} –cluster-context
${EKS_1_NAME}.${EKS_1_REGION} –host-cluster-context
${EKS_1_NAME}.${EKS_1_REGION} –v=2

kubefedctl join ${EKS_2_NAME}.${EKS_2_REGION} –cluster-context
${EKS_2_NAME}.${EKS_2_REGION} –host-cluster-context
${EKS_1_NAME}.${EKS_1_REGION} –v=2 

Next, we’ll use the command below to get the kubefed clusters:

kubectl -n kube-federation-system get kubefedclusters

kubefed_get_clusters

As an example we’ll complete the installation of a demo application by applying these files: 

kubectl apply -f echo-server/federated-namespace.yaml
kubectl apply -f echo-server/federated-deployment.yaml
kubectl apply -f echo-server/federated-service.yaml

Applying these files will install the echo-server application in both clusters. This happens to the first cluster we configured to have two replicas, and in the second cluster we will have three replicas of our application. 

For both applications, we’ll need to deploy our service as a LoadBalancer type. You will need to add the external name of service in Route53, and set up healthchecks for both endpoints to ensure that it will always route traffic to the healthiest endpoint. For testing, you can scale your first cluster application replicas to 0 to mimic that behavior. 

Conclusion

And that’s how you can create Kubernetes federated clusters in AWS. These steps are straightforward and this entire process shouldn’t take more than 20min to complete. A reminder, ensure you assign variables and naming conventions associated with your project for consistency.