Back to 2b site

Using AKS with workload identities in terraform

Gil Levy

Gil Levy

July 2023

The story around the solution

We all use Kubernetes on a daily basis, and the more we use it, the more it is apparent that Kubernetes alone will not be as fruitful as it will be with deeper integrations. One of these integrations is Microsoft Azure, which provides the ability to connect, use, and retrieve information from services on your behalf.

The problem: Why do we need this solution?

Today, working with Azure Kubernetes Service (AKS), you can assign managed identities at the pod-level, which has been a preview feature. This pod-managed identity allows the hosted workload or application access to resources through Azure Active Directory (Azure AD). For example, a workload stores files in Azure Storage, and when it needs to access those files, the pod authenticates itself against the resource as an Azure managed identity.

The solution:

The pod-level approach was replaced by a more “kubernetes native” authentication using Azure Active Directory (Azure AD) workload identities (preview), which integrates with the Kubernetes native capabilities to federate with any external identity providers. This approach is simpler to use, deploy, and overcome several limitations in Azure AD pod-managed identity:

There are several good reasons to use Azure AD workload identities:

  • Eliminates the performance issues by using mutating admission webhook.
  • Supports both Linux and Windows workloads, unlike managed identities at pod level.
  • Removes the need for Custom Resource Definitions and pods that intercept Azure Instance Metadata Service (IMDS) traffic.

Using AKS with workload identities in Terraform

Workload identities are an excellent way to assign permissions to resources in AKS clusters. These managed identities allow you to assign permissions to pods running in AKS clusters by using service accounts. Let’s now explore how to use AKS with workload identities in Terraform.

Prerequisites

Before we start, ensure that you have the following prerequisites:

  • An Azure account with the necessary permissions to create an AKS cluster and a virtual network (VNet).
  • Terraform is installed on your local machine.
  • The Azure Terraform provider installed.

Steps

1. First we need to enable the aks-preview extension by using the CLI.

2. Create an AKS cluster using Terraform. Note: in this example I’m using the least amount of configuration available

This configuration creates a resource group, an AKS cluster with a single node pool, a Linux-based node, and a system-assigned managed identity. For future use we also add some outputs to the configuration.

3. Create a managed identity for your workload by adding this to the code:

This code creates a user-assigned identity that you can assign to the serviceAccount and in the end to specific pods in your AKS cluster. For future use we also add some outputs to the configuration.

4. Create a federated identity credential. You can do this by adding the following code to your Terraform configuration:

Notes: The subject is created in this format: “system:serviceaccount:<namespace_name>:<serviceaccount_name>”. Each managed identity needs a federated_identity_credential to function, we use the identity resource id as the parent_id.

5. Let’s create a keyvault and add access permissions to our managed_identity.

6. Now we manually add a secret into the Keyvault using the UI or using Terraform. Note: the below code is optional

7. Let it build!

We’re almost done! Now we create a new folder and inside it we create a new Terraform file. We do this to avoid a known issue with Kubernetes infra and Kubernetes provider.

8. Inside the new terraform file we first add a reference to our first terraform project to get those important outputs we configured before

9. Next we create the workload ServiceAccount and the azure-workload-identity-system chart. Note: we use the same namespace and ServiceAccount name from above.

10. Now we test the entire project by provisioning a test pod that will retrieve our secret from the keyvault. Note: We use            “ignore_changes = all“ because the pod gets additional volumes and environment variables from the workload-identity chart. This is only for our test purposes.

11. Apply the newly created k8s.tf

12. We simply execute kubectl logs quick-start -f to see our secret value inside the pod logs. Note: You can easily retrieve the cluster credentials by going into the AKS resource page on the ‘Connect’ section.

Conclusion

Using AKS with workload identities, is a secure and straightforward way to manage access to resources in your AKS clusters. By following the steps outlined in this document, you can use AKS with workload identities in Terraform with ease.

More content you shouldn’t miss