Cloud Native

Amazon EKS and OIDC

February 16, 2021

by

Marc Boorshtein

TL;DR

  • Amazon EKS now supports OIDC for authentication
  • Any version greater then or equal to 1.16 is supported
  • Existing clusters can be converted
  • Works great with OpenUnison

Support for OpenID Connect

On Friday, February 12th 2021, Amazon announced the availability of OpenID Connect authentication for EKS. This makes integration with external identity systems much easier. Prior to this announcement, you had to rely on an impersonating reverse proxy to use an external identity provider. While this works well, there are some drawbacks. First, it made it impossible to use --as in kubectl for testing RBAC because the --as flag used impersonation and you can't layer impersonation on top of impersonation. In the case of OpenUnison, you couldn't use kubectl exec or port-forward because they rely on the old SPDY protocol that very few systems still support. Supporting OpenID Connect makes it easier for developers and admins to use the great tools that have been built off of the Kubernetes go client SDK and externalize authentication and authorization.

Why Externalize Identity in EKS?

Before we dive into the details, why would you want to externalize authentication from EKS? There are a few reasons. First, EKS requires your IAM user to assume a role that is mapped to an RBAC group or user in a static yaml file stored in a ConfigMap. To assign rights to a user you need to:

  1. Create an IAM User
  2. Create an IAM Role that user can impersonate
  3. Add the IAM Role to the mapRoles selement in the aws-auth ConfigMap in the kube-system namespace
  4. Create an Kubernetes RBAC RoileBinding or ClusterRoleBinding that binds the User or Group mapped to in the aws-auth ConfigMap to the desired permissions

This is a very error prone process that is difficult to automate. There's no syntax checking of the mapRoles element, making it difficult to debug. It also requires that adding a new user to a cluster requires updates to your account's IAM system which generally requires the team that manages AWS to intervene. This makes it harder to build multi-tenant systems with finer grained access controls because of the number of potential failure points. Compare this to the steps for on-boarding a user in an OIDC enabled cluster:

  • Add user to group in identity provider
  • Create Kubernetes ClusterRoleBinding or RoleBinding

The steps don't require involving the AWS team, and depending on your identity provider doesn't require their intervention either! It puts cluster owners in control of their cluster's authorizations and makes it easier to create least-privilege policies. This simpler path to control will ultimately lead to more secure clusters.

Another reason for using OIDC with your EKS clusters is a better UX. With AWS IAM authentication you need the aws cli tool to authenticate, then set your config, then use kubectl. With OIDC, you just need your configuration and you're ready to go. No special CLI tools required!

Finally, using an external authentication system can lead to better security. Too often IAM users are static tokens sitting in user's .aws directory. OpenID Connect is build to support short-lived tokens. I talked about the importance of short lived token in our last blog post. Using OIDC means that your tokens can be short lived transparently to your users without them having to think about it!

When Can't I Use OIDC With EKS?

In addition to the usual prerequisites Kubernetes has with OpenID Connect, your EKS masters must be able to access your IdP via your Ingress controller to retrieve the discovery JSON. Your IdP must also be signed by a well known commercial certificate authority. If either of these prerequisites fail, you'll continue to need to use OpenUnison with impersonation with your EKS clusters. If you have a self signed CA for your IdP, let Amazon know by commenting on the feature request!

Integrating Okta and EKS

In a past blog post, we integrated Okta into Kubernetes. For this post, we'll integrate Okta into our EKS cluster. First, create your application in Okta per out Okta blogpost. Next, we'll use the OpenUnison operator to deploy OpenUnison using the OIDC portal:

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.1.0/aio/deploy/recommended.yaml
$ helm repo add tremolo https://nexus.tremolo.io/repository/helm/
$ helm repo update
$ kubectl create ns openunison
$ helm install openunison tremolo/openunison-operator --namespace openunison

With the operator deployed, the next step is to create the Secret OpenUnison will use to store your Okta client secret. It will look something like:

apiVersion: v1
type: Opaque
metadata:
  name: orchestra-secrets-source
  namespace: openunison
data:
  OIDC_CLIENT_SECRET: aW0gYSBzZWNyZXQ=
  K8S_DB_SECRET: aW0gYSBzZWNyZXQ=
  unisonKeystorePassword: aW0gYSBzZWNyZXQ=
kind: Secret

The data.OIDC_SECRET object stores your base64 encoded client secret. The spec.unisonKeystorePassword and spec.K8S_DB_SECRET objects should be random junk data, with no ampersands (&). Once created, the next step is to deploy the orchestra helm chart. If you read our previous blog post about EKS authentication, we used impersonation. In this deployment we won't use impersonation anymore! Our values.yaml will have two specific changes:

network:
  .
  .
  .
  k8s_url: https://ABCD.gr7.us-east-1.eks.amazonaws.com

.
.
.
enable_impersonation: false

First, we set network.k8s_url to the URL of our API server endpoint. You can get this from the API Server Endpoint in your AWS console:

Get Your API Server Endpoint

We also set enable_impersonation to false. Don't worry about the API server certificate, OpenUnison will get that for you automatically! Finally, Deploy your helm chart:

$ helm install orchestra tremolo/openunison-k8s-login-oidc --namespace openunison -f /path/to/values-k8s-okta-eks.yaml

A few things are happening:

  1. A certificate is being generated for the dashboard
  2. Certificates used by OpenUnison are being generated
  3. If you're using NGINX as an Ingress controller, your Ingress object is being generated
  4. OpenUnison is being deployed

Once the OpenUnison pod is running, you will need to configure your EKS cluster. WARNING: this step has the same effect as upgrading your cluster. All your nodes will be redeployed. First, let's get our configuration information for EKS. The operator generates a ConfigMap with all the information you need to configure EKS:

kubectl describe configmap api-server-config  -n openunison
Name:         api-server-config
Namespace:    openunison
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: orchestra
              meta.helm.sh/release-namespace: openunison

Data
====
oidc-api-server-flags:
----
--oidc-issuer-url=https://k8sou.apps.abc.tremolo.dev/auth/idp/k8sIdp
--oidc-client-id=kubernetes
--oidc-username-claim=sub
--oidc-groups-claim=groups
--oidc-ca-file=/etc/kubernetes/pki/ou-ca.pem

Before we configured EKS, let's validate that Okta and OpenUnison are integrated properly. Go to the host you specified in network.openunison_host in your values.yaml file. You should be redirected to Okta, login and you should end up at a screen that looks like:

The last step is to configure EKS. In your cluster configuration, click on "Authentication" and then click "Associate Identity Provider".

Use the --oidc-issuer, --oidc-client-id, --oidc-username-claim, and --oidc-groups-claim. Once ready, Once ready, hit "Associate" and grab a cup of coffee. It will take about as long as a cluster upgrade takes. Once done you can click on the Kubernetes Dashboard badge in your portal and securely access your cluster directly from your browser!

Converting an Existing OpenUnison

Already running OpenUnison with EKS? Converting to using OpenID Connect couldn't be much easier. In your helm values.yaml, just update the network.k8s_url and enable_impersonation as described above and upgrade:

$ helm upgrade orchestra tremolo/openunison-k8s-login-oidc --namespace openunison -f ~/path/to/values-k8s-okta-eks.yaml

Then follow the same steps as a new deployment to integrate with EKS.

Why OpenUnison?

Since OIDC is a standard and Okta can talk OIDC, why bother with OpenUnison? OpenUnison gives you several UX and security improvements:

  1. Secure access to the dashboard directly from our browser
  2. No need to distribute certificates or configuration files, OpenUnison gives you a 100% standard kubectl command for both *nix and Windows to set your context
  3. The oulogin plugin lets you login to kubectl directly from your command line with only knowing the host of your OpenUnison server
  4. Tremolo Security keeps all these tools integrated and patched and patches become available
  5. Your tokens are short lived (1 minute), so a leaked token is much less likely to be abused without impacting the developer's UX

After getting your clusters integrated with OpenUnison, you can explore building a mult-cluster portal or integrate other applications from your development stack too!

Related Posts