Cloud Native

Pipelines and Kubernetes Authentication

January 26, 2021

by

Marc Boorshtein

TL;DR

  • Don't use ServiceAccount tokens outside of your cluster
  • Create service accounts inside of your authentication identity provider, assign RBAC privileges
  • Easy with Okta and OpenUnison

The Right Way To Authenticate to Your Clusters From Your CI/CD Pipelines

You have a shiny new cluster and new pipeline to automate the deployment of your applications! You're in DevOps heaven. Except. How are you talking to your API server? Are you using a ServiceAccount token? If your pipeline is running outside of your cluster, you shouldn't be. ServiceAccount tokens were not designed to be used from outside of your cluster, they're designed to provide an identity to the workloads running inside of your cluster. I've written about how you shouldn't use a ServiceAccount token for users, I've also written about how you shouldn't use certificates for authentication either. What's the correct way to access your cluster and why? The short answer is to use the same system your developers and admins use! Let's dive into the details as to why.

ServiceAccount Tokens - A Breach Waiting To Happen

A bit dramatic? Maybe, though being in security paranoia is part of the equation. That said, there's some very good reasons why you shouldn't be using a ServiceAccount token for accessing your cluster. First, let's create a token:

$ kubectl create sa blogpost
$ kubectl get secret blogpost-token-ffqfh -o json | jq -r '.data.token' | base64 -d
eyJhbGciOiJSUzI1NiIsImtpZCI6IlllUzR2Z0NHamJERFVIRm9YOWxrY0taX0NSelRFT1ozSDNtcjBGQ2FTLUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImJsb2dwb3N0LXRva2VuLWZmcWZoIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImJsb2dwb3N0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiOTM4MWI1YzMtNTFjMi00YmMwLWI0MmMtNGZkMjgxZmE0YmEyIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6YmxvZ3Bvc3QifQ.F3d0x4wEdJfJAhQJUx5mSH4KH3Q2VlZWEXDYUx1H2_BUGD3oJsHU7BhFRXxY1LSMyDEynsLk1DnrclFL61NdD68oWkeDOotbzxNPZYXuOC1XAB4ldqJgHsaQ235pf8O5MXC7pObLbUzHiQaOCbOu1FJY0e9xPRd3hQdCP_gynalrhNBdabKiVvhxdekXGfj7HEyyO29r3oNzQoHUTZMvRyaYNjNrX2ZRzksGvpworZD_fUsL95EJxep3_XWJ1v-ayy4NpHHIvj1LmBb_QN_Yck-LxcT-5TEgW0VPEcUHqAz2KJnpBsuXGd3HHmCz0UpsKSmhHxSu9mKVMCZBdj7mog

We now have a token we can use to access the cluster! Assuming we provide some RBAC bindings to this ServiceAccount all I need to do is embed the token in the Authorization header of each request to the API server or embed it into a kubectl configuration file for use with any of the Kubernetes client SDKs out there. That's because this token is known as a "Bearer token", as in the bearer of the token can do whatever the token is authorized to do. It's much like a ticket to your favorite sporting event or show. There's nothing tying the ticket to you, anyone who has it can use it. If you drop your ticket and someone else uses it, you're out of luck. This is in contrast to certificate authentication where the secret part, the private key, never leaves the client. The fact that bearer tokens travel across the wire means that every system that comes in contact with them is a potential breach. Accidentally log your tokens? Now your backup solution is a potential breach. Your CNI has a vulnerability? That's now a potential breach. Your cluster is many layers of virtual networks, proxies, logs, and that's even before you get to your pipeline! The question isn't if these layers will have bugs that can leak your tokens, it's when.

To mitigate this risk, bearer tokens should be short lived. Our deployments of OpenUnison deploy tokens with one minute life spans with the hope that should a token get leaked, by the time an attacker gets the token, recognizes it, and tries to use it it's already expired. Here's a typical token generated by OpenUnison for your cluster:

eyJraWQiOiJDPU15Q291bnRyeSwgU1Q9U3RhdGUgb2YgQ2x1c3RlciwgTD1NeSBDbHVzdGVyLCBPPU15T3JnLCBPVT1LdWJlcm5ldGVzLCBDTj11bmlzb24tc2FtbDItcnAtc2lnLUM9TXlDb3VudHJ5LCBTVD1TdGF0ZSBvZiBDbHVzdGVyLCBMPU15IENsdXN0ZXIsIE89TXlPcmcsIE9VPUt1YmVybmV0ZXMsIENOPXVuaXNvbi1zYW1sMi1ycC1zaWctMTYwOTQ1MjY2NjA1NyIsImFsZyI6IlJTMjU2In0.eyJpc3MiOiJodHRwczovL2s4c291LmFwcHMuMTkyLTE2OC0yLTE0OC5uaXAuaW8vYXV0aC9pZHAvazhzSWRwIiwiYXVkIjoia3ViZXJuZXRlcyIsImV4cCI6MTYxMTYyMzcyNCwianRpIjoiVHYyMGszM0NHanItellSU3N1ZzNGdyIsImlhdCI6MTYxMTYyMzY2NCwibmJmIjoxNjExNjIzNTQ0LCJzdWIiOiJtbW9zbGV5IiwibmFtZSI6IiBuYSIsImdyb3VwcyI6WyJDTj1rOHNfbG9naW5fY2t1c3Rlcl9hZG1pbnMsQ049VXNlcnMsREM9ZW50MmsxMixEQz1kb21haW4sREM9Y29tIiwiQ049b3VfYXVkaXRvcnMsQ049VXNlcnMsREM9ZW50MmsxMixEQz1kb21haW4sREM9Y29tIiwiQ049UG9ydGFsIFVzZXJzLENOPVVzZXJzLERDPWVudDJrMTIsREM9ZG9tYWluLERDPWNvbSJdLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJtbW9zbGV5IiwiZW1haWwiOiJtYXJjKzExMTFAdHJlbW9sby5pbyJ9.K7Q1aTi4SbRzPd5dCBA2j5ms3FHYKbOoRCfK_cyJ2pcbeAPhXz22qF9s0vd-1gVK_lod7mm9hm90hBkl07gc5pTz785-2sMRjHIMGTfXMzxLf5vjS2Z5O3cx7q9gLL30s86QVaZXL07SKIEMkGrHe7i68O-DwTJ60sgKOtlLDOTS9i7cYAqAbBxUNg1ehylMJgOFdXIlafSbmJd78iv7xcbwgCAg2be4S7vmlLhs7b21EyyiRGl2-RFV_8NfrqgBU0pQvu8D5xRodombpYpZclQPVfXtJOV3EoCqcnjiFkErtzEjG6YiqqVr0Kw5q3r5lIAwJTUmQ1rMFby5Jd6nmQ

This token, like the ServiceAccount token we generated earlier, is a JSON Web Token, or JWT. A JWT is a bit of digitally signed JSON that can be verified with the correct public key. This lets the API server, or anyone else, verify and trust the attributes in the JSON. My favorite tool for inspecting tokens is jwt.io. Dropping this token in shows the following "claims":

{
  "iss": "https://k8sou.apps.192-168-2-148.nip.io/auth/idp/k8sIdp",
  "aud": "kubernetes",
  "exp": 1611623724,
  "jti": "Tv20k33CGjr-zYRSsug3Fw",
  "iat": 1611623664,
  "nbf": 1611623544,
  "sub": "mmosley",
  "name": " na",
  "groups": [
    "CN=k8s_login_ckuster_admins,CN=Users,DC=ent2k12,DC=domain,DC=com",
    "CN=ou_auditors,CN=Users,DC=ent2k12,DC=domain,DC=com",
    "CN=Portal Users,CN=Users,DC=ent2k12,DC=domain,DC=com"
  ],
  "preferred_username": "mmosley",
  "email": "marc+1111@tremolo.io"
}

A "claim" in the JSON is something that the JWT is asserting. As in, this JSON is claiming I'm mmosley. The important claims are iat (initiated at), nbf (not before), exp (expires). All three of these claims are expressed as seconds since January 1, 1970 GMT. In this case, the token was initialized at Tuesday, January 26, 2021 1:14:24 AM UTC and expires at Tuesday, January 26, 2021 1:15:24 AM UTC one minute later. This means an attacker would need to get this token, recognize what it is, and exploit it within sixty seconds. That's a tall order.

Now lets drop our ServiceAccount token into jwt.io:

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "default",
  "kubernetes.io/serviceaccount/secret.name": "blogpost-token-ffqfh",
  "kubernetes.io/serviceaccount/service-account.name": "blogpost",
  "kubernetes.io/serviceaccount/service-account.uid": "9381b5c3-51c2-4bc0-b42c-4fd281fa4ba2",
  "sub": "system:serviceaccount:default:blogpost"
}

There's no iat, nbf, or exp claims! This token will NEVER expire. The only way to get Kubernetes to reject this token is to delete the Secret associated with the ServiceAccount, and if you're using Kubernetes' new oidc endpoint to get certificates for ServiceAccount verification you can get into real trouble! But that's another blog. The point here is that unless you have some very specific controls in place to constantly rotate your ServiceAccount tokens you're exposing your cluster to a potential break from multiple layers. Every system has bugs and you won't generally have control of that. You want to minimize the amount of time a token is useful. Want some more evidence of how scary ServiceAccount tokens can be? Take a look at darkbit.io's blog post on HoneyTokens.

This issue is one of the reasons why the TokenRequest API was created. The long term vision is to eliminate ServiceAccount token Secrets altogether in favor of short lived tokens that are generated as needed for specific workloads. Instead of a Secret being mounted into your Pod, each Pod will get a token mounted that has an exp claim that will tell Kubernetes how long to accept the token and the token won't be stored in Etcd. This api is still in beta and few applications know how to use these short lived tokens. The good news is OpenUnison will support them naively in our next release!

Authentication The Right Way

The short answer is use the same method as your developers and admins! Hopefully you're using OpenUnison, but that doesn't mean it's the only way to do this. Let's assume you are, and your users and groups are stored in Okta. If your pipelines get an oidc token from OpenUnison the same way your developers and admins do the "secret" for authentication can't be leaked by your cluster because your cluster will never see it, drastically cutting down on your attack surface. You still need to secure your CI/CD infrastructure, but that's another blog.

The goal for your pipeline is often to generate a kubectl configuration file that can be leveraged by the python, go, or some other SDK. Once you're generated that file the SDK does the rest of the work. The good news is OpenUnison does this for you already! If you login to your OpenUnison and click on the "Kubernetes Token" badge:

You'll see a screen much like this:

Depending on your browser, hit F12 for the developer tools and you'll see that your browser is just making a RESTful web services call to get your kubectl command:

You work with APIs all the time! So how to call this API from your pipeline? That really depends on how you store your users. The inspiration for this blog post was a customer using Okta, so we'll use that as an example. The basic steps are:

  1. Initialize your request by openning a connection to https://openunison.host.domain/k8stoken/token/user
  2. Follow the redirects to your provider's login page
  3. Submit a login
  4. Follow the redirects back to https://openunison.host.domain/k8stoken/token/user
  5. Extract the "kubectl Command" from the final JSON and run, giving you a kubectl configuration!

Step 3 is really the hardest part and very dependent on your identity provider. Some make this easier then others. Okta's was really easy! Here's the code we wrote to do it:

def get_openunison_token_from_okta(okta_domain,username,password,openunison_host):
    s = requests.Session() 
    # initiate the request
    response = s.get("https://" + openunison_host + "/k8stoken/token/user")
    # get the url from the okta login page
    parsed_url = urlparse(response.url)
    query_string = parsed_url.query
    query = parse_qs(query_string)
    # generate a redirect
    full_redirect_url = "https://" + okta_domain + query['fromURI'][0]
    
    # login to Okta
    okta_url = "https://" + okta_domain + "/api/v1/authn"

    payload = json.dumps({
    "password": password ,
    "username": username ,
    "options": {
    "warnBeforePasswordExpired": True,
    "multiOptionalFactorEnroll": False
    }
    })
    headers = {
    'Accept': 'application/json',
    'Content-Type': 'application/json'
    }
    response = s.post(okta_url, headers=headers, data = payload)

    okta_token = response.json()["sessionToken"]

    # finish login, redirect to token page

    finish_login_url = "https://" + okta_domain + "/login/sessionCookieRedirect?checkAccountSetupComplete=true&token=" + okta_token + "&redirectUrl=" + full_redirect_url
    response = s.get(finish_login_url)
    return json.loads(response.content)

First we initialize a session with Python's Requests library to track cookies. Next we start the process by accessing the token url. Once the redirects are finished, Okta has an API it uses to authenticate the user's credentials and then redirect you back to your original request. Finally, OpenUnison validates your session and generates a kubectl command for you, including everything you need such as certificates, URLs, etc. To see the code used:

# Load the token from openunison
openunison_token = get_openunison_token_from_okta("xyz.okta.com","user@domain.com","password","k8sou.apps.domain.com")

# get the command that will generate a full kubectl configuration file
kubectl_cmd = openunison_token['token']['kubectl Command']

# create the kubectl configuration
system(kubectl_cmd)

# do something with the python client

config.load_kube_config()

v1 = client.CoreV1Api()
print("Listing pods with their IPs:")
ret = v1.list_pod_for_all_namespaces(watch=False)
for i in ret.items:
    print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

The first thing we do is get a kubectl configuration command from OpenUnison using our Okta domain information and our service account username and password. We next get the kubectl configuration command and run it, generating a ready to go configuration file. Finally, we initialize our Python SDK using the configuration file. The great thing about this is that if your process takes more then a minute, the Python SDK will refresh your token for you! You can have your cake and eat it too. Use a service account for accessing your cluster without exposing it to the risks of a never expiring bearer token! The complete code snippet is available as a gist.

At this point you might ask "why not just use a standard API?". Because there isn't one. Each authentication system is different and can involve any number of different steps. The good news is most authentication pages these days are like Okta, where it's just a simple matter of inspecting their API in a browser.

Other Benefits

In addition to the direct security benefits of not exposing never-expiring-bearer-tokens from your cluster, this mechanism makes it easier to use RBAC. Instead of directly referencing a ServiceAccount object in the bindings, you can reference groups instead and add your pipeline service accounts to those groups. Just like with your developers and admins groups can be used to more easily audit your service account access.

What's Next?

If you want to start simplifying your cluster access, take a look at the multiple editions we have of the Orchestra Login Portal, including OIDC, SAML2, LDAP, and GitHub! If you want to learn more about the details of Kubernetes authentication and authorization you may be interested in the book I co-authored: Kubernetes and Docker: The Enterprise Guide!

Related Posts