That’s right, except for one single use case, you shouldn’t ever use certificates for authentication to Kubernetes clusters. Its simply not as secure as it appears, especially when OpenID Connect is available. In this blog post we’re going to walk through a quick refresher of public key cryptography, detail why using it for authentication to kubectl and the dashboard is a bad idea and show why you should be using OpenID Connect instead.
PKI & Certificate Authority Refresher
PKI (Public Key Infrastructure) is a mathematical way to generate one key for encryption and one key for decryption. You CAN’T decrypt a message with the same key that its encrypted with. The security of the “trap door” function comes from the fact that you NEVER let anyone have your private key, and only give out your public key. This lets you give someone a key to encrypt messages to you without anyone being able to decrypt them besides you.
The second component in establishing trust is the Certificate Authority or CA. A public key has no data of its own. Its just a huge number. To add metadata to the key and give it an identity its encapsulated in something called a certificate. A certificate contains a public key and information about who its for, who issued it and how long its valid for. How do you know this data can be trusted? Because a Certificate Authority signs (encrypts) the certificate with its private key. You can draw a cryptographic valid line from a certificate to its CA. If the CA is trusted, and you can draw that line (also known as a Certificate Chain) then you know the public key and other information in the certificate is valid and can also be trusted.
Hoe Does Certificate Authentication Work?
When you authenticate using your certificate, a few things happen:
- Your client (usually the kubectl command or a web browser) makes sure that it trusts the server by checking that the certificate chain that has been presented can be traced back to a trusted certificate
- The server makes sure it can trust you by checking the certificate chain that you present to it by attempting to trace it back to a trusted certificate
- You and the server do a handshake to prove to each other that the you and the server have the private keys that backs your certificates
Why Isn’t That Process Secure?
It is. Its probably the most secure authentication process that two systems talking to each other can do, when one single assumption is true – the user’s private key never leaves their possession. Certificate authentication is relied upon for high security authentication, such as applications managed by the US Federal Government or the US military. These certificates are not stored on the user’s local disk or in an s3 bucket. They’re stored on smart cards and when the handshake happens with the remote application the private key never leaves the user’s smart card. Also, most cards require some kind of PIN to unlock the authenticating key so its multi-factor as well.
Why Is This a Bad Idea for Kubernetes?
The first reason is that your private key is stored on insecure media (your disk). The kubectl command doesn’t yet support the standard needed to access a smart card, PKCS11. This means that at least for the time being, its impossible to use the kubectl command with a secure smart card. There’s also a good chance that your Kubernetes configuration was sent to you, rather then having you generate a local keypair and get it signed so even if your local disk were “secure” the fact is someone else has your private key. So much of the security of PKI is lost.
The next reason you shouldn’t use certificates for authentication to Kubernetes is that certificates are usually long lived. While there are some special use cases, most certificates are valid for about a year. What happens if your key is compromised? The CA infrastructure provides a way to revoke a certificate, to say “We no longer trust this key and neither should you” however because a certificate is self contained its up to the validating party (in this case the Kubernetes API server) to make sure the certificate hasn’t been revoked. Kubernetes does not support checking for revocation. So even if your key were compromised, there’s no way for Kubernetes to know at the authentication layer. It has been pointed out that you can use RBAC to sort of mimic a Certificate Rovocation List (CRL) by removing the subject of the certificate from all RBAC rules. This has multiple drawbacks. The first is that Kubernetes is still authenticating the key as valid. So even though its an invalid key Kubernetes is saying its OK. This opens the door to poorly written RBAC policies that can leave back doors. You could also have short lived certificates, such as 15 minutes, but then you still have a risk of compromise and no way to turn off access quickly.
Are you using the Kubernetes Dashboard? With certificate authentication you need to upload your Kubernetes configuration. You are giving the dashboard your private key to act on your behalf. This is also violates the cardinal rule of certificates to never give away your private key.
Finally, certificate authentication makes it very hard to use groups with RBAC. You can do it, but it means issuing a new certificate whenever rights change and since you can’t effectively revoke a certificate this functionality becomes mostly useless. There are multiple reasons that groups are a better way of managing RBAC access then adding subjects directly to PolicyBinding objects, but that’s a topic for another post.
Certificate Authentication Is Difficult On A Complex Network
In order for your client to authenticate a certificate, it MUST have a point-to-point connection with the API server. That’s how the handshake works. This means no reverse proxies or web application firewalls in front of your API server. Depending on your compliance requirements and infrastructure this can be a problem. You can work around this by using trusted headers, but then the API server isn’t authenticating the request anymore.
How Does OpenID Connect Solve These Problems?
OpenID Connect (OIDC) provides:
- Short lived tokens (I recommend 1 minute)
- A way to refresh tokens once they’ve expired
- A way to inject group information for easier RBAC management
- When using a reverse proxy that understands how to refresh tokens, a secured dashboard
- Can be used with almost any network topology
- A way to integrate multiple types of authentication, such as various multi-factor solutions (a demo of using Kubernetes with U2F keys)
You Said There Was One Use-case Where Certificates Were a Good Idea
Certificates are a good solution for “Break Glass In Case Of Emergency” situations where the OIDC provider is not reachable or OIDC hasn’t been configured yet. Kubeadm and OpenShift both do this by default, setting up certificates on the API masters so that kubectl can be used locally. That said, additional security layers should be used to protect the api servers. Examples are limiting access, treating access as privileged authentication and periodically re-certifying who should have access.
I understand why many users first go to certificate authentication. Its pretty easy to get going. Its also unfortunately very easy to do wrong and open your Kubernetes infrastructure up for attack and abuse. If you’re looking for an integrated way to integrate both the Dashboard and kubectl with OpenID Connect, take a look at our open source Kubernetes Login Portal that makes OIDC so much easier!