TL;DR
- Enable JetStack's OIDC Proxy for impersonation to support kubectl exec, cp, and port-forward with OpenUnison
- Deployment requires update to helm chart
- Inherits configuration from OpenUnison
- Directly integrated for authentication
Getting The Most From Impersonation
Kubernetes' impersonation feature is a powerful way to add authentication to clusters where you don't have the ability to configure OpenID Connect directly. This most often comes up in cloud based clusters where the administrators don't have the ability to set the parameters needed on the API server for a direct connection to their OpenID Connect identity provider. To enable external authentication in this scenario, a reverse proxy is used to intercept API requests and authenticate them on behalf of the API server. The API request is then forwarded to the API server with additional headers to tell the API server what user and groups should be used to run the API request as. OpenUnison has provided this capability out of the box for quite some time. Simply set enable_impersonation to true in your values.yaml and OpenUnison will setup this proxy for you. Everything is integrated. The major downside to this approach has been the lack of support for kubectl exec and port-forward. This is because SPDY, the protocol used by the client-go sdk and most popular Kubernetes tools, is an old protocol that is mostly unsupported in the industry outside of Kubernetes, including the web server that OpenUnison is built on. To work around this issue, we have operationalized the JetStack Kube OIDC Proxy. This popular tool lets you create a reverse proxy with the same parameters as the API server for configuring OpenID Connect. We integrated the oidc proxy into our operator, adding important operations features for security and usability. The proxy is pre-integrated directly into OpenUnison so there's no additional configuration or integration. Configure your values.yaml and the JetStack proxy is read to go!
Deploying the Kube OIDC Proxy
Regardless of which spin of OpenUnison you're using, integrating the oidc proxy is the same. After deploying your operator you simply need to add some settings to your values.yaml:
This will tell the helm chart to deploy the JetStack proxy instead of OpenUnison's integrated proxy. This is a typical scenario with EKS and a public signed certificate like Let's Encrypt. If you're using a self signed CA, such as for internal deployments, your configuration would look a little different:
In this case, explicit_certificate_trust is true and ca_secret_name is the name of a tls Secret in your openunison namespace that contains a tls.crt of your CA certificate chain. Deploy your spin's helm chart and now instead of using OpenUnison's built in reverse proxy for impersonation, you're now using JetStack's Kube OIDC Proxy!
Operationalizing the JetStack Proxy
Why use OpenUnison if you're not using our integrated reverse proxy? We went through the process of getting the JetStack proxy ready for production use:
- If using the helm chart's NetworkPolicy configuration, it automatically is applied to the proxy too
- If using the node_selectors in the helm chart, the proxy will as well
- The proxy no longer runs as root
- OpenUnison takes care of the ServiceAccount used by the proxy
- You can apply requests and limits directly to the proxy via the helm chart
There are two items that the JetStack proxy doesn't support but the OpenUnison proxy does:
- TokenRequest API support - OpenUnison can use short lived tokens to mitigate the security risks of a lost token. JetStack's proxy doesn't support it yet, but we're looking into adding support.
- Logging to stdout - OpenUnison logs all API requests in a way that can easily be piped directly into a SIEM tool. The JetStack proxy takes a route similar to the API server where audit data is sent to a webhook. Instead of complicating this process, we decided it was best to rely on the API server's own audit logs.
It's also important to note that your network infrastructure needs to support SPDY. This can be an issue when using modern network infrastructure like Amazon ALBs. More on that in the next section.
SPDY: The Legacy Protocol in Kubernetes
When you think about how kubectl exec and port forward works, you need a multi-direction protocol. You need to be able to send and receive input outside of the simple request and response model of HTTP. To achieve this connectivity, Kubernetes turned to SPDY, a prototype of the HTTP/2 protocol developed by Google. SPDY let Kubernetes build bi-directional communications over HTTP between users running kubectl and using internal components like between the kubelet and api server. The SPDY protocol paved the way for HTTP/2, which became a published standard in 2015. That same year, Google announced it was deprecating support for SPDY in Chrome and in 2016 it was removed entirely. With the removal of support for SPDY from browsers, web server and infrastructure vendors began removing support too. In fact, ourside of Kubernetes, it's almost impossible to find support in modern web servers or networking infrastructure and when it is present, it's marked with warnings saying you should be using HTTP/2. This also became an issue outside of golang based clients for Kubernetes, such as Python and Java that don't have native SPDY support anymore. To fix this issue, Kubernetes turned to WebSockets. Websockets provide a way to have bi-directional communications over HTTP. Its similar to HTTP/2 in that it requires an "upgrade" of a connection. The particulars of the protocol are very different though. That said, it's widely supported by network infrastructure and webservers. As an example, the terminal in the Kubernetes dashboard uses websockets to connect to a pod instead of SPDY. That's why the dashboard's terminal always worked with OpenUnison.
With the issue being in the client-go sdk, why not just fix it? Well, we started there. We began the work of adding websockets support in the client-go sdk, and even got it working for exec. The issue really comes with kubectl proxy and port-forward where we hit a stall. We also were worried that even once the support is in the SDK it could take quite some time for the tools compiled against the SDK to make use of the new support. There's also been talk of moving from SPDY to HTTP/2, but that has not seemed to go anywhere either. If you're interested in helping, we're happy to accept the help!
Since even if we were able to get websockets support into kubectl it could take some time to make its way to other tools, we decided to add the JetStack proxy to better support our customers and users. Long term the goal will be to "deprecate" its use once the SPDY technical debt is finally paid off. This will of course be transparent to you as the user!
Upgrading Existing Deployments
Upgrading your existing instance of OpenUnison is very simple. First, update your operator:
Once completed and your new instance of the operator is running, follow the instructions above and run an upgrade against your existing deployment via helm!