OK, sorry for the click bait style headline but for the first blog of the new year I thought I’d have some fun. I came across a blog post on twitter from @geek_king about this real nasty bug in the kernel that can give you root access to the host server, breaking out of your container. Is this a big deal if you are using a container orchestration system like OpenShift or Kubernetes? It really could be since both give you the ability to deploy a container and use a remote shell to login to that container. Before we go too much further a few points:
- The kernel has been patched by Red Hat (and I’m assuming other vendors) so PATCH YOUR SYSTEMS!
- You should be limiting which containers can be pulled onto your OpenShift/Kubernetes cluster via only allowing from trusted registries so that would help mitigate this as well
That being said, nothing is fool proof:
- There will be new exploits found that let you break out of containers because, well, thats just how these things go
- A private docker registry is just a webapp, nothing could possibly be comprisable there, right?
- People need to login to both systems to do development, deployments, maintenance, etc….
Lets first look at OpenShift and how a break out compromise could happen, what the impact would be and possible mitigations particular to identity management. Here’s a very basic OpenShift deployment:
This architecture assumes we’re using LDAPS for OpenShift to authenticate to FreeIPA AND you are using FreeIPA and SSSD on all of the servers supporting OpenShift. How would someone be able to exploit a container breakout? There are several points of entry:
- How well locked down are Active Directory and FreeIPA?
- Is the secure container registry, well, secure?
- Are users disabled when they no longer need access?
Maybe an employee or contractor gets caught in a phishing campaign and those credentials get them into your network and those credentials have access to OpenShift (but NOT the underlying servers). I know, many “ifs” but networks are complex and complexity is the enemy of security (well amongst others). I have these credentials and I login to OpenShift using the web console and I can get an rsh console into a container, a wordpress website used by one of the business orgs. Now that we’re in, we have the ability to get to the outside world and can pull in a binary with this exploit and use it to break out. From here we could cause all sorts of mayhem. Also, if there are private keys on the server that are associated with public keys in FreeIPA I could have access to other servers as well!
Now, where does identity management come in to help mitigate this disaster?
- Hopefully you are requiring authorization before anyone can access OpenShift
- If credentials are compromised and identity solution can disable them quickly
- An identity system would let you re-certify access to OpenShift, so if someone leaves the team or the company their access doesn’t leave open a hole that can be exploited
- After the breach occurs, you’ll want to know why the credential had access, and who granted it
- Multi factor authentication could also help
#1, #2 and potentially #3 could be handled by FreeIPA to a degree. FreeIPA can disable access quickly, remove keys, and be used to track some authorizations. What it can’t do is provide access recertification, workflow based approvals or direct integration with OpenShift authorizations. Thats where Unison or OpenUnison would come in. Setting up a self service solution for identity management that directly integrates with OpenShift would let you quickly see who has access and why, when access was approved and quickly disable access. It could also be used to re-certify access periodically and integrate with additional second factors not supported by FreeIPA (such as Symantec VIP).
What if you’re using standard Kubernetes? Everything still applies! If you’re using certificates for authentication its actually much worse because you can’t apply a CRL right now so unless you disable the CA there’s no mechanism to disable access. OpenID Connect integration would help mitigate this, but all of the issues with OpenShift described above would apply to Kubernetes as well. Identity management could really help mitigate this breach.
Even though the headline indicates its the end of the world, it isn’t. There are many ways break out attacks could work in the wild and they clearly wouldn’t be the first target of a script kiddie joy riding on a Friday night, but a determined attacker could sure use this as a vector to get into a larger network. I hope you enjoyed this thought exercise and if you’d like to try an identity management solution, we happen to have one. Try out Unison or the open source version OpenUnison!