A lot of work is going on behind the scenes right now so I thought I’d provide a little update because there are several big decisions being made and I’d like to make sure I’ve written down why these decisions were made, and welcome any additional feedback.
Moving from cert-manager to Pod Certificates, supporting SPIFFE SVID x509 client certificates & Service serving certificates
Podplane will use Kubernetes 1.37 stable support for Pod Certificates and Cluster Trust Bundles to handle automatically issuing Pods with a SPIFFE SVID x509 client certificate, and a serving certificate for Service SANs. The Podplane Operator will be the certificate signer for these. See CERTS spec for more.
Why? It greatly improves the security posture by keeping private keys out of etcd/netsy, and aligns closely to the expected future-state of Kubernetes.
Removing cert-manager, platform-certs, trust-manager, & platform-trust
The last remaining use case for cert-manager was ingress controller wildcard self-signed and ACME certificates.
Instead, Podplane Operator will handle ACME for ingress wildcard certificates, using lego (the package used by Treafik for ACME) due to its extensive and robust DNS-01 challenge DNS provider support.
Why? Less moving parts and coordination, a reduction in OCI dependencies by count and filesize, a significant reduction in initial cluster creation, and also keeping private keys out of etcd/netsy.
Replacing Treafik with Envoy Gateway
This was a tough decision to make and was not made lightly!
There are 4 reasons for this change:
- ClusterTrustBundle is supported by Envoy Gateway on Gateway API resources, and it’s been a stale PR for more than a month for Traefik. It’s unclear if Traefik is serious about keeping up with the latest Kubernetes features and best practices, which Envoy Gateway is very much aligned to.
- For the self-signed or ACME wildcard ingress certificates (as mentioned above), Traefik would require storing the private key in a Kubernetes Secrets - and as above, we want to avoid storing private keys in etcd/netsy. Envoy has the Secret Discovery Service (SDS) and Envoy Gateway supports SDS references.
- Envoy Gateway has first-class OIDC support via SecurityPolicy, which Traefik does not. Ideally we want applications to easily be protected behind a Truster deployment (note: we recently renamed Easy OIDC to Truster).
- Envoy & Envoy Gateway are licensed under the Apache 2.0 license, unlike Traefik which is MIT. Apache 2.0 is the preference license of the Podplane project, so moving to Envoy Gateway is better aligned on licensing.