I am looking into setting up security scanners across podplane and related projects to identify any CRITICAL / HIGH security concerns to enforce your intentions of the project being “secure by default”.
The security scanners I have used in the past are https://trivy.dev/ and https://www.checkov.io/ - of independent value. These can be setup as github CI and local verifiers to validate configurations and dependencies across a range of technologies within the projects:
Golang
Docker, Helm
Terraform / OpenTofu (IaC)
Kubernetes
I am wondering if there is any preferences around security you would want Ryan? Just noting, these scanners can provide output to a json convention SARIF - Static Analysis Results Interchange Format. This provides security findings in a unified structure which can be used inconjunction with other security tools.
My current aim is to configure this across components. It would then support ideas of triage across the breadth of the podplane + dependent projects. I’m not a security expert, but I do consider a good start for identifying, and managing visibility against security threats.
Hey Max, welcome to the forum, thank you for taking the time to post! And sorry for the delayed reply - it seems notifications don’t work how I’d expect, I need to look into that!
I think this is a great idea. Particularly once the MVP feature set is complete (I don’t think I have formalised this, but roughly: local, AWS, Google Cloud, and Proxmox should all work for all implemented CLI functions and relevant components).
So right now I’m prioritising closing that MVP feature gap myself, then will switch focus to bug squashing and doing a security audit, ideally ahead of a 1.0 release. Some things on my mind for improving the security posture: looking at distroless and minimus base images instead of stock base images for all components and templates, and reviewing vmconfig to ensure it aligns to Debian hardening best practices (maybe consider AppArmor too, for example). There’s also things like Kubernetes production best practices check lists and known configuration-based CVEs which need another review to ensure the latest Kubernetes component configuration continues to align to those.
My experience with security scanning tools is that signal-to-noise is critical - if the number of actionable items is lost in a sea of things which cannot be immediately actioned, it becomes difficult to prioritise them. So my main ask is that we find a way to bubble up the most important stuff that we can tackle first, rather than a boil-the-ocean approach. Obviously as the project progresses, as more contributors come onboard, we can dial this up!
So the question becomes, you have a bunch of output in JSON files like SARIF - what are we going to do with this? How do we convert this into a small number of prioritised issues? Assume each contributor tackles a small number each week. And ideally we can tune it such that it keeps a backlog roughly consistent in size - as total contributor velocity drops or increases, adjust accordingly. The number of CVEs is exploding with AI, so to make this impactful, prioritisation is the key. Of course, I’m willing to keep an open mind to any approach which improves the security posture!
Appreciate the feedback and I agree with you on your approach. I’ve put together:
Based on some analysis I’ve put together over the weekend based on your MVP approach. The only thing I’d want to assert is whether vmconfig is used within the deployment clusters? Or is it only used during the purpose of testing?
Scanners
On security scanners, I think your absolutely right you’d want to manage the signal. Your current posture is fairly strong with using distroless/static:nonroot images by default, so far so is AppArmor shouldn’t be strictly unless you’d be considering to cover included third-party components.
Interestingly kube-bench which can audit Kubernetes production best practices is managed by the parent company Aqua Security · GitHub under the Apache 2.0 license. This is the same company providing Trivy CLI for static analysis. A combination of approaches would be good to play around with which is my intent.
My original idea with SARIF is to collect any of the CRITICAL / HIGH CVEs across all netsy.dev, nstance.dev and podplane to combine for a formal triage. What that would look like - IDK. Simpler things like dependabot would be a more appropriate first steps too. I’ll continue looking into what you recommend.
My appraoch
Personally I’d like to setup a scanner and kube-bench formally via Github CI which can be managed cheaply using terraform-aws-github-runner. It’s something I’ve setup for myself before as a low-cost way to manage Github Actions. I can share this once I’ve got this tidy within a personal Github Org scope.
I’ll need to start deploying to all cloud targets to verify any changes don’t break backwards compatibility. If you have any recommendations for how you have set this up for yourself in a way that’s easy, I’d be all ears. I typically use direnv + a`sercets-manager` for managing deployments to different clouds, but am without specific techniques to validate any breaking changes across your different targets other than hard up/down across each environment.
Once Podplane actually works for anything other than local clusters, it would be good to set up some kind of CI job which does a fresh deploy e.g. daily. Is that what you’re thinking?
Looking forward to seeing what you put together here.