--- title: "QuenchWorks: Building a 0-CVE Container Image and Helm Chart Catalog" lang: "en" author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/post/quenchworks-zero-cve-catalog --- ![Blog post image for QuenchWorks: Building a 0-CVE Container Image and Helm Chart Catalog - How a from-scratch catalog replaced Bitnami with 150+ container images and 120+ Helm charts built from source on Wolfi, gated to zero fixable CVEs, signed, and pinned by digest.](/_astro/hero.BfjMKoMg_2mDXh4.webp) [Home](/)›[Case studies](/case-studies)›[All Categories](/case-studies/categories)›[Security](/case-studies/categories/security) Case studies [Security](/case-studies/categories/security)[DevOps](/case-studies/categories/devops)[Cloud Native](/case-studies/categories/cloud-native) # QuenchWorks: Building a 0-CVE Container Image and Helm Chart Catalog [Mohammad Abu Mattar](/authors/mohammad-abu-mattar)Published: 15 Jul 202608 Mins read12 Mins listen [Markdown for AI(opens in a new tab)](/post/quenchworks-zero-cve-catalog/index.md "Open the plain-Markdown version of this page, for pasting into an AI tool") TL;DR How a from-scratch catalog replaced Bitnami with 150+ container images and 120+ Helm charts built from source on Wolfi, gated to zero fixable CVEs, signed, and pinned by digest. Series [QuenchWorks](/series/quenchworks)1/1 All posts in this series (1) Case Studies1 1. [QuenchWorks: Building a 0-CVE Container Image and Helm Chart CatalogYou are here](/case-studies/post/quenchworks-zero-cve-catalog) ### QuenchWorks: Building a 0-CVE Container Image and Helm Chart Catalog Contents [Impact](#impact)[The problem](#the-problem)[Constraints](#constraints)[Architecture](#architecture)[Implementation](#implementation)[Results](#results)[Lessons](#lessons)[Frequently Asked Questions](#frequently-asked-questions)[References](#references) When Bitnami moved its long-trusted catalog behind a paid tier, thousands of teams woke up to a supply-chain problem they didn’t choose. The free images they had pinned in production would stop getting updates, and the migration clock started that morning. QuenchWorks is my answer to that. It is a catalog of container images and Helm charts, built entirely from source, hardened under a strict zero-CVE build gate, signed, and free. This is how it’s put together and why each decision earns its place. ## [Impact](#impact) QuenchWorks is real and in production use, not a proof of concept. It replaced Bitnami for common workloads with a catalog that is built from source, provable, and free. Everything below is verifiable: pull any image and check its signature, SBOM, and provenance yourself. The build gate stays green because the base is small enough that there is almost nothing to be vulnerable in, so the numbers hold instead of drifting the week after launch. 0+images built 0-CVE from source 0+production Helm charts 0%cosign-signed, SBOM + provenance ## [The problem](#the-problem) The Bitnami catalog was popular for good reasons. It was broad, it was versioned, and it was maintained well enough that most teams never thought about it. Its weaknesses only became obvious once access changed: you didn’t control the build, you couldn’t prove what was inside a given image, and continued free access was never actually guaranteed. That last point is the one that bites. When an upstream catalog changes its terms, every `image:` line you pinned becomes a liability at once. You either pay, fork, or scramble. And even before that day comes, an opaque image is its own quiet risk. If you can’t see how a layer was produced, you can’t reason about what a scanner finds inside it, and you can’t answer a security review with anything better than “we trust the vendor.” Most hardened-image alternatives fix one slice of this and charge for the rest. I wanted the whole thing: a catalog broad enough to actually replace Bitnami for common workloads, provable rather than “trust us,” and free with no pull limits and no lock-in. If it couldn’t be all three, it wasn’t worth building. ## [Constraints](#constraints) A handful of hard limits shaped every later decision. - **Zero fixable CVEs, enforced by the build.** Not a nightly report someone reads later. A gate that fails the build so a vulnerable image never ships in the first place. - **Built from source.** No repackaging of someone else’s opaque binary layers. If it’s in the image, we produced it. - **Provable.** Every image needs a bill of materials and build provenance that a consumer can verify without trusting me. - **Free to run and maintain.** The whole system builds on free CI, so cost can never be the reason it slips behind a paywall later. - **Multi-arch.** amd64 and arm64, because production is both now, not one or the other. Those constraints pull against each other. Zero fixable CVEs across 150+ images sounds impossible if you picture a fat base image. Building everything from source sounds slow. The architecture is what makes them coexist. ## [Architecture](#architecture) The catalog is a pipeline, not a pile of Dockerfiles. Each image is declared as an `apko` plus `melange` spec, built from source on Wolfi, scanned against a zero-fixable-CVE gate, signed, and only then published pinned by digest. Charts sit on a shared library chart and reference those images by digest. From declarative spec to signed, digest-pinned image and chart, with a nightly rescan self-heal loop. The single decision that makes the zero-CVE gate realistic is the base. QuenchWorks builds on Wolfi, a glibc Linux undistro designed for containers. Most images start with no shell, no package manager, and a tiny set of packages. There’s simply very little in the image that can be vulnerable, so keeping the gate green is a fight you can actually win instead of an endless race against a bloated base. The image itself is assembled declaratively. `melange` builds signed APK packages from source, and `apko` composes those packages plus the Wolfi base into an OCI image with no Dockerfile involved. Because the whole thing is declared, the contents are known, reproducible, and easy to record as a bill of materials. melange builds APKs from source; apko assembles them with the Wolfi base into a nonroot, multi-arch image and an SBOM. Tip The zero-CVE gate and the minimal base are the same decision viewed twice. You don’t reach zero fixable CVEs by patching harder. You reach it by shipping so little that there’s almost nothing to patch. A catalog is never done, though, because CVEs are disclosed against packages long after an image ships. So the pipeline runs in reverse on a schedule. A nightly Trivy rescan checks every published image, and when a fix lands upstream the affected image rebuilds, re-enters the gate, gets re-signed, and republishes under a new digest. The catalog trends toward zero drift without anyone babysitting it. Published images are rescanned nightly; a new fixable CVE triggers an automatic rebuild, re-gate, and re-sign. ## [Implementation](#implementation) Each image is a pair of specs. `melange` describes how to build the package from source, and `apko` describes how to assemble the final image. Here’s the shape of both, trimmed for clarity. melange.yaml ``` 1package:2 name: my-app3 version: 1.2.34environment:5 contents:6 packages:7 - build-base8pipeline:9 - uses: fetch10 with:11 uri: https://example.com/my-app-${{package.version}}.tar.gz12 expected-sha256: '...'13 - uses: autoconf/configure14 - uses: autoconf/make15 - uses: autoconf/make-install ``` apko.yaml ``` 1contents:2 repositories:3 - https://packages.wolfi.dev/os4 packages:5 - my-app6 - ca-certificates-bundle7accounts:8 users:9 - username: nonroot10 uid: 6553211 run-as: 6553212archs:13 - x86_6414 - aarch6415entrypoint:16 command: /usr/bin/my-app ``` The gate is one Trivy call, and it’s deliberately strict about what counts. It only fails on CVEs that have a fix available, because a vulnerability with no upstream patch isn’t something a rebuild can clear. Everything fixable has to be at zero before the image is allowed out. 0-CVE gate ``` # Fail the build if any FIXABLE HIGH/CRITICAL vulnerability is presenttrivy image --ignore-unfixed --severity HIGH,CRITICAL \ --exit-code 1 ghcr.io/quenchworks/my-app:latest ``` Once an image passes, it gets signed and attested before it’s pushed for real. Signing is keyless with Cosign, so there’s no long-lived private key to leak, and the SBOM and SLSA provenance ride along as attestations. 1. **Build from source.** `melange` produces signed APKs; `apko` assembles the image with the Wolfi base, nonroot user, and read-only root filesystem defaults. 2. **Gate on zero fixable CVEs.** Trivy scans the full image. One fixable HIGH or CRITICAL fails the pipeline, so a vulnerable image never reaches the registry. 3. **Sign and attest.** Cosign signs the image keyless, then attaches an SPDX SBOM and a SLSA build-provenance attestation. 4. **Publish pinned by digest.** The image is pushed, and the Helm charts reference it by `sha256:` digest, never by a movable tag. 5. **Rescan nightly.** A scheduled Trivy run watches for new fixes and triggers the self-heal rebuild loop. On the delivery side, charts are the second half of the story. Every chart builds on a shared `quench-common` library chart, so common concerns like security context, probes, and labels live in one place instead of being copy-pasted 120 times. Each chart pins its image by digest. App charts depend on the quench-common library chart and reference images by digest, which Helm then installs into the cluster. Pinning by digest instead of tag is what makes the catalog trustworthy in practice. A tag can be moved; a digest can’t. When a consumer pins a QuenchWorks chart, they get exactly the bytes that passed the gate. values.yaml ``` 1image:2 repository: ghcr.io/quenchworks/postgresql3 # Pinned by digest, not tag. This is the exact image that passed the gate.4 digest: 'sha256:abc123...' ``` The proof only matters if consumers can check it, so verification gets its own documented step. Anyone can verify an image’s signature and attestations before it runs, and an admission policy can enforce that in the cluster so unsigned or unverifiable images never schedule. Verify before you run ``` cosign verify \ --certificate-identity-regexp '^https://github.com/quenchworks/' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ ghcr.io/quenchworks/postgresql@sha256:abc123... ``` A consumer pulls by digest, verifies the keyless signature and the SBOM/SLSA attestations, and only then does admission allow the deploy. ## [Results](#results) Everything below is running today, not staged for a launch. - 150+ container images built from source on Wolfi, each gated to zero fixable CVEs. - 120+ production Helm charts on the shared `quench-common` library, every one pinned to its image by digest. - Every image cosign-signed with an SPDX SBOM and SLSA provenance, published under an ArtifactHub verified-publisher organization. - Multi-arch (amd64 and arm64), nonroot, and read-only root filesystem by default. - A nightly rescan and self-heal rebuild loop that keeps the catalog current as upstream ships fixes. - Free and independent. There is no subscription, no registry pull limit, and nothing that locks you in. The counts above are current catalog figures. Per-image build times and scan times vary by package, so I’d treat any single number there as indicative rather than a benchmark. ## [Lessons](#lessons) The biggest lesson is that the base image choice decides everything downstream. Trying to reach zero CVEs on a fat base is a treadmill; starting from Wolfi’s minimal surface turns the gate into something you can keep green for months. If I’d started anywhere else, the self-heal loop would be firing constantly and the whole thing would feel like bailing water. The second lesson is that provenance costs far less than it’s worth. Signing and generating SBOMs added very little build time, but they change the catalog’s whole posture. It stops being “trust me” and becomes “verify it yourself,” which is the entire point of replacing an opaque upstream. If I were doing it again, I’d wire verification into the consumer docs even earlier, because an unverified signed image is only half the value. The one thing I’d watch more carefully next time is chart sprawl. The `quench-common` library chart paid for itself immediately, but library conventions need to be locked down early. Once a few charts drift from the shared patterns, every future change gets more expensive. ## [Frequently Asked Questions](#frequently-asked-questions) It’s zero _fixable_ CVEs, and it’s mostly a consequence of the base. Wolfi images ship with almost nothing beyond what the app needs, so there’s very little surface for a vulnerability to live in. The Trivy gate then fails any build with a fixable HIGH or CRITICAL, so a vulnerable image can’t ship. Vulnerabilities with no upstream fix are tracked but don’t block, because a rebuild can’t clear them. A tag is a movable pointer; a digest is the content itself. If you pin `:latest` or even `:1.2.3`, the bytes behind that tag can change. Pinning `sha256:...` guarantees you get exactly the image that passed the gate and was signed. It’s the difference between “probably the right image” and “provably the right image.” Use `cosign verify` with the QuenchWorks certificate identity and OIDC issuer, as shown above. That checks the keyless signature against the transparency log. You can also verify the SBOM and SLSA provenance attestations, and enforce all of it in-cluster with an admission policy so nothing unsigned ever schedules. The nightly Trivy rescan catches it. If the CVE is fixable, the affected image rebuilds from source, goes back through the gate, gets re-signed, and republishes under a new digest. You pick up the fix by moving your pin to the new digest. Nobody has to notice the CVE manually for the loop to run. It’s free, with no subscription and no registry pull limits. The catch, if you call it one, is that you verify and pin things yourself rather than outsourcing trust to a vendor relationship. That’s a feature for most teams: you get provenance you can audit instead of a support contract you have to believe. Yes. The charts and images are independent. You can pull a single hardened image by digest, or install one chart, without buying into everything. The `quench-common` library chart is an implementation detail of the charts, not something you have to adopt in your own repos. ## [References](#references) - [QuenchWorks catalog and website](https://quench-works.com/) - [QuenchWorks on GitHub](https://github.com/quenchworks) - [Wolfi undistro](https://github.com/wolfi-dev) - [apko](https://github.com/chainguard-dev/apko) and [melange](https://github.com/chainguard-dev/melange) - [Trivy vulnerability scanner](https://trivy.dev/) - [Sigstore Cosign](https://docs.sigstore.dev/) - [SPDX](https://spdx.dev/) and [SLSA provenance](https://slsa.dev/) - [ArtifactHub](https://artifacthub.io/) Was this useful? ## Tags [#Containers](/case-studies/tags/containers)[#Wolfi](/case-studies/tags/wolfi)[#Helm](/case-studies/tags/helm)[#Supply Chain Security](/case-studies/tags/supply-chain-security)[#Trivy](/case-studies/tags/trivy)[#Cosign](/case-studies/tags/cosign)[#SLSA](/case-studies/tags/slsa)[#ArtifactHub](/case-studies/tags/artifacthub) ## Share [Facebook](https://facebook.com/sharer/sharer.php?u=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fquenchworks-zero-cve-catalog "Share on Facebook")[Twitter](https://twitter.com/intent/tweet/?text=QuenchWorks%3A%20Building%20a%200-CVE%20Container%20Image%20and%20Helm%20Chart%20Catalog&url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fquenchworks-zero-cve-catalog "Share on Twitter")[LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fquenchworks-zero-cve-catalog&title=QuenchWorks%3A%20Building%20a%200-CVE%20Container%20Image%20and%20Helm%20Chart%20Catalog&summary=How%20a%20from-scratch%20catalog%20replaced%20Bitnami%20with%20150%2B%20container%20images%20and%20120%2B%20Helm%20charts%20built%20from%20source%20on%20Wolfi%2C%20gated%20to%20zero%20fixable%20CVEs%2C%20signed%2C%20and%20pinned%20by%20digest.&source=https://mkabumattar.com "Share on LinkedIn")[WhatsApp](https://wa.me/?text=QuenchWorks%3A%20Building%20a%200-CVE%20Container%20Image%20and%20Helm%20Chart%20Catalog%20https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fquenchworks-zero-cve-catalog "Share on WhatsApp")[Telegram](https://t.me/share/url?url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fquenchworks-zero-cve-catalog&text=QuenchWorks%3A%20Building%20a%200-CVE%20Container%20Image%20and%20Helm%20Chart%20Catalog "Share on Telegram")[Reddit](https://www.reddit.com/submit?url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fquenchworks-zero-cve-catalog&title=QuenchWorks%3A%20Building%20a%200-CVE%20Container%20Image%20and%20Helm%20Chart%20Catalog "Share on Reddit")[Hacker News](http://news.ycombinator.com/submitlink?u=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fquenchworks-zero-cve-catalog&t=QuenchWorks%3A%20Building%20a%200-CVE%20Container%20Image%20and%20Helm%20Chart%20Catalog "Share on Hacker News")[Pinterest](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fquenchworks-zero-cve-catalog&media=&description=How%20a%20from-scratch%20catalog%20replaced%20Bitnami%20with%20150%2B%20container%20images%20and%20120%2B%20Helm%20charts%20built%20from%20source%20on%20Wolfi%2C%20gated%20to%20zero%20fixable%20CVEs%2C%20signed%2C%20and%20pinned%20by%20digest. "Share on Pinterest")[Email]() ## Comments ## You might also enjoy More posts on similar topics [![Migrating a Monolith to Kubernetes Without a Big-Bang Cutover](/_astro/hero.CAKh7bXG_Z1PnEBt.webp)](/case-studies/post/monolith-to-kubernetes-strangler-migration) ## [Migrating a Monolith to Kubernetes Without a Big-Bang Cutover](/case-studies/post/monolith-to-kubernetes-strangler-migration) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [DevOps](/case-studies/categories/devops) - [Cloud Native](/case-studies/categories/cloud-native) - [Architecture](/case-studies/categories/architecture) Almost every failed "let's move off the monolith" project shares one detail: the plan was a big-bang cutover. Rewrite in parallel, pick a weekend, flip the switch, and pray. This is the opposite of th [#Kubernetes](/case-studies/tags/kubernetes)[#EKS](/case-studies/tags/eks)[#Migration](/case-studies/tags/migration)+4 tags [read more](/case-studies/post/monolith-to-kubernetes-strangler-migration) [![Building an Internal Developer Platform on Backstage and GitOps](/_astro/hero.Dq3xrist_1gdSPN.webp)](/case-studies/post/internal-developer-platform-backstage-gitops) ## [Building an Internal Developer Platform on Backstage and GitOps](/case-studies/post/internal-developer-platform-backstage-gitops) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [DevOps](/case-studies/categories/devops) - [Platform Engineering](/case-studies/categories/platform-engineering) - [Cloud Native](/case-studies/categories/cloud-native) Product teams were spending more time waiting on the platform team than building features. Spinning up a new service meant opening a ticket and waiting for someone to provision a repo, wire up CI, wri [#Backstage](/case-studies/tags/backstage)[#GitOps](/case-studies/tags/gitops)[#Argo CD](/case-studies/tags/argo-cd)+4 tags [read more](/case-studies/post/internal-developer-platform-backstage-gitops) [![Cutting a SaaS AWS Bill 41% Without Slowing Delivery](/_astro/hero.DJTB593d_Z1iASgs.webp)](/case-studies/post/aws-cost-optimization-saas-case-study) ## [Cutting a SaaS AWS Bill 41% Without Slowing Delivery](/case-studies/post/aws-cost-optimization-saas-case-study) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Cloud Computing](/case-studies/categories/cloud-computing) - [DevOps](/case-studies/categories/devops) - [Cloud Native](/case-studies/categories/cloud-native) A growing SaaS ran on EKS with a full GitOps pipeline, and it was over its AWS budget nearly every month. The reflex from leadership was the usual one: freeze features until the bill comes down. That [#AWS](/case-studies/tags/aws)[#EKS](/case-studies/tags/eks)[#Kubernetes](/case-studies/tags/kubernetes)+7 tags [read more](/case-studies/post/aws-cost-optimization-saas-case-study) [![Zero-Downtime PostgreSQL Major-Version Upgrade at Scale](/_astro/hero.C03RcOLI_141vsK.webp)](/case-studies/post/zero-downtime-postgres-upgrade) ## [Zero-Downtime PostgreSQL Major-Version Upgrade at Scale](/case-studies/post/zero-downtime-postgres-upgrade) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [DevOps](/case-studies/categories/devops) - [Databases](/case-studies/categories/databases) - [Cloud Computing](/case-studies/categories/cloud-computing) A multi-terabyte PostgreSQL 12 database was reaching end of life, and the business ran around the clock, so the usual answer of "schedule a maintenance window" was off the table. We upgraded it to Pos [#PostgreSQL](/case-studies/tags/postgresql)[#Logical Replication](/case-studies/tags/logical-replication)[#Zero Downtime](/case-studies/tags/zero-downtime)+3 tags [read more](/case-studies/post/zero-downtime-postgres-upgrade) [![Multi-Region Active-Active for a Payments API](/_astro/hero.RPbRRCdE_flG0g.webp)](/case-studies/post/multi-region-active-active-payments) ## [Multi-Region Active-Active for a Payments API](/case-studies/post/multi-region-active-active-payments) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Architecture](/case-studies/categories/architecture) - [Cloud Computing](/case-studies/categories/cloud-computing) - [Reliability](/case-studies/categories/reliability) A payments API that moves real money had been running comfortably in a single AWS region for years. It was reliable until the day it was not: a regional control-plane incident took the whole service o [#Multi Region](/case-studies/tags/multi-region)[#Active Active](/case-studies/tags/active-active)[#Payments](/case-studies/tags/payments)+5 tags [read more](/case-studies/post/multi-region-active-active-payments) 5 related posts