Mercure 0.23.5: Helm chart hardening

Mercure v0.23.5 just landed, and the dominant theme is the Helm chart. If you run hubs on Kubernetes, especially in HA or multi-tenant mode, this release tightens defaults and adds the kind of policy templates that previously required forking the chart or templating policies outside it.

The story behind the release: we audited a production Mercure Cloud Kubernetes cluster. The findings were straightforward (root containers, no NetworkPolicy, missing PodSecurity hardening), and the practical question was where to fix them. Most belonged in the OSS Helm chart, because every chart user on a multi-tenant or HA cluster runs into the same constraints!

Opt-in NetworkPolicy and CiliumNetworkPolicy templates

The chart now ships two new templates, both disabled by default. A standard NetworkPolicy for clusters whose CNI enforces it (Calico, Cilium, …). A CiliumNetworkPolicy for Cilium-specific features like FQDN-pinned egress and L7 rules. You enable whichever your CNI supports and pass through the rule lists:

ciliumNetworkPolicy:
  enabled: true
  ingress:
    - fromEntities:
        - ingress
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
  egress:
    - toEntities:
        - kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*"
    - toFQDNs:
        - matchName: redis.example.com
      toPorts:
        - ports:
            - port: "6379"
              protocol: TCP

When ingress and egress are both populated, Cilium treats anything outside the lists as denied. That gives you per-tenant default-deny without having to write your own policies. The chart’s endpointSelector is scoped to app.kubernetes.io/component: server, so the helm test pod (which inherits selector labels but not the component label) stays unscoped and helm test keeps working.

readOnlyRootFilesystem out of the box

Until now, setting securityContext.readOnlyRootFilesystem: true would crash the pod on first write because Caddy autosaves config under XDG_CONFIG_HOME=/config. The chart now mounts /config and /tmp as emptyDir unconditionally, and /data as a writable PVC when persistence.enabled: true.

The Go library also got a fix: bolt.NewBoltTransport now creates the parent directory before opening the database, so a fresh empty /data no longer crashes the hub on first write.

A working rootless values.yaml snippet for v0.23.5, with the hub binding to an unprivileged port and all capabilities dropped:

podSecurityContext:
  runAsNonRoot: true
  runAsUser: 1000
  runAsGroup: 1000
  fsGroup: 1000
  seccompProfile:
    type: RuntimeDefault
securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop: [ALL]
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000
service:
  targetPort: 8080            # not needed on runtimes with ip_unprivileged_port_start=0

Modern container runtimes (containerd 1.5+, cri-o, Docker 20.10+) set net.ipv4.ip_unprivileged_port_start=0 inside the container, so an unprivileged process can bind any port directly without any capability. service.targetPort: 8080 is still useful as a fallback for older runtimes that still keep ip_unprivileged_port_start at 1024, but it is no longer required.

Restricted PodSecurity Standard by default

The chart now ships defaults aligned with the restricted PSS where it can do so without breaking existing users:

• serviceAccount.automount: false (Mercure does not call the Kubernetes API).
• enableServiceLinks: false on the hub Pod (no neighbour-Service env leak in shared namespaces).
• podSecurityContext.seccompProfile.type: RuntimeDefault (engages the runtime seccomp profile).
• The helm test pod is also fully hardened: pinned busybox:1.37, RuntimeDefault seccomp, drop ALL caps, RO rootfs, non-root, wget -q --spider so the response body is not written to disk.

The container-level securityContext (drop ALL caps, RO rootfs, runAsNonRoot, etc.) stays opt-in via the securityContext map, since flipping those defaults is a real breaking change for users with custom images. That one is left for a future chart minor.

Topic-selector cache capped at 100k entries

The default topic-selector cache size was 2,560,000 entries. At roughly 100 bytes per entry, a busy hub could push the cache to ~256 MB, which, on a small pod, sat right against GOMEMLIMIT and put Go’s runtime in a gcBgMarkWorker thrashing loop. Pods spent more than 90% of their CPU in GC.

The new default is 100,000 entries (~10 MB at the same per-entry cost). You can resize per workload via topic_selector_cache in the Caddyfile, or set it to -1 to disable caching entirely.

Mercure Cloud and Mercure Enterprise

If you run Mercure Cloud, every default in this release is already applied for you, because we manage the cluster on your behalf. You also get the production transports (Redis, Kafka, Pulsar, Postgres), managed multi-tenant isolation, and an SLA-backed offering.

If you want the same hardening on-premise plus the HA transports and priority support, Mercure Enterprise has you covered. Contact contact@mercure.rocks for the managed cloud offering, on-premise licenses, custom development, consulting, and training.

The release notes are on GitHub: v0.23.4 (v0.23.5 was tagged shortly after and contains a hotfix).