Highly available CoreDNS on Scaleway Kapsule

Scaleway Kapsule runs the cluster DNS on a single CoreDNS pod. When you drain the node where this pod runs, DNS lookups fail across the whole cluster until a new pod is ready.

A managed service is not automatically a redundant service. You cannot solve this by scaling Scaleway's CoreDNS, because Kapsule changes it back. What does work is a second CoreDNS Deployment of your own, behind Scaleway's DNS Service. You do not need to change anything that Scaleway owns.

We have run this setup on the three clusters of Heystaq, our own product, since 28 October 2026. This article describes the full setup, including the parts we have not proven yet.

Why one CoreDNS pod is a problem

On 27 October 2026, a node roll drained the node that ran the only CoreDNS pod. DNS lookups to the DNS Service address failed with read: connection refused.

A short DNS outage during a node roll may not sound serious. In our case it was, because of Grafana. Grafana saw the failed lookups as a permanent error and dropped alert notifications after a single attempt. So routine maintenance caused alerts that never arrived.

Node rolls happen all the time. Every node pool upgrade or node replacement drains nodes. With only one DNS pod, each of these events puts DNS at risk.

Kapsule undoes any change to its CoreDNS

The obvious fix is to add more replicas. On Kapsule this does not last, and the cluster itself shows you why.

The addon manager owns the CoreDNS Deployment and Service. kube-system/coredns has the label addonmanager.kubernetes.io/mode=Reconcile. Because of this label, the addon manager reverts every change to these objects: a new replica count, a patch, or an autoscaler. The ConfigMaps work differently. The addon manager creates coredns and coredns-resolvconf with the mode EnsureExists, and after that it leaves them alone.

The names are also not what you may expect. The Kubernetes documentation calls the DNS Service kube-dns. On Kapsule it is called coredns, and it selects pods with the label k8s-app=kube-dns. If you run kubectl get svc kube-dns on Kapsule, you get NotFound. A runbook written for upstream Kubernetes will fail here.

Scaleway also does not document a supported way to run more replicas. The shared responsibility model and the managed Kubernetes service definition say nothing about scaling CoreDNS. Both list CoreDNS as a system add-on that Scaleway manages. The responsibility model also says that responsibility moves to you when you "modify or replace preinstalled components".

If you scale the Deployment by hand, it looks fixed. A few minutes later, the addon manager reverts your change.

The fix: a second Deployment that Scaleway's Service already selects

Scaleway's Service selects its pods by label. Every pod in kube-system with the label k8s-app=kube-dns receives traffic on the DNS Service address. It does not matter which Deployment the pod belongs to.

So the fix is to create your own Deployment, called coredns-ha. Its pods have this label, and they mount Scaleway's own coredns ConfigMap. Scaleway's Service sends traffic to them, and both Deployments always use the same Corefile.

This is the whole change: one Deployment and a PodDisruptionBudget for it. Scaleway's Deployment, Service and ConfigMaps stay as they are.

There is a cost: the two new pods are your responsibility. After every Kapsule upgrade, you must keep their image the same as Scaleway's. After every node roll, you must check which zones they run in.

First, inspect your cluster

Start with five checks. They only read from the cluster and change nothing. Run them on every cluster and save the output, because you need the image and the zones for your manifests.

# 1. How many DNS pods, on which nodes
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
 
# 2. The zones of your nodes
kubectl get nodes -L topology.kubernetes.io/zone
 
# 3. Scaleway's image (it differs per region, because each region pulls from its own cache)
kubectl -n kube-system get deploy coredns -o jsonpath='{.spec.template.spec.containers[0].image}'
 
# 4. The full pod spec to mirror
kubectl -n kube-system get deploy coredns -o yaml
 
# 5. Which objects the addon manager owns, and how
kubectl -n kube-system get deploy,svc,cm -L addonmanager.kubernetes.io/mode

If your cluster has only one zone, the zone rules below still work. But they cannot protect you when that one zone goes down.

The manifests copy Scaleway's pod spec

You need three files in a kustomize base, plus one overlay per cluster to set the image. The Deployment copies Scaleway's pod spec as we found it on Kapsule 1.36 on 28 October 2026. We copy it on purpose: when both Deployments run the same image with the same Corefile, it does not matter which pod the Service chooses. Before you use it, compare it with your output from check 4, because a Kapsule upgrade can change Scaleway's spec.

base/deployment.yaml:

# A second CoreDNS Deployment beside Scaleway's one-pod coredns. It selects on its own
# label, because Scaleway's selector (k8s-app=kube-dns) also matches these pods.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns-ha
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    app.kubernetes.io/name: coredns-ha
spec:
  replicas: 2
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: coredns-ha
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
  template:
    metadata:
      labels:
        k8s-app: kube-dns
        app.kubernetes.io/name: coredns-ha
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        runAsGroup: 65532
        seccompProfile:
          type: RuntimeDefault
      topologySpreadConstraints:
        # Hard, over our own pods: they never share a zone.
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          nodeTaintsPolicy: Honor
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: coredns-ha
        # Preference over every DNS pod, Scaleway's included.
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              k8s-app: kube-dns
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: coredns-ha
      containers:
        - name: coredns
          image: coredns   # set per cluster in the overlay
          args: ["-conf", "/etc/coredns/Corefile"]
          imagePullPolicy: IfNotPresent
          livenessProbe:
            httpGet: {path: /health, port: 8080, scheme: HTTP}
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 5
          readinessProbe:
            httpGet: {path: /ready, port: 8181, scheme: HTTP}
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          ports:
            - {containerPort: 53, name: dns, protocol: UDP}
            - {containerPort: 53, name: dns-tcp, protocol: TCP}
            - {containerPort: 9153, name: metrics, protocol: TCP}
          resources:
            requests: {cpu: 100m, memory: 70Mi}
            limits: {memory: 170Mi}
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              add: [NET_BIND_SERVICE]
              drop: [all]
            readOnlyRootFilesystem: true
          volumeMounts:
            - {name: config-volume, mountPath: /etc/coredns, readOnly: true}
            - {name: custom-config-volume, mountPath: /etc/coredns/custom, readOnly: true}
            - {name: resolv-conf, mountPath: /run/systemd/resolve/resolv.conf, readOnly: true}
      dnsPolicy: Default
      priorityClassName: system-cluster-critical
      serviceAccountName: coredns
      terminationGracePeriodSeconds: 30
      tolerations:
        - {key: node-role.kubernetes.io/master, effect: NoSchedule}
        - {key: CriticalAddonsOnly, operator: Exists}
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items: [{key: Corefile, path: Corefile}]
        - name: custom-config-volume
          projected:
            sources:
              - configMap: {name: coredns, optional: true, items: [{key: empty, path: empty.override}]}
              - configMap: {name: coredns, optional: true, items: [{key: empty, path: empty.server}]}
              - configMap: {name: coredns-resolvconf, items: [{key: resolv-override.conf, path: resolv-override.conf}]}
              - configMap: {name: coredns-custom, optional: true}
        - name: resolv-conf
          hostPath: {path: /run/systemd/resolve/resolv.conf, type: File}

base/pdb.yaml selects only your own pods, never Scaleway's:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: coredns-ha
  namespace: kube-system
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: coredns-ha

base/kustomization.yaml lists deployment.yaml and pdb.yaml. Each overlays/<cluster>/kustomization.yaml sets the image you found in check 3. This is the overlay for Amsterdam:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: coredns
    newName: cache.k8s.nl-ams.scw.cloud/registry.k8s.io/coredns/coredns
    newTag: v1.14.7

There is one intentional difference from Scaleway's spec: the pod securityContext. Scaleway leaves it empty. Ours declares the user that the image already runs as, which also fixes the Trivy finding KSV-0118. Each pod requests 100m CPU and 70Mi of memory.

Two pods in two zones

Your two pods never run in the same zone. If you lose one zone, at least one of them keeps running. Every setting in the table below protects this, and we learned each one from an attempt that failed.

The Kubernetes documentation on topology spread constraints explains how cordoned nodes are handled. With nodeTaintsPolicy: Honor, the skew only counts nodes without taints, plus tainted nodes that the pod tolerates. If you do not set the field, it behaves as Ignore. The field has been beta since Kubernetes 1.26 and GA since 1.33.

Only a server-side dry run finds the duplicate zone rule

The API server identifies each entry in topologySpreadConstraints by the combination of topologyKey and whenUnsatisfiable. If you have two DoNotSchedule rules on topology.kubernetes.io/zone, the API server rejects them as duplicates, and Flux cannot apply the manifest. The Kubernetes documentation allows only one constraint per combination.

kustomize build, our tests and our code review all missed this. Only the server-side dry run found it.

That is why the manifest has exactly one hard zone rule, for your own pods, and one soft zone rule for all DNS pods. The downside is that keeping your pods out of Scaleway's zone is only a preference, not a guarantee.

Roll it out and watch DNS while you do it

1. Run a dry run on every cluster before you open the merge request. It saves nothing, and it finds anything the API server would reject:

kustomize build overlays/<cluster> | kubectl apply --server-side --dry-run=server --field-manager=kustomize-controller -f -

Both lines must say serverside-applied (server dry run).

2. Apply it with Flux. Use one Flux Kustomization per cluster that points to the overlay, with wait: true and a health check on the coredns-ha Deployment. If the apply fails, Flux then shows it in red, instead of quietly keeping the old revision running.

3. Watch while it applies. Before you merge, start a loop that runs once per second. It reads the Ready endpoints behind the DNS Service, and any Pending pods:

kubectl -n kube-system get endpointslices -l kubernetes.io/service-name=coredns -o jsonpath='{range .items[*].endpoints[*]}{.conditions.ready}{" "}{end}'
kubectl -n kube-system get pods -l app.kubernetes.io/name=coredns-ha

You should see at least two Ready endpoints the whole time: Scaleway's pod and one of yours. If a sample shows no pods at all, the read timed out. That is not a DNS outage.

4. Check the zones after the rollout. Run kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide and compare the result with the node zones from check 2. Your two pods must be in different zones.

5. Test with real lookups. From a pod in the cluster, run getent hosts <a name you resolve> in a loop every 0.5 seconds. While the loop runs, delete or drain one DNS pod. The loop should show no failures. Before the fix, deleting the single pod caused one failed lookup.

Our results at Heystaq on 28 October 2026: on all three clusters, the rollout took about one minute, there were always at least two Ready endpoints, and one pod was Pending for one second.

Check the spread after every upgrade

The scheduler only checks these rules when it places a pod. It never moves a pod that is already running. So the spread is correct after each rollout, but it can change during drains and node replacements, until the next rollout. The Kubernetes documentation says the same, and recommends the Descheduler to rebalance pods.

This gives you a short maintenance list:

• After a Kapsule upgrade, check Scaleway's image again (check 3). If it changed, update your overlay, so that both Deployments run the same CoreDNS version. The Corefile needs no change, because both Deployments use the same one.
• After a node pool upgrade or node roll, check the zones of the DNS pods. If both of your pods ended up in one zone, spread them again by hand with kubectl -n kube-system rollout restart deploy/coredns-ha. Do not add an annotation in git to force a rollout, because then your manifest contains a trigger that exists only for rebalancing.
• Drains are safe. The disruption budget lets only one of your pods go down at a time, and the Kapsule upgrade waits for it. Because the hard zone rule ignores cordoned nodes, it can never block an upgrade forever. Note that a disruption budget only covers voluntary disruptions such as drains, not a node that fails (Kubernetes documentation).
• Monitoring works without changes. If you scrape CoreDNS with a ServiceMonitor that selects k8s-app=kube-dns on port metrics, it also finds your pods.
• The overlapping selectors are not a problem. Scaleway's Deployment selector also matches your pods. But Deployments adopt pods through ReplicaSet owner references and the pod-template-hash label, so neither Deployment takes the other's pods.

What we have not proven yet, and mistakes to avoid

What we have not proven yet:

• A real drain with zero failed lookups. At Heystaq we have tested rollouts and pod deletions, but not yet a drain of a node that runs a DNS pod. Deleting one pod is not the same as losing a node. Before the fix, DNS was down for 1 second when we deleted the pod, but for 7 seconds when a node was replaced. So test with a real drain.
• What Scaleway thinks of this setup. The documentation does not say if adding a Deployment to kube-system counts as changing a managed add-on. The shared responsibility model asks you to leave kube-system unchanged. We have not opened a support ticket yet, so ask Scaleway support before you depend on this setup. If Scaleway offers a supported way to scale its CoreDNS, use that instead.
• Automatic rebalancing. We plan to use the Descheduler (RemovePodsViolatingTopologySpreadConstraint, evicting pods through the disruption budget) instead of rebalancing by hand. We have not built this yet.

Mistakes to avoid:

• Changing the replica count of Scaleway's coredns Deployment by hand. It looks like a fix, but the addon manager reverts it within minutes.
• Copying the pod spec without a pod securityContext. Trivy then reports KSV-0118.
• Relying on ScheduleAnyway alone. It is a score, not a guarantee, and both pods can end up on the same node.

Conclusion

On Kapsule, "managed" means that Scaleway decides what CoreDNS looks like, including the single pod. When you change it, Scaleway changes it back. So instead of changing Scaleway's objects, work with them. Scaleway's Service already selects pods by label. A Deployment of your own with that label runs DNS on more than one node and in more than one zone, and you do not change a single object that Scaleway owns.

One important test is still missing: the drain test. This is the test that matters most, because the old outage was 7 seconds when a node was replaced, and only 1 second when a pod was deleted. And if Scaleway publishes a supported way to scale its CoreDNS, use that instead.

If you do not use Kapsule, the most useful lesson here is the dry run. Our two hard zone rules passed kustomize build, our tests and our review, and only the API server rejected them. If your pipeline does not check your manifests against the API server before you merge, you only find these errors after the merge.