Skip to content

ADR-0005: Cilium as the CNI from day one (no Canal provisional CNI)

Date: 2026-08-29 Status: Decided by operator; implementation pending

Context

RKE2 installs a bundled Canal (Flannel VXLAN + Calico policy) CNI by default. PLATFORM-007 already names Cilium as the CNI and PLATFORM-008 names Hubble as the network-observability base. This decision is recorded in the decisions log as PLATFORM-018 (the next free identifier; PLATFORM-008 was already assigned to Hubble), and the master architecture plan (section 14.4) fixes the first-version baseline as Cilium with VXLAN encapsulation and Hubble enabled, explicitly without BGP or native routing (PLATFORM-017). A provisional Canal bootstrap — installing Canal first and replacing it with Cilium later — would force every node through a CNI migration (restart of all pods, datapath teardown, stale flannel state) and build NetworkPolicy experience on a stack the platform never intends to keep. The official Cilium RKE2 documentation offers exactly two paths: the bundled rke2-cilium chart or RKE2 with cni: none plus the official Cilium Helm chart. The cluster does not exist yet, so the CNI choice costs nothing today and an extra migration later. Pod CIDR 10.64.0.0/14 and Service CIDR 10.68.0.0/16 are set at the RKE2 level; Cilium must not re-derive them. Finally, Argo CD does not exist yet at first bootstrap, so the first cluster addon cannot be managed by GitOps from minute one.

Decision

  • Cilium is the CNI from the first RKE2 node. No Canal provisional CNI: the rke2-server configuration must set cni: none, and the future RKE2 Ansible role must keep the bundled Canal disabled.
  • Install the official Cilium chart from oci://quay.io/cilium/charts/cilium, pinned to chart version 1.20.1 (chart versions carry no v prefix; appVersion v1.20.1) into kube-system, not the rke2-cilium addon chart, so the Helm release and values stay under this repository's control.
  • Baseline values (master plan 14.4): routingMode: tunnel with tunnelProtocol: vxlan; IPAM cluster-pool with clusterPoolIPv4PodCIDRList: [10.64.0.0/14] and clusterPoolIPv4MaskSize: 24, exactly mirroring the RKE2 pod CIDR; kubeProxyReplacement: false; hubble.enabled: true with hubble.relay.enabled: true and hubble.ui.enabled: false (documented flip-on option). No BGP, no native routing, no egress features initially.
  • CIDR ownership stays with RKE2: RKE2 owns the pod CIDR 10.64.0.0/14 and the Service CIDR 10.68.0.0/16 (the latter stays owned by kube-proxy); the Cilium IPAM pool only consumes the pod range and must never diverge from the rke2-server configuration.
  • Keep the Cilium MTU at its default (auto-computed, about 1450 on the MTU-1500 node LAN 10.72.0.0/23, VLAN 4, single NIC eth0 per node); no explicit MTU unless a measured reason appears.
  • Node-to-node firewall paths must allow VXLAN UDP 8472 and Hubble Relay TCP 4244.
  • Bootstrap-time install from wetopi-ops01: the addon is installed with Helm during the apply phase (ansible/playbooks/cilium.yml in wetopi-platform/infrastructure, with pinned values in kubernetes/cilium/values.yaml). This is a bootstrap-time install only; once Argo CD exists, the addon moves under Argo CD management and the playbook is retired.
  • Enabling kube-proxy replacement, Hubble UI by default, BGP or native routing are explicitly deferred; each needs its own reviewed decision.

Consequences

  • No CNI migration is ever needed: the cluster is born with its final CNI.
  • The egress path depends on RKE2's kube-proxy staying installed and healthy; replacing kube-proxy later is a deliberate, separate change.
  • VXLAN tunnels ride the node LAN 10.72.0.0/23, so UDP 8472 (and TCP 4244 for Hubble Relay) must stay reachable node-to-node on VLAN 4.
  • The Cilium IPAM pool must always mirror the RKE2 pod CIDR; divergent clusterPoolIPv4PodCIDRList values are a standing misconfiguration risk to guard in review.
  • Until Argo CD exists, the addon is reconciled by hand (the pinned one-liner in kubernetes/cilium/README.md), so drift between Git values and the installed release is possible in the interim window.
  • The apply-phase validation checklist (Cilium status and connectivity, Hubble flows, DNS, default-deny baseline) lives in kubernetes/cilium/VALIDATION.md in the infrastructure repository.

References

  • Master architecture plan, sections 14.3 (CIDRs) and 14.4 (Cilium baseline).
  • Decisions log, section 32 (PLATFORM-018 — CNI Cilium desde el día uno).
  • Cilium v1.20.1 Helm chart values and the official Cilium installation notes for RKE2.