Skip to content

Cilium network — CNI, Hubble and edge-node datapath

Purpose

The pod/service network of the RKE2 cluster: who assigns pod IPs, how nodes talk across the LAN, and how the dual-NIC edge workers keep their cluster datapath strictly on the LAN. Decision record: ADR-0005 (accepted) — Cilium is the CNI from day one; there was never a Canal provisional phase.

Design

RKE2 boots with cni: none so the bundled Canal manifest never deploys; the official Cilium chart is installed as the first addon and owns the datapath.

  • Chart: pinned cilium:1.20.1 from oci://quay.io/cilium/charts (values in Git: kubernetes/cilium/values.yaml in the infrastructure repo).
  • Mode: VXLAN encapsulation (routingMode: tunnel, tunnelProtocol: vxlan); no BGP, no native routing.
  • IPAM: cluster pool mirroring the RKE2 pod CIDR 10.64.0.0/14 exactly (clusterPoolIPv4PodCIDRList), /24 allocated per node.
  • kube-proxy coexistence: kubeProxyReplacement: false for v1 — the service CIDR 10.68.0.0/16 stays owned by kube-proxy.
  • Hubble: enabled with Hubble Relay; relay UI off at bootstrap.

CIDR ownership (must never diverge from the rke2-server config):

Range Owner Cilium role
10.64.0.0/14 RKE2 (pod CIDR) IPAM cluster pool mirrors it
10.68.0.0/16 RKE2 / kube-proxy (service CIDR) none

Underlay and ports

Tunnels ride the node LAN (10.72.0.0/23, single-NIC nodes, MTU 1500). The Cilium MTU is left at its auto-computed default (~1450, 50-byte VXLAN overhead). Node-to-node matrix that must stay allowed on the LAN:

VXLAN tunnels:  UDP 8472
Hubble Relay:   TCP 4244
kubelet:        TCP 10250

The edge workers carry these ports on their LAN interface only (edge_lb role asserts they never appear on the WAN interface).

Edge-node datapath pin (WAN Edge v1)

The chart leaves devices unset, so the Cilium agent auto-detects its interface. On the single-NIC nodes that picks eth0 (the LAN) — correct. On the dual-NIC edge workers eth0 is the WAN; auto-detection would attach the datapath to it. The fix is a namespaced CiliumNodeConfig (kubernetes/cilium/edge-node-config.yaml, applied after the chart install by the bootstrap playbook and reconciled by the Cilium Argo Application):

spec:
  nodeSelector:
    matchLabels:
      wetopi.io/role: edge
  defaults:
    devices: eth1    # eth1 = LAN NIC on edge nodes (WAN Edge v1 plan §F1)

It selects only wetopi.io/role=edge nodes; the six other nodes keep the auto-detect default.

GitOps ownership

The Cilium release is declared in Git as Argo CD Application cilium (kubernetes/cilium/application.yaml): Helm source (pinned 1.20.1 + Git values) plus a directory source carrying exactly edge-node-config.yaml, so the release and its edge override share one ownership/pruning policy. Prune=false and ignoreDifferences protect the chart-generated TLS secrets (cilium-ca, hubble-relay-client-certs, hubble-server-certs) whose live .data must never be replaced by Helm material. Like argo-edge, the Application itself still follows the operator-bootstrap path (kubectl apply from ops01) — it is not yet anchored in the root app's path (see gitops.md).

Invariants

  • Pod IPs only from 10.64.0.0/14; the Cilium pool must always mirror the rke2-server cluster-cidr.
  • Edge-node datapath is pinned to eth1 (LAN); a CiliumNodeConfig change or label drift that moves it to eth0 is an incident (cluster traffic on the WAN — ADR-0006 violation).
  • VXLAN/Hubble/kubelet ports LAN-only on edge nodes (asserted by the edge_lb role).
  • No BGP/native routing in v1.

Verified state (2026-08-31, live from ops01)

Cilium 1.20.1 pods Running in kube-system on all nodes (DaemonSet + operator), Hubble Relay up, one CiliumNode per node, the wetopi-edge-devices CiliumNodeConfig present and matching the edge label. Dated observation — re-read kubectl -n kube-system get pods,daemonset,ciliumnode before relying on it.

Dependencies

  • Depends on: the RKE2 node set and CIDRs (rke2-cluster.md), the node-LAN underlay, the edge label contract (edge.md).
  • Depends on this: the edge plane (Traefik ingress scheduling tolerates the node.cilium.io/agent-not-ready taint), GitOps handoff.

Failure considerations

  • Cilium down on a node: that node's pod traffic is down; with cni: none below it, the node is NotReady — no fallback CNI exists (by design).
  • VXLAN MTU surprises: symptom is black-holed large packets; the auto MTU keeps headroom, do not pin it manually.
  • Edge datapath mis-attachment: detect via cilium status/connectivity checks per VALIDATION.md in the infrastructure repo.

Live inventory (verified 2026-09-06, BookStack mirror — T1.6)

  • v1.20.1 (chart oci://quay.io/cilium/charts), tunnel vxlan, IPAM cluster-pool 10.64.0.0/14/24; kubeProxyReplacement: false (kube-proxy convive); Hubble + Relay ON, UI OFF.
  • UDP 8472 (VXLAN) + TCP 4244 (Hubble Relay) entre nodos.
  • Gotcha verificado: netpol con ipBlock 0.0.0.0/0 en ns nuevo bloquea SYN de pods remotos (drop silencioso pre-veth) — patrón casa: SIN netpols para apps edge (ver REC-20260906-KDOCPANE).
  • Traefik edge corre hostNetwork: su tráfico aparece como identity host (no kube-system pod).

Repo: infrastructure/kubernetes/cilium.