Skip to content

ADR-0004: RKE2 API endpoint and routing to Ceph

Date: 2026-08-29 Status: Proposed — API endpoint in force; the Ceph routing decision below is superseded by ADR-0007 (operator decision 2026-08-30: storage-native attachment, no routing, no natgw involvement). The API endpoint design (keepalived VIP 10.72.0.20) is unaffected and remains in force.

Context

RKE2 needs one stable control-plane endpoint, and the infrastructure README gates RKE2 installation on that endpoint and on routing to Ceph being accepted. Joining servers and agents register against port 9345 (server: https://<endpoint>:9345); kube-apiserver serves 6443. A VIP is not automatically included in the apiserver serving certificate, so it must be added with tls-san. The SAN list is fixed at bootstrap; adding a SAN later forces per-node certificate rotation. After a node joins, its rke2 agent load-balances locally (port 6444) across the apiserver endpoints discovered from etcd, so the configured server: URL matters only for initial or full re-registration.

No DNS is controlled anywhere in the platform (VMs use public resolvers), so bootstrap-safe endpoints are limited to an IP URL or /etc/hosts entries. The node LAN address plan already reserves 10.72.0.20 as the stable RKE2 API VIP. Nothing routes node-LAN traffic to Ceph 192.168.96.0/20 today: the gateway has no VLAN 2 interface, no static route, and its nftables forward policy drops everything except LAN-to-WAN. Candidate endpoints:

  • Gateway VIP 10.72.0.1 on wetopi-natgw01: rejected. It couples API availability to a 2 GiB egress gateway VM, whose loss would take the API and node egress down together, contradicts the reserved 10.72.0.20, and would flap the API with future gateway VRRP changes.
  • Round-robin DNS over the control-plane IPs: needs DNS control that does not exist; the /etc/hosts variant is unreliable (glibc typically returns only the first matching entry) and must be kept in sync on eight nodes.
  • TCP load balancer on wetopi-ops01: workable without Proxmox firewall changes, but makes the deliberately minimal recovery bastion the API single point of failure and adds a long-lived service to it.
  • Direct server URL to 10.72.0.21 with documented manual failover: zero extra SANs and the lowest operational burden, but fragile exactly when re-registration coincides with a downed control plane; upstream warns node IPs may change over time.
  • Dedicated API VIP 10.72.0.20 on the three control planes via keepalived: already reserved in the address plan, lives inside the cluster failure domain, adds no VM or new single point of failure, availability tracks local apiserver readiness including etcd connectivity, and costs one new Ansible role.

The design set carries a standing proposal for this endpoint: "2 x HAProxy + Keepalived, private VIP" (master architecture plan, section 16 "Endpoint HA de la API RKE2"; decisions log, section 29.2). Its status is OPEN, to be closed before the definitive cluster install (action NOW-05); it is described as conservative and compatible with existing operational knowledge, but not yet approved as a final decision.

Decision

  • Adopt the dedicated API VIP: a keepalived VRRP group on VLAN 4 across rke-cp01/02/03 owns 10.72.0.20.
  • This supersedes the standing proposal and closes it: the CP-owned keepalived VIP, with no separate HAProxy boxes, is the endpoint design that satisfies NOW-05. Rationale: the HAProxy pair would add two new VMs and roles for a capability the VIP alone already provides at this scale; the three control planes terminate 9345 and 6443 directly, and both ports are served by the same process set, so a TCP load balancer in front adds a hop and a failure domain without adding correctness; keeping the VIP on the control planes keeps the endpoint inside the cluster failure domain, in line with the v1 constraint of simplicity over sophistication (no unnecessary new infrastructure roles). Trade-off acknowledged: without a separate LB layer there is no place for per-backend health-aware balancing beyond VRRP failover granularity; accepted for v1 at this scale.
  • rke-cp01 is MASTER by priority, failing over to rke-cp02, then rke-cp03. VRID and priority values are implementation parameters; no other VRRP is documented on VLAN 4.
  • Holding the VIP is gated on local health: a control plane keeps the VIP only while its local apiserver is ready (/readyz, which includes etcd-backed readiness and fails when the local apiserver cannot reach etcd as part of a quorum) and its local registration service on 9345 is reachable. Exact check commands, scripts, and timings are implementation parameters.
  • For 6443 (kubectl and kubeconfig clients) and 9345 (new node registration) alike, this places the VIP only on a control plane whose apiserver reports ready including etcd connectivity: a strictly stronger signal than a TCP listen check on 6443, which proves neither etcd-backed readiness nor a live 9345.
  • It is not a global quorum oracle. With one control plane down, the remaining owners are etcd-ready and keep serving both ports. With two down, the last control plane's /readyz fails, it releases the VIP, and the endpoint is down, which is correct: a usable API requires etcd quorum (at least two control planes). The check does not create quorum; it avoids pointing the endpoint at an apiserver that already reports not ready.
  • Agents and joining servers use server: https://10.72.0.20:9345; kubeconfig and kubectl clients use https://10.72.0.20:6443. IP-only, no hostname SAN: no DNS is controlled, and this avoids a split-horizon hazard.
  • Set tls-san identically on all three servers to the VIP plus all control-plane IPs: 10.72.0.20, 10.72.0.21, 10.72.0.22, 10.72.0.23, so any control-plane IP works for direct registration or clients.
  • Bootstrap order: because every control-plane IP is in the SAN list, initial bring-up may register directly against 10.72.0.21:9345 before keepalived is deployed, then switch server: URLs to the VIP; no certificate rotation is needed either way.
  • ~~Route, do not NAT, node-LAN traffic to Ceph 192.168.96.0/20 (per ADR-0001), using gateway-side routing on wetopi-natgw01.~~ SUPERSEDED by ADR-0007 (operator decision 2026-08-30): the Ceph clients (the 3 platform workers) attach directly to VLAN 2 / vmbr2 with addresses from the reserved 192.168.98.0/25 — no routing, no NAT, no return-routes, no natgw involvement. The bullet below is retained as the historical record of the superseded design; its shape never left the default-off state. Shape (superseded):
  • give the gateway a third interface on VLAN 2 / vmbr2 with an address in 192.168.96.0/20 (netplan; the OpenTofu VM module already supports extra NICs), making the Ceph subnet a directly connected route with no next-hop;
  • add no routes on the node LAN: the default gateway 10.72.0.1 already covers the destination once the gateway owns the connected route;
  • Ceph-side return path: hosts in 192.168.96.0/20 need a route for 10.72.0.0/23 via the gateway's VLAN 2 address, shaped as ip route 10.72.0.0/23 via <natgw VLAN 2 address>;
  • add nftables accept rules in the gateway forward chain for iifname <lan_if> ip daddr 192.168.96.0/20 and the return direction oifname <lan_if> ip saddr 192.168.96.0/20.
  • Keep MTU at 1500 per ADR-0001, to be confirmed against the live VLAN 2 MTU (a pre-implementation gate). MTU 1500 is carried over unchanged by ADR-0007.

Failure modes

Effects are stated for each scenario on: API reachability (6443 via the VIP for kubectl and new kubeconfigs), new node registration (9345 join), already-joined agents (client-side LB on 6444 across surviving control planes), Internet egress (nodes' default gateway 10.72.0.1 on wetopi-natgw01), and Ceph connectivity (since ADR-0007: the workers' direct VLAN 2 attachment — storage no longer depends on the gateway at all; the pre-ADR-0007 routing remarks below are retained as the historical record).

  • VIP-owner CP death (rke-cp01 dies). VRRP fails the VIP over to rke-cp02 within the failover time: a brief interruption of API reachability and of new node registration, then both served by the new owner. Etcd keeps quorum (2 of 3). Already-joined agents are unaffected: their 6444 LB already spans the surviving control planes. Internet egress and Ceph routing are unaffected.
  • Non-owner CP death (rke-cp03 dies while not holding the VIP). No VIP movement. API reachability and new node registration are unaffected; etcd keeps quorum (2 of 3). Already-joined agents re-balance around the dead control plane on 6444. Internet egress and Ceph routing are unaffected.
  • Endpoint service degradation (apiserver process up, etcd degraded or quorum lost). With the /readyz-gated check, a degraded owner does not hold the VIP: if etcd degrades on the owner only, the VIP moves to a healthy owner and API reachability and registration continue. If quorum is lost cluster-wide, no control plane passes the check, none holds the VIP, and the API is intentionally unavailable regardless of VIP placement: a usable API requires etcd quorum, and the endpoint must not present an API that cannot serve. Already-joined agents see failing API requests for as long as quorum is lost (an etcd failure, not an endpoint failure) and resume when quorum returns. Internet egress and Ceph routing are unaffected.
  • wetopi-natgw01 death. The VIP lives on the node LAN, so API reachability and new node registration are unaffected (this is exactly why the gateway VIP was rejected), and already-joined agents are unaffected. Internet egress is down for all nodes (default gateway 10.72.0.1 lost). ~~Ceph routing is down (the sole proposed route rides the gateway's VLAN 2 leg).~~ Since ADR-0007, Ceph connectivity is UNAFFECTED by a natgw death: the workers reach Ceph directly on VLAN 2. (Historical record: under the superseded routing design the gateway would have been a single point of failure for both Internet egress and Ceph routing; gateway redundancy stays out of scope either way, see Consequences.)

Consequences

  • VIP loss blocks new node registrations and new kubeconfig clients; already-joined agents keep working through their client-side load balancer (port 6444) against the surviving control planes, and running workloads are unaffected.
  • Rollback is cheap: revert server: URLs to a control-plane IP (for example https://10.72.0.21:9345) and uninstall keepalived. No certificate rotation is needed because all control-plane IPs are already in the SAN list.
  • The stable endpoint survives control-plane renumbering or reprovisioning, kubeconfigs never change, and accepting this ADR satisfies the infrastructure README gate, letting RKE2 installation proceed.
  • VRRP is a new operational surface on the node LAN (VRID collision care), and keepalived on the control planes adds guest configuration that must stay Ansible-owned per ADR-0003.
  • A later move to a DNS-name endpoint would need a SAN addition plus certificate rotation.
  • ~~Pre-implementation gates for the Ceph route~~ — closed by ADR-0007 (the route itself is superseded; the address allocation is now the operator-reserved 192.168.98.0/25, and no return-route owner, gateway firewall path or rp_filter verification is needed — there is no gateway on the path. The live VLAN 2 MTU check (1500, ADR-0001) and the VPN CIDR overlap validation survive as apply-time gates in infrastructure/kubernetes/ceph-csi/VALIDATION.md). Historical record of the superseded gates:
  • VLAN 2 address allocation: the exact address for the gateway's third NIC in 192.168.96.0/20, confirmed against live Ceph allocations;
  • Ceph-side return-route owner: which L3 device or hosts on the Ceph side own the return route for 10.72.0.0/23 via the gateway's VLAN 2 address;
  • firewall path: the nftables accept rules needed in both directions, including any Ceph-side firewall;
  • rp_filter behavior: the gateway enforces rp_filter=1, and asymmetric replies could be dropped; must be field-verified;
  • live MTU on VLAN 2: actual MTU and jumbo-frame support verified against the 1500 kept per ADR-0001;
  • VPN CIDR overlap validation for 10.64.0.0/14 and 10.68.0.0/16.
  • Out of scope, to be decided separately: gateway keepalived redundancy for 10.72.0.1, any DNS infrastructure, MTU changes, and storage design.