Skip to content

ADR-0007: Ceph client access via direct storage-network attachment

Date: 2026-08-30 Status: Accepted (operator decision recorded 2026-08-30 — the 192.168.98.0/25 allocation, .10/.11/.12 = the 3 platform workers — on build card t_7415a1d6; accepted status set after the round-1 review cycle of infrastructure PR #13 / docs PR #12, verdict t_4ed11dad)

Context

ADR-0004 bundled a second decision with the RKE2 API endpoint: node-LAN traffic to the Ceph cluster (192.168.96.0/20, VLAN 2 / vmbr2) would be routed via the NAT gateway — wetopi-natgw01 gaining a third interface on VLAN 2 so the Ceph subnet became a directly connected route on the gateway, with a Ceph-side return route for 10.72.0.0/23 and forward-chain firewall rules. That design was accepted as design only, gated behind pre-implementation gates (address allocation, return-route owner, firewall path, rp_filter, MTU, CIDR overlap), and shipped default-off in the nat_gateway Ansible role.

Field reality (2026-08-30) invalidated the routing hypothesis:

  • No path exists today: from a worker, 192.168.96.10 routes via the LAN default gateway 10.72.0.1 with 100% loss — the natgw has no VLAN 2 leg, no static route, and a forward policy that drops everything except LAN-to-WAN.
  • The LAN gateway is not ours to depend on for storage: routing via 10.72.0.1 means every Ceph I/O depends on a non-managed default-gateway hop (plus, in the ADR-0004 shape, on a Ceph-side return route nobody owns).
  • The dedicated network is already there: vmbr2 / VLAN 2 is present on the Proxmox hosts — the Ceph storage network is a directly connected L2 domain one NIC away from every VM.

The operator decided (Carlos, 2026-08-30): "storage on the native network; routing into Ceph is unnecessary overhead."

Decision

The 3 platform workers (rke-platform01/02/03 — the PVC-carrying nodes) get a third NIC on VLAN 2 / vmbr2 with addresses in 192.168.96.0/20, attaching them natively to the Ceph storage network — the standard Ceph client pattern:

  • Operator allocation (2026-08-30): the sub-CIDR 192.168.98.0/25 is reserved for the platform's storage consumers — rke-platform01=192.168.98.10, rke-platform02=192.168.98.11, rke-platform03=192.168.98.12 (all with the /20 mask per the connected subnet); the rest of the /25 stays free for future storage consumers.
  • No routing, no NAT, no return-routes, no natgw involvement: the /20 connected route appears automatically (scope link) when the address is assigned; the storage NIC carries no gateway and no routes. The default route stays exclusively 10.72.0.1 on the LAN NIC (unchanged). The ADR-0001 no-NAT invariant is trivially preserved (there is no intermediary to NAT).
  • Control planes stay single-NIC in v1: CPs are tainted (node-role.kubernetes.io/control-plane:NoSchedule) and carry no PVC workloads; their storage-NIC shape is documented (commented) in the tfvars for future need.
  • Firewall: the workers' storage interface admits only the Ceph client ports from 192.168.96.0/20 — mons TCP 3300 (msgr2) + 6789 (msgr1), OSDs TCP 6800-7300 — inbound plus established/related (ceph_storage Ansible role, default-off until the reviewed apply). LAN semantics stay permissive (identical to the pre-change no-firewall state); the workers carry no WAN surface.
  • MTU stays 1500 (ADR-0001), verified against the live VLAN 2 MTU as an apply-time gate.
  • This ADR supersedes the Ceph routing decision section of ADR-0004 (its last Decision bullet and the corresponding failure-mode/consequence text). ADR-0004's API-endpoint decision (keepalived VIP 10.72.0.20) is UNAFFECTED and remains in force.

Consequences

  • 3 addresses consumed in the storage LAN (.10-.12 from the reserved /25); the allocation is now explicit and recorded, replacing ADR-0004's open "VLAN 2 address allocation" gate.
  • The apply is MUTATING on live workers: adding a NIC requires a serial drain/reboot per node (the Phase W pattern, already proven). The ops sequence (plan gate → per-worker drain/apply/reboot/verify/uncordon → firewall enable) is documented in infrastructure/kubernetes/ceph-csi/VALIDATION.md §2.
  • Storage traffic is isolated from the management LAN: Ceph I/O never touches vmbr4 or the natgw; the gateway's failure can no longer affect storage at all (ADR-0004 had accepted the natgw as a Ceph-routing SPOF).
  • The ADR-0004 routing artifacts are retired in the infrastructure repo: the natgw_ceph_* enablement is reduced to a permanent default-false gate + cleanup task; the natgw carries no VLAN 2 leg, ever.
  • Prerequisites for Ceph CSI shrink to: address allocation (done), Ceph pool + dedicated user (client.kubernetes-csi, never admin), fsid/mons read-only, csi-rbd-secret.
  • Worker VMs become dual-homed; operational care points (documented in the runbook): the storage NIC must never acquire a default route (asserted), and node identity stays LAN-only.

Alternatives considered

  • Gateway-side routing (ADR-0004 original): rejected — indirection with no benefit, a natgw SPOF on the storage path, an unowned Ceph-side return route, and 100% loss today because no path exists. The pre-implementation gates could never be satisfied cheaply.
  • Routing via the LAN gateway without a natgw VLAN 2 leg: rejected — depends on a non-managed gateway that currently drops the path (100% loss); still requires Ceph-side return routes; storage I/O coupled to egress infrastructure.
  • Moving Ceph storage traffic to the management LAN (10.72.0.0/23): rejected — worse isolation (storage and management share fate and bandwidth) and touches the untouched-Wetopi-Ceph-LAN principle in the wrong direction.
  • NAT to Ceph: never considered — violates the ADR-0001 no-NAT invariant.
  • ADR-0001 (node LAN, no-NAT invariant, Ceph LAN untouched)
  • ADR-0004 (API endpoint — unaffected; Ceph routing section superseded by this ADR)
  • ADR-0003 (tofu owns VM/NIC shape, Ansible owns guest config — both applied here)
  • infrastructure PR feat/ceph-storage-native (the build: tfvars, ceph_storage role, ADR-0004 artifact retirement, README/VALIDATION rewrite)