RKE2 apply runbook — phases 0-G¶

This is the single reviewed procedure a human follows to bring the RKE2 cluster up, with validation gates between phases. Every phase ends in an explicit STOP gate: do not proceed past a failing gate. Each gate result is recorded in the apply log with a date.

Status (2026-08-30, post-Phase-D update): the live apply has RESUMED and executed through Phase D. Executed 2026-08-29/30, fully green: B (cp01 bootstrap, infra PR #6 fix d553643), C (cp02+cp03 quorum, PR #7), F6-1..3 (CP-only acceptance: quorum recorded, one-CP failure drill via VIP failover, etcd snapshot + real restore), E (Cilium 1.20.1 + Hubble, PR #8), and D (all 3 platform workers joined via rke2-workers.yml at infra PR #9 f841ae7 — serial gates green, per-worker kubectl wait + exact-label asserts passed incl. the /var/lib/rancher/rke2/agent/kubelet.kubeconfig data-dir gate; live-verified on rke-platform01). Cluster state: 6/6 nodes Ready, CP taints enforced (node-role.kubernetes.io/control-plane:NoSchedule), workers labelled wetopi.io/role=platform, kube-system pods Running, in-cluster DNS resolving (10.68.0.0/16), scheduling test landed on a platform worker, CPs workload-free (rke2-workers-validate.yml rack green — details its runbook section describes for D2 remain next). Next in executable order: F (Ceph, blocked on operator-held facts, t_80ca13af); G (Argo CD) has since APPLIED with recorded deviations (see the post-Phase-G update below and the Phase G section). Earlier halt notice (retained for history): the live apply (task t_a88bb36b) was HALTED BEFORE PHASE B and FROZEN per operator directive #38 — the 2026-08-29 23:17-23:26 UTC attempt failed at rke2-server : Create the RKE2 server configuration with Destination directory /etc/rancher/rke2 does not exist on rke-cp01 (neither rke2 role created the config dir); fix was infrastructure PR #6 (merged main d553643). cp01 carried keepalived (active+enabled) + RKE2 binaries from that pre-halt attempt — roles proved idempotent (RKE2 install guarded by creates:) when B/C re-ran clean on 2026-08-30 (tasks t_b0eebfac, t_58bd66bb, t_f7099d9d, t_b3841920). Verified against merged infrastructure main @ f841ae7 (PRs #1-#9) and docs main @ 7cc387a.

Phase order (the authorized executable order — stated IDENTICALLY in this banner, the F6 heading + STOP rule, the Phase W heading, the D2 handoff, the DoD mapping, and the provenance section): B → C → STOP → F6-1..3 → E → D → D2 → W → F → G. Concretely: A (VIP) is folded into the B/C gates (no standalone A apply) → B (cp01 bootstrap + VIP bring-up) → C (quorum cp02+cp03 incl. VIP failover) → STOP (mandatory pause after the Phase C quorum gate: before any F6-2/F6-3 server stop, every F6 STOP prerequisite below must be PASS and dated in the apply log; historically this was the frozen-apply boundary of the 2026-08-29 halt — that freeze is superseded history, see the retained halt notice in the status banner above) → Fase-6 acceptance, CP-only portion: F6-1 (quorum recorded), F6-2 (one-CP failure drill), F6-3 (etcd snapshot + REAL restore test) — these three execute on the CP-only cluster (3 nodes, no workers) BEFORE any addon, per master plan §33 (Fase 6 before Fase 7), and are gated by the hard F6 STOP prerequisites (external verified snapshot copy, SHA-256 match, quorum evidence, explicit human authorization) → E (Cilium install + basic install validation ONLY) → D (workers join, registration + exact-label gates only) → D2 (full post-Cilium cluster validation on the FINAL 6-node topology, incl. the full Cilium acceptance) → W = Phase W — the worker lifecycle drill (drain + reboot; separately named; former F6-4/F6-5) — worker drain/reboot happens HERE, after D2, NEVER before addons → F (Ceph, PR-only, default-off) → G (Argo CD). Known tension with master plan §33, resolved EXPLICITLY: §33 orders Fase 5 (workers) and Fase 6 (which includes node drain/reboot) BEFORE Fase 7 (Cilium), but ADR-0005 requires Cilium as the IMMEDIATE first addon with NO pre-Cilium worker-Ready/DNS gates — with cni: none, workers can never pass Ready/drain/reboot gates before Cilium exists. Resolution: the CP-only gates of Fase 6 (quorum, CP failure, snapshot + real restore) run before the first addon, and the worker-lifecycle drill is the separately-named Phase W landing after D2 on the final 6-node topology; the DoD maps drain/reboot to that POST-D2 evidence, never to pre-addon acceptance. Do NOT count Phase W as completed pre-addon acceptance. With cni: none at the RKE2 level, no node can be Ready before Cilium exists: health before E = /readyz + etcd member counts ONLY; nodes NotReady is EXPECTED before E and there are NO worker-Ready/CoreDNS/DNS expectations before Cilium. The FULL Cilium acceptance (connectivity test, Hubble flows, DNS, NetworkPolicy, CiliumNode per node) runs in D2 on the final 6-node topology, not in E.

0. Preconditions (before Phase B)¶

Work through this checklist in order. Each item has its concrete verification.

ADRs accepted — docs/decisions/ADR-0004-rke2-api-endpoint.md and docs/decisions/ADR-0005-cilium-first-addon.md merged in docs main (merge commit 5740404).
Implementation PRs merged to infrastructure main — verified at d553643 (current main, 2026-08-30):
PR #1 (Argo CD bootstrap, merged as 8f3e0e8)
PR #2 (feature/cilium-hubble-addon: ansible/playbooks/cilium.yml, kubernetes/cilium/* — merged as e283e24)
PR #3 (Ceph routing + CSI, merged as 76782dc)
PR #4 (feature/rke2-server-role: rke2-server + keepalived roles, ansible/playbooks/rke2-cp.yml, scripts/gen-rke2-token.sh — merged as b15a087)
PR #5 (feature/rke2-agent-role: rke2-agent role, ansible/playbooks/rke2-workers.yml, ansible/playbooks/rke2-workers-validate.yml — merged as 8d836e8)
PR #6 (fix/rke2-config-dir-create, head 3ff2ee8 — merged as d553643): creates /etc/rancher/rke2 (root:root 0750) before templating in BOTH rke2 roles. This is the fix for the config-directory defect that halted the 2026-08-29 apply attempt (Destination directory /etc/rancher/rke2 does not exist at rke2-server : Create the RKE2 server configuration; byte-identical defect in rke2-agent). Without it, phases B and D fail at config templating.

STEP 0 gate: git -C infrastructure log --oneline -1 on the checkout used for apply must show d553643 or a verified descendant — 8d836e8 (pre-PR#6) is NO LONGER sufficient: resuming phases B/C against a checkout without the config-dir fix reproduces the halt. Verify the fix is present: git -C infrastructure show d553643 --stat mentions both rke2-server and rke2-agent tasks. git grep secrets-encryption still returns nothing on d553643 (the explicit hardening line is still absent — see item 8 for the current measured-evidence gate rule). 3. Branch protection — main is the reviewed line; all changes land via PR with wetopi-approver review; never commit directly to main. 4. Token — on the controller (the infrastructure checkout that will run the playbooks — the evidenced controller is the WORKSTATION, not ops01; see item 9), run scripts/gen-rke2-token.sh (umask 077, openssl rand -hex 16, 32-char hex). It writes ansible/files/rke2/node-token, mode 0600, gitignored. Verify: stat -c '%a' ansible/files/rke2/node-token → 600. NEVER commit it. Both rke2-cp.yml and rke2-workers.yml pre-flight plays hard-fail if the file is missing or not 0600. 5. Inventory — ./scripts/render-ansible-inventory.sh regenerates ansible/inventories/prod/hosts.json from the tofu ansible_inventory output (mode 600, gitignored); the generated hostvars set ansible_ssh_common_args = "-o ProxyJump=ubuntu@92.222.245.91" for every non-bastion VM — that ProxyJump form is the inventory's internal contract consumed by Ansible. Bastion is wetopi-ops01 = 92.222.245.91 (WAN) / 10.72.0.10, SSH account ubuntu (bastion SSH policy allows only ubuntu; the workstation keeps all private keys; ops01 is a ProxyJump target only). Human reachability check from the workstation uses the documented double-key form: ssh -i /opt/data/.ssh/bigardo -o ProxyCommand="ssh -i /opt/data/.ssh/bigardo -W %h:%p ubuntu@92.222.245.91" <user>@<host>, or plain ping 10.72.0.21 from ops01. 6. Proxmox snapshots (rollback plane) — before Phase B, snapshot every VM that phases B-G will touch, from the Proxmox node hosting it (standard Proxmox CLI qm snapshot <vmid> pre-rke2-apply):

VM	vm_id	Proxmox node	IP
wetopi-ops01	8001	ns3249468	10.72.0.10
rke-cp01	8011	ns3243421	10.72.0.21
rke-cp02	8012	ns3226939	10.72.0.22
rke-cp03	8013	ns3214108	10.72.0.23
rke-platform01	8021	ns31826929	10.72.0.41
rke-platform02	8022	ns3226931	10.72.0.42
rke-platform03	8023	ns31826925	10.72.0.43
wetopi-natgw01	8031	ns3243421	10.72.0.2

Rollback = qm rollback <vmid> pre-rke2-apply (VM must be stopped). These are the Proxmox-level safety net; the per-phase rollbacks below are the first line. 7. Cluster facts (from ADR-0004) — API VIP 10.72.0.20 (vmbr4/VLAN 4 node LAN 10.72.0.0/23, never Internet); server URL https://10.72.0.20:9345; API https://10.72.0.20:6443; tls-san = 10.72.0.20/.21/.22/.23 (IP-only, no DNS SAN); cluster-cidr 10.64.0.0/14; service-cidr 10.68.0.0/16; CNI none at RKE2 level (Cilium is the first addon, ADR-0005). 8. At-rest encryption gate — ONE current statement (reconciled 2026-08-30, Phase G live evidence, card t_e0cace4b). The earlier "honest gap" assumption (that a missing secrets-encryption line in the config template means Secrets sit PLAINTEXT in etcd) is superseded: it was wrong for this RKE2 build. Measured fact: RKE2 v1.36.3+rke2r1 enables AES-CBC at-rest encryption BY DEFAULT — the config template renders no secrets-encryption line, yet rke2 secrets-encrypt status measured on ALL THREE CPs shows Encryption Status: Enabled / Current Rotation Stage: start / Server Encryption Hashes: All hashes match / Active * AES-CBC aescbckey, and raw etcd probes (approved kubectl-exec etcdctl form, below) of argocd/argocd-initial-admin-secret (316B) and argocd/repo-infra-https (685B) returned k8s:enc:aescbc:v1:aescbckey:<opaque ciphertext> — NOT plaintext. CURRENT GATE RULE (this is the operative rule; it supersedes every older wording elsewhere in this runbook, including any residual plaintext framing): current protection is determined by MEASURED EVIDENCE, not by config-template contents — for any Secret at any gate, the measured checks below. What the explicit secrets-encryption: true line is: a SEPARATE, tracked, future reviewed hardening/rotation prerequisite (explicit declarative intent in the rke2-server role template, needed BEFORE any future key rotation). It is NOT an unlock condition for any gate while the measured checks pass — a build with default-on encryption and no explicit line is PROTECTED, not blocked. Measured-protection checks (the form used at any [encryption-gate]): - (i) rke2 secrets-encrypt status on all three CPs, each showing:

Encryption Status: Enabled
Current Rotation Stage: start
Server Encryption Hashes: All hashes match

plus an Active key row of the shape * AES-CBC aescbckey... (the * marks the active key); - (ii) an exact raw-etcd ciphertext check, reusing the approved kubectl-exec etcdctl form (Phase B gate) with the 17-arg pattern:

/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n kube-system exec etcd-rke-cp01 -- env ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt --key=/var/lib/rancher/rke2/server/tls/etcd/client.key get /registry/secrets/<namespace>/<NAME>

Expected: the value shows the encrypted form (k8s:enc:aescbc ciphertext — opaque), NOT the plaintext returned by kubectl get secret <name> -n <namespace> -o jsonpath='{.data.<KEY>}' | base64 -d. STOP: plaintext visible → protection NOT confirmed → the [encryption-gate] in question stays STOPped. Status of this cluster (2026-08-30, Phase G): measured PROTECTED — (i) verified on all 3 CPs, (ii) verified by raw etcd ciphertext probes on both argocd Secrets. The future hardening/rotation prerequisite (reviewed secrets-encryption: true merged to infrastructure main, then sequential restart cp01→cp02→cp03 one at a time with each returning Ready before the next, re-verifying (i) and (ii)) is NOT yet merged (git grep secrets-encryption on infrastructure main still returns empty) — it must land via its own reviewed PR BEFORE any future encryption-key rotation; it does not block F2/G while the measured checks pass. Future shape: a reviewed secrets-encryption: true setting in /etc/rancher/rke2/config.yaml (rendered by the rke2-server role), merged to infrastructure main. NOTE: there is no manually-managed key file — RKE2's documented behavior (rancher/rke2-docs, docs/security/secrets_encryption.md) is that once secrets encryption is enabled, RKE2 automatically generates an AES-CBC key and the encryption-provider config file at /var/lib/rancher/rke2/server/cred/encryption-config.json and passes it to the kube-apiserver as --encryption-provider-config. The config file is node-local state auto-generated and managed by RKE2 itself — never commit it to Git. 9. Controller and execution environment — define and verify BEFORE the first phase command. Direct evidence from the 2026-08-29 apply (t_a88bb36b): wetopi-ops01 had NO Ansible venv, NO infrastructure checkout, and NO kubeconfig; the playbooks ran from the WORKSTATION, which holds the SSH private keys (/opt/data/.ssh/bigardo). ops01 is a ProxyJump target only (item 5): it holds no private keys and cannot SSH to the LAN VMs by itself. This runbook therefore fixes the execution context explicitly instead of assuming it: - Controller = the workstation (the Ansible phases B, C, D, D2 — playbooks rke2-cp.yml, rke2-workers.yml, rke2-workers-validate.yml run from the controller and reach the VMs over SSH through the bastion). Verify on the controller BEFORE phases B/C (record each check with a date): - clean infrastructure checkout at d553643 or a verified descendant (item 2 STEP 0 gate): git -C infrastructure log --oneline -1 shows d553643 or a descendant, and git -C infrastructure status --porcelain is empty; - Ansible venv per requirements-dev.txt (pins ansible-core==2.21.3): python3 -m venv .venv then .venv/bin/pip install -r requirements-dev.txt; verify .venv/bin/ansible-playbook --version reports ansible-core 2.21.3; - inventory: ./scripts/render-ansible-inventory.sh regenerates ansible/inventories/prod/hosts.json (item 5) — or use the synthesized gitignored inventory shape the 2026-08-29 apply verified against outputs.tf; EITHER form must include the ungrouped localhost host with ansible_connection: local, ansible_become: false (the roles' delegate_to: localhost token checks fail on a sudo-less controller without it — evidenced 2026-08-29); - token file (item 4): stat -c '%a' ansible/files/rke2/node-token → 600; - SSH path: the double-key ProxyCommand form (item 5) opens a sudo-capable shell on cp01 (sudo -v succeeds there). - Ops01 provisioning — verify BEFORE Phase E (ops01 hosts the connection: local Helm op of cilium.yml, the cilium/hubble CLI acceptance in D2, and the F2/G commands; NONE of this existed at apply time): - infrastructure checkout on ops01 at d553643 or a verified descendant: git clone https://github.com/wetopi-platform/infrastructure.git ~/wetopi-infrastructure then git checkout main and verify the head; - Ansible venv on ops01: python3 -m venv .venv && .venv/bin/pip install -r requirements-dev.txt (Phase E invokes .venv/bin/ansible-playbook there); - inventory on ops01: ansible/inventories/prod/hosts.json is gitignored — either render it on ops01 (requires the tofu state) or copy the controller's verified copy over (same relay as the kubeconfig below); record which was done; - binaries on ops01: helm version --short (v3.x; the chart reference was validated with Helm 3.19), kubectl version --client, and the cilium + hubble CLIs (GitHub releases cilium/cilium-cli and cilium/hubble; verify with cilium version and hubble version; record exact versions — the cilium CLI must be compatible with the pinned chart 1.20.1); - current admin kubeconfig, transferred AFTER the F6-3 restore test (Phase E is the first consumer; a pre-restore transfer is stale): 1. on cp01: sudo cat /etc/rancher/rke2/rke2.yaml > /tmp/rke2-admin-kubeconfig && chmod 600 /tmp/rke2-admin-kubeconfig; 2. from the workstation (relay — ops01 cannot pull from cp01):

scp -i /opt/data/.ssh/bigardo -o ProxyCommand="ssh -i /opt/data/.ssh/bigardo -W %h:%p ubuntu@92.222.245.91" ubuntu@10.72.0.21:/tmp/rke2-admin-kubeconfig /tmp/rke2-admin-kubeconfig

; 3. workstation → ops01 (direct to the bastion): scp -i /opt/data/.ssh/bigardo /tmp/rke2-admin-kubeconfig ubuntu@92.222.245.91:~/secrets/rke2-admin-kubeconfig; 4. remove the /tmp copies on cp01 and the workstation; 5. on ops01:

sed -i 's|https://127.0.0.1:6443|https://10.72.0.20:6443|' ~/secrets/rke2-admin-kubeconfig && chmod 600 ~/secrets/rke2-admin-kubeconfig

(the RKE2 admin kubeconfig points at 127.0.0.1 by default; the VIP is in tls-san per ADR-0004, so the apiserver cert validates), then verify: KUBECONFIG=~/secrets/rke2-admin-kubeconfig kubectl get nodes lists the 3 CPs. NEVER commit the kubeconfig — it is cluster-admin credentials. - Two canonical kubectl invocation forms (every command block below uses one of these): - CP-local (executed ON a control-plane node): /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml ... — plain kubectl is NOT in PATH on CP nodes; only the RKE2-bundled binary is (same form the playbooks use, e.g. the rke2_kubectl fact in rke2-workers-validate.yml); - Remote (ops01 or the workstation): kubectl ... with KUBECONFIG pointing at the transferred admin kubeconfig (ops01 provisioning above).

Phase B — first control plane (cp01 bootstrap, incl. VIP bring-up)¶

Keepalived is NOT a standalone phase: it is the first role of both RKE2 plays inside ansible/playbooks/rke2-cp.yml (role order per CP: keepalived, then rke2-server). Play 1 pre-flights all CPs (asserts VIP+addresses in tls-san, CIDRs defined, VIP == 10.72.0.20, token file 0600). The bootstrap play (play 2, hosts rke-cp01, serial: 1, any_errors_fatal: true) renders /etc/rancher/rke2/config.yaml via the rke2-server template (token; server: only on non-bootstrap nodes; tls-san list; cluster-cidr; service-cidr; cni: none).

Keepalived design facts (from ansible/roles/keepalived/): interface eth0; VIP 10.72.0.20/23; VRID 51; unicast peers = the other two CPs; priorities cp01=150 (initial MASTER), cp02=100, cp03=50; advert_int 1; health-check script /etc/keepalived/check_rke2_api.sh probes the LOCAL API only: curl -sf --cacert /var/lib/rancher/rke2/server/tls/server-ca.crt --cert .../client-kube-apiserver.crt --key .../client-kube-apiserver.key https://127.0.0.1:6443/readyz plus a TCP probe of 127.0.0.1:9345; interval 2, fall 3, rise 2.

Consequence (fail-safe by design): a CP only holds the VIP while its own local API is healthy — before cp01's API is up there is deliberately NO VIP holder, so ping 10.72.0.20 failing before Phase B completes is EXPECTED, not a bug. There is NO standalone Phase-A apply; VIP acceptance is folded into the B/C gates below.

Command (from the CONTROLLER — the workstation per §0 item 9, in its infrastructure checkout):

ANSIBLE_CONFIG=ansible/ansible.cfg \
  .venv/bin/ansible-playbook -i ansible/inventories/prod/hosts.json \
  ansible/playbooks/rke2-cp.yml

The playbook is one invocation for B AND C — play 1 pre-flights all CPs, play 2 bootstraps cp01, play 3 joins cp02/cp03. Phase B is complete when play 2's built-in gates pass: wait_for 127.0.0.1:6443 (600s timeout), /readyz via the VIP (Ansible uri with client certs, retries 30 × 10s, expects status 200), and etcd member list == 1 (see gate B below).

Validation gate B (includes former "Phase A" VIP acceptance)¶

curl -k https://10.72.0.20:6443/readyz → ok (API via VIP works; VIP now held by cp01)
systemctl is-active keepalived on cp01 ONLY (Phase B applies cp01; the all-three-CP keepalived check belongs to gate C, where cp02/cp03 are applied) → active
ip -4 addr show eth0 on each CP: exactly one node lists 10.72.0.20/23 (the current owner; cp01 when healthy)
ping -c 3 10.72.0.20 from wetopi-ops01 → 0% loss

etcd member list (EXACT form the playbook uses, executed as a kubectl exec into the etcd pod on cp01):

/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n kube-system exec etcd-rke-cp01 -- env ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt --key=/var/lib/rancher/rke2/server/tls/etcd/client.key member list

→ exactly 1 member, started.

Node Ready state is NOT a Phase B gate: with cni: none, the CP-local form (§0 item 9) /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes on cp01 shows cp01 NotReady — EXPECTED. CP health at this stage = /readyz via VIP + etcd member count only.

STOP: any playbook play fails (any_errors_fatal aborts the whole run — do not re-run blindly; read the failure first); /readyz via VIP fails; etcd member count != 1; keepalived not active on cp01.

Phase C — quorum (cp02 + cp03, incl. VIP failover test)¶

Same playbook, play 3 (hosts rke-cp02:rke-cp03, serial: 1, roles keepalived + rke2-server, joining via server: https://10.72.0.20:9345; per-node expected etcd member counts: after cp02 = 2, after cp03 = 3). It continues automatically once Phase B gates pass.

Validation gate C¶

etcd member list (exact kubectl-exec form, run from cp01):

/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n kube-system exec etcd-rke-cp01 -- env ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt --key=/var/lib/rancher/rke2/server/tls/etcd/client.key member list

→ exactly 3 members, all started; quorum = 2 of 3

curl -k https://10.72.0.20:6443/readyz → ok
systemctl is-active keepalived on all three CPs (the all-three-CP keepalived check lives HERE, in Phase C where cp02/cp03 are applied) → active
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes (CP-local form, on cp01) → rke-cp01..03 all NotReady — EXPECTED (pre-CNI); do NOT treat this as a failure
VIP failover test (only now, with quorum): on the current owner (cp01; find via ip -4 addr show eth0 on each CP) run sudo systemctl stop keepalived; expect the VIP to move to the highest-priority healthy peer (cp02) within roughly 10 s (advert_int 1 × fall 3, plus check rise 2 × interval 2 on the receiver); verify from ops01: ping -c 3 10.72.0.20 still 0% loss and curl -k https://10.72.0.20:6443/readyz still ok; then sudo systemctl start keepalived on cp01 — cp01 (priority 150) reclaims the VIP by preemption; confirm with ip -4 addr again.
Documented one-CP-outage test: on ONE non-VIP-owner CP (cp02 or cp03): sudo systemctl stop rke2-server; then from ops01 curl -k https://10.72.0.20:6443/readyz must STILL return ok (2 members keep quorum; API served via VIP on remaining CPs) and the etcd member list (kubectl-exec form, from cp01) still lists 3 members; restart with sudo systemctl start rke2-server. STOP the whole run if the API via VIP fails during this test (that is a quorum/VIP defect, not a node defect).
CP workload check (CP-local form, on each CP): /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml describe node rke-cp0X | grep -i taint — record the taint state on all three CPs in the apply log. The no-platform-workloads-on-CPs guarantee is enforced by the workers playbook play 2, which applies node-role.kubernetes.io/control-plane=true:NoSchedule on all three CPs (idempotency decided from the API pre-state, not kubectl stdout), so ordinary pods can never schedule onto a CP; system components tolerate the taint by upstream default. Do not proceed if /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n kube-system get pods -o wide (same form) shows any platform-labeled pod scheduled on a CP.

STOP: member count != 3; /readyz fails; keepalived not active on all three CPs; API via VIP fails with one CP stopped or during VIP failover; any platform workload found on a CP.

Phase F6 / Fase-6 acceptance — CP-only portion (F6-1..3), gated by hard STOP prerequisites¶

Part of the authorized executable order (identical to the banner, the Phase W heading, the D2 handoff, the DoD mapping, and the provenance section): B → C → STOP → F6-1..3 → E → D → D2 → W → F → G. This phase is the CP-only portion of the Fase-6 acceptance: it runs immediately after the Phase C quorum gate and BEFORE any addon (Cilium E, workers D, Ceph F, Argo CD G), on the CP-only cluster (3 nodes, cni: none, no workers joined). Master plan §33 places Fase 6 before Fase 7 (Cilium/Hubble); its CP-only gates — F6-1 (quorum recorded), F6-2 (one-CP failure drill), F6-3 (etcd snapshot + REAL restore test) — run here, gated by the hard F6 STOP prerequisites below. The worker-lifecycle drill (drain + reboot) is NOT part of this phase: it is the separately-named Phase W and executes AFTER D2 on the final 6-node topology (ADR-0005 forbids pre-Cilium worker-Ready gates; see the banner for the §33 tension and its explicit resolution). Do NOT count Phase W as completed pre-addon acceptance. Each test has a concrete command, expected output, and STOP condition; record every result with a date in the apply log. Node Ready state is NOT a gate in this phase (CPs are NotReady pre-CNI — EXPECTED). See the execution notes per test.

F6-1 — quorum recorded¶

Command: etcd member list via the kubectl-exec form (Phase B gate), from cp01.
Expected: exactly 3 members, all started.
STOP: member count != 3 or any member not started.

F6 STOP prerequisites — HARD GATES (ALL must be PASS, dated in the apply log, BEFORE F6-2/F6-3 stop any rke2-server)¶

These are GATES, not suggestions. F6-3's restore test stops ALL THREE servers — full cluster API downtime — and replaces the etcd peer databases; F6-2 stops one server. Until every prerequisite below is PASS, this phase halts after the read-only F6-1 and NO systemctl stop rke2-server may be issued.

P1 — Quorum and API evidence (immediately before any stop): F6-1 recorded with today's date (member list == 3 members, all started), plus curl -k https://10.72.0.20:6443/readyz → ok; record both outputs.
P2 — Snapshot saved AND copied OUT of cp01's failure domain (on cp01, as root):
```
rke2 etcd-snapshot save --name pre-addons-fase6
rke2 etcd-snapshot ls
```
Expected: save reports the snapshot file written under /var/lib/rancher/rke2/server/db/snapshots/ (on-demand snapshots default to on-demand-<node>-<unix-ts>; --name sets the base name to pre-addons-fase6); ls lists the new snapshot with a size and timestamp. Record both outputs. Then copy the snapshot file OFF cp01 to external storage — e.g. wetopi-ops01 ~/rke2-snapshots/ or the workstation's disk — using the relay documented in §0 item 9 (on cp01 place it under /tmp readable by ubuntu; pull it with the workstation's ProxyCommand form; ops01 cannot pull from cp01 itself because it holds no private keys). The external copy MUST live outside /var/lib/rancher/rke2/server/db AND outside cp01's failure domain: NOT on cp01, NOT on any CP node, NOT inside the live db tree, and preferably not on cp01's Proxmox node (ns3243421) either — wetopi-ops01 (ns3249468) or the workstation satisfies this. cp01 keeps the copy the reset consumes; the external copy is the recovery plane if the reset damages cp01's db.
P3 — SHA-256 digests match, independently recomputed: sha256sum /var/lib/rancher/rke2/server/db/snapshots/<snapshot-file> on cp01, and — independently, ON the host holding the external copy (not via a shell relayed through cp01) — sha256sum <external-path>/<snapshot-file>. The two digests MUST be identical; record both in the apply log. Mismatch → re-copy and re-verify before anything else.
P4 — EXPLICIT human authorization for full API downtime: a named human explicitly authorizes the F6-3 restore window (all three servers stopped, cluster API fully down; plan tens of minutes for reset + peer rejoin), recorded in the apply log with name, date, scope, and the expected window — BEFORE the first systemctl stop rke2-server of F6-3. No recorded authorization line = gate NOT passed.

STOP (prerequisites): any P1-P4 item not PASS → do NOT run F6-2/F6-3; the cluster stays up (read-only F6-1 is the only permitted test); escalate to the lead.

F6-2 — one-CP failure (recorded acceptance test; requires the F6 STOP prerequisites P1-P4 ALL PASS — this test stops a server)¶

Command: sudo systemctl stop rke2-server on cp02, then curl -k https://10.72.0.20:6443/readyz from ops01, then sudo systemctl start rke2-server on cp02.
Expected: /readyz returns ok throughout (2 members keep quorum); API served via VIP on remaining CPs; after restart, member list still 3.
STOP: API via VIP fails while one CP is stopped (quorum/VIP defect — stop the whole apply).

F6-3 — REAL restore test (restore probado) — requires the F6 STOP prerequisites P1-P4 ALL PASS; snapshot per P2¶

This is a REAL, EXECUTED acceptance test — not a documented-only procedure. Master plan §35.1 requires "restore probado"; a snapshot-save-only check does NOT satisfy it. The restore test must be executed and recorded HERE, before Phase E (Cilium). If the restore test cannot be executed, the DoD marks it UNMET and the runbook mandates STOP before all subsequent phases (E onward). The snapshot itself was saved and externally checksum-verified in the F6 STOP prerequisites P2/P3 above; this test consumes the cp01-local copy.

The snapshot file is the one saved and externally verified in prerequisites P2/P3 above (cp01-local path /var/lib/rancher/rke2/server/db/snapshots/<file>; the verified external copy is the recovery plane — if the cp01-local file is damaged, re-copy from the external copy and re-verify its SHA-256 before use). EXECUTE the restore procedure below as the acceptance test and record the evidence (expected-output lines, member list 3/3, /readyz, and the post-restore journal checks in step 6).

Restore procedure — `rke2 server --cluster-reset` (multi-server, 3 CPs)¶

CRITICAL: there is NO rke2 etcd-snapshot restore subcommand. Verified from the authoritative RKE2 docs (rancher/rke2-docs, docs/datastore/backup_restore.md): the etcd-snapshot subcommands are delete / ls (list, l) / prune / save ONLY. Restore is the server cluster-reset flow.

Snapshot on N1 = cp01; peers N2/N3 = cp02/cp03:

Stop RKE2 on ALL server nodes: systemctl stop rke2-server on cp01, cp02, cp03.
On N1 (cp01): rke2 server --cluster-reset --cluster-reset-restore-path=<PATH-TO-SNAPSHOT> (path = the snapshot file recorded by rke2 etcd-snapshot ls, under /var/lib/rancher/rke2/server/db/snapshots/). Expected output on success:
```
Managed etcd cluster membership has been reset, restart without --cluster-reset flag now.
Backup and delete ${datadir}/server/db on each peer etcd server and rejoin the nodes.
```
(datadir = /var/lib/rancher/rke2)
On N1 (cp01): systemctl start rke2-server.
On N2/N3 (cp02/cp03): rm -rf /var/lib/rancher/rke2/server/db/ then systemctl start rke2-server — each peer rejoins the restored cluster by re-downloading the db.
Re-verify: etcd member list (kubectl-exec form, from cp01) == 3 members started; curl -k https://10.72.0.20:6443/readyz → ok; the CP-local form /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes (on cp01) shows 3/3 CPs registered.
Post-restore journal checks on EVERY CP (explicit gate — record all three outputs with a date in the apply log):
```
sudo journalctl -u rke2-server --since "-30 min" --no-pager | grep -E 'level=(error|fatal)' || echo CLEAN
```
Failure criteria (explicit): on ANY of the three CPs, any level=error/level=fatal line mentioning corruption, corrupt, checksum, panic, or db/data-dir damage — OR a recurring failed to start persisting more than ~2 minutes after that node's start — means the restore test is FAILED: apply the STOP-and-escalate rule below; do NOT retry. Transient retry/election/timeout lines during peer rejoin that stop once quorum re-forms are acceptable. Required to PASS: CLEAN output (or only transient lines that cease) on all three CPs.

Mechanics (from the RKE2 docs): the current etcd db files are moved to ${data-dir}/server/db/etcd-old-$TIMESTAMP/; the snapshot checksum is verified; all etcd members except the restore node are removed from the cluster, and peers rejoin by re-downloading the db.

Reset-flag safety mechanism (rancher/rke2-docs, docs/datastore/backup_restore.md; semantics verified 2026-08-30 against the k3s source vendored by the pinned rke2_version: v1.36.3+rke2r1 — k3s commit 4ca29faac89e, pkg/cluster/managed.go + pkg/etcd/etcd.go): when RKE2 resets the cluster it creates an empty file at /var/lib/rancher/rke2/server/db/reset-flag to prevent accidental repeated cluster resets; the flag is deleted only when RKE2 starts normally (a normal start of the managed database clears it). Version-exact consequences: (a) the flag is written after the snapshot-restore step succeeds — an early failure (bad snapshot path, download/checksum/restore error) happens with NO flag present; a late failure (post-reset etcd startup) happens WITH the flag present; (b) the flag blocks only a bare rke2 server --cluster-reset (no restore path) — the pinned version does NOT consult the flag when --cluster-reset-restore-path is set (source comment: "If we're restoring from a snapshot, don't check the reset-flag - just reset and restore"); (c) the pre-restore db is moved aside to db/etcd-old-<timestamp>/ by the restore step, so a failed attempt may leave the db dir absent or partially restored.

STOP — reset/restore failure is an immediate STOP-and-escalate condition. If the reset invocation fails, does not print the expected output, the db dir is not re-provisioned, peers fail to rejoin, quorum is not re-established, or /readyz is not OK: do NOT blindly retry — the failure state is flag- and db-dependent per (a)-(c) above, and all CPs are stopped. Escalate to the lead with the full output; the cluster stays STOPped. No addons or workloads until recovery is complete and quorum is re-verified.
Only documented, version-supported recovery — every option below is a lead-approved, apply-log-recorded escalated action, never an operator default; a second reset attempt is permitted ONLY after a successful normal cp01 start or a coordinated rollback decision:
Normal start on cp01: systemctl start rke2-server — the documented flag-clearing mechanism. It starts from whatever db state the failed attempt left: if the snapshot-restore step succeeded and only the startup failed, the node comes up as the restored single member; if the restore step itself failed (db moved aside, restore incomplete), the node may fail to start or come up uninitialized. Record the actual observed state (flag present? db/etcd-old-* present? member list, /readyz) and STOP again if anything is unexpected. A successful normal start is the precondition for considering any second reset attempt.
Second attempt, only after (1) succeeded or a coordinated rollback decision: re-attempt the documented restore with a KNOWN-GOOD snapshot (stop rke2-server on all CPs → rke2 server --cluster-reset --cluster-reset-restore-path=<PATH> on cp01 → normal start cp01 → peers rm -rf /var/lib/rancher/rke2/server/db/ + start). Version fact for the record: the flag check applies only to a bare --cluster-reset; the pinned version does not consult the flag when a restore path is set. This runbook never performs a bare reset — do not remove reset-flag manually; if it is ever encountered blocking a start, that is itself a STOP-and-escalate condition.
Coordinated rollback: Proxmox qm rollback <vmid> pre-rke2-apply applied to ALL CPs together — the snapshot predates Phase B, so this returns the cluster to the pre-B state and phases B/C must be re-run.
Escalate to Rancher support with the exact rke2 version and full logs.
A restore is a STOP-and-reassess event: after the test, re-run F6-1/F6-2 gates and only then proceed to Phase E.

STOP (F6 overall): any F6 STOP prerequisite P1-P4 not PASS (no server may be stopped); any F6-1/F6-2/F6-3 gate failing — the cluster is not accepted; fix before Phase E (these three are the pre-addon CP-only gates and are the ONLY Fase-6 gates blocking E). The worker-lifecycle drill is the separately-named Phase W of the same executable order (B → C → STOP → F6-1..3 → E → D → D2 → W → F → G); it runs after D2 and its gates are defined there. The Fase-6 acceptance ledger is COMPLETE only when F6-1..3 AND Phase W's W-1/W-2 all have dated PASS entries; W passes POST-D2 by design (ADR-0005 — see banner), never pre-addon.

Phase E — Cilium install + basic install validation (first addon)¶

Runs as the immediate first addon after the CP-only Fase-6 gates (F6-1..3) — and BEFORE workers join: with Cilium already present, newly joined workers can reach Ready as soon as their Cilium agent starts. (The worker-lifecycle drill — drain/reboot — is the separately-named Phase W and executes later in the executable order, AFTER D2; it does not block E.) This runbook splits Cilium work in two: Phase E = install + basic install validation ONLY; the full Cilium acceptance (connectivity test, Hubble flows, DNS, NetworkPolicy, one CiliumNode per node) runs in Phase D2 on the FINAL 6-node topology (see D2).

Playbook ansible/playbooks/cilium.yml (infrastructure PR #2, merged as e283e24): runs from the operations host (ops01) with connection: local so the Helm op executes exactly once; chart pinned oci://quay.io/cilium/charts/cilium --version 1.20.1 (full OCI reference; the --repo oci:// form is NOT valid); values kubernetes/cilium/values.yaml (VXLAN tunnel, kubeProxyReplacement false, Hubble enabled).

Gate semantics are FAIL-CLOSED: every kubectl probe uses --ignore-not-found and requires rc 0 — any rc != 0 (API error, connection refused, auth/TLS failure) ABORTS the run; absence of a resource is proven ONLY by rc 0 with EMPTY stdout. An error is never "absent": it is a STOP.

Ordering gates (fail-closed, per ADR-0005): the play refuses to run if a CNI is already deployed (Canal would fight Cilium), requires kube-proxy present (RKE2 with cni: none provides it), and aborts unless the Cilium agent is absent (first-install check).

Command (from ops01 — AFTER the §0 item 9 ops01 provisioning is verified: checkout at d553643 or a verified descendant, Ansible venv, inventory, helm, and the current transferred kubeconfig in KUBECONFIG; this playbook targets the operations group with connection: local, so the Helm op runs on ops01 itself):

ANSIBLE_CONFIG=ansible/ansible.cfg \
  .venv/bin/ansible-playbook -i ansible/inventories/prod/hosts.json \
  ansible/playbooks/cilium.yml

Firewall prerequisites (node LAN must allow): UDP 8472 (VXLAN) between all nodes; TCP 4244 (Hubble Relay flow scraping). MTU stays Cilium default (~1450) on the MTU-1500 LAN.

Validation gate E (install-only)¶

Basic install validation on the CP-only (3-node) cluster, from ops01 with KUBECONFIG set to the transferred admin kubeconfig (§0 item 9):

cilium status --wait     # agents Ready on the 3 CPs; KVStore/Conntrack/ControllerStatus Ok, Hubble Ok/Enabled
kubectl -n kube-system get pods -l k8s-app=cilium    # cilium-ds Running+Ready on all 3 CPs (one per CP)
kubectl -n kube-system get pods -l io.cilium/app=operator   # cilium-operator Running+Ready

The following are deliberately NOT Phase E gates (they are D2 gates on the final 6-node topology): cilium connectivity test, Hubble flow observation, DNS resolution, NetworkPolicy enforcement, CiliumNode per-node completeness.

Also note (VALIDATION.md §7): pods that existed before Cilium must be restarted once to fall under the eBPF datapath (in this greenfield order the only such pods are the control-plane static/system pods; kube-proxy static pods and CoreDNS are handled by rollout/restart once nodes go Ready).

Rollback: helm uninstall cilium --namespace kube-system — documented in VALIDATION.md §8 with the warning that the cluster has NO other CNI after this (cni: none means RKE2 will not reinstall Canal), so only do this with a replacement plan.

STOP: any fail-closed probe rc != 0; cilium-ds pod not Running+Ready on each of the 3 CPs; cilium-operator not Running+Ready.

Phase D — platform workers join (registration + label gates only)¶

Playbook ansible/playbooks/rke2-workers.yml (infrastructure PR #5, merged as 8d836e8): play 1 pre-flights rke_platform (asserts rke2_server_url == https://10.72.0.20:9345, CIDRs inherited from shared group_vars, token file exists 0600 — no_log), play 2 (hosts rke-cp01) applies/verifies the control-plane NoSchedule taint node-role.kubernetes.io/control-plane=true:NoSchedule on all 3 CPs (idempotency decided from API pre-state), play 3 joins rke-platform01..03 (serial: 1, any_errors_fatal: true, rke2-agent role; config: token from the gitignored 0600 file, server: https://10.72.0.20:9345, node-label: [wetopi.io/role=platform], no taints).

Command (from the CONTROLLER — the workstation per §0 item 9):

ANSIBLE_CONFIG=ansible/ansible.cfg \
  .venv/bin/ansible-playbook -i ansible/inventories/prod/hosts.json \
  ansible/playbooks/rke2-workers.yml

Per-worker join gate (inside the serial join play): for each worker, the gate is control-plane-OBSERVED registration + exact label — kubectl get node <name> -o json delegated to rke-cp01 (retries 60 × 10s), then assert metadata.name == inventory_hostname and labels['wetopi.io/role'] == 'platform'. A failed gate aborts before the next worker joins. Phase D gate = registration + label ONLY.

Ready state after join (Cilium-before-workers): Cilium was installed in Phase E, so a newly joined worker SHOULD reach Ready once its Cilium agent (the cilium-ds DaemonSet pod) starts on it — NotReady is a TRANSIENT join state, NOT indefinitely expected. Ready/pod/DNS/scheduling validation is deferred to rke2-workers-validate.yml (Phase D2), which must prove: 6/6 nodes Ready, a cilium-ds pod per node, and a CiliumNode object per node, plus the full Cilium acceptance. A worker that stays NotReady indefinitely is a FAILURE to investigate (Cilium agent attachment issue), never "expected".

After the join play, the playbook reports the result; workers are expected to transition to Ready as their Cilium agents come up. Full validation runs next in D2.

Phase D2 — full post-Cilium cluster validation (rke2-workers-validate.yml + FULL Cilium acceptance, on the FINAL 6-node topology)¶

Playbook ansible/playbooks/rke2-workers-validate.yml (PR #5, merged as 8d836e8; hosts rke-cp01, become: true, any_errors_fatal: true, read-only queries + two short-lived uniquely-named test pods, always deleted). Run from the CONTROLLER (§0 item 9 — the play reaches cp01 over SSH) AFTER Cilium (Phase E) and after the workers have joined (Phase D). This phase ALSO carries the full Cilium acceptance (moved from former Phase E) on the final 6-node topology:

ANSIBLE_CONFIG=ansible/ansible.cfg \
  .venv/bin/ansible-playbook -i ansible/inventories/prod/hosts.json \
  ansible/playbooks/rke2-workers-validate.yml

Built-in gates (manual equivalents — run ON cp01 using the CP-local kubectl form, §0 item 9; define once for the shell: RKE2_KUBECTL="/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml"):

$RKE2_KUBECTL get nodes                          # 6 nodes, ALL Ready (3 CP + 3 platform)
$RKE2_KUBECTL get nodes -l wetopi.io/role=platform   # exactly 3, each carrying the exact label
$RKE2_KUBECTL -n kube-system get pods -l k8s-app=kube-dns            # Running+Ready
$RKE2_KUBECTL -n kube-system get pods -l k8s-app=kube-dns-autoscaler # Running+Ready
$RKE2_KUBECTL -n kube-system get pods -l app=rke2-metrics-server     # Running+Ready
$RKE2_KUBECTL -n kube-system get pods -l k8s-app=cilium              # Running+Ready, one per node
$RKE2_KUBECTL -n kube-system get pods -l io.cilium/app=operator      # Running+Ready
$RKE2_KUBECTL -n kube-system get pods -l k8s-app=hubble-relay         # Running+Ready
$RKE2_KUBECTL -n kube-system get pods -l component=kube-proxy         # Running+Ready, one per node
$RKE2_KUBECTL run rke2-dns-check-<suffix> --restart=Never \
  --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 \
  --command -- sleep 300
$RKE2_KUBECTL wait --for=condition=Ready pod/rke2-dns-check-<suffix> --timeout=300s
$RKE2_KUBECTL exec rke2-dns-check-<suffix> -- nslookup kubernetes.default.svc.cluster.local
$RKE2_KUBECTL run rke2-sched-check-<suffix> --restart=Never \
  --image=registry.k8s.io/e2e-test-images/busybox:1.36.1-1 \
  --overrides='{"spec":{"nodeSelector":{"wetopi.io/role":"platform"}}}' \
  --command -- sleep 300

Expected success:

6/6 nodes Ready (the playbook retries up to 60 × 10s);
every platform worker carries the exact label wetopi.io/role=platform (count == 3);
all required kube-system components Running+Ready with the selectors above (CoreDNS k8s-app=kube-dns and autoscaler k8s-app=kube-dns-autoscaler min 1 each; metrics-server app=rke2-metrics-server min 1; Cilium agent k8s-app=cilium one per node; cilium-operator io.cilium/app=operator min 2; hubble-relay k8s-app=hubble-relay min 1; kube-proxy component=kube-proxy one per node);
DNS gate: nslookup resolves kubernetes.default.svc.cluster.local to answer address(es) in 10.68.0.0/16 — the playbook parses the ANSWER addresses (the Address lines AFTER the Name: kubernetes.default... line) into a CSV string and requires every entry to match the octet-bounded regex ^10\.68\.\d{1,3}\.\d{1,3}$; the resolver's own Server: 10.68.0.10 line appears BEFORE the Name line and must never satisfy the gate;
scheduling gate: the test pod lands on a node of group rke_platform (placement on a CP = FAIL — investigate taints/labels).

D2 full Cilium acceptance (final 6-node topology, from kubernetes/cilium/VALIDATION.md; cilium/hubble CLIs on ops01 — §0 item 9 ops01 provisioning — with `KUBECONFIG` set to the transferred admin kubeconfig; the `kubectl` lines in THIS block are the ops01 remote form, not CP-local)¶

cilium status --wait            # all agents ready, KVStore/Conntrack/ControllerStatus Ok, Hubble Ok/Enabled
cilium connectivity test        # all tests pass, incl. cross-node pod-to-pod, pod-to-service, north-south
hubble status                   # flows counters increasing, sane uptime
hubble observe --since 5m --from-cluster   # real flows with identities and VERDICTs, not empty
kubectl run dns-utils --image=busybox:1.36 --restart=Never -- sleep 60
kubectl wait --for=condition=Ready pod/dns-utils --timeout=120s
kubectl exec dns-utils -- nslookup kubernetes.default    # resolves into 10.68.0.0/16
kubectl delete pod dns-utils
kubectl get ciliumnodes.cilium.io    # one CiliumNode object per node (6/6); podCIDR /24s inside 10.64.0.0/14

NetworkPolicy sample (VALIDATION.md §5):

kubectl create namespace cilium-validation
kubectl -n cilium-validation run web --image=nginx:1.27 --restart=Never
kubectl -n cilium-validation expose pod web --port 80
kubectl apply -f kubernetes/cilium/test-default-deny.yaml     # default-deny Ingress in that namespace
kubectl -n cilium-validation run probe --image=busybox:1.36 --restart=Never \
  -- wget -qO- --timeout=5 http://web                          # MUST fail (dropped)
hubble observe --namespace cilium-validation --since 5m        # shows VERDICT Dropped
kubectl delete -f kubernetes/cilium/test-default-deny.yaml
kubectl delete namespace cilium-validation

One cilium-ds pod per node: kubectl -n kube-system get pods -l k8s-app=cilium -o wide → exactly 6 Running+Ready pods, one on each of the 6 nodes (3 CP + 3 platform).
One CiliumNode object per node: kubectl get ciliumnodes.cilium.io → exactly 6, one per node, each podCIDR inside 10.64.0.0/14.

STOP (D2 overall): any node not Ready after the retry window; any selector returning pods that are not all Running+Ready (or missing entirely); DNS gate failing (rc != 0, zero parsed answers, or any answer outside 10.68.0.0/16); sched-check not placed on a platform-labeled node; cilium connectivity test failures; hubble observe empty; NetworkPolicy probe NOT dropped; any podCIDR outside 10.64.0.0/14; fewer than 6 CiliumNode objects or missing cilium-ds pod on any node.

The executable order (identical to the banner, the F6 heading + STOP rule, the Phase W heading, the DoD mapping, and the provenance section) is B → C → STOP → F6-1..3 → E → D → D2 → W → F → G: the CP-only Fase-6 gates (F6-1 quorum, F6-2 one-CP failure, F6-3 snapshot + REAL restore test) sat BEFORE this phase; Phase W — the worker lifecycle drill (drain + reboot) — runs NEXT, after these D2 gates pass, on the final 6-node topology. Do NOT count Phase W as completed pre-addon acceptance.

Phase W — worker lifecycle drill (drain + reboot; former F6-4/F6-5)¶

Part of the authorized executable order (identical to the banner, the F6 heading + STOP rule, the D2 handoff, the DoD mapping, and the provenance section): B → C → STOP → F6-1..3 → E → D → D2 → W → F → G. This is a SEPARATE named phase — the worker-lifecycle portion of the Fase-6 acceptance — running AFTER the D2 gates pass on the final 6-node topology, BEFORE Phase F, and NEVER before the addons: drain/reboot evidence is POST-D2 by design (ADR-0005: with cni: none, workers cannot pass Ready — and therefore cannot pass drain/reboot gates — before Cilium exists; see the banner for the §33 tension and its explicit resolution). Results are recorded in the Fase-6 acceptance ledger under their historical aliases F6-4 (drain) / F6-5 (reboot), which map to master plan §35.1/§35.2 drain/reboot recovery. Do NOT count this phase as completed pre-addon acceptance.

W-1 — drain one worker (former F6-4)¶

Command (CP-local form on cp01 — or the ops01 remote form with the transferred kubeconfig, §0 item 9):

/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml drain rke-platform01 --ignore-daemonsets --delete-emptydir-data
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml uncordon rke-platform01

Expected: drain evicts/reschedules eligible pods (kube-system DaemonSet pods stay — expected); node shows SchedulingDisabled during the drain; after uncordon the node returns to Ready/Schedulable.
STOP: drain hangs on un-evictable pods (investigate PodDisruptionBudgets), node not Schedulable after uncordon.

W-2 — reboot one worker + Ready return (former F6-5)¶

Command: sudo reboot on rke-platform02 (via the documented SSH form, §0 item 5), then on cp01 (CP-local form):

/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get node rke-platform02 -w    # wait for Ready

Expected: node cycles NotReady → Ready within ~2-3 min of boot (kubelet re-registers, Cilium agent re-attaches); label wetopi.io/role=platform still present; no NotReady residue after ~5 min.
STOP: node not Ready after 10 min, label lost, or the Cilium agent on that node not Running+Ready after reboot.

STOP (W overall): either W-1/W-2 gate failing — fix before Phase F. With W-1/W-2 recorded PASS (post-D2) alongside F6-1..3 (pre-addon), the Fase-6 acceptance ledger is COMPLETE; the DoD maps drain/reboot recovery to this POST-D2 evidence, never to pre-addon acceptance.

Phase F — Ceph routing + RBD CSI¶

Source artifacts (all on merged main): ansible/roles/nat_gateway (vars natgw_ceph_routing_enabled: false, natgw_ceph_address: "" — default-off until gates), kubernetes/ceph-csi/ (chart ceph-csi-rbd pinned 3.17.1, StorageClass ceph-rbd, secret.example.yaml), ansible/playbooks/ceph-csi.yml, kubernetes/ceph-csi/VALIDATION.md, kubernetes/ceph-csi/README.md.

Phase F has TWO independent sub-phases; the CSI part does not block the cluster (RKE2 installs with no storage class).

F0 — tofu NIC change (NEW precondition for F1)¶

The third NIC for the Ceph leg lives in tofu/proxmox/environments/prod/terraform.tfvars.example: a commented-out block at the end of the wetopi-natgw01 nic list with the ADR-0004 comment above it (VLAN 2 / vmbr2, address from 192.168.96.0/20; guest-side config via the nat_gateway role's natgw_ceph_* vars, NOT cloud-init). F1 requires it enabled via a reviewed PR, not an apply-time edit:

Open a PR uncommenting the block with the allocated address:
```
# {
#   bridge       = "vmbr2"
#   ipv4_address = "192.168.96.<allocated>/20"
#   firewall     = false
# }
```
→ { bridge = "vmbr2", ipv4_address = "192.168.96.<allocated>/20", firewall = false }.
Merge it (wetopi-approver review), then on ops01 check out a clean main and run:
```
tofu -chdir=tofu/proxmox/environments/prod plan
```
tofu plan gate (record the output): the plan must show ADD of exactly one NIC on wetopi-natgw01 and ZERO destroys (no other resource touched). Any destroy = STOP.
tofu apply = HUMAN gate — a human explicitly reviews the plan and runs the apply; record evidence (plan summary + apply output) in the apply log.

STOP (F0): plan shows any destroy; plan touches any other VM/resource; apply fails.

F1 — Routing¶

ADR-0004 shape: route, not NAT; third Nic on wetopi-natgw01, VLAN 2, into Ceph LAN 192.168.96.0/20.

Preconditions (human, from VALIDATION.md §1 — each gate verified and recorded BEFORE flipping the vars):

Gate	Check
(a) VLAN 2 address allocation	pick a free address in 192.168.96.0/20 — check live allocations (`ip -4 route`, Ceph-side `ceph mon dump`)
(b) Ceph-side return-route owner	who installs `ip route 10.72.0.0/23 via <natgw VLAN 2 addr>` on Ceph hosts (not managed by us — confirm with Ceph owner, record mechanism)
(c) firewall path both directions	`nft list chain inet wetopi_filter forward` shows the two ceph_net accept rules; `nft list table ip wetopi_nat` still scopes NAT to oifname WAN only
(d) rp_filter=1 behavior	`sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.default.rp_filter`; asymmetric drops = missing gate b
(e) MTU 1500 on VLAN 2	from a CP: `ping -c 3 -M do -s 1472 <natgw-ceph-addr>` passes, `-s 1473` fails
(f) no CIDR overlap	10.64.0.0/14 + 10.68.0.0/16 vs 192.168.96.0/20 + 10.72.0.0/23 (`ip -4 route` review; `ip route get <ceph-host>` takes the gateway)

Apply: via reviewed PR, not an apply-time edit — open a PR setting natgw_ceph_routing_enabled: true + natgw_ceph_address: <chosen> in ansible/inventories/prod/group_vars/all.yml, merge it, then check out a clean main and re-run the baseline playbook (idempotent; its first play applies nat_gateway to wetopi-natgw01):

ANSIBLE_CONFIG=ansible/ansible.cfg \
  .venv/bin/ansible-playbook -i ansible/inventories/prod/hosts.json \
  ansible/playbooks/bootstrap.yml

Gateway checks (on wetopi-natgw01, VALIDATION.md §2): ip -4 route shows 192.168.96.0/20 dev eth2 ... src <natgw_ceph_address>; ip route get 192.168.96.10 via eth2; ip route get 10.72.0.21 still via eth0 src 10.72.0.2. Ceph-side (prerequisite, VALIDATION.md §3): ip route get 10.72.0.21 on a Ceph host must go via the natgw VLAN 2 address.

CP-node connectivity (from 10.72.0.21, VALIDATION.md §4):

ping -c 3 192.168.96.10          # Ceph monitor; 0% loss
traceroute -n 192.168.96.10      # one hop 10.72.0.1 (gateway), then the mon host
nc -vz -w 3 192.168.96.10 3300   # msgr2 open
nc -vz -w 3 192.168.96.10 6789   # msgr1 open
timeout 3 bash -c 'echo > /dev/tcp/192.168.96.10/3300' && echo OK   # fallback if nc absent

F2 — CSI install¶

Human prerequisites (README OPEN list): dedicated Ceph pool (suggested kubernetes); dedicated Ceph user + caps (e.g. client.kubernetes-csi, never admin); live clusterID (fsid) + monitor addresses replacing the placeholders in kubernetes/ceph-csi/values.yaml + storageclass.yaml — via reviewed PR, then merge and a clean main checkout (placeholders are tracked files; never edit them at apply time); csi-rbd-secret in kube-system created BY HAND from that user's keyring per secret.example.yaml (key material never enters Git; Vault flow later).

STOP [encryption-gate] before creating csi-rbd-secret: the Ceph key stored in a Kubernetes Secret must be confirmed PROTECTED at rest by the measured-protection checks of §0 item 8 (the operative gate rule — current protection is determined by measured evidence, not by config-template contents; this build's default-on AES-CBC state is the baseline, re-verify, do not assume). Before creating the Secret: rke2 secrets-encrypt status on all three CPs showing Encryption Status: Enabled / Current Rotation Stage: start / Server Encryption Hashes: All hashes match and an Active * AES-CBC aescbckey... key (§0 item 8 check (i)). Then create csi-rbd-secret per the prerequisites above and immediately run check (ii) on it: etcdctl ... get /registry/secrets/kube-system/csi-rbd-secret via the approved kubectl-exec 17-arg form (§0 item 8) showing k8s:enc:aescbc ciphertext, NOT the plaintext from kubectl get secret csi-rbd-secret -n kube-system -o jsonpath='{.data.<KEY>}' | base64 -d. STOP if either check fails or shows plaintext — do not run the csi playbook. (The explicit reviewed secrets-encryption: true line is the separately tracked future hardening/rotation prerequisite per §0 item 8 — it is not an unlock condition here while the measured checks pass.) Then:

ANSIBLE_CONFIG=ansible/ansible.cfg \
  .venv/bin/ansible-playbook -i ansible/inventories/prod/hosts.json \
  ansible/playbooks/ceph-csi.yml

(Run from the controller or ops01 — the play targets rke_control_plane with connection: local + run_once: true, so helm/kubectl talk to the cluster API from wherever ansible runs; kubeconfig resolution: -e kubeconfig= → $KUBECONFIG → /etc/rancher/rke2/rke2.yaml — on a non-CP host the transferred kubeconfig must be in KUBECONFIG, §0 item 9.)

The playbook fails early on placeholders or missing secret (by design); it targets rke_control_plane with connection: local, run_once: true, kubeconfig resolution: -e kubeconfig= → $KUBECONFIG → /etc/rancher/rke2/rke2.yaml; helm upgrade --install ceph-csi-rbd ceph-csi-rbd --repo https://ceph.github.io/csi-charts --version 3.17.1 -n kube-system --values kubernetes/ceph-csi/values.yaml; then applies storageclass.yaml.

PVC acceptance (VALIDATION.md §5, maps master plan §35.3):

kubectl -n kube-system get pods -l app=ceph-csi-rbd     # all Running
kubectl get storageclass ceph-rbd                       # provisioner rbd.csi.ceph.com, allowVolumeExpansion true
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-rbd-test
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: ceph-rbd
  resources:
    requests:
      storage: 1Gi
EOF
kubectl get pvc ceph-rbd-test -w     # Bound within ~1 min
# (b) mount+write: kubectl run ceph-rbd-test --rm -it --restart=Never --image=nginx:alpine \
#   --overrides='{"spec":{"containers":[{"name":"ceph-rbd-test","image":"nginx:alpine","command":["sh"],"stdin":true,"tty":true,"volumeMounts":[{"name":"data","mountPath":"/mnt/data"}]}],"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"ceph-rbd-test"}}]}}'
#   inside: date > /mnt/data/marker.txt && cat /mnt/data/marker.txt && exit
# (c) persistence: delete the pod, re-run the SAME kubectl run, marker still readable with the same timestamp
# (d) failure test: kubectl delete pvc ceph-rbd-test; on a Ceph host: rbd ls kubernetes  → no csi-vol- image for the PVC
# (e) performance: no CSI errors/timeouts in kubectl -n kube-system logs deploy/ceph-csi-rbd-provisioner (and node plugin logs) across (a)-(d); provisioning within ~1 min

Rollback F: routing — revert natgw_ceph_routing_enabled: false (and clear the address) via reviewed PR and re-run bootstrap.yml (the role removes the leg; the default-off state is the reviewed baseline). CSI — inverse of the install: helm uninstall ceph-csi-rbd -n kube-system, kubectl delete -f kubernetes/ceph-csi/storageclass.yaml, delete test PVCs first; keep csi-rbd-secret out of Git always. (This rollback line is runbook-authored inverse of the playbook install; the parent README documents no uninstall procedure.)

STOP (F0): plan shows destroys or extra resource churn. STOP (F1): missing connected route, wrong next-hop, ping loss, TCP 3300/6789 closed, MTU probe failing. STOP (F2): [encryption-gate] measured-protection checks failing (§0 item 8); PVC Pending (describe it; check provisioner logs — usual causes: placeholder clusterID/monitors, wrong userKey, routing regression), any acceptance step (b)-(e) failing.

Phase G — Argo CD¶

Source: infrastructure PR #1 (MERGED, main): kubernetes/gitops/ + docs/gitops.md. Pinned Argo CD v3.4.8, non-HA, vendored manifest, namespace argocd. Run from ops01 with a clean checkout of infrastructure main at d553643 or a verified descendant (§0 item 9 ops01 provisioning; every kubectl below is the ops01 remote form with KUBECONFIG set to the transferred admin kubeconfig). docs/gitops.md (infrastructure repo) is the authoritative step-by-step procedure and has been reconciled to the deployed state — its steps 1-19 now match what was applied: HTTPS repository Secret, split/server-side CRD apply, current encryption status, drift deferral. The commands below are the same procedure restated; where wording differs, follow docs/gitops.md.

Install (docs/gitops.md steps 1-6); SSH to ops01 (the bastion itself, 92.222.245.91) is DIRECT — no ProxyCommand (reserve the ProxyCommand form for LAN targets behind ops01):

ssh -i /opt/data/.ssh/bigardo ubuntu@92.222.245.91
cd ~/wetopi-infrastructure
git fetch origin && git checkout main && git pull --ff-only
sha256sum kubernetes/gitops/bootstrap/argocd-install-v3.4.8/install.yaml   # optional integrity check
kubectl apply -f kubernetes/gitops/bootstrap/argocd-install-v3.4.8/00-namespace.yaml
# The pinned install.yaml is a 59-document manifest. The CRD
# applicationsets.argoproj.io (1385776 bytes raw; ~341KB serialized) exceeds
# kubectl's 262144-byte client-side last-applied-configuration annotation
# cap, so a broad client-side apply of the whole manifest FAILS on that
# object ("metadata.annotations: Too long"). The other large CRD,
# applications.argoproj.io, serializes to ~191KB and stays under the cap —
# only applicationsets needs special handling. REQUIRED METHOD (validated
# for both first install and re-apply; no CRD-presence branch, no extra
# tools — plain awk): split the CRD out, apply it SERVER-SIDE (idempotent
# whether the CRD is absent or already present), apply the remaining 58
# objects client-side. Same form for installs, re-applies after upgrades,
# and the Phase G rollback path.
awk '
  function flush() {
    if (buf == "") { return }
    if (want) { print buf > "/tmp/argocd-install-appsets-crd.yaml" }
    else {
      if (n++) print "---" > "/tmp/argocd-install-rest.yaml"
      print buf > "/tmp/argocd-install-rest.yaml"
    }
    buf = ""; want = 0; kind = 0
  }
  /^---$/ { flush(); next }
  /^kind: CustomResourceDefinition$/ { kind = 1 }
  /^  name: applicationsets\.argoproj\.io$/ { if (kind) want = 1 }
  { buf = buf $0 "\n" }
  END { flush() }
' kubernetes/gitops/bootstrap/argocd-install-v3.4.8/install.yaml
# sanity check before applying (expected counts on the pinned manifest):
grep -c '^apiVersion:' /tmp/argocd-install-appsets-crd.yaml   # 1
grep -c '^apiVersion:' /tmp/argocd-install-rest.yaml           # 58
kubectl apply --server-side -f /tmp/argocd-install-appsets-crd.yaml
kubectl apply -n argocd -f /tmp/argocd-install-rest.yaml
rm /tmp/argocd-install-appsets-crd.yaml /tmp/argocd-install-rest.yaml
kubectl apply -f kubernetes/gitops/bootstrap/argocd-install-v3.4.8/01-cmd-params.yaml

Validation (steps 7-12): kubectl -n argocd get pods (application-controller StatefulSet 1 replica by pin, applicationset-controller, dex (idle; SSO OPEN), redis, repo-server, server — all Running/Ready); port-forward kubectl -n argocd port-forward svc/argocd-server 9090:443 --address=127.0.0.1; install CLI curl -sSLf -o /tmp/argocd https://github.com/argoproj/argo-cd/releases/download/v3.4.8/argocd-linux-amd64 + chmod +x + sudo mv /usr/local/bin + argocd version --short (expect v3.4.8 client+server); initial password kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d; echo (never commit); argocd login 127.0.0.1:9090 --username admin --insecure. UI from workstation via: ssh -i /opt/data/.ssh/bigardo -L 8443:127.0.0.1:9090 ubuntu@92.222.245.91 (direct to the bastion, NO ProxyCommand — ops01 IS the bastion; the ProxyCommand form stays reserved for LAN targets behind ops01).

[encryption-gate] on the repo credential (steps 13-17) — ORDERED GATE PROCEDURE (status-check BEFORE, ciphertext-verify AFTER, cleanup STOP on failure). The Argo CD repo credential is stored as a Kubernetes Secret, and — key semantic — creating the labeled repository Secret IS the registration: Argo CD auto-discovers any Secret carrying the label argocd.argoproj.io/secret-type=repository, so a create-without-registering sequence does not exist. The gate therefore does not "hold registration until after the check"; it verifies the at-rest protection of the Secret that already exists, per the measured-protection checks of §0 item 8 (the operative gate rule: current protection = measured evidence, not config-template contents). Execute in this exact order:

Check (i) FIRST — before creating the Secret: rke2 secrets-encrypt status on all three CPs, each showing Encryption Status: Enabled / Current Rotation Stage: start / Server Encryption Hashes: All hashes match / Active * AES-CBC aescbckey key (§0 item 8 check (i)). Fails → STOP here, before any credential material enters the cluster; re-verify before continuing.
Create/update the labeled Secret argocd/repo-infra-https exactly per the "Repo credential" block below (protected read -rs token input; stdin-only kubectl -n argocd apply -f - heredoc; label argocd.argoproj.io/secret-type=repository; fields type: git, url, username, password). Creation auto-registers the credential (Argo CD label auto-discovery).
Immediately run check (ii) on the created Secret: raw-etcd ciphertext probe via the approved kubectl-exec etcdctl form (§0 item 8) — get /registry/secrets/argocd/repo-infra-https must return k8s:enc:aescbc ciphertext, NOT the plaintext/base64 of the token.
Check (ii) fails or shows plaintext → CLEANUP STOP: kubectl -n argocd delete secret repo-infra-https (this also removes the auto-registered credential), unset ARGOCD_REPO_TOKEN, then STOP — do not proceed to the root Application — and revoke/rotate the token on GitHub as appropriate (a credential that sat readable in etcd must be treated as exposed). Do not re-attempt from step 2 until the at-rest encryption fault is found, fixed, and re-measured (checks (i) and (ii) green again).
Only after check (ii) passes: the GATE of step 17 — argocd repo list shows https://github.com/wetopi-platform/infrastructure.git CONNECTION STATUS Successful — and only then proceed to the root Application (steps 18-19).

This gate does NOT block the Argo CD install itself (steps 1-12 above), only the private-repo credential. APPLY-TIME VERDICT (2026-08-30, card t_e0cace4b): gate PASSED on measured evidence, in exactly the ordered sequence above — check (i) verified on all 3 CPs BEFORE the credential Secret existed; the labeled Secret argocd/repo-infra-https was then created, which auto-registered the credential via Argo CD's label auto-discovery; raw-etcd check (ii) immediately probed the created Secret and returned k8s:enc:aescbc:v1:aescbckey: ciphertext (685B total, NOT plaintext; argocd/argocd-initial-admin-secret also probed: 316B ciphertext); only after that verification did execution proceed to the repo-list gate (argocd repo list → Successful) and the root Application. The cleanup STOP path was not needed.

Repo credential (steps 13-17; private repo → mandatory before first sync). APPLY-TIME DEVIATION (lead decision, card t_e0cace4b) — the deploy-key path is SUPERSEDED: GitHub org policy DISABLES deploy keys on wetopi-platform/infrastructure (API returns HTTP 422 "Deploy keys are disabled for this repository", org-level setting), so the deployed mechanism is an HTTPS read credential — the token-based repository Secret below, auto-discovered by Argo CD. The exact step-by-step procedure (steps 13-16, secret-safe form) lives in docs/gitops.md in the infrastructure repo, "Private repository credential (apply-phase, before first sync)" — read token from GitHub into a shell variable via read -rs (protected input, never typed into a command or history); create/update Secret argocd/repo-infra-https in the argocd namespace by kubectl -n argocd apply -f - heredoc referencing ${ARGOCD_REPO_TOKEN} (the manifest exists only on stdin — never in Git, never a file on disk) with the required label argocd.argoproj.io/secret-type=repository and required stringData fields type: git, url: https://github.com/wetopi-platform/infrastructure.git, username: box-daxter, password: <the variable>; unset ARGOCD_REPO_TOKEN after; rotation = re-run the same block, cleanup = kubectl -n argocd delete secret repo-infra-https + revoke the token on GitHub. Do NOT instead use inline argocd repo add --password ... — an inline add was tried at apply time and registered an empty credential (STATUS Failed); the labeled Secret is the only registration. When Infisical lands, this credential migrates to the vault contract.

GATE (step 17, mandatory): argocd repo list shows https://github.com/wetopi-platform/infrastructure.git CONNECTION STATUS Successful. Do not proceed to the root Application until this passes. PASSED at apply time (t_e0cace4b) via the Secret auto-discovery above (one recorded correction: an initial inline-password argocd repo add registered an empty credential, STATUS Failed — entry removed; the labeled Secret is now the only registration).

First app sync (steps 18-19):

kubectl apply -f kubernetes/gitops/app.project.yaml -f kubernetes/gitops/app.root.yaml
argocd app list       # one row: root, project=platform, in-cluster, Synced/Healthy
argocd app get root   # STATUS Synced, HEALTH Healthy, source infrastructure main @ kubernetes/platform
argocd app sync root  # no-op sync (target path holds placeholders only)
argocd app wait root --health --sync --timeout 120
kubectl get ns platform    # created by the sync via CreateNamespace=true

Drift check (documented expectations): Git is desired state; manual kubectl edit/apply/delete inside the cluster is drift and shows OutOfSync; root runs automated.prune=true, selfHeal=true so drift self-heals within the reconcile interval and every correction lands in argocd app history root. Drift procedure to record: make a manual change in the platform namespace, confirm argocd app get root shows OutOfSync then self-heals back, confirm the correction in argocd app history root; preferred rollback is always a Git revert (Argo reconciles it) — argocd app rollback <id> exists for emergencies but must be followed by a Git fix or selfHeal re-applies Git state. Note the Argo CD control plane itself is apply-only from pinned manifests (procedural exclusion, re-apply install.yaml after reviewed upgrades). APPLY-TIME DEVIATION — drift check DEFERRED (card t_e0cace4b): structurally untestable at bootstrap. kubernetes/platform/ holds a README placeholder only (0 tracked resources), so the Application has no desired state to diverge from: a probe ConfigMap created manually in the platform ns kept Synced/Healthy (Argo cannot flag an object outside its tree; OutOfSync/selfHeal/prune have nothing to act on). Probe deleted; cluster left clean. The automated.prune=true, selfHeal=true manifest evidence IS factual (verified in the app JSON spec). The check is trivially testable once ≥1 real workload exists under kubernetes/platform/ (e.g. the first app-of-apps child of a later phase) — re-run the drift procedure above then, and test drift by EDITING or DELETING one of the Git-managed objects in the cluster (never by creating another unmanaged object outside the Application's tree, which is invisible to Argo by design).

Apply-time deviations (2026-08-30, card t_e0cace4b — reconciled into this runbook and the repo): 1. Server-side CRD apply (required method): applicationsets.argoproj.io CRD (1385776 bytes raw; ~341KB serialized) exceeds kubectl client-side apply's 262144-byte last-applied-configuration annotation cap — a broad client-side apply of the whole install manifest fails with metadata.annotations: Too long. Required apply method: awk-split that single CRD out of the pinned manifest and kubectl apply --server-side -f it (recorded serverside-applied; idempotent on first install and re-apply — no CRD-presence branch needed); the remaining 58 objects stay client-side apply per this runbook. Any re-apply of the pinned manifest must use the same split form. 2. HTTPS credential instead of SSH deploy key (lead decision): both kubernetes/gitops/app.project.yaml and app.root.yaml were applied with repoURL/sourceRepos git@github.com:wetopi-platform/infrastructure.git → https://github.com/wetopi-platform/infrastructure.git (deploy keys disabled at org level); credential = Secret argocd/repo-infra-https (see "Repo credential" above). The infrastructure repo is reconciled to the applied state by PR (infrastructure) — repoURL lines updated to HTTPS; the app yamls now match what is deployed. 3. Drift self-heal check deferred — structural reason and re-test condition as recorded above.

Execution evidence (t_e0cace4b, 2026-08-30): Argo CD v3.4.8 installed (ops01 checkout at main f841ae7, PIN SHA-256 e223122b… matched docs/gitops.md); 7/7 pods Running/Ready; CLI v3.4.8 client+server match; encryption gate PASSED on measured AES-CBC evidence in the ordered sequence (check (i) pre-credential, check (ii) ciphertext on the created repo-infra-https Secret, cleanup path not needed); repo credential Successful; argocd app list → one row root, project platform, in-cluster, https://github.com/wetopi-platform/infrastructure.git @ kubernetes/platform @ main, SYNCPOLICY Auto-Prune; first sync Succeeded (no-op, placeholders only), root Synced to main (b610209) Healthy; ns platform Active; history ID 0.

STOP: pods not all Running; [encryption-gate] measured-protection checks failing per the ordered procedure (check (i) failing at step 1, or check (ii) failing/showing plaintext at step 3 → cleanup STOP: delete argocd/repo-infra-https, unset ARGOCD_REPO_TOKEN, revoke/rotate the token, stop — §0 item 8 and the ordered gate procedure above); argocd repo list not Successful; root app not Synced+Healthy after argocd app wait. (Drift-check STOP superseded by the deferral above: no drift gate applies while kubernetes/platform/ tracks 0 resources — re-test when the first workload lands.)

Rollback story (general, per phase)¶

On ANY phase failure — stop, do not proceed; read the playbook/task failure output first (all plays run any_errors_fatal); the run is serial, so a failed phase leaves the cluster in a known partial state. Order of escalation: (1) fix forward and re-run the phase playbooks (roles are idempotent, serial-safe); (2) per-node rollback (below); (3) etcd snapshot restore via the cluster-reset flow (systemctl stop rke2-server on all CPs → rke2 server --cluster-reset --cluster-reset-restore-path=<PATH-TO-SNAPSHOT> on cp01 → start cp01 → rm -rf /var/lib/rancher/rke2/server/db/ + start on cp02/cp03 → re-verify quorum + /readyz — see Phase F6-3, INCLUDING its reset-failure rule: a failed reset is an immediate STOP-and-escalate condition, NOT a retry — reset-flag blocks repeated resets; there is NO rke2 etcd-snapshot restore subcommand — and the F6 STOP prerequisites P1-P4 gate ANY cluster-reset restore, including from this rollback path) for cluster-state corruption; (4) Proxmox qm rollback <vmid> pre-rke2-apply (last resort — cluster-level state like etcd membership makes node-level rollback of ONE CP risky once quorum exists: prefer etcd member removal for a node you are rebuilding rather than snapshot rollback of a single CP mid-quorum; snapshot rollback is safest applied to ALL CPs together or before Phase B).

Per-phase specifics (from parent docs; the per-phase STOP blocks above remain the authoritative inline version):

B/C (from the CP runbook): stop RKE2 systemctl stop rke2-server; per node serially remove /etc/rancher/rke2, /var/lib/rancher/rke2, /var/lib/kubelet; keepalived may stay (role idempotent) or be removed; re-run the playbook from scratch after regenerating the token; do NOT manually delete etcd members unless removing a node permanently — then remove the member with the same kubectl-exec etcdctl form (Phase B gate) with member remove <ID>. Rebuild-a-node criterion: node whose config/state is unrecoverable (repeated join failures, corrupt etcd data dir) — snapshot-rollback or recreate via tofu, then re-run the CP join play.
D (from the worker runbook): per node serially systemctl stop rke2-agent, remove /etc/rancher/rke2, /var/lib/rancher/rke2, /var/lib/kubelet, then kubectl delete node rke-platform0X; re-run the workers playbook after fixing the root cause (idempotent).
E: helm uninstall cilium --namespace kube-system (documented; no other CNI behind it — replacement plan required).
W: no install state to roll back — if interrupted mid-drill, uncordon (CP-local form) and re-verify the node Ready; a worker that fails W-2 recovery follows the D bullet (agent wipe + node delete + re-join).
F: F0 — revert the tfvars PR (reviewed); routing — revert the natgw vars PR to default-off + re-run bootstrap.yml; CSI inverse (uninstall chart, delete StorageClass) as authored above.
G: GitOps plane state is rebuildable from Git — re-apply the pinned manifests using the SAME split form as the install (awk-split the applicationsets.argoproj.io CRD out, kubectl apply --server-side -f the CRD file, kubectl apply -n argocd -f the remaining-objects file; a broad client-side apply of the whole install.yaml fails on that CRD by design), re-register the repo credential (the argocd/repo-infra-https labeled Secret per docs/gitops.md steps 13-16), re-sync; app-level rollback = Git revert (argocd app history root records syncs). Full cluster loss = rebuild from the tofu/ansible phases, then this runbook top to bottom.

Definition of done — master plan §35.1/35.2 mapped to concrete commands¶

Plan item	Concrete check in this runbook
§35.1 pérdida de un CP no rompe API	Phase C one-CP-outage test + Phase F6-2 (`systemctl stop rke2-server` on cp02, `curl -k https://10.72.0.20:6443/readyz` → ok)
§35.1 etcd quorum correcto	kubectl-exec etcdctl member list == 3 members (Phase B/C gates + F6-1)
§35.1 nodes reproducibles desde IaC	phases B-E run purely from tofu-generated inventory + ansible roles (re-run of the same playbooks reproduces the state; snapshot rollback to pre-rke2 then re-run = full reproduction test)
§35.1 no customer workloads en CP	workers playbook play 2 taint enforcement (`node-role.kubernetes.io/control-plane=true:NoSchedule` on all CPs) + Phase C taint record + `get pods -o wide` scan + D2 scheduling gate placement
§35.1 snapshot funciona / restore probado	F6 STOP prerequisite P2 (`rke2 etcd-snapshot save --name pre-addons-fase6` + `rke2 etcd-snapshot ls`, external copy outside cp01's failure domain) + P3 (SHA-256 match on cp01 and the external copy), then F6-3: the EXECUTED cluster-reset restore test (`systemctl stop rke2-server` all CPs → `rke2 server --cluster-reset --cluster-reset-restore-path=<PATH-TO-SNAPSHOT>` on cp01 → start cp01 → `rm -rf /var/lib/rancher/rke2/server/db/` + start on cp02/cp03 → re-verify quorum 3/3 + `/readyz` + the post-restore journal checks on all three CPs). DoD maps "restore probado" to THIS executed-test evidence — snapshot-save-only is NOT restore probado. If the restore test cannot be executed, this DoD item is UNMET and the runbook mandates STOP before all subsequent phases (E onward).
§35.2 Pod-to-Pod + Pod-to-Service	`cilium connectivity test` (Phase D2, final 6-node topology)
§35.2 DNS	Phase D2 DNS gate (answer parsed into 10.68.0.0/16) + Phase D2 `dns-utils` nslookup (final topology)
§35.2 Hubble visibility	`hubble status` + `hubble observe` (Phase D2, final 6-node topology — real flows + VERDICTs)
§35.2 no CIDR collision	gate (f) `ip -4 route` review + `kubectl get ciliumnodes.cilium.io` podCIDR ⊂ 10.64.0.0/14 (Phase D2)
§35.2 Ceph reachable desde nodes	Phase F1 connectivity block (ping/traceroute/nc 3300+6789) + PVC Bound
§35.2 DSR demostrado por packet capture	NOT covered by phases 0-G (edge/DSR is later work per the master plan); recorded as an explicit gap
§35.3 Storage: PVC create/persistence/pod recreation/failure test	Phase F2 steps (a)-(d)
§35.3 Storage: performance	Phase F2 step (e)
§35.4 GitOps: Argo reconcilia desde Git	first sync block
§35.4 GitOps: drift detectable	drift procedure
§35.4 GitOps: Git conserva estado deseado	selfHeal + Git-revert rollback rule
§35.4 GitOps: rollback probado	Git revert is exercised only in a real incident — recorded as "procedure documented, not yet exercised"
§35.1/§35.2 node drain + reboot recovery	Phase W (W-1 drain, W-2 reboot + Ready return; former F6-4/F6-5) — the separately-named post-D2 phase, executed on the final 6-node topology in the executable order B → C → STOP → F6-1..3 → E → D → D2 → W → F → G. The DoD maps drain/reboot to this POST-D2 evidence, NOT to pre-addon acceptance: §33 places these Fase-6 items before Fase 7 (Cilium), but ADR-0005 requires Cilium as the immediate first addon with no pre-Cilium worker-Ready gates — with `cni: none` the drain/reboot gates are unexecutable before Cilium exists (see the banner for the §33/ADR-0005 tension and its explicit resolution). F6-1..3 (quorum, CP failure, snapshot + REAL restore test) remain the PRE-addon CP-only evidence.

Runbook complete when every phase gate above has a dated PASS entry in the apply log.

Provenance¶

Execution status (2026-08-30, post-Phase-D): the live apply has RESUMED and executed B, C, F6-1..3, E, and D fully green (details in the status banner; cluster 6/6 nodes Ready). Historical note (superseded): the 2026-08-29 23:17-23:26 UTC attempt (task t_a88bb36b, wetopi-ops) was HALTED BEFORE PHASE B and FROZEN (operator directive #38) after failing at rke2-server : Create the RKE2 server configuration with Destination directory /etc/rancher/rke2 does not exist on rke-cp01 — neither rke2 role created the config dir (byte-identical defect in rke2-agent). Infrastructure PR #6 (fix/rke2-config-dir-create, head 3ff2ee8, merged main d553643, 2026-08-29 23:56 UTC) is the fix: config-dir creation (root:root 0750) before templating in both roles. Residual pre-halt state on cp01 (keepalived installed+active, RKE2 binaries present, /etc/rancher absent, rke2-server inactive) was absorbed cleanly by the resumed run — the roles are idempotent (RKE2 install guarded by creates:). The former resume gates (infrastructure checkout at d553643 or verified descendant per the §0 STEP 0 gate, harness V2 validation, explicit lead unblock) were all satisfied on 2026-08-30 before resume; they remain historical prerequisites of that resume, not open conditions.
Verified against merged heads on 2026-08-30 (~00:15Z): infrastructure main @ d553643 (PR #6 merge — config-dir fix for both RKE2 roles; prior PRs #1 8f3e0e8, #2 e283e24, #3 76782dc, #4 b15a087, #5 8d836e8 all merged — no open-PR references remain), docs main @ 5740404 (ADR-0005 and platform runbooks in main).
Executable order (identical in the banner, the F6 heading + STOP rule, the Phase W heading, the D2 handoff, the DoD mapping, and here): B → C → STOP → F6-1..3 → E → D → D2 → W → F → G (STOP = the mandatory post-Phase-C pause before any F6-2/F6-3 server stop, gated by the F6 STOP prerequisites — historically the frozen-apply boundary of the superseded 2026-08-29 halt; W = Phase W, the separately-named worker lifecycle drill — former F6-4/F6-5). Master plan §33 orders Fase 5 (workers) and Fase 6 (incl. node drain/reboot) before Fase 7 (Cilium); ADR-0005 requires Cilium as the immediate first addon with NO pre-Cilium worker-Ready/DNS gates. Tension and resolution: with cni: none, workers cannot pass Ready (and therefore drain/reboot) gates before Cilium exists; the resolution adopted by this runbook is that Fase-6's CP-only gates (F6-1..3) run before the first addon per §33, and the worker-lifecycle drill is the separately-named Phase W landing after D2 on the final 6-node topology; the DoD maps drain/reboot to that post-D2 evidence, never to pre-addon acceptance. Phase W is not counted as completed pre-addon acceptance.
Playbook/role facts re-read on final main: ansible/playbooks/{rke2-cp,rke2-workers,rke2-workers-validate,cilium,ceph-csi,bootstrap}.yml, ansible/roles/{keepalived,rke2-server,rke2-agent,nat_gateway}/, kubernetes/{cilium,ceph-csi,gitops}/, docs/gitops.md, scripts/{gen-rke2-token.sh,render-ansible-inventory.sh}, tofu/proxmox/environments/prod/terraform.tfvars.example.
Execution environment (§0 item 9): the controller is the workstation (direct evidence from the t_a88bb36b apply: ops01 had no Ansible venv/repository; the playbooks ran from the workstation over the bastion); ops01 requires explicit provisioning (checkout, venv, inventory, helm/kubectl/cilium/hubble CLIs, and a transferred admin kubeconfig with the server endpoint rewritten to the VIP) before Phase E — needed because cilium.yml runs its Helm op on the operations host with connection: local. CP-local kubectl invocations use the RKE2-bundled binary path /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml, matching the playbooks' own rke2_kubectl form.
secrets-encryption: MEASURED PROTECTED, per the single operative gate rule (§0 item 8) — current protection is determined by measured evidence (default-on AES-CBC on this RKE2 build: secrets-encrypt status Enabled on all 3 CPs; raw etcd probes of both argocd Secrets returned k8s:enc:aescbc ciphertext), NOT by config-template contents. The explicit secrets-encryption: true role-template line is still absent from merged main (git grep secrets-encryption → empty) — it is the separately tracked future hardening/rotation prerequisite (must merge via its own reviewed PR before any key rotation), not an unlock condition for F2/G while the measured checks pass. F2's csi-rbd-secret gate (when Phase F executes) re-verifies the same measured checks on that Secret.
Design: docs/design/master-architecture-plan.md §33/§35 (Fase 6 = acceptance; CP-only portion BEFORE Fase 7 Cilium; worker-lifecycle portion after D2 per ADR-0005 — see the executable-order bullet above); docs/decisions/ADR-0004, docs/decisions/ADR-0005.
Restore procedure and reset-flag semantics: RKE2 docs, rancher/rke2-docs docs/datastore/backup_restore.md (etcd-snapshot subcommands = delete/ls/prune/save ONLY; restore = rke2 server --cluster-reset --cluster-reset-restore-path=<PATH> multi-server flow; reset creates /var/lib/rancher/rke2/server/db/reset-flag preventing repeated resets in succession, deleted only by a normal start — verified 2026-08-30 against the main branch of rancher/rke2-docs). Flag lifecycle additionally verified 2026-08-30 against the k3s source vendored by the pinned rke2_version: v1.36.3+rke2r1 (k3s commit 4ca29faac89e via rke2 go.mod; pkg/cluster/managed.go: the flag check is bypassed when --cluster-reset-restore-path is set and the flag blocks only a bare reset; pkg/etcd/etcd.go Reset(): the flag is written only after the snapshot-restore step succeeds).
Secrets-encryption behavior: RKE2 docs, rancher/rke2-docs docs/security/secrets_encryption.md (auto-generated AES-CBC key + /var/lib/rancher/rke2/server/cred/encryption-config.json; rke2 secrets-encrypt status output shape: Encryption Status / Current Rotation Stage / Server Encryption Hashes / key table).