Skip to content

Wetopi RKE2 Operations Handbook

Version: 0.2 — corrected per approver REQUEST_CHANGES (task t_f1aeb7f4 → fix t_6dcba59b), re-verified live 2026-08-30 03:37-03:40 UTC Status: command blocks in this handbook were re-verified read-only on the live cluster as ubuntu over the documented SSH path (rc recorded for each in the task evidence; see "Verification method" below). This is operational memory distilled from bring-up — it does not replace the runbook. Sources of truth (precedence): docs/platform/rke2-apply-runbook.md (merged main 7cc387a) → ADR-0004 → this handbook. Where this handbook and the runbook disagree, the runbook prevails. One known divergence is recorded in §3.2 (etcd certificate form: the runbook's kubectl-exec form references client.crt, which does not exist inside the etcd container at v1.36.3+rke2r1 — use the server-client form below and see the reconciliation note there).

Current state (dated observation — 2026-08-30 03:37 UTC, re-read via kubectl get nodes -L wetopi.io/role on cp01): 6/6 nodes Ready (3 CPs + 3 platform workers, all workers labeled wetopi.io/role=platform), RKE2 v1.36.3+rke2r1 on all nodes, Cilium v1.20.1 present on all 6 nodes (6 CiliumNode objects). Phases B, C, F6 (drills), E (Cilium) and D (workers) are COMPLETE; Phase D2 (full 6-node validation, task t_69d9d0bf) is IN FLIGHT (gates A-E green at 03:07-03:12 UTC, cilium connectivity test still running). Live state moves while D2 runs — re-read before relying on it.

Verification method (what "verified" means here): every command block marked verified below was executed read-only as the ubuntu user over the documented SSH path (§2) at 2026-08-30 03:37-03:40 UTC, with rc recorded. On the CPs/workers ubuntu has (ALL) NOPASSWD: ALL sudo (sudo -l evidence; sudo -n true → rc=0 works in non-interactive contexts — only sudo -v specifically demands a tty for password validation, rc=1). The rc=141 occasionally seen in evidence next to systemctl status invocations is SIGPIPE from piping the pager output into head — systemctl status rke2-server itself returns rc=0 when not piped (verified rc=0 unpiped).

1. Cluster identity
Distribution RKE2 v1.36.3+rke2r1 (pinned; binary rke2 version → v1.36.3+rke2r1 (c4f306e), kubelet v1.36.3+rke2r1 on every node)
Topology 3 CP (embedded etcd) + 3 platform workers + external bastion/recovery node
CPs rke-cp01 = 10.72.0.21, cp02 = .22, cp03 = .23 (LAN, VLAN 4)
Workers rke-platform01 = 10.72.0.41, 02 = .42, 03 = .43
Bastion wetopi-ops01 = 10.72.0.10 (WAN 92.222.245.91) — ProxyJump target + separately provisioned phase-E host (NOT the Ansible controller; see §2.1)
API VIP 10.72.0.20 (keepalived on CPs; server URL https://10.72.0.20:9345, API https://10.72.0.20:6443)
CNI none at bootstrap (by design) → Cilium+Hubble as first addon (phase E; live: cilium v1.20.1 DaemonSet 6/6)
Cluster CIDRs pods 10.64.0.0/14, services 10.68.0.0/16

2. Access patterns (verified 2026-08-30 03:37 UTC)

# Bastion direct (public SSH is rate-limited by in-VM nftables — by design)
ssh -i /opt/data/.ssh/bigardo ubuntu@92.222.245.91

# LAN VMs via ProxyCommand — NOTE: -J does NOT work (drops the -i for the jump host)
ssh -i /opt/data/.ssh/bigardo -o IdentitiesOnly=yes \
  -o ProxyCommand="ssh -i /opt/data/.ssh/bigardo -o IdentitiesOnly=yes -W %h:%p ubuntu@92.222.245.91" \
  ubuntu@10.72.0.21

2.1 Controller and host roles (corrected — matches runbook §0 item 9)

  • Controller = the workstation (the box that holds the SSH private keys and runs the Ansible playbooks for phases B/C/D/D2). Per runbook §0 item 9 it REQUIRES the pinned Ansible venv (requirements-dev.txt pins ansible-core==2.21.3; verify .venv/bin/ansible-playbook --version reports 2.21.3) to run the B/C/D/D2 playbooks — the playbooks are invoked as .venv/bin/ansible-playbook from the infrastructure checkout on the workstation. Live controller caveat (workstation, 2026-08-30): no sudo binary, private key /opt/data/.ssh/bigardo present. The runbook's controller caveat is about the workstation: become: true delegated to localhost fails there (sudo: not found) — the inventory fix is an ungrouped localhost host with ansible_connection: local, ansible_become: false.
  • wetopi-ops01 is NOT the controller. Per runbook §0 item 9 it is the ProxyJump target (bastion) plus a separately provisioned phase-E host (Cilium Helm op, D2 cilium/hubble CLI acceptance, F2/G commands). Live ops01 (2026-08-30 03:37 UTC): sudo IS present at /usr/bin/sudo (passwordless for ubuntu: sudo -n true → OK), Ansible venv PRESENT (~/wetopi-infrastructure/.venv/bin/ansible → core 2.21.3), infrastructure checkout at ~/wetopi-infrastructure head ce065f2, and the transferred admin kubeconfig present at ~/secrets/rke2-admin-kubeconfig (mode 600, 2954 bytes). kubectl lives at ~/bin/kubectl (v1.36.3, on PATH for login shells; use the absolute path or export PATH=$HOME/bin:$PATH in scripts). Also on ~/bin: cilium-cli v0.19.1, hubble v0.13.6, helm v3.19.0.
  • Ops quick-checks on ops01 (verified 2026-08-30 03:39 UTC, remote form, rc=0): 
    export KUBECONFIG=~/secrets/rke2-admin-kubeconfig
kubectl get nodes                     # 6/6 Ready, v1.36.3+rke2r1
kubectl -n kube-system get pods | grep -E 'etcd-|apiserver|coredns'
    (kubectl get nodes -L wetopi.io/role shows the 3 workers carrying platform.)

3. Verified commands (v1.36.3+rke2r1)

All blocks in this section executed read-only as ubuntu over the §2 SSH path, rc recorded (2026-08-30 03:37-03:39 UTC).

3.1 CP-local kubectl (the form the playbook gates use — REQUIRES sudo)

/etc/rancher/rke2/rke2.yaml is mode 600 root:root; the bundled kubectl is 755 but cannot read the kubeconfig as ubuntu: - plain kubectl get nodes as ubuntu → command not found (rc=127; kubectl is not in PATH) - KUBECONFIG=/etc/rancher/rke2/rke2.yaml /var/lib/rancher/rke2/bin/kubectl get nodes as ubuntu → error loading config file ...: permission denied (rc=1) - without KUBECONFIG, the bundled kubectl falls back to localhost:8080 and fails (rc=1)

The working CP-local form therefore prefixes sudo (verified, rc=0, 6/6 nodes listed):

sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes
sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes -L wetopi.io/role
sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n kube-system get pods

Health via the VIP (client kube-apiserver certs on the host, verified 200-equivalent ok):

sudo curl -s --cacert /var/lib/rancher/rke2/server/tls/server-ca.crt \
  --cert /var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt \
  --key  /var/lib/rancher/rke2/server/tls/client-kube-apiserver.key \
  https://10.72.0.20:6443/readyz          # → ok (rc=0)

# same check through the bundled kubectl (verified rc=0 → "ok")
sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml \
  --server=https://10.72.0.20:6443 get --raw /readyz

(Ansible runs these as root via become: true on the CP target, so the playbooks' un-prefixed form still works there — the sudo-less forms above are the human quick-check forms.)

3.2 etcd member list (THE version-specific lesson)

The live-working form (verified 2026-08-30 03:37 UTC, rc=0, 3 members all started):

sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n kube-system \
  exec etcd-rke-cp01 -- \
  etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt \
  --cert=/var/lib/rancher/rke2/server/tls/etcd/server-client.crt \
  --key=/var/lib/rancher/rke2/server/tls/etcd/server-client.key \
  member list

Pitfalls found live (do not repeat): - The etcd container neither contains client.crt/client.key nor the env binary. The runbook's kubectl-exec form (rke2-apply-runbook.md, e.g. its Phase B gate) passes env ETCDCTL_API=3 etcdctl ... --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt --key=.../client.key; executed exactly it fails FIRST at the env prefix itself (OCI runtime exec failed: exec: "env": executable file not found in $PATH, rc=1, verified 2026-08-30 03:39 UTC); and with the env prefix removed but client.crt kept, it fails inside the container with Error: open /var/lib/rancher/rke2/server/tls/etcd/client.crt: no such file or directory (rc=128, verified 03:37 UTC). The HOST directory /var/lib/rancher/rke2/server/tls/etcd/ does contain client.crt/client.key (plus peer-* and server-client.* pairs) — both absences are container-path/mount facts, not host facts. Until the runbook is amended (open PR docs#8 owns that file — follow-up noted, do not edit it here), the server-client form above is the operator-facing source of truth for v1.36.3+rke2r1. It has also been re-verified WITH the env ETCDCTL_API=3 prefix and server-client certs that the env prefix is the sole failure there (rc=1, same missing-env error) — etcdctl alone with full flags is the form that works; the minimal container also has NO sh/ls, so use kubectl-exec passing etcdctl directly, never shell utilities. - member list plain output includes a table header — for asserts use member list -w json and parse .members | length (verified rc=0, 3 members in JSON; the merged fix, infra PR#7 6d72da5).

3.3 Worker-side facts (verified on rke-platform01, 2026-08-30 03:38 UTC)

  • The agent's kubelet kubeconfig lives at /var/lib/rancher/rke2/agent/kubelet.kubeconfig (verified: PRESENT, mode 600 root:root, 464 bytes — contents never read for this check). Sibling agent kubeconfigs in the same dir: kubeproxy.kubeconfig, rke2controller.kubeconfig, plus client-kubelet.crt/.key, client-kube-proxy.crt/.key, client-rke2-controller.crt/.key.
  • On workers the same CP-local negative facts hold: plain kubectl rc=127; KUBECONFIG=/etc/rancher/rke2/rke2.yaml + bundled kubectl rc=1 (permission denied; no server kubeconfig on workers).
  • sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /var/lib/rancher/rke2/agent/kubelet.kubeconfig get nodes fails with Forbidden: User "system:node:rke-platform01" ... only its own Node object (rc=1) — the kubelet kubeconfig is node-scoped, NOT an admin credential. Never treat it as one.

3.4 keepalived / VIP

  • Health-check tied to rke2-server liveness → no VIP holder until a healthy API exists (a holder-less VIP at phase A/B is CORRECT, not drift).
  • Priority chain: cp01 150 > cp02 100 > cp03 50 (deterministic failover).
  • Failover drill PASSED live (F6-2): cp01 down → VIP to cp02 → cp01 reclaims (preemption) — seconds-scale, verified during B-C.
  • VIP /readyz through the surviving quorum after a one-CP outage: PASSED.
  • Current holder check (verified on cp01, rc=0): ip -4 addr show eth0 | grep -F 10.72.0.20 — the holder shows inet 10.72.0.20/23 scope global secondary eth0.

4. Bootstrap semantics learned

  • NotReady nodes are EXPECTED pre-CNI. Never "fix" them with workarounds; Cilium (phase E) is the fix. CoreDNS cannot start without a CNI — the "workers Ready before Cilium" ordering is an impossible deadlock (caught by approver ESCALATE; runbook now orders E before D).
  • The playbook is idempotent: after the gate-fix merge we re-ran rke2-cp.yml with cp01 already live — no reinstall, gates re-evaluated, joins proceeded.
  • Join token: 0600 file on the controller, read at runtime, no_log: true, never in Git. Token file existence is a pre-flight assert.
  • Ansible gate tasks must use the argv list form (ansible.builtin.command: argv: [...]) — cmd: with a YAML list becomes a literal string argv[0] (found live at phase B).
  • Config dir /etc/rancher/rke2 is created by our roles since infra PR#6 (d553643) — the official installer does NOT create it; without the role task, templating fails.
  • Infrastructure main at 1e8e689+ (d553643 + gate fix) is the verified baseline for all playbooks; live ops01 checkout sits at ce065f2 (kube-proxy gate fix), phase D/D2 apply used f841ae7.

5. Disaster-recovery semantics (F6 — from the runbook, which prevails)

The full procedure lives in the runbook (Phase F6, merged main 7cc387a): P1-P4 hard STOP prerequisites (external verified snapshot outside cp01's failure domain, independent SHA-256 match, quorum+/readyz evidence, explicit human downtime authorization) gate any server stop. The cluster-reset restore (rke2 server --cluster-reset --cluster-reset-restore-path=<snap>) is DISRUPTIVE — all servers stop, peer DBs are replaced, ~10-20 min full API downtime, NOT idempotent, and the reset-flag must NEVER be deleted manually.

If the reset FAILS: STOP-and-escalate. This is the runbook's fail-closed rule, not a retry ladder. The failure state is reset-flag- and DB-state-dependent: (a) the flag is written only after the snapshot-restore step succeeds (early failures leave NO flag; late failures leave it present), (b) the flag blocks only a bare --cluster-reset (not one with --cluster-reset-restore-path set), and (c) the pre-restore DB is moved aside to db/etcd-old-<timestamp>/ — a failed attempt may leave the DB dir absent or partially restored.

On any failure (reset invocation fails, expected output missing, DB not re-provisioned, peers not rejoining, quorum not re-established, /readyz not OK): 1. Do NOT retry blindly — all CPs are stopped; escalate to the lead with full output; the cluster stays STOPped. 2. The ONLY documented recovery actions are lead-approved, apply-log-recorded: (i) normal start on cp01 (systemctl start rke2-server) — the documented flag-clearing mechanism, which starts from whatever DB state the failed attempt left (restored single member if the restore step succeeded; may fail to start or come up uninitialized if it did not — an uninitialized cluster is an evidence-gathering STOP point, never a "fresh etcd is fine" outcome); (ii) a second reset attempt ONLY after a successful normal start or a coordinated rollback decision; (iii) coordinated Proxmox rollback (qm rollback <vmid> pre-rke2-apply on all CPs); (iv) Rancher support with exact version + logs. 3. A normal start is NOT a "restore to fresh etcd" escape hatch. Whatever DB state remains is what starts; cluster-state loss is only accepted as an explicit, lead-approved, recorded decision (with the F6 evidence trail), never as a default outcome of a failed reset. Now that Cilium and six nodes are live, treating "fresh etcd" as a viable fallback would silently discard cluster state.

6. Ops quick-reference (CP commands verified on cp01 and the worker-only rke2-agent command on worker rke-platform01 at 2026-08-30 03:39-03:40 UTC; rc's: kubectl/grep/journalctl lines and the worker systemctl status rke2-agent rc=0; on cp01 systemctl status rke2-agent returns rc=3 — the unit is absent on CPs, which is expected)

# define once per shell (CP-local admin form requires sudo — see §3.1)
RKE2_KUBECTL="sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml"
$RKE2_KUBECTL get nodes                                     # 6/6 Ready
$RKE2_KUBECTL get nodes -L wetopi.io/role                   # workers labeled platform
$RKE2_KUBECTL -n kube-system get pods | grep -E 'etcd-|apiserver|coredns'
# etcd health/members: use the §3.2 server-client kubectl-exec form (NOT client.crt)
# services
systemctl status rke2-server    # CPs
systemctl status rke2-agent     # workers ONLY (after phase D) — on CPs the unit is absent: rc=3 on cp01 (expected); rc=0 on worker rke-platform01
# keepalived / VIP holder
systemctl status keepalived; ip -4 addr | grep -F 10.72.0.20
# logs
journalctl -u rke2-server -n 50 --no-pager   # readable as ubuntu on CPs (rc=0 unprivileged)
# snapshot management (host binary; see runbook F6 for the full gated procedure)
sudo rke2 etcd-snapshot ls   # verified rc=0: lists pre-addons-fase6 snapshot; runbook F6 form: save --name <name>, then external copy + dual SHA-256

Remote form from ops01 (for addon/D2/F2/G operations; verified rc=0, 2026-08-30 03:39 UTC): 

export KUBECONFIG=~/secrets/rke2-admin-kubeconfig   # ops01 ~/bin/kubectl v1.36.3
kubectl get nodes -L wetopi.io/role

7. What is NOT in this handbook

Full phase-by-phase apply procedure → the merged runbook (docs/platform/rke2-apply-runbook.md). ADR-0004 for the endpoint decision and its rejection of alternatives. The Ansible roles themselves are the source for node config.

8. Changelog

  • 0.2 (2026-08-30): approver REQUEST_CHANGES round 1 (t_f1aeb7f4, fix t_6dcba59b) — all six blocking items: (1) every command block re-verified live with the sudo CP-local form / kubectl-exec etcd form, rc recorded; (2) controller/ops01 roles corrected to runbook §0 item 9 (controller = workstation; ops01 = ProxyJump target + provisioned phase-E host with sudo, venv, checkout ce065f2, kubeconfig); (3) worker kubelet kubeconfig path /var/lib/rancher/rke2/agent/kubelet.kubeconfig added, verified on rke-platform01; (4) dated current-state statement added (6/6 Ready, B/C/F6/E/D complete, D2 in flight); (5) reset-failure text replaced with the runbook's fail-closed, flag/DB-state-aware STOP-and-escalate semantics; (6) etcd cert guidance reconciled — server-client kubectl-exec form is the operator source of truth, client.crt absence scoped to the etcd container mount, runbook divergence recorded as a follow-up (do not edit the runbook here).
  • 0.1 (2026-08-30): first edition from live bring-up evidence: B-C quorum green (3 CPs, VIP failover + outage drills passed), gate-defect class (argv/certs/json-count, PR#7), no-CNI NotReady semantics, F6 drill procedure with P1-P4 gates.