Skip to content

PROC-PLATFORM-TAKEOVER: Taking over the platform

The platform map is ARCH-PLATFORM-TOPOLOGY — read it first. This guide is the procedure: access, truth sources, reading order, day-1 checks, the operations you will run most often, where to escalate, and what is never allowed. It links out for everything it does not need to duplicate.

Purpose and when to use

Use this when you are newly taking over operations of the platform — on a shift, a rotation, or an emergency handover. It is deliberately a composed guide of steps (the one allowed exception to the one-concept-per-pill rule); every fact it does not restate lives in a linked pill or runbook.

(a) Access — who has which credential, by location

Never search for a credential's value in this repo, the infrastructure repo, a ticket or a chat — only its location is documented (DocOps §8).

Access Who Credential location (never the value) Notes
SSH to the fleet operator holding the bigardo key the workstation's ~/.ssh/bigardo (key pair issued out-of-band; the public side is pushed to VMs by the ssh_keys role) ops01 (ubuntu@92.222.245.91) is the bastion; all LAN VMs are reached through it. ops01 itself holds no private keys — it cannot SSH to the LAN by itself.
Vault (human UI) the taking-over operator native Infisical account (username + password + TOTP) — the account is provisioned by the current operator at handover time; no value is documented https://vault.wetopi.net → project wetopi-harness. The UI's own login+TOTP is the gate (no basic-auth).
Vault (machine identities / Universal Auth) agent profiles / automation only client credentials live under project wetopi-harness in the Vault itself private-network-only: http://vault.vault.svc.cluster.local — never the public route, which fail-closes by design (ADR-0009).
kubectl on the cluster operator via ops01 kubeconfig fetched at apply-time from rke-cp01 (the ops_kubectl role; nothing persisted in Git) see ARCH-PLATFORM-RKE2 for the API VIP.
OpenTofu (PROD) single writer, by explicit task .env.proxmox on the phase-1 controller (0600); Cloudflare token on ops01 at /home/ubuntu/secrets/.cloudflare-wetopi-net.env (0600) pulled over SSH per session, never written to disk in the repo. State: terraform.tfstate 0600, never in Git (ARCH-INFRA-PROVIDERS).
Proxmox API the same .env.proxmox Proxmox VE HTTPS API token (env var) no SSH to hypervisors — API only.
Cloudflare DNS the Cloudflare token above Zone DNS Edit + Zone Read (both required) records are tofu-managed — do not edit in the dashboard.

Handover checklist for credentials: at the handover meeting the outgoing operator must (1) verify you can reach ops01 and one LAN node with your bigardo key, (2) hand you a working Vault human account (or provision one you can then rotate), (3) confirm the single-writer tofu lock is with you. If any of these fail, stop and escalate — do not improvise credentials.

(b) Sources of truth

Authority order (DocOps §1) — when sources disagree, the live one wins:

  1. Live state — kubectl (via ops01), Proxmox API, Cloudflare API, Ceph. "What IS happening."
  2. Desired state — the infrastructure repo (IaC: tofu/, ansible/, kubernetes/). A merge is a deploy.
  3. Curated docs — this repo (docs/). The map: ARCH-PLATFORM-TOPOLOGY.
  4. Kanban / work ledger — execution state of tasks (who is working what, which card a change belongs to). Live board, not documentation.
  5. tfstate — terraform.tfstate (0600, phase-1 controller) is the OpenTofu claim on PROD. Lost state → RUN-TOFU-STATE-LOSS, never a speculative apply.

(c) What to read, in order

  1. ARCH-PLATFORM-TOPOLOGY — the three-layer map; every node links to its pill.
  2. Decisions — ADR index; read ADR-0003 (tofu vs Ansible split), ADR-0004 (API endpoint), ADR-0005 (Cilium first), ADR-0006 (edge v1), ADR-0007 (Ceph storage-native), ADR-0009 (vault).
  3. Layer pills — infrastructure section (OVH/network → Proxmox → Ceph → providers), then platform section (RKE2 → Cilium → edge → GitOps).
  4. Service fiches — services section: one per deployed service with deploy path, dependencies and honest live state.
  5. Runbooks — runbooks index; the two load-bearing ones: RKE2 apply & restore (cluster bring-up/DR) and RUN-TOFU-STATE-LOSS (IaC state recovery).

(d) Day-1 operations — verify the platform is healthy

All from ops01 (kubeconfig in ~/.kube/config or fetched per the ops_kubectl model — never copied into Git):

  1. Cluster nodes: kubectl get nodes -o wide — expect 8/8 Ready (cp01/02/03, platform01-03, wan01/02; IPs per ARCH-PLATFORM-RKE2).
  2. GitOps plane: kubectl -n argocd get applications — expect root, cert-manager, docs-site, argo-edge Healthy+Synced; vault Synced — its Degraded health is the documented false positive of the fail-closed deny rules (verify pods 3/3 Running in vault ns and check SVC-VAULT "estado honesto" before treating it as a fault).
  3. Edge: kubectl -n kube-system get gateway,httproute — Gateway wetopi-edge Accepted+Programmed, listeners 8000/8443; kubectl get certificate -A — argo-tls, docs-tls, vault-tls (and edge-test-tls) Ready. Public probe: curl -sI https://docs.wetopi.net and https://argo.wetopi.net return 200/302 with a valid cert chain.
  4. DNS: the Cloudflare records for docs/argo/vault/edge-test .wetopi.net → 79.137.114.193/.194 (grey-cloud, TTL 300) — tofu-owned; verify with dig + the zone, don't trust the dashboard screenshot.
  5. IaC sanity: a tofu plan on PROD should be no-change; a plan wanting to CREATE everything means lost state → the state-loss runbook.
  6. Board: scan the kanban board for in-flight/blocked cards — understand what is mid-flight before touching anything.

(e) Frequent operations

  • DNS record change — never the Cloudflare dashboard: change the tofu/proxmox/environments/prod/ tfvars (0600) → PR in the infrastructure repo → merge is the deploy. The provider needs the Cloudflare token env (ops01, 0600). Records are prevent_destroy-annotated — deleting one is a recorded decision.
  • Deploy / change a service — GitOps only: edit kubernetes/<app>/ in the infrastructure repo → PR → review (blast-radius: a merge deploys) → merge; Argo CD syncs (prune + selfHeal). No kubectl apply of manifests from a workstation; no CI kubectl apply pipeline exists by design (ARCH-PLATFORM-GITOPS).
  • Secret rotation — via the Vault human UI (vault.wetopi.net, project wetopi-harness): rotate in Infisical, let the Infisical Operator sync to the cluster; SOPS+age is bootstrap/recovery only, not the daily vault (ADR-0009). The vault datastore backup/restore contract (pg_dump + age, rehearsed) is in the ADR AC8 — rehearse before calling it production.
  • Read-only inspection — any kubectl get/describe/logs via ops01. Read-only cluster reads are always allowed; anything mutating is a card with a gate.

(f) Escalation — existing runbooks

Situation Runbook / owner
Cluster node down, CP quorum loss, DR RKE2 apply & restore runbook
terraform.tfstate lost / empty plan wants to create everything RUN-TOFU-STATE-LOSS
A service is unhealthy its service fiche (deploy path + dependencies + honest state)
Something that matches no runbook file a kanban card for the right lane (ops/senior/lead); the board is the queue of truth

(g) No-go rules (NO-PO)

These are non-negotiable; violations are incidents, not "creative troubleshooting":

  1. Never tofu apply / kubectl apply from a card or on a hunch — mutations run only when the task body explicitly authorizes that specific apply with its gates; the board is the gate.
  2. Never put secrets in Git — no values, no keys, no htpasswd, no tfstate, no tokens: reference locations (vault path / Infisical path / 0600 file) only (DocOps §8).
  3. Never change PROD without the gate — a reviewed merge (or an explicitly gated, explicitly authorized apply). Single-writer discipline on the tofu state.
  4. Never edit Cloudflare DNS in the dashboard (tofu owns the records).
  5. Never assume a doc over live state — investigate the drift instead.

Validation

You have successfully taken over when: day-1 checks (d) all pass and you can name, for any node in the topology map, its pill, its credential location, and the runbook that applies to its failure mode.