Skip to content

ARCH-INFRA-PROVIDERS: IaC providers

The physical/virtual layer is owned by three tools with a strict responsibility split (ADR-0003 — OpenTofu and Ansible responsibility split). Everything below lives in the single GitOps repo https://github.com/wetopi-platform/infrastructure — there is exactly one infrastructure repo; docs explains it, never duplicates it.

Tool Owns Does NOT own
OpenTofu Proxmox VMs, node VLAN/bridge shape, base template clone, Cloudflare DNS records — via the HTTPS API Guest OS config, K8s workloads, secrets in HCL
cloud-init First-boot identity, network, SSH bootstrap only Anything after first boot
Ansible Ubuntu baseline, NAT gateway, RKE2, Ceph client firewall, edge VM identity/placement, K8s manifests
Argo CD K8s add-ons and workloads, after the cluster exists Infra VMs

No secrets are embedded in HCL or committed inventory; DR reconstructs infra and guest state from separate explicit sources of truth.

OpenTofu + Proxmox VE provider

  • Provider: bpg/proxmox pinned = 0.111.1 (tofu/proxmox/environments/prod/versions.tf), talking to the Proxmox HTTPS API (no SSH to the hypervisors).
  • Provider config (tofu/proxmox/environments/prod/provider.tf) reads PROXMOX_VE_ENDPOINT, PROXMOX_ve_API_TOKEN and PROXMOX_VE_INSECURE from the gitignored, repo-level .env.proxmox (mode 0600) — never from HCL. The controller is the phase-1 harness VM (operator decision), not ops01; see the state-loss runbook below.
  • Modules under tofu/proxmox/modules/: vm (clone-from-template + NICs), node-vlan (bridge/VLAN shape), cloudflare-dns-record (prevent_destroy per record).

Cloudflare provider

  • Provider: cloudflare/cloudflare pinned = 4.52.8 (modules/cloudflare-dns-record/versions.tf).
  • Reads CLOUDFLARE_API_TOKEN from the environment only — never in code, tfvars or state. On ops01 source /home/ubuntu/secrets/.cloudflare-wetopi-net.env (0600) before tofu init/plan/apply.
  • Token contract: both Zone - DNS - Edit and Zone - Zone - Read — the two-permission set the cert-manager Cloudflare DNS-01 solver requires (a DNS-Edit-only token fails zone discovery).
  • Records managed by tofu: see ovh-and-network.md.

Ansible

  • Lives at infrastructure/ansible/ (roles, inventories, playbooks). Key roles: common, ssh_keys, nat_gateway, rke2-server, rke2-agent, keepalived, ceph_storage, edge_lb, wan_bastion, ops_kubectl.
  • Ansible converges guest config idempotently; it never changes VM identity or placement (that is OpenTofu's job). The ops_kubectl role gives ops01 kubectl access via the API VIP (kubeconfig fetched at apply-time — no credentials in Git).

Where each state lives

State Location Mode / in Git?
terraform.tfstate (PROD) phase-1 controller (harness VM), …/tofu/proxmox/environments/prod/ 0600, gitignored — NEVER in Git
terraform.tfvars (real) same dir 0600, gitignored (.gitignore: *.tfvars except *.tfvars.example)
.env.proxmox (provider tokens) repo-level, controller tree 0600, gitignored
Cloudflare token /home/ubuntu/secrets/.cloudflare-wetopi-net.env on ops01 0600, not in Git
K8s kubeconfig fetched at apply-time from rke-cp01 by ops_kubectl role not persisted in Git
GitOps desired state infrastructure repo (committed) in Git, reviewed

Recovery if the PROD tfstate is lost: see RUN-TOFU-STATE-LOSS — OpenTofu PROD state loss & re-import recovery (read-only re-import from the live Proxmox API, tofu plan = No-changes gate). Never reconstruct state by speculative apply.

Secret handling

  • SOPS + age is bootstrap/recovery only (master plan §11): it answers "where do we keep the secrets needed to boot the system that keeps secrets". Two age keys — a bootstrap/automation key and an offline recovery key. Private age keys never go in Git.
  • The durable runtime vault is Infisical, self-hosted (ADR-0009 — Secrets vault harness); the vault.* contract + machine identities live there. SOPS+age is not the daily vault.
  • The rule "secrets used, not necessarily seen" (master plan §3.7) applies: tool runtimes and the vault, least-privilege — no plaintext in Git, prompts or logs.

Takeover notes

  • Before any tofu command: confirm you are in the controller dir, the .env.proxmox is sourced (0600), and no token is on disk in the tree.
  • tofu plan for tofu/proxmox/environments/prod should be a no-change when state is healthy; a plan that wants to CREATE all 28+ resources means the state is lost → run the state-loss runbook.
  • The GitOps repo is the single source of declarative truth; a merge is a deploy (Argo CD / tofu apply), so blast-radius review is required.