OpenTofu Proxmox PROD state loss and re-import recovery

Purpose

Recover full OpenTofu control of the 8 Wetopi Proxmox PROD VMs after the tfstate is lost, WITHOUT touching the hypervisors: the import path is read-only against Proxmox (GET API calls + tofu import only — never an apply), and the acceptance gate proves the recovered state matches live reality exactly.

The single executed instance of this runbook is task t_690e108f (wetopi-senior, 2026-08-30): 28 resources imported, tofu plan = No changes (0 add / 0 change / 0 destroy), verified twice. Evidence paths and the patch scripts it produced are listed under Evidence.

Trigger / symptoms

• tofu plan for tofu/proxmox/environments/prod wants to CREATE all 28 resources (empty or absent terraform.tfstate), or errors with "state data does not exist" / no state backend configured.
• Nobody can answer "which tfstate describes PROD?" — the controller location is unknown or its state file is missing/corrupt.

Impact

Without state, IaC has no claim on the live infrastructure: any speculative tofu apply would try to create duplicate VMs, template, network devices and storage. The RKE2 apply runbook (PROC-RKE2-APPLY) precondition "render ansible inventory from tofu output" is blocked. Blast radius is contained as long as no apply runs: Proxmox itself is unaffected.

Preconditions

• SSH key /opt/data/.ssh/bigardo with access to the ops01 bastion (ubuntu@92.222.245.91) — the secrets source.
• tofu on PATH (controller: /usr/local/bin/tofu).
• NO other operator is concurrently running tofu against PROD (single-writer discipline; the state is local, not locked by a remote backend).
• Read access to the infrastructure repo at current main.

Phase-1 tofu controller (PERMANENT location)

The phase-1 controller is this harness VM — NOT ops01 (operator decision 2026-08-30). State root:

/opt/data/kanban/workspaces/t_690e108f/infrastructure/tofu/proxmox/environments/prod/

Controller artifacts (all mode 0600, all gitignored via .gitignore):

Artifact	Role
terraform.tfstate	recovered state — 28 instances
terraform.tfvars	regenerated from the live Proxmox API (never commit; *.tfvars is gitignored)
provider env vars	session-only, pulled over SSH from ops01 (below) — never written to harness disk

Concurrency model (phase 1)

The phase-1 backend is the LOCAL state file terraform.tfstate — it has NO locking of any kind. Two concurrent tofu operations (even two plan runs) can corrupt it. Therefore only ONE tofu operation runs at a time, period.

Enforcement is BY CONVENTION, not by a technical gate:

• All tofu-touching tasks are routed through the wetopi-ops lane.
• Serialization is intended to be enforced by the dispatcher via max_in_progress_per_profile = 1 for tofu-touching tasks; the CURRENT configured value in this harness is 3 (not yet 1), so today the ACTUAL enforcement is that the lead dispatches tofu tasks one-at-a-time by design — never two tofu-touching tasks running simultaneously.
• Future hard gate: a per-board lock (only one tofu task claimable at a time) — DEFERRED. Until it exists, the one-at-a-time dispatch convention above is load-bearing.

Deferred backend transition:

Remote backend (SOPS-encrypted or Infisical-backed) arrives with the secrets phase — until then phase-1 local state with the serialization convention above.

Safety checks

1. Confirm you are in the phase-1 controller directory and the state file is present, mode 0600, and NOT tracked by git (git check-ignore terraform.tfstate → ignored).
2. Confirm no credentials are on disk in the controller tree: tokens are sourced over SSH per session only (see Procedure step 2).
3. Read-only recon of the live Proxmox API (GET only) to know what will be imported before importing: /nodes, /nodes/<n>/qemu, /nodes/<n>/qemu/<id>/config, /nodes/<n>/network, /storage, /nodes/<n>/storage/<ds>/content.

Procedure

1. Clone infrastructure at current main and enter the environment: git clone https://github.com/wetopi-platform/infrastructure, cd tofu/proxmox/environments/prod.
2. Source provider tokens (session-only, never on harness disk) and export:

eval "$(ssh -i /opt/data/.ssh/bigardo ubuntu@92.222.245.91 \
  'grep -E "^PROXMOX" /home/ubuntu/secrets/.env.proxmox; \
   grep -E "CLOUDFLARE" /home/ubuntu/secrets/.cloudflare-wetopi-net.env')"
export PROXMOX_VE_ENDPOINT PROXMOX_VE_API_TOKEN PROXMOX_VE_INSECURE=true CLOUDFLARE_API_TOKEN

PROXMOX_VE_INSECURE=true is REQUIRED when running from the harness VM (untrusted PVE certificate). The ops01 copy of the secret says false but ops01 trusts the cert — on the harness it must be true. 3. Regenerate terraform.tfvars from the LIVE API data (shape: terraform.tfvars.example; every VM field was verifiable 1:1 from /nodes/<n>/qemu/<id>/config). Mode 0600. 4. tofu init, then import the live resources (read-only vs Proxmox) with the bpg/proxmox 0.111.1 import ID formats:

Resource class	Import ID format	Example
VM (module.vm)	<node>/<vmid>	ns3249468/8001
network vlan/bridge (proxmox_virtual_environment_network)	<node>:<iface>	ns3235424:vlan_rke
download file (ISO)	<node>/<datastore>:<content>/<file>	ns3243421/local:iso/ubuntu-24.04-20260826-server-cloudimg-amd64.img
firewall options	vm/<node>/<vmid>	vm/ns3249468/8001
storage (proxmox_storage_directory)	<storage-id>	local

The full 28-resource inventory with per-import return codes is in Evidence. Declared-but-not-applied resources are NOT imported: edge VMs 8041/8042 (F1 build-only, create_vms=false) and the Cloudflare records edge-test/docs-new (F3/F5 pending — cloudflare_dns_records unset in tfvars).

Exact resource addresses (the 28 imports, verbatim from the executed run — tofu import <ADDR> <IMPORT-ID>):

Address derivation (module source, quoted):

• module "vm" — tofu/proxmox/environments/prod/main.tf lines 93-97: module "vm" { for_each = { for name, vm in var.vms : name => vm if try(var.create_vms[name], false) } … source = "../../modules/vm" }; the child resource is tofu/proxmox/modules/vm/main.tf line 1: resource "proxmox_virtual_environment_vm" "this" → per-VM address module.vm["<vm-name>"].proxmox_virtual_environment_vm.this.
• module "platform_network" — main.tf lines 1-3: module "platform_network" { for_each = var.manage_platform_network ? var.proxmox_nodes : {} … source = "../../modules/node-vlan" }.
• proxmox_virtual_environment_firewall_options.bastion — tofu/proxmox/environments/prod/bastion.tf lines 9-10: resource "proxmox_virtual_environment_firewall_options" "bastion" { for_each = local.bastion_vms.
• VM keys come from var.vms (the 8 live VM names); node placement (ns*) was read live from the Proxmox API during the run.

Per-VM imports (8) — one per live VM, key = name in var.vms:

tofu import 'module.vm["wetopi-ops01"].proxmox_virtual_environment_vm.this'      ns3249468/8001
tofu import 'module.vm["rke-cp01"].proxmox_virtual_environment_vm.this'          ns3243421/8011
tofu import 'module.vm["rke-cp02"].proxmox_virtual_environment_vm.this'          ns3226939/8012
tofu import 'module.vm["rke-cp03"].proxmox_virtual_environment_vm.this'          ns3214108/8013
tofu import 'module.vm["rke-platform01"].proxmox_virtual_environment_vm.this'    ns31826929/8021
tofu import 'module.vm["rke-platform02"].proxmox_virtual_environment_vm.this'    ns3226931/8022
tofu import 'module.vm["rke-platform03"].proxmox_virtual_environment_vm.this'    ns31826925/8023
tofu import 'module.vm["wetopi-natgw01"].proxmox_virtual_environment_vm.this'    ns3243421/8031

Non-VM imports (20) — template, ISO download, firewall options, storage, and the 16 per-node network devices:

tofu import 'proxmox_virtual_environment_vm.ubuntu_noble_template[0]'            ns3243421/9000
tofu import 'proxmox_download_file.ubuntu_noble[0]'                              ns3243421/local:iso/ubuntu-24.04-20260826-server-cloudimg-amd64.img
tofu import 'proxmox_virtual_environment_firewall_options.bastion["wetopi-ops01"]' vm/ns3249468/8001
tofu import 'proxmox_storage_directory.local'                                    local
tofu import 'module.platform_network["ns31826925"].proxmox_network_linux_bridge.platform' ns31826925:vmbr4
tofu import 'module.platform_network["ns31826925"].proxmox_network_linux_vlan.platform'   ns31826925:vlan_rke
tofu import 'module.platform_network["ns31826929"].proxmox_network_linux_bridge.platform' ns31826929:vmbr4
tofu import 'module.platform_network["ns31826929"].proxmox_network_linux_vlan.platform'   ns31826929:vlan_rke
tofu import 'module.platform_network["ns3214108"].proxmox_network_linux_bridge.platform'  ns3214108:vmbr4
tofu import 'module.platform_network["ns3214108"].proxmox_network_linux_vlan.platform'    ns3214108:vlan_rke
tofu import 'module.platform_network["ns3226931"].proxmox_network_linux_bridge.platform'  ns3226931:vmbr4
tofu import 'module.platform_network["ns3226931"].proxmox_network_linux_vlan.platform'    ns3226931:vlan_rke
tofu import 'module.platform_network["ns3226939"].proxmox_network_linux_bridge.platform'  ns3226939:vmbr4
tofu import 'module.platform_network["ns3226939"].proxmox_network_linux_vlan.platform'    ns3226939:vlan_rke
tofu import 'module.platform_network["ns3235424"].proxmox_network_linux_bridge.platform'  ns3235424:vmbr4
tofu import 'module.platform_network["ns3235424"].proxmox_network_linux_vlan.platform'    ns3235424:vlan_rke
tofu import 'module.platform_network["ns3243421"].proxmox_network_linux_bridge.platform'  ns3243421:vmbr4
tofu import 'module.platform_network["ns3243421"].proxmox_network_linux_vlan.platform'    ns3243421:vlan_rke
tofu import 'module.platform_network["ns3249468"].proxmox_network_linux_bridge.platform'  ns3249468:vmbr4
tofu import 'module.platform_network["ns3249468"].proxmox_network_linux_vlan.platform'    ns3249468:vlan_rke

These 28 addresses are exactly the output of tofu state list after the run (Evidence: state-list.txt), each with an rc=0 import receipt (Evidence: import-log.txt). 5. Patch the three known bpg 0.111.1 import gaps BEFORE gating. Each gap makes the plan diverge from the pinned HCL and must be patched by editing the LOCAL state JSON only (no Proxmox mutation): - proxmox_download_file: import leaves url, checksum, checksum_algorithm, verify, overwrite, upload_timeout NULL (read() only refreshes size from the datastore listing) → the plan wants to REPLACE the file AND cascade-replace template VM 9000 via its chained file_id. Patch the pinned HCL values into the state. - template VM disk[0].file_id: imports as "" → patch to the resolved volid local:iso/ubuntu-24.04-20260826-server-cloudimg-amd64.img (NOT the resource address — the address form also forces replacement). - VMs: stop_on_destroy imports false (HCL says true) and agent.wait_for_ip imports as an empty block (HCL: disabled=true) → patch all 8 VMs. After patching: bump serial, keep lineage. 6. Acceptance gate — full tofu plan (no -target): MUST be No changes (0 add / 0 change / 0 destroy). Verify twice.

Validation

• Final tofu plan output verbatim shows No changes. with the counts line empty (0/0/0). ANY destroy line = STOP (see Escalation).
• tofu state list returns the expected 28 instances.
• A tofu plan re-run immediately after the gate reproduces "No changes" (drift between the two runs means someone else is writing to Proxmox).

Rollback / recovery

• State corruption mid-procedure: delete the bad local state file and restart the export/import at step 4 — imports are idempotent and read-only against Proxmox.
• Every state write before the acceptance gate should keep a 0600 backup (terraform.tfstate.bak*); restore from the newest backup if a patch step goes wrong.
• The procedure never applies, so there is no infrastructure rollback surface — worst case is a clean restart of this runbook.

Failure modes

• Plan shows replacement of proxmox_download_file cascading into template VM 9000 → you skipped caveat 1 of step 5; patch the URL/checksum fields into state, do NOT let the plan run as an apply.
• Plan shows template disk file_id replacement even after patching → the file_id was patched to the resource address instead of the resolved volid (caveat 2).
• Plan shows all 8 VMs modified (stop_on_destroy, agent.wait_for_ip) → caveat 3 not applied to every VM.
• x509: certificate signed by unknown authority against the Proxmox API → PROXMOX_VE_INSECURE=true missing (harness requirement, step 2).
• Import of a network device fails → ID uses node name + :iface form (step 4 table), and VLAN/bridge must exist on that node (check recon).

Escalation conditions

• The acceptance plan shows ANY destroy → STOP, do not apply, escalate to the operator (wetopi-senior/lead) with the plan output.
• Live API recon disagrees with the 28-resource inventory above (VMs exist that are not declared, or declared VMs are absent) → STOP and reconcile with the operator before writing tfvars.
• Version gates — check the CLI and the provider as TWO SEPARATE assertions (they are different version domains and must never be compared to each other):
• OpenTofu CLI: tofu version must satisfy required_version = ">= 1.6.0" (quoted from tofu/proxmox/environments/prod/versions.tf line 2). The recovery was executed with tofu 1.12.6.
• bpg/proxmox provider: pinned = 0.111.1 — quoted from tofu/proxmox/environments/prod/versions.tf line 7 (version = "= 0.111.1") and confirmed by tofu/proxmox/environments/prod/.terraform.lock.hcl lines 4-6 (provider "registry.opentofu.org/bpg/proxmox" { version = "0.111.1", constraints = "0.111.1" }). Either assertion failing → STOP; the import ID formats and caveats are version-specific and must be re-verified before use.

Evidence

Executed run: task t_690e108f (wetopi-senior, 2026-08-30), records in the Kanban Memory Plane. Artifacts:

Path	Content
Per-import receipts: harness workspace …/t_690e108f/evidence/import-log.txt	28 import commands + rc=0 receipts
…/t_690e108f/evidence/plan-final.txt	acceptance gate: tofu plan = No changes (verified twice)
…/t_690e108f/evidence/state-list.txt	tofu state list — the 28 imported instances
…/t_690e108f/patch-state.py, patch-state2.py	the state-patch scripts that closed the step-5 caveats
…/t_690e108f/recon.py	reusable GET-only recon script (precondition check 3)

(the … prefix is /opt/data/kanban/workspaces/t_690e108f — the phase-1 controller workspace; all artifacts also attached to the t_690e108f board card.)

Related incidents

• 2026-08-30 state-loss incident and recovery: task t_690e108f (Memory Plane). This runbook is its repeatable pattern.