Raw \u2192 S3 \u00b7 Knowledge \u2192 BookStack \u00b7 Contract \u2192 Git \u00b7 Wisdom \u2192 Hindsight
\n| Almac\u00e9n | \nQu\u00e9 vive ah\u00ed | \nD\u00f3nde | \n
|---|---|---|
| Git (docs repo, ai-hub) | \nContratos: ADRs, fichas SVC, runbooks de plataforma, specs de objetivos. SSOT de decisiones | \nwetopi-platform/docs, ai-hub | \n
| BookStack (este sitio) | \nConocimiento vivo: borradores, evidencia, how-to del d\u00eda a d\u00eda | \nbook.wetopi.net | \n
| Artifact Store (S3/Ceph RGW) | \nEvidencia cruda: dumps, logs, outputs | \nPrerequisito: RGW desplegado (fase 2 del roadmap) | \n
| Hindsight (memory.wetopi.net) | \nWisdom: hechos con timestamp, aprendizajes | \nbanks personales (nativo) + corporate (plugin) | \n
id, title, type, status, owner, summary, scope, tags, related, last_verified.operations/runbooks/ (plataforma). Nunca dos copias.last_verified >30 d\u00edas tras cambio relevante \u2192 marcar stale en el \u00edndice.L0 Ef\u00edmero (/tmp, worktrees) \u2192 hoy, muere hoy\n \u2193 \u00bfimport\u00f3? \u2192 promociona\nL1 Workbench (BookStack, book 90) \u2192 borradores, evidencia, DRAFTs\n \u2193 \u00bfmadur\u00f3? \u2192 promociona\nL2 Formal (docs repo / ai-hub) \u2192 PR + review\nL3 Memoria (Hindsight + plugin) \u2192 hechos, no documentos\nLa memoria apunta, no almacena: el bank guarda el hecho esencial + timestamp + puntero (URL). Nunca el documento entero.
\n#draft#ready + PR al destino formal (Git) si procedePROMOVED \u2192 <link> + #superseded \u2014 la copia muere al promocionarCada book cubre un dominio (ver \u00edndice lateral). Los ADRs viven en Git\n(wetopi-platform/docs); el book 70 es solo mirror de lectura. Los IDs\ncan\u00f3nicos (OBJ/SVC/ADR/RUN/ART/REC) est\u00e1n registrados en\ndocs/reference/id-registry.md.
Skill de referencia en la flota: wetopi-docops (instalada en\ncarlos/joan/josep). C\u00e1rgala antes de crear, mover o promocionar documentos.
\n\nMigrado desde
\nwetopi-platform/docs@main(agents/harness-v31-governance.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
Harness V3.1 governance \u2014 roles, gates and fail-closed state machine
\nThis document is the operating model of the Harness V3.1 pilot. The\ndecisions live in ADR-0008\n(proposed); the pilot itself is\nOBJ-HARNESS-V31-001.\nNothing here is accepted governance until ADR-0008 is accepted after the\npilot verdict.
\nRoles and their boundaries, the review gates, degraded routing, and the\nfail-closed state machine that governs every Objective under V3.1.
\n| Role | \nActor | \nDelivers | \nHard boundary | \n
|---|---|---|---|
| THINK (Architect) | \narchitect-designated profile or the operator as architect | \nObjective Draft: WHAT / WHY / DONE (outcome, context, ACs, verification contract, constraints, boundaries, stop conditions) | \nNo Git write. Never pushes branches/commits. Never claims cards. Never judges its own draft. | \n
| OWN (Lead) | \nwetopi-lead | \nObjective acceptance, card decomposition, dispatch, Execution Ledger (single writer), documentation PRs | \nDoes not author WHAT/WHY/DONE it was not handed; does not judge deliveries. | \n
| DO (workers) | \nwetopi-build, wetopi-senior, wetopi-ops | \nCard implementation with self-validation and evidence | \nNo apply/destroy of real infrastructure from build cards; no self-approval; stays inside objective boundaries and stop conditions. | \n
| JUDGE (review) | \nwetopi-approver or any reviewer that did NOT implement | \nVerdicts against the verification contract: APPROVE / REQUEST_CHANGES / REJECT | \nDoes not implement or patch the delivery; does not merge without operator authority. | \n
The Architect defines WHAT, WHY and DONE \u2014 nothing else:
\ngoal_mode iteration may drive a worker's loop but\nNEVER substitutes for this gate.accepted and merged\nbefore any P2 work; the post-pilot step is operator confirmation, not\na second ADR acceptance.)| Condition | \nRouting | \n
|---|---|
| Reviewer (approver lane) unavailable | \nObjective delivery blocks (fail-closed). The operator may appoint a substitute judge that did not implement. No auto-approval, ever. | \n
| Qwen local fails its capability gate or economics gate, or is in cooldown | \nQwen local is excluded; work routes to available capable lanes or blocks. Exclusion never weakens any other gate and never blocks the role design. | \n
| Worker lane unavailable | \nThe Lead re-dispatches to another capable lane inside the same role boundaries; if none, the card blocks. | \n
| Git/PR unavailable (credential or platform failure) | \nObjective execution continues only if no gate needs the PR; otherwise blocks. No local \"shadow merges\". | \n
| Evidence incomplete at delivery | \nThe delivery is rejected back to DO; no partial credit. | \n
Qwen local (Ollama) may participate only if BOTH gates pass:
\nwall_time_validated for the direct\nGPT/Lead run and for the Qwen + Lead audit/rework run. Qwen is\nroutable only if 3/3 fixture ACs pass under Qwen, Qwen makes zero\ncloud/GPT fallback calls, and the aggregate Qwen time is\n\u2264 1.20\u00d7 the direct baseline. Any failure \u21d2 excluded (fail-closed\nrouting, above).goal_mode (automatic goal continuation on a worker) is DISABLED for\nevery real Objective until the Phase P2 smoke proves the full contract\nbelow. Like Qwen local (ADR-0008 Decisions 6/9), it is a capability-gated\nparticipant \u2014 never a replacement for the JUDGE gate (Decision 5).
Enablement gate: goal_mode may be enabled for real Objectives only\nafter a P2 smoke run on a non-mutating scratch card proves ALL of the\nfollowing, with recorded evidence:
| Smoke case | \nRequired (fail-closed) outcome | \nProhibited | \n
|---|---|---|
Judge DONE | \nNo promotion by goal_mode alone: DONE requires AC evidence + Lead audit + independent APPROVE | \nauto-done, auto-merge, auto-cutover | \n
Judge CONTINUE / FAIL | \nBounded turn budget honored; objective blocked/rejected as configured after the budget | \nunbounded loops, automatic re-dispatch | \n
| Provider timeout / offline | \nObjective lands Blocked with visible triage; goal_mode stays off | \nsilent retry storms, false DONE, silent cloud fallback | \n
| Worker/gateway restart | \nAt most one reconciled continuation run; smoke repeatable | \nduplicate runs/workers, lost-blocker resurrection | \n
| Turn budget exhaustion | \nTurns bounded by configuration; overflow \u21d2 Blocked | \nunlimited continuation | \n
| External-mutation watch | \nZero mutations outside the scratch card for every case | \nany write outside the scratch scope | \n
Fail-closed rule: on ANY failure, timeout, or unverifiable outcome in\nthis smoke \u2014 and on any later regression of the above behaviors \u2014\ngoal_mode remains disabled for real Objectives. Disabling it never\nweakens any other gate and never blocks the role design; enabling it\nwithout a fully green recorded smoke is a governance violation, and a\ngoal_mode-completed delivery judged without the JUDGE step is void.
stateDiagram-v2\n [*] --> Draft\n Draft --> ObjectiveReview: Architect hands off draft\n ObjectiveReview --> Accepted: reviewed\n ObjectiveReview --> Draft: review fails / REQUEST_CHANGES on the draft\n Accepted --> Executing: Lead acquires + decomposes\n Accepted --> ObjectiveReview: acquisition precondition lost\n Executing --> Delivering: cards complete with evidence\n Delivering --> DeliveryReview: verification contract exercised\n DeliveryReview --> Done: JUDGE approves\n DeliveryReview --> Executing: REQUEST_CHANGES\n Done --> AcceptedGovernance: operator confirms standing governance (pilot positive)\n Done --> Closed: operator rejects standing governance (lessons kept)\n ObjectiveReview --> Blocked: no reviewer / verdict\n Accepted --> Blocked: Lead acquisition unavailable\n Executing --> Blocked: dispatch fails at this phase\n Delivering --> Blocked: no verdict / evidence gap\n DeliveryReview --> Blocked: no reviewer / verdict\n Blocked --> ObjectiveReview: re-enter the gate this phase requires\n Blocked --> Abandoned: stop condition / rollback trigger\n Abandoned --> [*]\n Closed --> [*]\n AcceptedGovernance --> [*]\nFail-closed recovery rules:
\nBlocked always records which phase it arose in. Its only exit is\nback to the gate that phase requires \u2014 ObjectiveReview blocks return to\nObjectiveReview, post-acceptance blocks (acquisition/dispatch/\ndelivery) return to the review or dispatch gate of their own phase.\nThere is NO Blocked -> Executing bypass edge.Delivering blocked on missing reviewer/evidence goes to\nDeliveryReview only when a competent independent reviewer (JUDGE \u2260\nimplementer) is actually available; otherwise it stays Blocked.\nRecovery never re-enters Executing with unfixed evidence.Invariant: every non-happy path lands in Blocked or Abandoned,\nnever in a silently-accepted state. Abandonment and rollback follow the\ntriggers in ADR-0008 Decision 7.
\n\n\nMigrado desde
\nwetopi-platform/docs@main(agents/hermes-agents01-instances.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ARCH-HERMES-AGENTS01: Hermes agents on agents01
\nOne host, many disposable agents. The containers are cattle \u2014 the state\nis the bind mount. Everything an agent is (config, memory, skills, sessions,\nsecrets) lives on the host under /opt/hermes-<name>/data; everything the\ncontainer image or apt installs is reproducible from bootstrap.sh.
| Fact | \nValue | \n
|---|---|
| Public IP | \n92.222.245.89 | \n
| Hostname | \nagents01 (with s) | \n
| SSH access | \nssh -i <key> root@92.222.245.89 \u2014 root only (ubuntu@ is denied) | \n
| Resources | \n~8 GB RAM, 40 GB disk, Docker 29.x | \n
| Docker network | \nhermes-net \u2014 shared by all services + nginx | \n
Any Hermes instance reaches them via host.docker.internal (or container\nname on hermes-net):
| Service | \nEndpoint | \nUse | \n
|---|---|---|
| searxng | \nhttp://host.docker.internal:8888 | \nSEARXNG_URL env (web search) | \n
| firecrawl / crw | \nhttp://host.docker.internal:3000 | \nFIRECRAWL_API_URL env (scraping) | \n
| camofox-mcp | \nhttp://host.docker.internal:9378/mcp | \nCAMOFOX_MCP_URL env (browser automation) | \n
| lightpanda / crw-camofox | \ninternal | \nbrowser backends for the above | \n
| Instance | \nDashboard (host) | \nAPI (host) | \nPublic URL | \n
|---|---|---|---|
| hermes-carlos | \n9123 | \n8646 | \ncarlos.wetopi.net (HTTPS) | \n
| hermes-joan | \n9124 | \n8647 | \njoan.wetopi.net (HTTPS, LE cert) | \n
| hermes-josep | \n9125 | \n8648 | \njosep.wetopi.net (HTTPS, LE cert) | \n
Both joan/josep share ONE certificate (live/joan.wetopi.net/, SAN:\njoan + josep, HTTP-01 webroot, expires 2026-11-30, auto-renew via the\ncertbot container). Explicit A records for both subdomains \u2192 92.222.245.89\nwere created 2026-09-01 (the wildcard still points at .76).
Inside every container the ports are always 9119 (dashboard) and\n8642 (API server); only the host-side ports differ.
\n/opt/hermes-<name>/data \u2192 /opt/data (ALL agent state, rw bind)\n/opt/hermes-<name>/project \u2192 /home/hermes (workspace, rw bind)\n/opt/hermes-<name>/.env \u2192 API_SERVER_KEY (0600, per-instance secret)\nWhat lives inside data/ (= the container's /opt/data):\nconfig.yaml, profile.yaml, memories/, skills/, profiles/,\nsessions/, state.db, secrets/, bootstrap.sh.
Consequences:
\nrestart=always + Docker restarting on boot \u2192 instances survive host\nreboots; bind mounts remount automatically.docker rm + recreate is safe: all state is on the host. What is NOT\npreserved is the container layer (apt packages, /usr/local/bin binaries)\n\u2014 bootstrap.sh (persisted inside data/) reinstalls those on start.data/ loses the agent. Back it up like any stateful volume.Parametrized script at /opt/deploy-hermes-instance.sh on agents01\n(validated template lives in the lead profile skill wetopi-hermes-hosts):
/opt/deploy-hermes-instance.sh <name> <dashboard_port> <api_port> start\nSteps for a NEW instance (the script automates most of this):
\nstart \u2014 creates layout, generates .env, runs container\n(nousresearch/hermes-agent:latest, --restart always), joins\nhermes-net, waits for s6.bootstrap.sh into the fresh data/ (from\n/opt/hermes-carlos/data/bootstrap.sh) and run the bootstrap action \u2014\nfresh instances have none.dashboard: basic_auth block from a working instance's\nconfig.yaml and docker restart \u2014 without it the dashboard refuses to\nbind publicly (see pitfalls).OPENAI_API_KEY and no LLM access).Files: /opt/composes/nginx/conf.d/<name>.conf (compose in\n/opt/composes/nginx/docker-compose.yml; certs volume shared with the\ncertbot container, webroot /var/www/certbot).
proxy_pass http://hermes-<name>:9119 with websocket upgrade headers (the dashboard\nneeds Upgrade/Connection and long read/send timeouts).ssl_certificate /etc/nginx/certs/live/<name>.wetopi.net/fullchain.pem \u2014 ship it COMMENTED\nuntil certbot has issued the cert, then uncomment and reload.*.wetopi.net currently points at 92.222.245.76; each agent\nsubdomain needs an explicit A record \u2192 92.222.245.89 before HTTP-01\nvalidation can succeed.docker exec nginx-proxy nginx -t && docker exec nginx-proxy nginx -s reload.Current hermes-agent images refuse to bind the dashboard to 0.0.0.0\nwithout a registered auth provider. The working pattern (all three\ninstances): dashboard.basic_auth (username + password_hash) in each\ninstance's config.yaml. Hash via\npython -c \"from plugins.dashboard_auth.basic import hash_password; print(hash_password('...'))\".\nCredentials are per-agent and live only in each config.yaml \u2014 never in\ndocs.
\n\nARCHIVO \u2014 versi\u00f3n migrada cruda. La ficha estructurada con las 3 preguntas es la versi\u00f3n
\nARCH-*/SVC-*de este mismo chapter (actualizada con datos live 2026-09-06). Esta copia cruda se conserva por el historial.
\n\nMigrado desde
\nwetopi-platform/docs@main(architecture/infrastructure/ceph.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ARCH-INFRA-CEPH: Ceph storage LAN and client access
\nWetopi's primary storage is a pre-existing Ceph cluster that is\nhyperconverged with the legacy Proxmox/Ceph estate and lives on its own\ndedicated storage network. The master architecture plan is explicit: the\nCeph network is EXISTING \u2014 DO NOT MODIFY; it is never renumbered and\nnever NATed on any path (ADR-0001 invariants).
\n| Item | \nValue | \n
|---|---|
| Subnet | \n192.168.96.0/20 | \n
| VLAN / bridge | \nVLAN 2 / vmbr2 (trunk on every Proxmox node \u2014 see ovh-and-network.md) | \n
| Monitors (live, 2026-08-31) | \n192.168.111.9-15 (msgr1 port 6789; msgr2 port 3300), 5 mons | \n
| Cluster fsid | \nb60e7cbb-e83f-42ef-8a4c-395e22cbb96f | \n
| OSD ports | \nTCP 6800-7300 | \n
| Owner | \nPre-existing cluster \u2014 not managed by the Wetopi rebuild | \n
Monitors and OSDs are outside the rebuild's scope; the rebuild only\nconsumes this cluster via Ceph CSI. Live cluster facts above were read\nfrom the deployed CSI config (ConfigMap ceph-csi-config) on 2026-08-31.
The access design is fixed by\nADR-0007 \u2014 Ceph client access via direct\nstorage-network attachment\n(operator decision 2026-08-30, superseding the ADR-0004 routing section):
\nrke-platform01/02/03, the PVC-carrying nodes)\ncarry a third NIC on vmbr2/VLAN 2 \u2014 the Ceph subnet becomes a\ndirectly connected L2 domain, the standard Ceph client pattern.192.168.98.0/25 reserved for\nplatform storage consumers \u2014 .10/.11/.12 = workers 01/02/03\n(all /20); the rest of the /25 is free for future consumers./20 connected route\nappears automatically. The default route stays exclusively 10.72.0.1\non the LAN NIC.Verified on the live workers (2026-08-31, all three):
\neth1 UP 192.168.98.10/20 # rke-platform01 (.11/.12 on 02/03)\nip route get 192.168.111.9\n -> 192.168.111.9 dev eth1 src 192.168.98.10 # storage NIC, direct\ndefault via 10.72.0.1 dev eth0 # LAN-only default\nflowchart LR\n subgraph workers[\"rke-platform01/02/03\"]\n E0[eth0 \u2014 vmbr4 / 10.72.0.x/23<br/>default via 10.72.0.1]\n E1[eth1 \u2014 vmbr2 / 192.168.98.10-12/20<br/>no gateway, no routes]\n end\n subgraph ceph[\"Ceph LAN 192.168.96.0/20 (VLAN 2, pre-existing)\"]\n M[mons 192.168.111.9-15<br/>6789 / 3300]\n O[OSDs 6800-7300]\n end\n E1 -->|directly connected L2| M\n E1 --> O\n E0 -.->|\"mgmt only; never Ceph I/O\"| LAN[10.72.0.0/23]\nThe RKE2 cluster gets RBD-backed PVCs from this cluster via the\nceph-csi-rbd chart artifacts in\ninfrastructure kubernetes/ceph-csi/.\nDeployed reality (verified 2026-08-31): the live driver is an\nOUT-OF-CHART install of upstream ceph-csi v3.12.0 raw manifests, applied\nby the operator (not via the repo chart, not via Helm \u2014 helm ls shows no\nceph-csi release) during the VAULT apply unblock (2026-08-31). The repo\nchart artifacts remain the declared desired state for a future chart-based\nadoption/sync, pinned to match the live version.
Status as verified 2026-08-31 (kubectl via ops01): deployed and\nrunning. The earlier bootstrap state \u2014 CSI pods stuck in\nContainerCreating waiting for the operator-delivered inputs\n(clusterID/monitors and the ceph-config / ceph-csi-encryption-kms\nConfigMaps, operator inputs tracked in card t_8eee26e4) \u2014 has been\nresolved: the ConfigMaps (ceph-config, ceph-csi-config,\nceph-csi-encryption-kms-config) exist in kube-system and the CSI pods\nare Running:
| Component | \nState (2026-08-31) | \n
|---|---|
csi-rbdplugin DaemonSet | \n3/3 Running (on the 3 platform workers) | \n
csi-rbdplugin-provisioner Deployment | \n3/3 Running (replicas 3, raw-manifest default) | \n
StorageClass ceph-rbd | \npool kubernetes, ext4, Immediate, reclaim Delete, volume expansion enabled; secret ceph-csi-secret; NO imageFeatures | \n
| CSI image (live) | \nquay.io/cephcsi/cephcsi:v3.12.0 | \n
| Sidecars (live) | \nprovisioner v5.0.1, snapshotter v8.0.1, attacher v4.6.1, resizer v1.11.1, registrar v2.11.1 | \n
| RBAC naming (live) | \nraw-manifest names (rbd-csi-nodeplugin, rbd-csi-provisioner) | \n
| Vault PVCs | \nvault-postgres-data 5Gi + vault-redis-data 1Gi \u2014 Bound on ceph-rbd | \n
Drift resolved (2026-08-31, LEAD decision, card t_f091f122): the live\nCSI image v3.12.0 was found drifted against the repo pin\n3.17.1 (a design-time pin written before any deployment). Canonical =\nlive v3.12.0 (validated in prod: vault PVCs Bound on it); the repo pin\nwas realigned to 3.12.0 in infrastructure PR\n#30, which\nalso documents the live-vs-repo deltas that any future chart-based sync\nmust reconcile first (Secret name ceph-csi-secret vs csi-rbd-secret,\nprovisioner replicas 3 vs 2, adoption of the out-of-chart objects).\nUpstream upgrade to 3.17.x is DEFERRED, deliberate, operator-gated\nmaintenance (ADR-0007: Ceph prod untouched outside operator sessions).
ceph -s on a mon\n(operator access to the pre-existing cluster) \u2014 not from the K8s side.ceph_storage Ansible role admits only Ceph client\nports (3300/6789 + 6800-7300) from the Ceph /20 on eth1, in a\ndedicated nftables table \u2014 see\ninfrastructure/ansible/roles/ceph_storage.../storage/; this pill covers the physical LAN and the access path.\n\nARCHIVO \u2014 versi\u00f3n migrada cruda. La ficha estructurada con las 3 preguntas es la versi\u00f3n
\nARCH-*/SVC-*de este mismo chapter (actualizada con datos live 2026-09-06). Esta copia cruda se conserva por el historial.
\n\nMigrado desde
\nwetopi-platform/docs@main(architecture/infrastructure/ovh-and-network.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ARCH-INFRA-OVH: OVH hosting, vRack and network blocks
\nWetopi runs on 8 OVH dedicated servers in a single vRack (OVH's\nprivate L2 fabric that interconnects the servers' vRack ports). Proxmox VE\nruns on all 8; the VMs and the RKE2 cluster live on top of it. The whole\nstack is provisioned from the GitOps repo\nhttps://github.com/wetopi-platform/infrastructure\n(tofu/proxmox/environments/prod/ \u2014 authoritative source;\nthe committed terraform.tfvars.example documents the shape, the real\nterraform.tfvars is gitignored/0600).
Node names are the OVH server identifiers used throughout tfvars\n(proxmox_nodes) and live Proxmox (cluster wetopi7):
| Proxmox node | \nTrunk NIC (live-verified) | \n
|---|---|
| ns31826925 | \nenp10s0f1np1 | \n
| ns31826929 | \nenp10s0f1np1 | \n
| ns3214108 | \nenp2s0f1np1 | \n
| ns3226931 | \nenp2s0f1np1 | \n
| ns3226939 | \nenp2s0f1np1 | \n
| ns3235424 | \nenp10s0f1np1 | \n
| ns3243421 | \nenp10s0f1np1 | \n
| ns3249468 | \nenp10s0f1np1 | \n
Each node carries a single 802.1Q trunk NIC (the vlan-raw-device\nverified live per node via the Proxmox API) through which the vRack VLANs\nbelow flow. VM placement across the 8 nodes spreads the control planes and\nworkers over distinct failure domains (see\nproxmox-cluster.md).
Two OVH-allocated public blocks are attached to the servers and reachable\non the WAN bridge vmbr3:
| Block | \nAddresses in use (verified) | \n
|---|---|
92.222.245.64/27 | \n92.222.245.91 = wetopi-ops01 (bastion, direct WAN); 92.222.245.83 = wetopi-natgw01 (WAN egress); gateway 92.222.245.94 | \n
79.137.114.192/27 | \n79.137.114.193 = rke-wan01; 79.137.114.194 = rke-wan02 (edge v1, ADR-0006); gateway 79.137.114.222 | \n
vmbr3 carries untagged public IPs on the VM NICs (no VLAN tag, no IP\non the hypervisor). The ops01 .91 address is the SSH bastion for the\nwhole internal network (every other VM is reached through it); natgw .83\nis the LAN's NAT egress.
Three L2 domains exist on the vRack, each a bridge on every Proxmox node:
\n| Bridge | \nVLAN | \nSubnet | \nPurpose | \n
|---|---|---|---|
vmbr4 | \n4 | \n10.72.0.0/23 (MTU 1500, from reserved 10.72.0.0/20 \u2014 ADR-0001) | \nRKE2 node LAN: all 8 cluster VMs. No IP on the hypervisors. | \n
vmbr2 | \n2 | \n192.168.96.0/20 | \nPre-existing Ceph storage LAN (see ceph.md). | \n
vmbr3 | \n(untagged) | \npublic blocks above | \nWAN attachment. | \n
Address plan on 10.72.0.0/23 (from tfvars ansible_host + NIC configs):\n.1 LAN gateway (non-managed default gateway hop), .2 natgw, .10\nops01, .20 RKE2 API VIP (keepalived, ADR-0004), .21-23 control\nplanes, .41-43 platform workers, .51-52 edge nodes. VMs resolve names\nvia public resolvers (1.1.1.1, 1.0.0.1, domain wetopi.priv) \u2014 there\nis no internal DNS in the platform; stable endpoints are IP-based.
flowchart TB\n subgraph OVH[\"OVH (8 dedicated servers, one vRack)\"]\n direction TB\n subgraph WAN[\"vmbr3 \u2014 WAN (untagged)\"]\n O91[ops01 92.222.245.91<br/>bastion]\n N83[natgw 92.222.245.83<br/>LAN egress]\n W1[rke-wan01 79.137.114.193]\n W2[rke-wan02 79.137.114.194]\n end\n subgraph LAN[\"vmbr4 \u2014 VLAN 4, 10.72.0.0/23\"]\n CP[rke-cp01/02/03<br/>.21/.22/.23 + VIP .20]\n PL[rke-platform01/02/03<br/>.41/.42/.43]\n end\n subgraph CEPH[\"vmbr2 \u2014 VLAN 2, 192.168.96.0/20\"]\n C1[\"ceph mons/OSDs (pre-existing)\"]\n PL2[\"workers eth1 192.168.98.10-12/20\"]\n end\n end\n Internet((Internet)) --- WAN\n WAN ---|LAN-to-WAN NAT| LAN\n LAN --- CP & PL\n C1 --- CEPH\n CEPH --- PL2\n O91 -.-|\"SSH bastion to all VMs\"| LAN\nwetopi.net)The zone is managed on Cloudflare (zone id\n6728a5c341cfadd578c1539ccb28c9ad \u2014 non-secret, read via the CF API) and\nWetopi-new service records are declared in OpenTofu (tofu/proxmox/environments/prod/cloudflare-dns.tf +\nvar.cloudflare_dns_records), one module instance per record with\nprevent_destroy. Verified live via the Cloudflare API on 2026-08-31:
| Name | \nRecords | \nShape | \n
|---|---|---|
edge-test.wetopi.net | \nA \u00d7 79.137.114.193 / .194 | \nTTL 300, proxied=false (grey-cloud) | \n
docs-new.wetopi.net | \nA \u00d7 79.137.114.193 / .194 | \nTTL 300, grey-cloud | \n
docs.wetopi.net | \nA \u00d7 79.137.114.193 / .194 | \nTTL 300, grey-cloud (F4 cutover) | \n
argo.wetopi.net | \nA \u00d7 79.137.114.193 / .194 | \nTTL 300, grey-cloud | \n
Scope rule (WAN Edge v1 plan \u00a74): only Wetopi-new service records are\nin tofu scope. Legacy records (wildcards, agg, carlos-*,\ndevops-wan-*, \u2026) and _acme-challenge TXT records (cert-manager) are\ndeliberately out of scope \u2014 never imported, declared or touched.\nedge-test was the Let's Encrypt staging gate (F3); docs-new the\npre-cutover staging name; docs and argo are the live v1-window edge\nservices (ADR-0006: DNS round-robin, reduced availability, operator-driven\nremediation).
tofu plan on\ntofu/proxmox/environments/prod (no-changes expected), CF API\nGET /zones/<id>/dns_records, and kubectl get nodes via ops01.terraform.tfstate, terraform.tfvars or provider tokens\n(.env.proxmox, .cloudflare-wetopi-net.env) \u2014 see\nproviders.md for where each secret lives.\n\nARCHIVO \u2014 versi\u00f3n migrada cruda. La ficha estructurada con las 3 preguntas es la versi\u00f3n
\nARCH-*/SVC-*de este mismo chapter (actualizada con datos live 2026-09-06). Esta copia cruda se conserva por el historial.
\n\nMigrado desde
\nwetopi-platform/docs@main(architecture/infrastructure/providers.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ARCH-INFRA-PROVIDERS: IaC providers
\nThe physical/virtual layer is owned by three tools with a strict\nresponsibility split\n(ADR-0003 \u2014 OpenTofu and Ansible responsibility\nsplit). Everything below\nlives in the single GitOps repo\nhttps://github.com/wetopi-platform/infrastructure\n\u2014 there is exactly one infrastructure repo; docs explains it, never\nduplicates it.
| Tool | \nOwns | \nDoes NOT own | \n
|---|---|---|
| OpenTofu | \nProxmox VMs, node VLAN/bridge shape, base template clone, Cloudflare DNS records \u2014 via the HTTPS API | \nGuest OS config, K8s workloads, secrets in HCL | \n
| cloud-init | \nFirst-boot identity, network, SSH bootstrap only | \nAnything after first boot | \n
| Ansible | \nUbuntu baseline, NAT gateway, RKE2, Ceph client firewall, edge | \nVM identity/placement, K8s manifests | \n
| Argo CD | \nK8s add-ons and workloads, after the cluster exists | \nInfra VMs | \n
No secrets are embedded in HCL or committed inventory; DR reconstructs\ninfra and guest state from separate explicit sources of truth.
\nbpg/proxmox pinned = 0.111.1\n(tofu/proxmox/environments/prod/versions.tf), talking to the Proxmox\nHTTPS API (no SSH to the hypervisors).tofu/proxmox/environments/prod/provider.tf) reads\nPROXMOX_VE_ENDPOINT, PROXMOX_ve_API_TOKEN and PROXMOX_VE_INSECURE\nfrom the gitignored, repo-level .env.proxmox (mode 0600) \u2014 never\nfrom HCL. The controller is the phase-1 harness VM (operator decision),\nnot ops01; see the state-loss runbook below.tofu/proxmox/modules/: vm (clone-from-template + NICs),\nnode-vlan (bridge/VLAN shape), cloudflare-dns-record\n(prevent_destroy per record).cloudflare/cloudflare pinned = 4.52.8\n(modules/cloudflare-dns-record/versions.tf).CLOUDFLARE_API_TOKEN from the environment only \u2014 never in\ncode, tfvars or state. On ops01 source\n/home/ubuntu/secrets/.cloudflare-wetopi-net.env (0600) before\ntofu init/plan/apply.Zone - DNS - Edit and Zone - Zone - Read \u2014 the two-permission set the cert-manager Cloudflare DNS-01 solver\nrequires (a DNS-Edit-only token fails zone discovery).infrastructure/ansible/ (roles, inventories, playbooks).\nKey roles: common, ssh_keys, nat_gateway, rke2-server,\nrke2-agent, keepalived, ceph_storage, edge_lb, wan_bastion,\nops_kubectl.ops_kubectl role\ngives ops01 kubectl access via the API VIP (kubeconfig fetched at\napply-time \u2014 no credentials in Git).| State | \nLocation | \nMode / in Git? | \n
|---|---|---|
terraform.tfstate (PROD) | \nphase-1 controller (harness VM), \u2026/tofu/proxmox/environments/prod/ | \n0600, gitignored \u2014 NEVER in Git | \n
terraform.tfvars (real) | \nsame dir | \n0600, gitignored (.gitignore: *.tfvars except *.tfvars.example) | \n
.env.proxmox (provider tokens) | \nrepo-level, controller tree | \n0600, gitignored | \n
| Cloudflare token | \n/home/ubuntu/secrets/.cloudflare-wetopi-net.env on ops01 | \n0600, not in Git | \n
| K8s kubeconfig | \nfetched at apply-time from rke-cp01 by ops_kubectl role | \nnot persisted in Git | \n
| GitOps desired state | \ninfrastructure repo (committed) | \nin Git, reviewed | \n
Recovery if the PROD tfstate is lost: see\nRUN-TOFU-STATE-LOSS \u2014 OpenTofu PROD state loss &\nre-import recovery\n(read-only re-import from the live Proxmox API, tofu plan = No-changes\ngate). Never reconstruct state by speculative apply.
vault.* contract + machine identities live there. SOPS+age is not\nthe daily vault.tofu command: confirm you are in the controller dir, the\n.env.proxmox is sourced (0600), and no token is on disk in the tree.tofu plan for tofu/proxmox/environments/prod should be a no-change\nwhen state is healthy; a plan that wants to CREATE all 28+ resources\nmeans the state is lost \u2192 run the state-loss runbook.tofu apply), so blast-radius review is required.\n\nARCHIVO \u2014 versi\u00f3n migrada cruda. La ficha estructurada con las 3 preguntas es la versi\u00f3n
\nARCH-*/SVC-*de este mismo chapter (actualizada con datos live 2026-09-06). Esta copia cruda se conserva por el historial.
\n\nMigrado desde
\nwetopi-platform/docs@main(architecture/infrastructure/proxmox-cluster.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ARCH-INFRA-PROXMOX: Proxmox cluster wetopi7
\nThe RKE2 platform and all its VMs run on a single Proxmox VE cluster\nnamed wetopi7, spread over the 8 OVH dedicated servers (see\novh-and-network.md). OpenTofu owns the VMs and the\nnode VLAN/bridge shape through the Proxmox HTTPS API (ADR-0003 \u2014\nOpenTofu and Ansible responsibility split).
This pill does not restate the tfvars. The authoritative declarative\nsource is\ntofu/proxmox/environments/prod/\nin the GitOps repo\nhttps://github.com/wetopi-platform/infrastructure.\nThe committed terraform.tfvars.example documents the shape; the real\nterraform.tfvars is gitignored and mode 0600 (never in Git). The table\nbelow is a point-in-time restatement of that tfvars (verified\n2026-08-31) so a new engineer can read the inventory without the ignored\nfile; the tfvars always wins on any drift.
8 Proxmox nodes (= the 8 OVH servers), each with one 802.1Q trunk NIC\ncarrying the vRack VLANs (vmbr3 WAN, vmbr4 node LAN, vmbr2 Ceph). Trunk\ndevices (proxmox_nodes in tfvars, verified live via the Proxmox API):
| Node | \nTrunk NIC | \n
|---|---|
| ns31826925, ns31826929, ns3235424, ns3243421, ns3249468 | \nenp10s0f1np1 | \n
| ns3214108, ns3226931, ns3226939 | \nenp2s0f1np1 | \n
| Item | \nValue | \nSource | \n
|---|---|---|
| Base VM template | \nvmssd, VMID 9000 on node ns3243421 | \ntemplate_vm_id = 9000, template_node_name = \"ns3243421\" | \n
| VM datastore | \nvmssd (NVMe SSD) | \ndatastore_id per VM | \n
| Template download datastore | \nlocal | \ntemplate_download_datastore_id = \"local\" | \n
| Guest cloud-init identity | \nUbuntu 24.04, first-boot SSH/network | \ntemplate + ssh_public_key_file | \n
New VMs are cloned from template 9000 and then converged by cloud-init\n(first boot) and Ansible (guest config) \u2014 OpenTofu sets identity/placement\nand never touches guest OS state.
\nVMIDs 8001-8042. \"Datastore\" is the VM disk location; LAN IPs are on\nvmbr4/10.72.0.0/23; WAN IPs (where present) are on vmbr3.
| Name | \nVMID | \nNode | \nRole | \nvCPU/RAM/Disk | \nDatastore | \nLAN IP | \nWAN IP | \n
|---|---|---|---|---|---|---|---|
| wetopi-ops01 | \n8001 | \nns3249468 | \noperations (bastion) | \n2 / 4096 MB / 40 GB | \nvmssd | \n10.72.0.10 | \n92.222.245.91 | \n
| rke-cp01 | \n8011 | \nns3243421 | \ncontrol-plane | \n4 / 8192 MB / 50 GB | \nlocal | \n10.72.0.21 | \n\u2014 | \n
| rke-cp02 | \n8012 | \nns3226939 | \ncontrol-plane | \n4 / 8192 MB / 50 GB | \nlocal | \n10.72.0.22 | \n\u2014 | \n
| rke-cp03 | \n8013 | \nns3214108 | \ncontrol-plane | \n4 / 8192 MB / 50 GB | \nlocal | \n10.72.0.23 | \n\u2014 | \n
| rke-platform01 | \n8021 | \nns31826929 | \nplatform worker | \n4 / 8192 MB / 60 GB | \nvmssd | \n10.72.0.41 | \n\u2014 | \n
| rke-platform02 | \n8022 | \nns3226931 | \nplatform worker | \n4 / 8192 MB / 60 GB | \nvmssd | \n10.72.0.42 | \n\u2014 | \n
| rke-platform03 | \n8023 | \nns31826925 | \nplatform worker | \n4 / 8192 MB / 60 GB | \nvmssd | \n10.72.0.43 | \n\u2014 | \n
| wetopi-natgw01 | \n8031 | \nns3243421 | \nnat-gateway (egress) | \n2 / 2048 MB / 20 GB | \nvmssd | \n10.72.0.2 | \n92.222.245.83 | \n
| rke-wan01 | \n8041 | \nns3235424 | \nedge (WAN) | \n2 / 4096 MB / 20 GB | \nvmssd | \n10.72.0.51 | \n79.137.114.193 | \n
| rke-wan02 | \n8042 | \nns3226931 | \nedge (WAN) | \n2 / 4096 MB / 20 GB | \nvmssd | \n10.72.0.52 | \n79.137.114.194 | \n
Roles and notes. wetopi-ops01 is the SSH bastion (bastion = true) \u2014\nthe only VM with a direct public address other than the edge/natgw; all\nother VMs are reached through it. The three control planes own the keepalived\nRKE2 API VIP 10.72.0.20 (ADR-0004).\nwetopi-natgw01 is the LAN's NAT egress (LAN\u2192WAN forward policy only).\nrke-wan01/02 are the edge v1 nodes (ADR-0006);\nthey were initially built as wetopi3-wan1/2 and renamed to rke-wan* on\n2026-08-30 (VMIDs 8041/8042 unchanged).
Storage NICs (ADR-0007 \u2014 verified live 2026-08-31). The three platform\nworkers additionally carry a third NIC on vmbr2/VLAN 2 (the Ceph\nstorage network) with the operator-reserved addresses:\nrke-platform01=192.168.98.10/20, rke-platform02=192.168.98.11/20,\nrke-platform03=192.168.98.12/20 \u2014 no gateway, no default route (storage\ntraffic is directly connected; see ceph.md). The control planes\nstay single-NIC in v1. This third NIC was the ADR-0007 mutating apply; it is\npresent in the live VMs and the OpenTofu state (the base table above reflects\nthe tfvars snapshot and lists only the LAN/WAN NICs).
flowchart TB\n subgraph cluster[\"Proxmox cluster wetopi7 \u2014 8 nodes\"]\n direction TB\n subgraph nA[\"ns3249468\"]\n ops01[\"wetopi-ops01 \u00b7 8001 \u00b7 bastion\"]\n end\n subgraph nB[\"ns3243421\"]\n cp01[\"rke-cp01 \u00b7 8011 \u00b7 control-plane\"]\n natgw[\"wetopi-natgw01 \u00b7 8031 \u00b7 NAT egress\"]\n tpl[\"base template vmssd \u00b7 VMID 9000\"]\n end\n subgraph nC[\"ns3226939\"]\n cp02[\"rke-cp02 \u00b7 8012 \u00b7 control-plane\"]\n end\n subgraph nD[\"ns3214108\"]\n cp03[\"rke-cp03 \u00b7 8013 \u00b7 control-plane\"]\n end\n subgraph nE[\"ns31826929\"]\n p01[\"rke-platform01 \u00b7 8021 \u00b7 worker (+Ceph NIC)\"]\n end\n subgraph nF[\"ns3226931\"]\n p02[\"rke-platform02 \u00b7 8022 \u00b7 worker (+Ceph NIC)\"]\n wan2[\"rke-wan02 \u00b7 8042 \u00b7 edge\"]\n end\n subgraph nG[\"ns31826925\"]\n p03[\"rke-platform03 \u00b7 8023 \u00b7 worker (+Ceph NIC)\"]\n end\n subgraph nH[\"ns3235424\"]\n wan1[\"rke-wan01 \u00b7 8041 \u00b7 edge\"]\n end\n end\n ops01 -.->|\"SSH bastion to all VMs\"| cp01\ntofu plan on tofu/proxmox/environments/prod (no-changes\nexpected) and qemu-pending/qemu-status per VM via the Proxmox API or\nops01.ops01 node ns3249468 also hosts the Proxmox API endpoint used by\nOpenTofu (PROXMOX_VE_ENDPOINT); see\nproviders.md.\n\nMigrado desde
\nwetopi-platform/docs@main(architecture/overview-topology.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ARCH-PLATFORM-TOPOLOGY: Platform topology \u2014 global three-layer map
\nThe handover map: one diagram, three layers, every node linked to its\npill. This is the page a new technician reads first; from here they\nnavigate the entire platform \u2014 physical infrastructure, the K8s platform\nlayer, and the deployed services \u2014 without guessing where anything lives.
\nlast_verified. Live state always wins (DocOps \u00a71) \u2014 re-read it with\nkubectl before relying on it.flowchart TB\n subgraph L1[\"Layer 1 \u2014 INFRASTRUCTURE (OVH \u00b7 Proxmox \u00b7 Ceph)\"]\n direction TB\n OVH[\"OVH \u2014 8 dedicated servers \u00b7 one vRack\"]\n PX[\"Proxmox VE cluster wetopi7 \u2014 8 nodes \u00b7 VMs 8001-8042\"]\n IAC[\"IaC \u2014 OpenTofu (bpg) + Cloudflare + Ansible \u00b7 state 0600, never in Git\"]\n BAST[\"wetopi-ops01 \u2014 SSH bastion \u00b7 92.222.245.91\"]\n NAT[\"natgw \u2014 LAN egress \u00b7 92.222.245.83\"]\n CEPH[\"Ceph LAN 192.168.96.0/20 \u2014 VLAN 2 \u00b7 pre-existing\"]\n OVH --> PX\n OVH --> BAST\n OVH --> NAT\n IAC -.->|\"Proxmox HTTPS API\"| PX\n end\n subgraph L2[\"Layer 2 \u2014 PLATFORM (RKE2 \u00b7 Cilium \u00b7 Edge \u00b7 GitOps)\"]\n direction TB\n K8S[\"RKE2 cluster wetopi3 \u2014 3 CP + 3 platform + 2 edge workers \u00b7 API VIP 10.72.0.20\"]\n CILIUM[\"Cilium CNI + Hubble (no Canal \u2014 ADR-0005)\"]\n EDGE[\"WAN edge v1 \u2014 Traefik on rke-wan01/02 \u00b7 Gateway wetopi-edge\"]\n GITOPS[\"GitOps \u2014 Argo CD v3.4.8 \u00b7 AppProject platform\"]\n K8S --- CILIUM\n K8S --- EDGE\n K8S --- GITOPS\n end\n subgraph L3[\"Layer 3 \u2014 SERVICES (deployed via GitOps)\"]\n direction LR\n SVCARGO[\"Argo CD \u2014 argo.wetopi.net\"]\n SVCVAULT[\"Vault / Infisical \u2014 vault.wetopi.net\"]\n SVCTRAEFIK[\"Traefik edge \u2014 public routing\"]\n SVCCSI[\"Ceph CSI \u2014 StorageClass ceph-rbd\"]\n SVCCERT[\"cert-manager \u2014 Let's Encrypt DNS-01\"]\n SVODOCS[\"Docs site \u2014 docs.wetopi.net\"]\n end\n PX -->|\"VMs 8001-8042 host the nodes\"| K8S\n BAST -->|\"SSH + kubectl (kubeconfig from rke-cp01)\"| K8S\n CEPH -->|\"workers eth1 \u00b7 ADR-0007\"| SVCCSI\n GITOPS -->|\"syncs kubernetes/ tree\"| L3\n EDGE --- SVCTRAEFIK\n SVCTRAEFIK --- SVCARGO\n SVCTRAEFIK --- SVCVAULT\n SVCTRAEFIK --- SVODOCS\n SVCCERT -->|\"argo-tls \u00b7 docs-tls \u00b7 vault-tls\"| EDGE\n SVCCSI -->|\"PVCs on ceph-rbd\"| SVCVAULT\n\n click L1 \"infrastructure section map\" \"infrastructure/index.html\"\n click OVH \"ARCH-INFRA-OVH\" \"infrastructure/ovh-and-network.html\"\n click PX \"ARCH-INFRA-PROXMOX\" \"infrastructure/proxmox-cluster.html\"\n click IAC \"ARCH-INFRA-PROVIDERS\" \"infrastructure/providers.html\"\n click BAST \"bastion ops01 (in ARCH-INFRA-OVH)\" \"infrastructure/ovh-and-network.html\"\n click NAT \"natgw (in ARCH-INFRA-OVH)\" \"infrastructure/ovh-and-network.html\"\n click CEPH \"ARCH-INFRA-CEPH\" \"infrastructure/ceph.html\"\n click L2 \"platform section map\" \"platform/index.html\"\n click K8S \"ARCH-PLATFORM-RKE2\" \"platform/rke2-cluster.html\"\n click CILIUM \"ARCH-PLATFORM-CILIUM\" \"platform/cilium-network.html\"\n click EDGE \"ARCH-PLATFORM-EDGE\" \"platform/edge.html\"\n click GITOPS \"ARCH-PLATFORM-GITOPS\" \"platform/gitops.html\"\n click SVCARGO \"SVC-ARGOCD\" \"../services/argocd.html\"\n click SVCVAULT \"SVC-VAULT\" \"../services/vault.html\"\n click SVCTRAEFIK \"SVC-TRAEFIK-EDGE\" \"../services/traefik-edge.html\"\n click SVCCSI \"SVC-CEPH-CSI\" \"../services/ceph-csi.html\"\n click SVCCERT \"SVC-CERT-MANAGER\" \"../services/cert-manager.html\"\n click SVODOCS \"SVC-DOCS-SITE\" \"../services/docs-site.html\"\nEvery node of the diagram resolves to the pill below. Relative links work\nfrom this page; the rendered site mirrors the same mapping.
\n| Layer | \nNode | \nPill | \nWhat it covers | \n
|---|---|---|---|
| 1 \u2014 infra | \nOVH, bastion (ops01), natgw | \nARCH-INFRA-OVH | \nProvider, vRack, WAN blocks, VLAN/bridge model, Cloudflare DNS. | \n
| 1 \u2014 infra | \nProxmox cluster wetopi7 | \nARCH-INFRA-PROXMOX | \n8 nodes, template, datastores, VM inventory (8001\u20138042). | \n
| 1 \u2014 infra | \nIaC toolchain | \nARCH-INFRA-PROVIDERS | \nOpenTofu/Cloudflare/Ansible split, where state lives, secret handling. | \n
| 1 \u2014 infra | \nCeph LAN | \nARCH-INFRA-CEPH | \nPre-existing storage LAN, storage-native access (ADR-0007). | \n
| 2 \u2014 platform | \nRKE2 cluster wetopi3 | \nARCH-PLATFORM-RKE2 | \nTopology, control plane, API VIP, node roles. | \n
| 2 \u2014 platform | \nCilium | \nARCH-PLATFORM-CILIUM | \nCNI, Hubble, edge-node datapath. | \n
| 2 \u2014 platform | \nWAN edge v1 | \nARCH-PLATFORM-EDGE | \nDirect Traefik exposure, Gateway API, nftables posture. | \n
| 2 \u2014 platform | \nGitOps plane | \nARCH-PLATFORM-GITOPS | \nArgo CD bootstrap, AppProject, app layout. | \n
| 3 \u2014 services | \nArgo CD | \nSVC-ARGOCD | \nGitOps control plane, argo.wetopi.net. | \n
| 3 \u2014 services | \nVault / Infisical | \nSVC-VAULT | \nSecrets system, vault.wetopi.net UI, Universal Auth (private). | \n
| 3 \u2014 services | \nTraefik edge | \nSVC-TRAEFIK-EDGE | \nPublic routing for docs/argo/vault hostnames. | \n
| 3 \u2014 services | \nCeph CSI | \nSVC-CEPH-CSI | \nRBD storage, StorageClass ceph-rbd. | \n
| 3 \u2014 services | \ncert-manager | \nSVC-CERT-MANAGER | \nACME/Let's Encrypt DNS-01, the edge certificates. | \n
| 3 \u2014 services | \nDocs site | \nSVC-DOCS-SITE | \ndocs.wetopi.net, the site you are reading. | \n
Section entry points: infrastructure section \u00b7\nplatform section \u00b7 services section \u00b7\ndecisions \u00b7 runbooks.
\nDated handover snapshot \u2014 re-read live before relying on it:
\nwetopi-edge Accepted+Programmed with the web :8000 and\nhttps :8443 listeners.root, cert-manager, docs-site,\nargo-edge (all Healthy/Synced) and vault (Synced; its Degraded\nhealth is the documented false positive of the fail-closed deny\nrules \u2014 pods 3/3 Running, see SVC-VAULT);\n4 certificates Ready (argo-tls, docs-tls, vault-tls, edge-test-tls).This map is the shape. The procedure is\ntaking-over-the-platform.md\n(PROC-PLATFORM-TAKEOVER): access by location, sources of truth, day-1\nverification, frequent operations, escalation and the no-go rules.
\n" }, { "id": 40, "title": "cilium network", "slug": "cilium-network", "book_id": 3, "chapter_id": 16, "description": null, "created_at": "2026-09-06T13:38:55.000000Z", "updated_at": "2026-09-06T21:48:07.000000Z", "markdown": "> **ARCHIVO \u2014 versi\u00f3n migrada cruda.** La ficha estructurada con las 3 preguntas es la versi\u00f3n `ARCH-*`/`SVC-*` de este mismo chapter (actualizada con datos live 2026-09-06). Esta copia cruda se conserva por el historial.\n\n> Migrado desde `wetopi-platform/docs@main` (architecture/platform/cilium-network.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.\n\n Cilium network \u2014 CNI, Hubble and edge-node datapath\n\n## Purpose\n\nThe pod/service network of the RKE2 cluster: who assigns pod IPs, how nodes\ntalk across the LAN, and how the dual-NIC edge workers keep their cluster\ndatapath strictly on the LAN. Decision record:\n[ADR-0005](../../decisions/ADR-0005-cilium-first-addon.md) (accepted) \u2014 Cilium\nis the CNI from day one; there was never a Canal provisional phase.\n\n## Design\n\nRKE2 boots with `cni: none` so the bundled Canal manifest never deploys; the\nofficial Cilium chart is installed as the first addon and owns the datapath.\n\n- **Chart**: pinned `cilium:1.20.1` from `oci://quay.io/cilium/charts`\n (values in Git: `kubernetes/cilium/values.yaml` in the\n [infrastructure](https://github.com/wetopi-platform/infrastructure) repo).\n- **Mode**: VXLAN encapsulation (`routingMode: tunnel`,\n `tunnelProtocol: vxlan`); no BGP, no native routing.\n- **IPAM**: cluster pool mirroring the RKE2 pod CIDR `10.64.0.0/14` exactly\n (`clusterPoolIPv4PodCIDRList`), `/24` allocated per node.\n- **kube-proxy coexistence**: `kubeProxyReplacement: false` for v1 \u2014 the\n service CIDR `10.68.0.0/16` stays owned by kube-proxy.\n- **Hubble**: enabled with Hubble Relay; relay UI off at bootstrap.\n\nCIDR ownership (must never diverge from the rke2-server config):\n\n| Range | Owner | Cilium role |\n|---|---|---|\n| `10.64.0.0/14` | RKE2 (pod CIDR) | IPAM cluster pool mirrors it |\n| `10.68.0.0/16` | RKE2 / kube-proxy (service CIDR) | none |\n\n## Underlay and ports\n\nTunnels ride the node LAN (`10.72.0.0/23`, single-NIC nodes, MTU 1500). The\nCilium MTU is left at its auto-computed default (~1450, 50-byte VXLAN\noverhead). Node-to-node matrix that must stay allowed on the LAN:\n\n```text\nVXLAN tunnels: UDP 8472\nHubble Relay: TCP 4244\nkubelet: TCP 10250\n```\n\nThe edge workers carry these ports on their **LAN** interface only\n(`edge_lb` role asserts they never appear on the WAN interface).\n\n## Edge-node datapath pin (WAN Edge v1)\n\nThe chart leaves `devices` unset, so the Cilium agent auto-detects its\ninterface. On the single-NIC nodes that picks `eth0` (the LAN) \u2014 correct. On\nthe dual-NIC edge workers `eth0` is the **WAN**; auto-detection would attach\nthe datapath to it. The fix is a namespaced `CiliumNodeConfig`\n(`kubernetes/cilium/edge-node-config.yaml`, applied after the chart install\nby the bootstrap playbook and reconciled by the Cilium Argo Application):\n\n```yaml\nspec:\n nodeSelector:\n matchLabels:\n wetopi.io/role: edge\n defaults:\n devices: eth1 # eth1 = LAN NIC on edge nodes (WAN Edge v1 plan \u00a7F1)\n```\n\nIt selects only `wetopi.io/role=edge` nodes; the six other nodes keep the\nauto-detect default.\n\n## GitOps ownership\n\nThe Cilium release is declared in Git as Argo CD Application `cilium`\n(`kubernetes/cilium/application.yaml`): Helm source (pinned 1.20.1 + Git\nvalues) plus a directory source carrying exactly\n`edge-node-config.yaml`, so the release and its edge override share one\nownership/pruning policy. `Prune=false` and `ignoreDifferences` protect the\nchart-generated TLS secrets (`cilium-ca`, `hubble-relay-client-certs`,\n`hubble-server-certs`) whose live `.data` must never be replaced by Helm\nmaterial. Like `argo-edge`, the Application itself still follows the\noperator-bootstrap path (`kubectl apply` from ops01) \u2014 it is not yet anchored\nin the root app's path (see [gitops.md](gitops.md)).\n\n## Invariants\n\n- Pod IPs only from `10.64.0.0/14`; the Cilium pool must always mirror the\n rke2-server `cluster-cidr`.\n- Edge-node datapath is pinned to `eth1` (LAN); a `CiliumNodeConfig` change\n or label drift that moves it to `eth0` is an incident (cluster traffic on\n the WAN \u2014 ADR-0006 violation).\n- VXLAN/Hubble/kubelet ports LAN-only on edge nodes (asserted by the\n `edge_lb` role).\n- No BGP/native routing in v1.\n\n## Verified state (2026-08-31, live from ops01)\n\nCilium 1.20.1 pods Running in `kube-system` on all nodes (DaemonSet +\noperator), Hubble Relay up, one `CiliumNode` per node, the\n`wetopi-edge-devices` CiliumNodeConfig present and matching the edge label.\nDated observation \u2014 re-read `kubectl -n kube-system get pods,daemonset,ciliumnode`\nbefore relying on it.\n\n## Dependencies\n\n- Depends on: the RKE2 node set and CIDRs ([rke2-cluster.md](rke2-cluster.md)),\n the node-LAN underlay, the edge label contract\n ([edge.md](edge.md)).\n- Depends on this: the edge plane (Traefik ingress scheduling tolerates the\n `node.cilium.io/agent-not-ready` taint), GitOps handoff.\n\n## Failure considerations\n\n- Cilium down on a node: that node's pod traffic is down; with `cni: none`\n below it, the node is NotReady \u2014 no fallback CNI exists (by design).\n- VXLAN MTU surprises: symptom is black-holed large packets; the auto MTU\n keeps headroom, do not pin it manually.\n- Edge datapath mis-attachment: detect via `cilium status`/connectivity\n checks per [VALIDATION.md](https://github.com/wetopi-platform/infrastructure/blob/main/kubernetes/cilium/VALIDATION.md)\n in the infrastructure repo.", "html": "\n\nARCHIVO \u2014 versi\u00f3n migrada cruda. La ficha estructurada con las 3 preguntas es la versi\u00f3n
\nARCH-*/SVC-*de este mismo chapter (actualizada con datos live 2026-09-06). Esta copia cruda se conserva por el historial.
\n\nMigrado desde
\nwetopi-platform/docs@main(architecture/platform/cilium-network.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
Cilium network \u2014 CNI, Hubble and edge-node datapath
\nThe pod/service network of the RKE2 cluster: who assigns pod IPs, how nodes\ntalk across the LAN, and how the dual-NIC edge workers keep their cluster\ndatapath strictly on the LAN. Decision record:\nADR-0005 (accepted) \u2014 Cilium\nis the CNI from day one; there was never a Canal provisional phase.
\nRKE2 boots with cni: none so the bundled Canal manifest never deploys; the\nofficial Cilium chart is installed as the first addon and owns the datapath.
cilium:1.20.1 from oci://quay.io/cilium/charts\n(values in Git: kubernetes/cilium/values.yaml in the\ninfrastructure repo).routingMode: tunnel,\ntunnelProtocol: vxlan); no BGP, no native routing.10.64.0.0/14 exactly\n(clusterPoolIPv4PodCIDRList), /24 allocated per node.kubeProxyReplacement: false for v1 \u2014 the\nservice CIDR 10.68.0.0/16 stays owned by kube-proxy.CIDR ownership (must never diverge from the rke2-server config):
\n| Range | \nOwner | \nCilium role | \n
|---|---|---|
10.64.0.0/14 | \nRKE2 (pod CIDR) | \nIPAM cluster pool mirrors it | \n
10.68.0.0/16 | \nRKE2 / kube-proxy (service CIDR) | \nnone | \n
Tunnels ride the node LAN (10.72.0.0/23, single-NIC nodes, MTU 1500). The\nCilium MTU is left at its auto-computed default (~1450, 50-byte VXLAN\noverhead). Node-to-node matrix that must stay allowed on the LAN:
VXLAN tunnels: UDP 8472\nHubble Relay: TCP 4244\nkubelet: TCP 10250\nThe edge workers carry these ports on their LAN interface only\n(edge_lb role asserts they never appear on the WAN interface).
The chart leaves devices unset, so the Cilium agent auto-detects its\ninterface. On the single-NIC nodes that picks eth0 (the LAN) \u2014 correct. On\nthe dual-NIC edge workers eth0 is the WAN; auto-detection would attach\nthe datapath to it. The fix is a namespaced CiliumNodeConfig\n(kubernetes/cilium/edge-node-config.yaml, applied after the chart install\nby the bootstrap playbook and reconciled by the Cilium Argo Application):
spec:\n nodeSelector:\n matchLabels:\n wetopi.io/role: edge\n defaults:\n devices: eth1 # eth1 = LAN NIC on edge nodes (WAN Edge v1 plan \u00a7F1)\nIt selects only wetopi.io/role=edge nodes; the six other nodes keep the\nauto-detect default.
The Cilium release is declared in Git as Argo CD Application cilium\n(kubernetes/cilium/application.yaml): Helm source (pinned 1.20.1 + Git\nvalues) plus a directory source carrying exactly\nedge-node-config.yaml, so the release and its edge override share one\nownership/pruning policy. Prune=false and ignoreDifferences protect the\nchart-generated TLS secrets (cilium-ca, hubble-relay-client-certs,\nhubble-server-certs) whose live .data must never be replaced by Helm\nmaterial. Like argo-edge, the Application itself still follows the\noperator-bootstrap path (kubectl apply from ops01) \u2014 it is not yet anchored\nin the root app's path (see gitops.md).
10.64.0.0/14; the Cilium pool must always mirror the\nrke2-server cluster-cidr.eth1 (LAN); a CiliumNodeConfig change\nor label drift that moves it to eth0 is an incident (cluster traffic on\nthe WAN \u2014 ADR-0006 violation).edge_lb role).Cilium 1.20.1 pods Running in kube-system on all nodes (DaemonSet +\noperator), Hubble Relay up, one CiliumNode per node, the\nwetopi-edge-devices CiliumNodeConfig present and matching the edge label.\nDated observation \u2014 re-read kubectl -n kube-system get pods,daemonset,ciliumnode\nbefore relying on it.
node.cilium.io/agent-not-ready taint), GitOps handoff.cni: none\nbelow it, the node is NotReady \u2014 no fallback CNI exists (by design).cilium status/connectivity\nchecks per VALIDATION.md\nin the infrastructure repo.\n\nARCHIVO \u2014 versi\u00f3n migrada cruda. La ficha estructurada con las 3 preguntas es la versi\u00f3n
\nARCH-*/SVC-*de este mismo chapter (actualizada con datos live 2026-09-06). Esta copia cruda se conserva por el historial.
\n\nMigrado desde
\nwetopi-platform/docs@main(architecture/platform/edge.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
WAN edge v1 \u2014 direct Traefik exposure without L4 layer
\nHow the platform publishes services to the Internet in the v1 window: the\nbounded exception that lets the first public services run without the\nfull L4 edge stack (VRRP VIP + IPVS + DSR) that the master plan requires for\nproduction. Decision record:\nADR-0006 (accepted\n2026-08-30, with the 2026-08-30 UPDATE adding argo.wetopi.net to the\nservice list).
rke-wan01 (79.137.114.193) and\nrke-wan02 (79.137.114.194), range 79.137.114.192/27. NIC1 = WAN\n(vmbr3, eth0), NIC2 = node-LAN (vmbr4, eth1) \u2014 the reverse of the\nsingle-NIC nodes; the edge_lb Ansible role owns the interface contract.\nThese are also cluster workers: wetopi.io/role=edge\n(rke2-cluster.md).HelmChartConfig rke2-traefik, hostNetwork: true) runs on edge nodes\nonly and binds the chart-native entryPoints 8000 (web) / 8443\n(websecure) in the host netns. Public 80/443 is bridged by in-VM\nnftables REDIRECT 80\u21928000, 443\u21928443 (the edge_lb role is the single\nenforcement point).Gateway API (v1) on the Traefik Gateway controller:
\n| Object | \nNamespace | \nRole | \n
|---|---|---|
GatewayClass wetopi-edge | \ncluster-scoped | \ncontrollerName: traefik.io/gateway-controller (chart provider enabled) | \n
Gateway wetopi-edge | \nkube-system | \nlisteners web :8000 (HTTP), https :8443 (Terminate); https allowedRoutes.namespaces.from: All to admit cross-namespace routes | \n
HTTPRoute argo (+ argo-http-redirect) | \nkube-system | \nargo.wetopi.net \u2192 argocd/argocd-server:80 behind Middleware argo-basic-auth (ReferenceGrant in argocd); port 80 route 308-redirects to https | \n
HTTPRoute docs-site | \ndocs-site | \ndocs.wetopi.net \u2192 docs-site Deployment (basic-auth Middleware), cross-namespace attachment via the from: All widening | \n
HTTPRoute vault | \nvault | \nvault.wetopi.net \u2192 Infisical Web UI; no basic-auth Middleware \u2014 the UI's own login+TOTP is the gate (operator decision) | \n
Certificates argo-tls, docs-tls, vault-tls | \nkube-system | \nLet's Encrypt PROD, DNS-01 via cert-manager; all four certs (incl. staging edge-test-tls) Ready live 2026-08-31 | \n
Secret wetopi-edge-default-cert | \nkube-system | \nbootstrap self-signed fallback cert (superseded per-name by the LE certs) | \n
CertificateRefs are additive and fail-safe ordered: a missing referenced\nSecret would make the whole https listener ResolvedRefs=False; every\ncert addition therefore follows \"Secret Ready first, Gateway after\".
flowchart LR\n I[\"internet client\"] -->|A x2 TTL 300| D[\"DNS round-robin<br/>79.137.114.193 / .194\"]\n D --> W1[\"rke-wan01<br/>nftables 80/443 REDIRECT \u2192 8000/8443\"]\n D --> W2[\"rke-wan02<br/>nftables 80/443 REDIRECT \u2192 8000/8443\"]\n W1 & W2 --> T[\"Traefik (rke2-traefik, hostNetwork)<br/>entryPoints web :8000 / https :8443\"]\n T --> G[\"Gateway wetopi-edge (kube-system)<br/>https listener :8443 \u2014 TLS Terminate\"]\n G -->|HTTPRoute argo| M1[\"Middleware argo-basic-auth\"] --> S1[\"argocd/argocd-server:80\"]\n G -->|HTTPRoute docs-site| S2[\"docs-site Deployment :80\"]\n G -->|HTTPRoute vault| S3[\"Infisical Web UI (vault ns)\"]\nThe v1 exception covers exactly docs.wetopi.net + argo.wetopi.net\n(both recorded operator decisions; each further name is its own decision).\nvault.wetopi.net was added 2026-08-31 for the in-cluster Infisical Web UI\n(OBJ-VAULT-001 / ADR-0009)\nthrough the same edge machinery \u2014 see its ADR for the recorded decision.
The ADR's hard conditions:
\n10.72.0.5x, kubelet, Cilium/VXLAN on eth1.Edge objects are split: the Gateway/GatewayClass are live-verified objects in\nkubernetes/edge/ (F2 landing; the Gateway carries a strict\ncertificateRefs coordination note), and the per-service routes/certs live in\nkubernetes/edge-argo/ (Application argo-edge) and the service\ndirectories (docs-site/, vault/). See\ngitops.md for the app inventory.
edge_lb.Accepted=False).One Gateway wetopi-edge (kube-system) Accepted=True + Programmed=True\n(plus the wetopi-edge GatewayClass); 3 HTTPRoutes (docs-site, argo,\nvault) attached; 4 Certificates Ready (docs-tls, argo-tls, vault-tls\nPROD + edge-test-tls staging). Dated observation \u2014 re-read\nkubectl -n kube-system get gateway,httproute and\nkubectl get certificate -A before relying on it.
\n\nARCHIVO \u2014 versi\u00f3n migrada cruda. La ficha estructurada con las 3 preguntas es la versi\u00f3n
\nARCH-*/SVC-*de este mismo chapter (actualizada con datos live 2026-09-06). Esta copia cruda se conserva por el historial.
\n\nMigrado desde
\nwetopi-platform/docs@main(architecture/platform/gitops.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
GitOps plane \u2014 Argo CD v3.4.8 bootstrap and app layout
\nThe deployment plane for everything that runs inside the cluster. The\nownership boundary (ADR-0003 split, made operational): Ansible = node\nbootstrap only; Argo CD = everything in-cluster after node bootstrap.\nThere is deliberately no CI kubectl apply pipeline \u2014 Git is the\ndeployment path.
argoproj/argo-cd v3.4.8 manifest, upstream SHA-256 recorded in\ndocs/gitops.md of the\ninfrastructure repo)\nand vendored bit-identical in\nkubernetes/gitops/bootstrap/argocd-install-v3.4.8/ (namespace file,\nserver.insecure=\"true\" params, install.yaml).wetopi-ops01 during apply runbook Phase G\n(2026-08-30), with two recorded apply-time deviations: the org disables\ndeploy keys (repo credential is therefore a read-only HTTPS token in\nrepository Secret argocd/repo-infra-https), and the\napplicationsets.argoproj.io CRD exceeds kubectl's client-side annotation\ncap and is applied server-side (split manifest).server.insecure: \"true\" \u2014 the Argo CD server\nserves plain HTTP; TLS for argo.wetopi.net terminates at the edge\nGateway (edge.md).stable URL.One AppProject, platform\n(kubernetes/gitops/app.project.yaml):
sourceRepos: the private infrastructure repo over HTTPS (token in\nargocd/repo-infra-https) + quay.io/jetstack/charts (public OCI Helm\nregistry for cert-manager; nothing outside that exact path is admitted).destinations: in-cluster only, namespace: '*' \u2014 AppProject\ndestinations support globs but not negation, so the argocd-namespace\nself-management exclusion is procedural (the Argo control plane is\ninstalled only from the pinned manifests by human apply; Applications are\nreviewed before apply; mechanical enforcement = admission control, which\ndoes not exist at bootstrap \u2014 OPEN). ResourceQuota/LimitRange kinds are\nblacklisted.clusterResourceWhitelist: */* (cluster-scoped resources needed at\nbootstrap).The layout is a minimal viable app-of-apps:
\nflowchart TB\n subgraph GIT[\"GitHub (private)\"]\n R[\"wetopi-platform/infrastructure<br/>kubernetes/ (desired state)\"]\n end\n subgraph CL[\"cluster (ops01 kubectl for bootstrap only)\"]\n RC[\"AppProject platform\"]\n ROOT[\"Application root<br/>path kubernetes/platform/\"]\n ROOT --> CM[\"Application cert-manager<br/>\u2192 Helm chart + F3 manifests\"]\n ROOT --> DS[\"Application docs-site<br/>\u2192 6 workload manifests\"]\n ROOT --> VT[\"Application vault<br/>\u2192 Infisical manifests (ns vault)\"]\n OE[\"Application argo-edge (not anchored)<br/>\u2192 kube-system edge-argo manifests\"]\n CL2[\"Application cilium (not anchored)<br/>\u2192 Helm 1.20.1 + edge-node-config\"]\n end\n R -->|\"HTTPS + repo Secret argocd/repo-infra-https\"| RC\n RC --> ROOT\n RC -. \"bootstrap apply (ops01)\" .-> OE & CL2\nkubernetes/platform/, so they survive cluster state loss via\nroot prune+selfHeal): root, cert-manager, docs-site, vault.\nAnchored children carry the resources-finalizer.argocd.argoproj.io\nfinalizer \u2014 deleting an anchoring merge cascades foreground (children's\nworkloads are deleted before the Application CR disappears), so a revert\nnever orphans live state.argo-edge, cilium.root, argo-edge,\ncert-manager, docs-site, vault. vault is Degraded: its two\noperator-provisioned Secrets (postgres/infra credentials) predate S0\ninputs and must exist on-cluster before first sync \u2014 expected, tracked by\nOBJ-VAULT-001. Per-service facts live in services/.The broader App-of-Apps vs ApplicationSet vs per-domain split remains OPEN\n(master architecture plan \u00a79.4); this layout is built to stay evolvable.
\nkubectl from ops01 is\nfor recovery/surgery and must be reconciled back into Git.argo-basic-auth,\ndocs-site-basic-auth, the vault credentials, the repo token) exist only\non-cluster \u2014 Argo with default sync options neither prunes nor replaces\nthem.Argo CD v3.4.8 7/7 pods Running/Ready; 5 Applications in project\nplatform as listed above; argocd/repo-infra-https present. Dated\nobservation \u2014 re-read argocd app list before relying on it.
\n\nARCHIVO \u2014 versi\u00f3n migrada cruda. La ficha estructurada con las 3 preguntas es la versi\u00f3n
\nARCH-*/SVC-*de este mismo chapter (actualizada con datos live 2026-09-06). Esta copia cruda se conserva por el historial.
\n\nMigrado desde
\nwetopi-platform/docs@main(architecture/platform/rke2-cluster.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
RKE2 cluster \u2014 topology, control plane and node roles
\nHow the Kubernetes cluster layer is built on the Wetopi node LAN: node\ntopology, the HA control plane and its stable API endpoint, and the node-role\ncontract that the CNI, edge and GitOps planes all depend on.
\n3 control planes + 3 platform workers + 2 edge workers = 8 nodes, all\nReady (live-verified from wetopi-ops01, 2026-08-31). RKE2 v1.36.3+rke2r1\npinned on every node. All node identity and cluster-plane traffic rides the\nnode LAN (VLAN 4, 10.72.0.0/23); edge nodes additionally carry a WAN NIC\n(see edge.md).
flowchart LR\n subgraph LAN[\"Node LAN \u2014 VLAN 4, 10.72.0.0/23 (vmbr4)\"]\n direction TB\n subgraph CP[\"Control plane (embedded etcd)\"]\n CP1[\"rke-cp01<br/>10.72.0.21 (bootstrap)\"]\n CP2[\"rke-cp02<br/>10.72.0.22\"]\n CP3[\"rke-cp03<br/>10.72.0.23\"]\n end\n VIP[\"API VIP 10.72.0.20<br/>keepalived VRRP\"]\n subgraph PW[\"Platform workers<br/>wetopi.io/role=platform\"]\n P1[\"rke-platform01<br/>10.72.0.41\"]\n P2[\"rke-platform02<br/>10.72.0.42\"]\n P3[\"rke-platform03<br/>10.72.0.43\"]\n end\n subgraph EW[\"Edge workers<br/>wetopi.io/role=edge (dual-NIC)\"]\n E1[\"rke-wan01<br/>LAN 10.72.0.51 / WAN 79.137.114.193\"]\n E2[\"rke-wan02<br/>LAN 10.72.0.52 / WAN 79.137.114.194\"]\n end\n end\n VIP --- CP1 & CP2 & CP3\n CP1 -->|server :9345 / API :6443| P1 & P2 & P3 & E1 & E2\n| Node | \nLAN IP | \nRole / label | \nNotes | \n
|---|---|---|---|
rke-cp01 | \n10.72.0.21 | \ncontrol-plane (bootstrap) | \nnode-role.kubernetes.io/control-plane:NoSchedule taint | \n
rke-cp02 | \n10.72.0.22 | \ncontrol-plane | \nkeepalived priority 100 | \n
rke-cp03 | \n10.72.0.23 | \ncontrol-plane | \nkeepalived priority 50 | \n
rke-platform01..03 | \n10.72.0.41-43 | \nwetopi.io/role=platform | \nhost internal platform services | \n
rke-wan01 | \n10.72.0.51 | \nwetopi.io/role=edge | \nWAN 79.137.114.193 (eth0); node-ip bound to LAN | \n
rke-wan02 | \n10.72.0.52 | \nwetopi.io/role=edge | \nWAN 79.137.114.194 (eth0) | \n
Node sizing and datastore placement (Proxmox nodes, vCPU/RAM/disk, local\nvs vmssd) are physical-infra facts \u2014 see the legacy\nbootstrap VMs record until the\ninfra section reorg lands.
10.72.0.20 owned by a keepalived VRRP group (VRID 51, unicast\npeers) on the CP nodes: cp01 priority 150 (MASTER), cp02 100, cp03 50. The\nhealth check probes the local kube-apiserver https://127.0.0.1:6443\nand the RKE2 supervisor on 127.0.0.1:9345, so a failing CP drops the VIP\nto the next priority.https://10.72.0.20:9345,\nAPI endpoint https://10.72.0.20:6443 (kubeconfig clients). tls-san is\nIP-only: the VIP + the three CP IPs, no DNS SAN.The decision record and failure-mode analysis:\nADR-0004.
\nThe cluster is built by Ansible from the\ninfrastructure repo \u2014\nAnsible owns node bootstrap only; everything inside the cluster is Argo CD's\n(see gitops.md):
\n| Piece | \nPath in infrastructure | \nRole | \n
|---|---|---|
rke2-server role + rke2-cp.yml | \nansible/roles/rke2-server/, ansible/playbooks/rke2-cp.yml | \nrenders /etc/rancher/rke2/config.yaml on CPs (bootstrap/join split), tls-san, CIDRs, cni: none | \n
keepalived role | \nansible/roles/keepalived/ | \nVRRP VIP 10.72.0.20 + local /readyz/9345 health check | \n
rke2-agent role + rke2-workers.yml | \nansible/roles/rke2-agent/, ansible/playbooks/rke2-workers.yml | \nworker join (platform + edge), exact-label gates | \n
rke2-workers-validate.yml | \nansible/playbooks/ | \npost-join validation (data-dir, labels, scheduling) | \n
| node token | \nansible/files/rke2/node-token | \nmode 0600, gitignored, never committed | \n
RKE2 configuration: cni: none (no Canal; Cilium is the CNI \u2014\ncilium-network.md), cluster-cidr: 10.64.0.0/14,\nservice-cidr: 10.68.0.0/16. Edge workers additionally carry\nnode-ip: 10.72.0.5x (LAN-bound node identity, ADR-0006 condition 4) and the\nedge label/taint consumed by the edge Traefik placement.
NoSchedule: control planes host no platform workloads.wetopi.io/role=platform / wetopi.io/role=edge\n(asserted at join time; the CNI edge override and edge scheduling both key\noff them).8/8 nodes Ready; RKE2 v1.36.3+rke2r1 on all nodes; embedded etcd quorum on\nthe three CPs. Dated observation \u2014 re-read kubectl get nodes before\nrelying on it (DocOps \u00a71: live state outranks this pill).
\n\nMigrado desde
\nwetopi-platform/docs@main(decisions/ADR-0001-rke2-node-lan.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ADR-0001: Dedicated RKE2 node LAN
\nDate: 2026-08-29\nStatus: Accepted, pending implementation
\nThe existing 192.168.96.0/20 network is used by Ceph and legacy VMs. The new\nRKE2 foundation must not increase coupling with that network or require\nrenumbering when hosting workers grow.
10.72.0.0/20 for Wetopi platform networks.10.72.0.0/23 for the initial RKE2 node LAN.vlan_rke and access bridge vmbr4 on each\nProxmox node.vmbr4 on the hypervisors.10.64.0.0/14 and Service CIDR 10.68.0.0/16 separate.vmbr2,\n192.168.96.0/20) with addresses from the operator-reserved\n192.168.98.0/25 \u2014 no routing and no NAT on the Ceph path. The\ninvariants this bullet protected are PRESERVED and unchanged: the Ceph\nnetwork is still never NATed on any path, and 192.168.96.0/20 is\nstill never renumbered; only the access model (routed vs directly\nattached) is replaced./24 blocks from the reserved /20.This decision supersedes earlier drafts that left the node LAN undefined or\nimplicitly reused 192.168.96.0/20.
\n\nMigrado desde
\nwetopi-platform/docs@main(decisions/ADR-0002-bootstrap-sizing.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ADR-0002: Minimal expandable bootstrap sizing
\nDate: 2026-08-29\nStatus: Accepted, pending implementation
\nThe RKE2 foundation needs production-shaped HA without reserving the larger\nresources proposed in early planning documents before actual utilization is\nknown.
\nThis decision replaces the larger initial allocations in earlier bootstrap\ndrafts while retaining their intended node roles and failure-domain separation.
\n" }, { "id": 46, "title": "ADR 0003 opentofu ansible", "slug": "adr-0003-opentofu-ansible", "book_id": 8, "chapter_id": 28, "description": null, "created_at": "2026-09-06T13:38:57.000000Z", "updated_at": "2026-09-06T13:38:57.000000Z", "markdown": "> Migrado desde `wetopi-platform/docs@main` (decisions/ADR-0003-opentofu-ansible.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.\n\n ADR-0003: OpenTofu and Ansible responsibility split\n\n**Date:** 2026-08-29\n**Status:** Accepted\n\n## Context\n\nProxmox resources and Ubuntu configuration have different lifecycles. Using\none tool for both would either weaken infrastructure state management or rely\non fragile remote provisioners.\n\n## Decision\n\n- OpenTofu owns Proxmox resources through its HTTPS API.\n- cloud-init owns only the first-boot identity, network, and SSH bootstrap.\n- Ansible owns the Ubuntu baseline, NAT gateway, and RKE2 configuration.\n- Argo CD owns Kubernetes add-ons and workloads after the cluster exists.\n- SSH access to Proxmox hypervisors is not required for the selected resource\n operations.\n\n## Consequences\n\n- OpenTofu plans and records virtual infrastructure lifecycle.\n- Ansible can repeatedly converge guest configuration without changing VM\n identity or placement.\n- No secrets are embedded in HCL or committed inventory.\n- Disaster recovery can reconstruct both infrastructure and guest state from\n separate, explicit sources of truth.", "html": "\n\nMigrado desde
\nwetopi-platform/docs@main(decisions/ADR-0003-opentofu-ansible.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ADR-0003: OpenTofu and Ansible responsibility split
\nDate: 2026-08-29\nStatus: Accepted
\nProxmox resources and Ubuntu configuration have different lifecycles. Using\none tool for both would either weaken infrastructure state management or rely\non fragile remote provisioners.
\n\n\nMigrado desde
\nwetopi-platform/docs@main(decisions/ADR-0004-rke2-api-endpoint.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ADR-0004: RKE2 API endpoint and routing to Ceph
\nDate: 2026-08-29\nStatus: Proposed \u2014 API endpoint in force; the Ceph routing decision\nbelow is superseded by ADR-0007\n(operator decision 2026-08-30: storage-native attachment, no routing, no\nnatgw involvement). The API endpoint design (keepalived VIP 10.72.0.20)\nis unaffected and remains in force.
\nRKE2 needs one stable control-plane endpoint, and the infrastructure README\ngates RKE2 installation on that endpoint and on routing to Ceph being\naccepted. Joining servers and agents register against port 9345\n(server: https://<endpoint>:9345); kube-apiserver serves 6443. A VIP is\nnot automatically included in the apiserver serving certificate, so it must\nbe added with tls-san. The SAN list is fixed at bootstrap; adding a SAN\nlater forces per-node certificate rotation. After a node joins, its rke2\nagent load-balances locally (port 6444) across the apiserver endpoints\ndiscovered from etcd, so the configured server: URL matters only for\ninitial or full re-registration.
No DNS is controlled anywhere in the platform (VMs use public resolvers),\nso bootstrap-safe endpoints are limited to an IP URL or /etc/hosts\nentries. The node LAN address plan already reserves 10.72.0.20 as the\nstable RKE2 API VIP. Nothing routes node-LAN traffic to Ceph\n192.168.96.0/20 today: the gateway has no VLAN 2 interface, no static\nroute, and its nftables forward policy drops everything except LAN-to-WAN.\nCandidate endpoints:
10.72.0.1 on wetopi-natgw01: rejected. It couples API\navailability to a 2 GiB egress gateway VM, whose loss would take the API\nand node egress down together, contradicts the reserved 10.72.0.20, and\nwould flap the API with future gateway VRRP changes./etc/hosts variant is unreliable (glibc typically returns\nonly the first matching entry) and must be kept in sync on eight nodes.wetopi-ops01: workable without Proxmox firewall\nchanges, but makes the deliberately minimal recovery bastion the API\nsingle point of failure and adds a long-lived service to it.10.72.0.21 with documented manual failover: zero\nextra SANs and the lowest operational burden, but fragile exactly when\nre-registration coincides with a downed control plane; upstream warns\nnode IPs may change over time.10.72.0.20 on the three control planes via\nkeepalived: already reserved in the address plan, lives inside the\ncluster failure domain, adds no VM or new single point of failure,\navailability tracks local apiserver readiness including etcd\nconnectivity, and costs one new Ansible role.The design set carries a standing proposal for this endpoint: \"2 x HAProxy
\nrke-cp01/02/03 owns 10.72.0.20.rke-cp01 is MASTER by priority, failing over to rke-cp02, then\nrke-cp03. VRID and priority values are implementation parameters;\nno other VRRP is documented on VLAN 4./readyz, which includes\netcd-backed readiness and fails when the local apiserver cannot reach\netcd as part of a quorum) and its local registration service on 9345\nis reachable. Exact check commands, scripts, and timings are\nimplementation parameters.\n/readyz fails, it releases the\nVIP, and the endpoint is down, which is correct: a usable API\nrequires etcd quorum (at least two control planes). The check does\nnot create quorum; it avoids pointing the endpoint at an apiserver\nthat already reports not ready.server: https://10.72.0.20:9345;\nkubeconfig and kubectl clients use https://10.72.0.20:6443. IP-only, no\nhostname SAN: no DNS is controlled, and this avoids a split-horizon\nhazard.tls-san identically on all three servers to the VIP plus all\ncontrol-plane IPs: 10.72.0.20, 10.72.0.21, 10.72.0.22,\n10.72.0.23, so any control-plane IP works for direct registration or\nclients.10.72.0.21:9345 before\nkeepalived is deployed, then switch server: URLs to the VIP; no\ncertificate rotation is needed either way.192.168.96.0/20 (per\nADR-0001), using gateway-side routing on wetopi-natgw01.vmbr2 with addresses from the reserved\n192.168.98.0/25 \u2014 no routing, no NAT, no return-routes, no natgw\ninvolvement. The bullet below is retained as the historical record of\nthe superseded design; its shape never left the default-off state.\nShape (superseded):\nvmbr2 with an\naddress in 192.168.96.0/20 (netplan; the OpenTofu VM module already\nsupports extra NICs), making the Ceph subnet a directly connected\nroute with no next-hop;10.72.0.1 already\ncovers the destination once the gateway owns the connected route;192.168.96.0/20 need a route for\n10.72.0.0/23 via the gateway's VLAN 2 address, shaped as\nip route 10.72.0.0/23 via <natgw VLAN 2 address>;iifname <lan_if> ip daddr 192.168.96.0/20 and the return direction\noifname <lan_if> ip saddr 192.168.96.0/20.Effects are stated for each scenario on: API reachability (6443 via the\nVIP for kubectl and new kubeconfigs), new node registration (9345 join),\nalready-joined agents (client-side LB on 6444 across surviving\ncontrol planes), Internet egress (nodes' default gateway 10.72.0.1 on\nwetopi-natgw01), and Ceph connectivity (since ADR-0007: the workers'\ndirect VLAN 2 attachment \u2014 storage no longer depends on the gateway at\nall; the pre-ADR-0007 routing remarks below are retained as the\nhistorical record).
rke-cp01 dies). VRRP fails the VIP over to\nrke-cp02 within the failover time: a brief interruption of API\nreachability and of new node registration, then both served by the new\nowner. Etcd keeps quorum (2 of 3). Already-joined agents are\nunaffected: their 6444 LB already spans the surviving control planes.\nInternet egress and Ceph routing are unaffected.rke-cp03 dies while not holding the VIP). No\nVIP movement. API reachability and new node registration are\nunaffected; etcd keeps quorum (2 of 3). Already-joined agents\nre-balance around the dead control plane on 6444. Internet egress and\nCeph routing are unaffected./readyz-gated check, a degraded owner\ndoes not hold the VIP: if etcd degrades on the owner only, the VIP\nmoves to a healthy owner and API reachability and registration\ncontinue. If quorum is lost cluster-wide, no control plane passes the\ncheck, none holds the VIP, and the API is intentionally unavailable\nregardless of VIP placement: a usable API requires etcd quorum, and\nthe endpoint must not present an API that cannot serve.\nAlready-joined agents see failing API requests for as long as quorum\nis lost (an etcd failure, not an endpoint failure) and resume when\nquorum returns. Internet egress and Ceph routing are unaffected.wetopi-natgw01 death. The VIP lives on the node LAN, so API\nreachability and new node registration are unaffected (this is\nexactly why the gateway VIP was rejected), and already-joined agents\nare unaffected. Internet egress is down for all nodes (default\ngateway 10.72.0.1 lost). server: URLs to a control-plane IP (for\nexample https://10.72.0.21:9345) and uninstall keepalived. No\ncertificate rotation is needed because all control-plane IPs are already\nin the SAN list.192.168.98.0/25, and no return-route owner,\ngateway firewall path or rp_filter verification is needed \u2014 there is no\ngateway on the path. The live VLAN 2 MTU check (1500, ADR-0001) and the\nVPN CIDR overlap validation survive as apply-time gates in\ninfrastructure/kubernetes/ceph-csi/VALIDATION.md). Historical record\nof the superseded gates:\n192.168.96.0/20, confirmed against live Ceph allocations;10.72.0.0/23 via the gateway's VLAN 2\naddress;rp_filter behavior: the gateway enforces rp_filter=1, and\nasymmetric replies could be dropped; must be field-verified;10.64.0.0/14 and 10.68.0.0/16.10.72.0.1, any DNS infrastructure, MTU changes, and storage design.\n\nMigrado desde
\nwetopi-platform/docs@main(decisions/ADR-0005-cilium-first-addon.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ADR-0005: Cilium as the CNI from day one (no Canal provisional CNI)
\nDate: 2026-08-29\nStatus: Decided by operator; implementation pending
\nRKE2 installs a bundled Canal (Flannel VXLAN + Calico policy) CNI by default.\nPLATFORM-007 already names Cilium as the CNI and PLATFORM-008 names Hubble as\nthe network-observability base. This decision is recorded in the decisions log\nas PLATFORM-018 (the next free identifier; PLATFORM-008 was already assigned\nto Hubble), and the master\narchitecture plan (section 14.4) fixes the first-version baseline as Cilium\nwith VXLAN encapsulation and Hubble enabled, explicitly without BGP or native\nrouting (PLATFORM-017). A provisional Canal bootstrap \u2014 installing Canal first\nand replacing it with Cilium later \u2014 would force every node through a CNI\nmigration (restart of all pods, datapath teardown, stale flannel state) and\nbuild NetworkPolicy experience on a stack the platform never intends to keep.\nThe official Cilium RKE2 documentation offers exactly two paths: the bundled\nrke2-cilium chart or RKE2 with cni: none plus the official Cilium Helm\nchart. The cluster does not exist yet, so the CNI choice costs nothing today\nand an extra migration later. Pod CIDR 10.64.0.0/14 and Service CIDR\n10.68.0.0/16 are set at the RKE2 level; Cilium must not re-derive them.\nFinally, Argo CD does not exist yet at first bootstrap, so the first cluster\naddon cannot be managed by GitOps from minute one.
cni: none, and the future RKE2\nAnsible role must keep the bundled Canal disabled.oci://quay.io/cilium/charts/cilium, pinned to chart version 1.20.1\n(chart versions carry no v prefix; appVersion v1.20.1) into\nkube-system, not the rke2-cilium addon chart, so the Helm release and\nvalues stay under this repository's control.routingMode: tunnel with\ntunnelProtocol: vxlan; IPAM cluster-pool with\nclusterPoolIPv4PodCIDRList: [10.64.0.0/14] and\nclusterPoolIPv4MaskSize: 24, exactly mirroring the RKE2 pod CIDR;\nkubeProxyReplacement: false; hubble.enabled: true with\nhubble.relay.enabled: true and hubble.ui.enabled: false (documented\nflip-on option). No BGP, no native routing, no egress features initially.10.64.0.0/14 and\nthe Service CIDR 10.68.0.0/16 (the latter stays owned by kube-proxy);\nthe Cilium IPAM pool only consumes the pod range and must never diverge\nfrom the rke2-server configuration.10.72.0.0/23, VLAN 4, single NIC eth0 per node);\nno explicit MTU unless a measured reason appears.8472 and Hubble Relay\nTCP 4244.wetopi-ops01: the addon is installed with\nHelm during the apply phase (ansible/playbooks/cilium.yml in\nwetopi-platform/infrastructure, with pinned values in\nkubernetes/cilium/values.yaml). This is a bootstrap-time install only;\nonce Argo CD exists, the addon moves under Argo CD management and the\nplaybook is retired.10.72.0.0/23, so UDP 8472 (and TCP 4244\nfor Hubble Relay) must stay reachable node-to-node on VLAN 4.clusterPoolIPv4PodCIDRList values are a standing misconfiguration risk\nto guard in review.kubernetes/cilium/README.md), so drift between Git values\nand the installed release is possible in the interim window.kubernetes/cilium/VALIDATION.md in the infrastructure repository.\n\nMigrado desde
\nwetopi-platform/docs@main(decisions/ADR-0006-edge-v1-wan-exception.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ADR-0006: Edge v1 WAN exception \u2014 direct Traefik, no L4 layer
\naccepted (2026-08-30 \u2014 approver r4 verification t_02a006fc; operator authorization recorded in the WAN Edge v1 plan review trail)
\nThe master plan (\u00a717 EdgeOps) preserves VRRP + IPVS + DSR as the production edge architecture and requires a DSR PoC (\u00a717.4) before production. The rebuild has now reached the point where the first internet-facing service (docs.wetopi.net \u2014 serving the platform's own documentation statics, with HTTPS + basic auth) is desired BEFORE the full L4 edge stack is built.
\nThe operator decided (2026-08-30, recorded in the WAN Edge v1 plan doc): to ease testing and debugging, v1 enters WITHOUT the L4 load-balancing layer \u2014 two edge nodes with Traefik exposed directly, DNS round-robin between two public A records.
\nThe approver's plan-review (t_a79fed0e, t_2d5385cc) correctly held that this conflicts with the prevailing master plan and that an \"ADR-lite\" cannot reconcile the conflict: a published internet endpoint is not an \"internal\" service. Hence this ADR \u2014 the accepted mechanism (same as ADR-0004 superseding the \u00a716 endpoint design).
\nFor a bounded v1 window, the edge consists of exactly:
\nThe v1 exception applies ONLY to the services explicitly listed in this ADR:
\nNo other service name is covered; each addition to this list is its own recorded operator decision (the security rule \"nada m\u00e1s sin necesidad demostrada y decisi\u00f3n expl\u00edcita\" holds).
\nThe exception is bounded by these hard conditions:
\nOperator directive (verbatim, 2026-08-30 ~17:00, goal session docs.wetopi.net thread):
\n\n\n\"en cuanto podamos, tambien quiero argo.wetopi.net securizado a internet, y sacado con https, para poder ver el estado de argo desde webui\"
\n
This is the explicit decision the platform security rule requires (\"nada m\u00e1s sin necesidad demostrada y decisi\u00f3n expl\u00edcita\") \u2014 recorded here verbatim per that rule. It adds the SECOND (and so far last) v1-window service:
\nScope:
\nSecurity posture for argo (TWO auth layers + edge TLS):
\nargo-basic-auth, htpasswd Secret operator-managed on the cluster, never in Git).argo-tls, DNS-01 via the cert-manager chain validated staging-first by the F3 edge-test gate) \u2014 a control-plane UI in daily operator use does not take a staging-cert detour.\n\nMigrado desde
\nwetopi-platform/docs@main(decisions/ADR-0007-ceph-storage-native.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ADR-0007: Ceph client access via direct storage-network attachment
\nDate: 2026-08-30\nStatus: Accepted (operator decision recorded 2026-08-30 \u2014 the\n192.168.98.0/25 allocation, .10/.11/.12 = the 3 platform workers \u2014\non build card t_7415a1d6; accepted status set after the round-1\nreview cycle of infrastructure PR #13 / docs PR #12, verdict\nt_4ed11dad)
ADR-0004 bundled a second decision with the RKE2 API endpoint: node-LAN\ntraffic to the Ceph cluster (192.168.96.0/20, VLAN 2 / vmbr2) would be\nrouted via the NAT gateway \u2014 wetopi-natgw01 gaining a third\ninterface on VLAN 2 so the Ceph subnet became a directly connected route\non the gateway, with a Ceph-side return route for 10.72.0.0/23 and\nforward-chain firewall rules. That design was accepted as design only,\ngated behind pre-implementation gates (address allocation, return-route\nowner, firewall path, rp_filter, MTU, CIDR overlap), and shipped\ndefault-off in the nat_gateway Ansible role.
Field reality (2026-08-30) invalidated the routing hypothesis:
\n192.168.96.10 routes via the\nLAN default gateway 10.72.0.1 with 100% loss \u2014 the natgw has no VLAN 2\nleg, no static route, and a forward policy that drops everything except\nLAN-to-WAN.10.72.0.1 means every Ceph I/O depends on a non-managed default-gateway\nhop (plus, in the ADR-0004 shape, on a Ceph-side return route nobody\nowns).vmbr2 / VLAN 2 is present\non the Proxmox hosts \u2014 the Ceph storage network is a directly connected\nL2 domain one NIC away from every VM.The operator decided (Carlos, 2026-08-30): \"storage on the native\nnetwork; routing into Ceph is unnecessary overhead.\"
\nThe 3 platform workers (rke-platform01/02/03 \u2014 the PVC-carrying nodes)\nget a third NIC on VLAN 2 / vmbr2 with addresses in\n192.168.96.0/20, attaching them natively to the Ceph storage network\n\u2014 the standard Ceph client pattern:
192.168.98.0/25\nis reserved for the platform's storage consumers \u2014\nrke-platform01=192.168.98.10, rke-platform02=192.168.98.11,\nrke-platform03=192.168.98.12 (all with the /20 mask per the connected\nsubnet); the rest of the /25 stays free for future storage consumers.10.72.0.1 on the LAN NIC (unchanged). The\nADR-0001 no-NAT invariant is trivially preserved (there is no\nintermediary to NAT).node-role.kubernetes.io/control-plane:NoSchedule) and carry no PVC\nworkloads; their storage-NIC shape is documented (commented) in the\ntfvars for future need.192.168.96.0/20 \u2014 mons TCP 3300 (msgr2) + 6789 (msgr1),\nOSDs TCP 6800-7300 \u2014 inbound plus established/related (ceph_storage\nAnsible role, default-off until the reviewed apply). LAN semantics stay\npermissive (identical to the pre-change no-firewall state); the workers\ncarry no WAN surface.10.72.0.20) is\nUNAFFECTED and remains in force..10-.12 from the reserved\n/25); the allocation is now explicit and recorded, replacing ADR-0004's\nopen \"VLAN 2 address allocation\" gate.infrastructure/kubernetes/ceph-csi/VALIDATION.md \u00a72.natgw_ceph_* enablement is reduced to a permanent\ndefault-false gate + cleanup task; the natgw carries no VLAN 2 leg, ever.client.kubernetes-csi, never admin), fsid/mons\nread-only, csi-rbd-secret.feat/ceph-storage-native (the build:\ntfvars, ceph_storage role, ADR-0004 artifact retirement,\nREADME/VALIDATION rewrite)\n\nMigrado desde
\nwetopi-platform/docs@main(decisions/ADR-0008-harness-v31-objective-governance.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ADR-0008: Harness V3.1 objective governance \u2014 versioned Git Objectives with separated THINK, OWN, DO and JUDGE
\nDate: 2026-08-30\nStatus: Accepted (2026-08-30). Lifecycle completed per the build-p1 spec v2:\nthe ADR opened as proposed for independent review; the independent content\nreview reached APPROVE (round 2, t_f5f1fb23) with green CI, and this narrow\nin-PR transition updates front matter, this human status line and the decisions\nindex atomically, with all DocOps gates re-run. Landing the pilot is NOT\nautomatic standing governance \u2014 V3.1 becoming standing governance is recorded\nonly after the accepted ADR is merged and the pilot verdict is in, per the\npromotion matrix of\nOBJ-HARNESS-V31-001.
Wetopi is built and operated by an AI-agent harness (Hermes Kanban lanes:\ndiscovery / build / ops / senior / approver) supervised by one human\noperator. Two structural risks have surfaced during Phases 0\u20134:
\nHarness V3.1 is the pilot that addresses both: versioned Git Objectives\nplus a strict role separation. This ADR records the governance decisions;\nARCH-HARNESS-V31-GOVERNANCE\ndocuments the operating model (roles, gates, degraded routing, fail-closed\nstate machine); OBJ-HARNESS-V31-001\nis the first Objective exercising the mechanism end-to-end.
\nThis ADR follows the accepted decision conventions (ADR-0003 among them)\nand does not modify any infrastructure responsibility split \u2014 it is\nagent-governance only. It is consistent with ADR-0003's principle of\nseparate, explicit sources of truth per lifecycle.
\nRole separation THINK / OWN / DO / JUDGE. Harness work is split into\nfour non-overlapping roles:
\n| Role | \nLane / actor | \nResponsibility | \nExplicitly NOT allowed | \n
|---|---|---|---|
| THINK (Architect) | \narchitect capability (e.g. an architect-designated profile or the operator acting as architect) | \nDefine WHAT, WHY and DONE as an Objective Draft: outcome, context, acceptance criteria, verification contract, constraints, boundaries, stop conditions | \nGit write, execution, card claiming, review of own draft | \n
| OWN (Lead) | \nwetopi-lead | \nAcquire the accepted Objective, decompose it into cards, dispatch, maintain the Execution Ledger, own the documentation handoff | \nAuthoring the Objective's WHAT/WHY/DONE it did not receive; judging deliveries | \n
| DO (workers) | \nwetopi-build, wetopi-senior, wetopi-ops (read-only ops) | \nImplement cards within objective boundaries, self-validate, deliver evidence | \nApplying/destroying real infrastructure from build cards; approving own work | \n
| JUDGE (independent review) | \nwetopi-approver or a reviewer that did not implement | \nVerify deliveries against the Objective's verification contract; record verdicts (APPROVE / REQUEST_CHANGES / REJECT) | \nImplementing; merging without operator authority | \n
The Architect has no Git write. The Architect produces the Objective\nDraft (WHAT/WHY/DONE) and hands it to the Lead. The draft enters Git only\nthrough a separate Lead/Docs task that opens the documentation PR.\nThis PR \u2014 not the draft itself \u2014 is the reviewed, mergeable artifact.\nThe Architect may propose, comment, and answer questions, but never\npushes branches or commits.
\nLead acquires and maintains execution. Once an Objective is accepted,\nthe Lead OWNs its execution: decomposition into cards, assignment to\nlanes, the Execution Ledger (single-writer, Lead-owned), and the\ndocumentation PRs that land Objective artifacts in Git. The Kanban board\nand Work Ledger remain the operational state (level 1); the Git\nObjective is the versioned contract (level 2). The ledger references\nthe Objective (commit/PR SHA); the Objective never copies ledger state.\nNever copied into Git: live snapshots, .env, auth.json, tokens,\nconnection URLs, credential hashes, pool output, or any secret \u2014 Git\nholds contracts and evidence references, never live operational state\n(DocOps \u00a71 authority model preserved).
Build/PR is separated from apply. No build or documentation card\napplies or destroys real infrastructure. Apply runs only from explicitly\nauthorized ops cards following reviewed runbooks with the human gate \u2014\nthe standing platform rule. The Objective layer does not change it: an\nObjective's verification contract may REQUIRE an apply as a follow-up\nops card, but the build/PR deliverable itself never executes one.
\nReview is conditional and fail-closed. A delivery is NOT done until\nits verification contract passes independent review. If the reviewer is\nunavailable, the verdict is missing, or evidence is incomplete, the\noutcome is blocked, never silently accepted. goal_mode\n(automatic goal continuation) never substitutes for review \u2014 it may\ndrive a worker's iteration budget, but the JUDGE step remains a\nseparate, human-supervised gate. goal_mode is additionally\ncapability-gated like Qwen local: it stays DISABLED for real\nObjectives until a non-mutating P2 smoke proves the fail-closed\ncontract \u2014 DONE requires AC evidence + Lead audit + independent\nAPPROVE (never auto-done); CONTINUE/FAIL honor a bounded turn budget;\nprovider timeout/offline and any failure land Blocked and leave\ngoal_mode disabled; restarts deduplicate to a single reconciled run;\nno external mutation in any case. It remains disabled on any\nfailure; the full enablement gate and smoke contract are specified\nin ARCH-HARNESS-V31-GOVERNANCE
Qwen local is capability-gated and may remain excluded. Local-model\nparticipants (Qwen via Ollama) may take part in the pilot only if a\ncapability gate passes (adequate reasoning/verbosity for the assigned\nrole) AND the economics gate of Decision 9 passes. If either fails or\navailability degrades (e.g. multi-day provider cooldowns), Qwen local\nis simply excluded \u2014 exclusion never weakens any other gate and\nnever blocks the role design; work routes to available capable lanes\nor blocks, fail-closed.
\nBackup, restoration and rollback triggers. The level-1 operational\nstores (Kanban DB, Work Ledger) are backed up on a defined schedule with\na documented restore procedure (ledger template carries the\nbackup/restore locator field). The level-2 Git contract is its own\nbackup (branch/PR history). Rollback triggers \u2014 any of these\ninvalidates the pilot state and invokes recovery: (a) a secret or live\ncredential detected in Git \u2192 immediate revert of the offending commit\nplus credential rotation; (b) an unreviewed or unverified merge reached\nmain \u2192 revert, re-review; (c) an Objective or its ADR contradicts an\naccepted ADR or the master plan \u2192 block the objective, amend or abandon;\n(d) the generated catalog drifts from the source front matter \u2192\nregenerate and re-run gates before further merges.
ADR lifecycle and merge gate. This ADR opens as proposed for\nindependent review of the real PR diff. After an independent APPROVE\nverdict, the Lead authorizes a narrow in-PR transition to\naccepted: front matter status, the human Status line and the\ndecisions index updated atomically, all DocOps gates re-run. Only after\nthat transition (with a green branch check and a current approval) is\nthe PR mergeable \u2014 and even then this build card does not merge; the\nmerge is a separate operator-gated act. No P2 work starts until main\ncontains the accepted ADR with consistent index/navigation.
Qwen economics gate (routability, optional participant). Qwen local\nis additionally subject to an economics gate before routing any pilot\nwork to it: a paired benchmark on the same three non-mutating\nfixtures recording wall_time_validated for the direct GPT/Lead run\nand for the Qwen + Lead audit/rework run. Qwen is routable only if ALL\nhold: 3/3 fixture ACs pass under Qwen; Qwen invokes zero cloud/GPT\nfallback calls; and the aggregate Qwen (including audit/rework) is\n\u2264 1.20\u00d7 the direct baseline's validated wall time. Any failure \u21d2 Qwen\nexcluded \u2014 exclusion never blocks the role design.
Status discipline. Even after accepted, the pilot still has to\nrun: Harness V3.1 becomes the standing governance only if the operator\nconfirms it after the pilot verdict (promotion matrix of\nOBJ-HARNESS-V31-001). Nothing in this document retires the existing\nlane model, and no V3.1 mechanism may be treated as accepted\nprecedent for other work until then.
| Alternative | \nWhy rejected | \n
|---|---|
| Objectives as Kanban cards only (no Git contract) | \nKeeps WHAT/WHY/DONE in mutable operational state; no reviewed contract; no versioned audit trail. | \n
| Full independence: every card independently reviewed | \nProhibitive review cost for the pilot's size; Objectives batch the review at the contract and delivery boundaries instead. | \n
goal_mode auto-completion as the acceptance signal | \nSubstitutes a heuristic judge for independent review; explicitly rejected (Decision 5). | \n
| A dedicated objectives database / RAG store | \nHeavier than the pilot needs; Git + generated catalog already provide versioning and retrieval; revisit if objective count grows. | \n
| Wait for agent-services planes before governing | \nGovernance gaps identified in Phases 0\u20134 exist today; the pilot is deliberately lightweight so it runs on the current Kanban harness. | \n
main contains the accepted\nADR with consistent index/navigation. The Lead executes\nOBJ-HARNESS-V31-001 end-to-end under the mechanism; deviations and\nevidence are recorded in the Execution Ledger (level 1), not here.completed/) or the\npilot is closed with lessons recorded.\n\nMigrado desde
\nwetopi-platform/docs@main(decisions/ADR-0009-secrets-vault-harness.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ADR-0009: Secrets vault harness \u2014 Infisical self-hosted in-cluster (ns vault), identidades por perfil y contrato ai-hub
\nDate: 2026-08-31\nStatus: Accepted (2026-08-31). Lifecycle: el ADR abri\u00f3 como proposed;\nel operador decidi\u00f3 el 2026-08-31 dejar Infisical \"totalmente operativo\ninternamente por el cluster service endpoint, con el service provider\ninstalado en Kubernetes\", lo que sustituye la VM dedicada por un despliegue\nIN-CLUSTER (namespace vault) y trae el Operator/CRDs v1beta1 a ESTA fase.\nEsta transici\u00f3n in-PR (front matter, status line, decisiones 1 y 8 y el\n\u00edndice de decisions, actualizados at\u00f3micamente, patr\u00f3n ADR-0008) registra\nesa decisi\u00f3n del operador con su fecha; el despliegue concreto se materializa\ny revisa independiente en infra PR #27 (feat/vault-infra).
Hoy hay ocho ficheros plaintext bajo ops01:~/secrets/, copias manuales en\n.env de perfiles, credenciales GHCR de alcance excesivo, dos credenciales\nexpuestas pendientes de revocaci\u00f3n (t_b904ba3f), un acceso argocd admin\nest\u00e1tico y ausencia de trazabilidad de lecturas. El master plan \u00a710 ya\nseleccion\u00f3 Infisical self-hosted, SOPS+age \u00fanicamente para bootstrap/recovery\ny el contrato estable vault.get/put/update/delete/list/exists en ai-hub. La\nvalidaci\u00f3n de licencia, machine identities e integraci\u00f3n Kubernetes (\u00a710.6,\nt_7459a827) estaba en curso al redactar el handoff y es gate obligatorio\nantes de producci\u00f3n; su veredicto se registra bajo\nEstado de la validaci\u00f3n \u00a710.6.
vault de\nwetopi3 (decisi\u00f3n del operador, 2026-08-31: Infisical totalmente\noperativo internamente v\u00eda el service endpoint del cl\u00faster, con el\nservice provider instalado en Kubernetes). Esta decisi\u00f3n SUSTITUYE la\nVM dedicada fuera del cl\u00faster del borrador proposed: el datastore\nsoportado (PostgreSQL + Redis noeviction persistentes) corre en el\nmismo cl\u00faster, con backup cifrado off-host, restauraci\u00f3n ensayada y\nUniversal Auth privado (s\u00f3lo service interno). El trade-off de fallo\ncorrelacionado con wetopi3 se acepta expl\u00edcitamente y se compensa con\nel contrato de recovery (pg_dump + age ensayado, AC8) \u2014 la alternativa\nde VM dedicada queda registrada como alternativa considerada y descartada\npor decisi\u00f3n del operador con fecha 2026-08-31.vault.wetopi.net detr\u00e1s del edge aprobado, con TLS LE production\ny autenticaci\u00f3n nativa Infisical siempre obligatoria. La ruta p\u00fablica sirva\nS\u00d3LO la UI humana: el API de machine identities (Universal Auth) NO es\nalcanzable por el hostname p\u00fablico \u2014 enforcement verificable en el edge\n(path-filter con DENY expl\u00edcito del resto, infra #27). Basic Auth se eval\u00faa\ns\u00f3lo como defensa adicional de la UI humana; no puede sustituir la\nautenticaci\u00f3n del vault ni interceptar/romper el API de machine identities.\nSi no puede separarse con rutas inequ\u00edvocas o un hostname administrativo\ndistinto, no se activa..env; el access token derivado es corto, s\u00f3lo en memoria y nunca se\nregistra. Client ID/referencias no secretas pueden declararse en\nconfiguraci\u00f3n.proposed): el Operator v0.11.8 con\nCRDs v1beta1 (InfisicalAuth + InfisicalStaticSecret; la superficie\nInfisicalSecret v1alpha1 est\u00e1 deprecated upstream y NO se usa en\ninstalaciones nuevas) se instala junto al vault in-cluster y sincroniza\nsecrets hacia los K8s Secrets de los workloads v\u00eda Universal Auth por el\nendpoint interno.wetopi3: la versi\u00f3n proposed de este ADR (2026-08-31\npor la ma\u00f1ana) lo descart\u00f3 por fallo correlacionado y p\u00e9rdida de acceso\ndurante recuperaci\u00f3n del cl\u00faster, y propuso una VM dedicada. El operador\nreconsider\u00f3 esa premisa el mismo d\u00eda (chat 2026-08-31: dejar Infisical\ntotalmente operativo internamente por el cluster service endpoint, service\nprovider instalado en Kubernetes) y la decisi\u00f3n vigente es IN-CLUSTER; el\ntrade-off de fallo correlacionado se mitiga con el contrato de recovery\n(decisi\u00f3n 1) y queda registrado aqu\u00ed para trazabilidad..env: prohibido como mecanismo normal; s\u00f3lo referencias\nno secretas y una credencial bootstrap local por perfil.El vault in-cluster y su backup pasan a ser infraestructura Tier-0; se exige\nhardening (securityContexts non-root, secretos nunca en Git), monitorizaci\u00f3n,\nparcheo y runbook de recuperaci\u00f3n. El fallo correlacionado con wetopi3 es\nun riesgo aceptado expl\u00edcitamente (decisi\u00f3n 1) \u2014 la disponibilidad del vault\nqueda ligada a la del cl\u00faster, y el recovery ensayado es el compensatorio\nobligatorio. Se a\u00f1ade un adaptador ai-hub y una pol\u00edtica por identidad, pero\nlos agentes quedan desacoplados.\nLas autoridades quedan expl\u00edcitas: Infisical runtime; Git L2; registro Hermes\npara perfiles; SOPS+age s\u00f3lo bootstrap/recovery; auditor\u00eda append-only para\nevidencia; Kanban no es fuente de verdad del dise\u00f1o.
El architecture pack normativo (boundary y topolog\u00eda, namespaces y\nreferencias, contrato vault.*, matriz de identidades y permisos, bootstrap\ny ciclo de credenciales, rotaci\u00f3n, backup y recovery, migraci\u00f3n del estado\nvivo, auditor\u00eda, riesgos y diagramas) vive como pill de arquitectura en\nARCH-VAULT-SYSTEM; el ADR\nregistra aqu\u00ed s\u00f3lo la decisi\u00f3n y sus consecuencias.
El addendum de integraci\u00f3n del handoff THINK (que registraba la validaci\u00f3n\nen curso) se sustituye por el veredicto final de la discovery\n(t_7459a827, 2026-08-31): GO para Infisical self-hosted como vault del\nharness, con TRES condicionales. Informe completo:\nfindings-infisical-selfhosted.md (adjunto a las cards t_7459a827 y\nt_e2292367).
| # | \nHecho | \nFuente oficial | \n
|---|---|---|
| 1 | \nLicensing: core MIT Expat; directorio ee/ bajo Infisical Enterprise License; activaci\u00f3n self-hosted por LICENSE_KEY (online u offline) | \nhttps://github.com/Infisical/infisical/blob/main/LICENSE \u00b7 https://github.com/Infisical/infisical/blob/main/backend/src/ee/LICENSE.md \u00b7 https://infisical.com/docs/self-hosting/ee | \n
| 2 | \nMachine identities / Universal Auth completo en MIT core: clientId/secret \u2192 JWT de corta vida, TTL/renew, rotaci\u00f3n y revocaci\u00f3n de client secrets, expiry alerts | \nhttps://infisical.com/docs/documentation/platform/identities/machine-identities \u00b7 https://infisical.com/docs/documentation/platform/identities/universal-auth | \n
| 3 | \nAudit logs en self-hosted es feature de pago (Enterprise license) | \nhttps://infisical.com/docs/documentation/platform/audit-logs | \n
| 4 | \nAudit log streams (SIEM/Splunk/Datadog) tambi\u00e9n paid EE | \nhttps://infisical.com/docs/documentation/platform/audit-log-streams/audit-log-streams | \n
| 5 | \nPoint-in-Time Recovery (rollback de folder/env a snapshot) es paid EE | \nhttps://infisical.com/docs/documentation/platform/pit-recovery | \n
| 6 | \nTrusted IPs sobre client secret / access token de identidades es paid EE | \nhttps://infisical.com/docs/documentation/platform/identities/universal-auth | \n
| 7 | \nContrato vault.* mapea 1:1 a la API v4 (/api/v4/secrets*), verificada contra el OpenAPI oficial | \nhttps://app.infisical.com/api/docs/json | \n
| 8 | \nSecrets Operator K8s vivo (v0.11.8; CRDs v1beta1 recomendados; InfisicalSecret v1alpha1 deprecated) | \nhttps://github.com/Infisical/kubernetes-operator/releases \u00b7 https://infisical.com/docs/integrations/platforms/kubernetes/infisical-static-secret-crd | \n
| 9 | \nDatastore: PostgreSQL es la \u00fanica BD soportada y Redis es dependencia dura y persistente (noeviction, AOF/RDB) | \nhttps://infisical.com/docs/self-hosting/configuration/requirements.md | \n
| 10 | \nRecovery SOPS+age viable v\u00eda pg_dump + age; no existe export cifrado nativo program\u00e1tico | \nhttps://infisical.com/docs/self-hosting/configuration/requirements.md \u00b7 https://infisical.com/docs/self-hosting/guides/rotating-the-encryption-key | \n
| 11 | \nSecret versioning base (MIT) con lectura por ?version=N | \nhttps://infisical.com/docs/documentation/platform/secret-versioning | \n
Audit logs (EE, paid) \u2014 DECISI\u00d3N: NO-GO sin licencia EE aceptada; sin\ncontrol compensatorio admisible. El audit append-only sin valores es\nrequisito normativo (AC7) y la alternativa a nivel ingress/infraestructura\nno cubre actor identity, decisi\u00f3n allow/deny ni retenci\u00f3n 365 d\u00edas.\nPor tanto, la compra de la licencia Enterprise (LICENSE_KEY) que\nhabilita el audit es condici\u00f3n de cierre de S1. Si el operador no la\nacepta, S1 es NO-GO y el objetivo se detiene antes de S3.\nFuente: https://infisical.com/docs/documentation/platform/audit-logs
PIT recovery (EE, paid) \u2014 DECISI\u00d3N: control compensatorio; NO compra.\nEl PIT de Infisical (rollback por folder/env a snapshot) no es requisito\ndel contrato: la ruta de recovery normativa es backup/restore SOPS+age\nv\u00eda pg_dump + age con restore drill aislado (AC8), y la lectura de\nversiones hist\u00f3ricas usa el versioning base MIT. PIT queda como mejora\nopcional futura. Reapertura: si en S7 el restore drill demostrara que la\nventana de recuperaci\u00f3n exigida no es alcanzable sin PIT, vuelve aqu\u00ed la\ndecisi\u00f3n \u2014 compra de licencia EE o NO-GO, nunca promesa abierta.\nFuente: https://infisical.com/docs/documentation/platform/pit-recovery
Trusted IPs (EE, paid) \u2014 DECISI\u00d3N: control compensatorio de red; NO\ncompra. La restricci\u00f3n de IP nativa de Universal Auth no se compra: el\nbinding de origen se compensa a nivel de red. Con la decisi\u00f3n de\ndespliegue in-cluster (2026-08-31) la forma del compensatorio es el\nAISLAMIENTO DE RED DEL SERVICE: Universal Auth s\u00f3lo es alcanzable por el\nservice interno vault.vault.svc.cluster.local; la ruta p\u00fablica\nvault.wetopi.net sirve exclusivamente la UI humana con DENY expl\u00edcito\nde los endpoints de API (infra #27), y los StorageClasses PVC no exponen\nnada a Internet.
\n\nNOTA (2026-08-31) \u2014 RESUELTO por la decisi\u00f3n del operador: la\nexposici\u00f3n de Universal Auth qued\u00f3 decidida como private-network-only\nin-cluster ( Universal Auth s\u00f3lo por
\nvault.vault.svc.cluster.local,\nruta p\u00fablica = s\u00f3lo UI humana con DENY de API; chat del operador\n2026-08-31 e infra #27). Este era el OPEN \"a resolverse en S0 antes del\nprimer identity bootstrap\"; la decisi\u00f3n de arriba y el enforcement de\ninfra #27 lo cierran. La eficacia del control (login p\u00fablico rechazado /\nlogin interno autorizado) se demuestra antes del primer bootstrap.
Consecuencia normativa para S1: la etapa exige firma conjunta\napprover+operador que fije versi\u00f3n/edici\u00f3n/componentes y acepte\nexpl\u00edcitamente licencia/coste (licencia base + audit EE, seg\u00fan las\ndecisiones de arriba). No se puede degradar AC7 para usar una edici\u00f3n\ngratuita. Si el condicional audit no queda cubierto por una licencia\naceptada, el resultado es NO-GO y se detiene antes de S3.
\nt_7459a827 (arriba); pendiente la aceptaci\u00f3n expl\u00edcita de licencia/coste\npor el operador (gate S1).\n\nMigrado desde
\nwetopi-platform/docs@main(decisions/ADR-0014-edge-source-ip-allowlist.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
ADR-0014: Edge source-IP allow-list (capa VPN del operador)
\nACCEPTED (2026-09-01, directriz del operador durante la operaci\u00f3n;\nimplementado y verificado en vivo el mismo d\u00eda \u2014 infra PR #36, merge\nb369b0a).
\nAmpl\u00eda ADR-0009 (re-cierre de su decisi\u00f3n 2 por otra v\u00eda de\nenforcement) y la postura de ADR-0006 (exposici\u00f3n de servicios por el\nedge WAN): el edge pasa de \"auth HTTP\" a \"filtro de red por origen\".
\nLa decisi\u00f3n 2 de ADR-0009 prohib\u00eda que el API de machine identities\n(Universal Auth) fuera alcanzable por vault.wetopi.net, con\nenforcement por path-DENY en el edge (infra#27). El filtro result\u00f3\ndemasiado ancho: bloqueaba tambi\u00e9n POSTs same-origin de la propia UI\nhumana (/api/v1/auth/universal-auth/identities/{id}), rompiendo la\nconfiguraci\u00f3n interactiva de identidades. El deny se retir\u00f3 (infra#35,\n2026-09-01) y el API completo qued\u00f3 expuesto detr\u00e1s de la \u00fanica auth\nde Infisical, dejando la Ley 1 de ADR-0009 en deuda abierta con un\ncompensating control prometido: \"el filtrado de m\u00e1quinas pasa a la\ncapa VPN/red\".
En paralelo, el operador dictamin\u00f3 retirar el HTTP Basic Auth del edge\n(\"quitamos los authbasic\"): la autenticaci\u00f3n por htpasswd duplicaba la\nauth nativa de las apps (Argo login, Infisical login+TOTP), no expresa\nautorizaci\u00f3n por secreto (alternativa ya descartada en ADR-0009) y su\ngesti\u00f3n de credenciales (Secrets operator-side con rotaci\u00f3n manual) es\ndeuda operativa sin valor a\u00f1adido si existe un control de red previo.
\nLos hostnames de control-plane del edge (argo.wetopi.net,\nvault.wetopi.net, docs.wetopi.net) son de uso exclusivo del\noperador; no hay raz\u00f3n para que internet pueda ni siquiera intentar\nlogin contra ellos.
ipAllowList\n(edge-ip-allowlist, una instancia por namespace de ruta:\nkube-system, vault, docs-site) con sourceRange: 51.254.106.87/32\n\u2014 el egress de la VPN del operador. Cualquier otro origen recibe\n403 Forbidden en el edge, antes de alcanzar backend, auth de\naplicaci\u00f3n o API.docs-basic-auth) se conserva temporalmente SOLO porque el\nrunbook de rollback del cutover docs-runtime (CUTOVER.md) restaura\nla ruta pre-incidente desde git history con ese filtro; se elimina\nen el paso de cutover.docs.wetopi.net/hooks/rebuild, ruta docs-runtime-webhook) NO\nlleva filtro IP: su cliente es GitHub (IPs no controlables) y su\nauth es la firma HMAC-SHA256 (ADR-0011) validada timing-safe por el\nreceiver. Un endpoint de m\u00e1quina autenticado por criptograf\u00eda no\nnecesita ni debe heredar el gate de origen humano.middleware-ipallowlist.yaml\n\u00d73 + el valor en el runbook del apply): cuando exista el pool VPN\ndefinitivo se extiende sourceRange con su CIDR (un cambio\nreviewed por Git). Hasta entonces, 51.254.106.87/32 es el egress VPN\nvigente del operador (NO es la WAN del basti\u00f3n ops01, que es\n92.222.245.91).ipAllowList\nrechaza con 403 lo no listado; si el middleware no resuelve, la ruta\nfalla cerrada (500), nunca abierta.git revert (Argo selfHeal aplica el rollback)./api/v1/auth/universal-auth/login desde internet); el edge\nqueda con una postura uniforme \"control de red + auth nativa de la\napp\"; un solo mecanismo para argo/vault/docs.docs-basic-auth\nlegacy persiste hasta el cutover (objeto inerte pero vivo).middleware-ipallowlist.yaml (PR revisado, Argo sincroniza). Las\napps include-only exigen que cualquier fichero nuevo est\u00e9 en la\ninclude list de su Application (lecci\u00f3n del caso PR#36, en\nwetopi-gitops-ops).| Alternativa | \nPor qu\u00e9 se descart\u00f3 | \n
|---|---|
| Restaurar el path-DENY en vault (Ley 1 por paths) | \nYa demostr\u00f3 ser fr\u00e1gil: rompi\u00f3 la UI misma (infra#27\u2192#35); cada endpoint nuevo del API reabre la herida | \n
| Mantener basic-auth como capa extra | \nDuplicaba la auth nativa, credenciales fuera del vault, sin valor tras el gate IP (directriz expl\u00edcita del operador) | \n
| WireGuard/VPN propia ya (Ley 1 en L3) | \nEs el objetivo final pero requiere dise\u00f1ar/desplegar la VPN; el ipAllowList entrega HOY el mismo enforcement con el egress existente del operador | \n
| Cloudflare Access / SSO en el edge | \nA\u00f1ade dependencia externa y cambia el modelo de acceso; no requerido por el operador | \n
feat/edge-ip-allowlist,\nmerge b369b0a, 2026-09-01), revisi\u00f3n en 2 rondas (r1 REQUEST_CHANGES\npor include lists de las apps include-only, corregido; r2 APPROVE).argo-edge no est\u00e1 anclada a la root app \u2014 su Application se\nre-aplic\u00f3 por el patr\u00f3n bootstrap documentado), ruta legacy de docs\nparcheada out-of-band (objeto restore deliberadamente no gestionado\npor Argo; re-own por Git la resucitar\u00eda tras el cutover \u2014 invariante\nRC4), Secret argo-basic-auth borrado.sourceRange cuando exista el pool VPN\n(ampliaci\u00f3n trivial); el webhook pasar\u00e1 a estar routable con el flip\nde docs-runtime (card t_ffe4d39b, secrets de webhook pendientes del\noperador).\n\nMigrado desde
\nwetopi-platform/docs@main(design/ai-first-harness-plan.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
Fecha: 28 de agosto de 2026
\nEstado: Dise\u00f1o inicial consolidado
\n\u00c1mbito: Reconstrucci\u00f3n de Wetopi sobre Kubernetes con una arquitectura AI-first y una flota interna de sistemas ag\u00e9nticos.
Wetopi se va a reconstruir sobre Kubernetes con un enfoque AI-first desde el primer d\u00eda.
\nLa intenci\u00f3n no es construir una plataforma tradicional y a\u00f1adir agentes despu\u00e9s. La nueva Wetopi debe asumir desde su base que:
\nLa primera capa que se desplegar\u00e1 en la nueva infraestructura ser\u00e1 precisamente la infraestructura ag\u00e9ntica com\u00fan.
\nLos componentes iniciales ser\u00e1n:
\nAgent Runtime Layer
\nMemory Plane
\nWork Coordination Plane
\nSecrets Plane
\nDocumentation Plane
\nAI Hub
\nEl objetivo es que estos servicios formen la capa base de operaci\u00f3n inteligente de Wetopi antes de reconstruir encima el resto de servicios de negocio y hosting.
\nConstruir una nueva plataforma interna de Wetopi sobre Kubernetes donde los agentes sean ciudadanos de primera clase.
\nLa plataforma debe permitir que cualquier agente:
\nCada servicio deber\u00e1 responder afirmativamente a esta pregunta:
\n\n\n\u00bfPuede un agente consumir este sistema de forma program\u00e1tica, segura y auditable sin depender de una interfaz humana?
\n
Se priorizar\u00e1n:
\nLos humanos seguir\u00e1n teniendo UI:
\nPero la UI nunca ser\u00e1 el \u00fanico canal de acceso.
\n| Dominio | \nSource of truth | \n
|---|---|
| C\u00f3digo | \nGit | \n
| Skills / tools / componentes AI | \nwetopi-ai-hub | \n
| Documentaci\u00f3n | \nMarkdown en Git | \n
| Trabajo y coordinaci\u00f3n | \nWork Ledger | \n
| Conocimiento factual | \nMemory Plane | \n
| Secrets | \nOpenBao | \n
| Estado real de infraestructura | \nAPIs del sistema correspondiente | \n
| Preferencias personales | \nMemoria local de Hermes | \n
La memoria guarda conocimiento \u00fatil, no sustituye a los sistemas autoritativos.
\nEjemplo:
\nKubernetes replicas = source of truth real\n\nFleet Memory =\n\"este servicio suele correr con 5 replicas\"\n+\nhist\u00f3rico\n+\ndecisiones\n+\nprocedimientos\nSi hay contradicci\u00f3n, gana la fuente viva autoritativa.
\nCada agente tendr\u00e1 \u00fanicamente:
\nLas acciones importantes deben poder responder:
\n\u00bfQu\u00e9 ocurri\u00f3?\n\u00bfQui\u00e9n lo hizo?\n\u00bfPor qu\u00e9?\n\u00bfSobre qu\u00e9 recurso?\n\u00bfCon qu\u00e9 ticket?\n\u00bfCon qu\u00e9 tool?\n\u00bfCu\u00e1ndo?\n\u00bfCu\u00e1l fue el resultado?\nLos servicios compartidos no deben dise\u00f1arse exclusivamente para ADK o Hermes.
\nSe debe poder a\u00f1adir en el futuro:
\nnuevo runtime\nnuevo coding agent\nnuevo autonomous agent\nsin cambiar el dise\u00f1o institucional.
\n HUMANS\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502 \u2502\n \u25bc \u25bc \u25bc\n Hermes Web UIs Git / CLI\n \u2502\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 AGENTIC SERVICES LAYER \u2502\n \u2502 \u2502\n \u2502 Memory Work Ledger Secrets Docs \u2502\n \u2502 \u2502 \u2502 \u2502 \u2502 \u2502\n \u2502 \u2502 \u2502 \u2502 \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 AI HUB / contracts \u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502\n \u25bc \u25bc\n ADK Fleet Hermes Fleet\n \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u25bc\n Kubernetes\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc \u25bc \u25bc\n Infra Hosting Business\nModelo com\u00fan:
\nActor\n\u251c\u2500\u2500 Human\n\u251c\u2500\u2500 Hermes\n\u251c\u2500\u2500 ADK Agent\n\u2514\u2500\u2500 Future Agent Runtime\nCada actor debe tener como m\u00ednimo:
\nactor_id:\nactor_type:\nruntime:\ncrew:\nrole:\nuser_id:\nenvironment:\nEjemplos:
\nactor_id: hermes-developer\nactor_type: hermes\nruntime: hermes\ncrew: development\nactor_id: platform-observability-manager\nactor_type: agent\nruntime: adk\ncrew: platform\nrole: manager\nLos primeros servicios funcionales que deben existir son:
\n1. Kubernetes base\n2. Identity / workload identity\n3. OpenBao\n4. Memory Plane\n5. Work Coordination Plane\n6. Documentation Plane\n7. wetopi-ai-hub\n8. Hermes\n9. ADK Fleet\nTodos los dem\u00e1s servicios internos podr\u00e1n despu\u00e9s apoyarse en esta base.
\nLa interfaz Kanban es para humanos.
\nEl sistema real es un:
\n\n\nShared Work Ledger / Work Coordination Plane
\n
Su misi\u00f3n es evitar que Hermes, ADK y humanos ejecuten trabajos contradictorios.
\n\n\nNing\u00fan agente modifica un recurso compartido sin consultar primero el Work Ledger cuando la acci\u00f3n pueda interferir con otro actor, requiera seguimiento o tenga impacto operativo.
\n
No toda lectura genera ticket.
\nBACKLOG\nREADY\nCLAIMED\nIN_PROGRESS\nBLOCKED\nREVIEW\nDONE\nCANCELLED\nid: OPS-381\n\ntitle: Upgrade ingress\n\nowner: hermes-sre\n\nstatus: IN_PROGRESS\n\nscope:\n - k8s:cluster:k3s01\n - k8s:deployment:ingress\n\nrisk: medium\n\nlease_until: ...\n\nactivity:\n - inspected current deployment\n - updating configuration\nEjemplos:
\ngithub:repo:wetopi-api\nk8s:cluster:k3s01\nk8s:namespace:agents\nk8s:deployment:support-manager\nceph:cluster:wetopi\nwordpress:site:12345\ndns:zone:wetopi.com\nmarketing:campaign:summer\nshared\nexclusive\nLos locks deben tener TTL.
\nclaim\n \u2193\nlease\n \u2193\nheartbeat\n \u2193\nrenew\nSi el agente desaparece, el recurso vuelve a quedar disponible.
\nAntes de ejecutar cambios relevantes:
\nintent:\n action: restart\n target: support-manager\n reason: memory leak\n expected_effect: rolling restart\nNo guardar chain-of-thought.
\nGuardar hechos operativos:
\n11:02 claimed OPS-381\n11:03 inspected deployment\n11:05 rollout started\n11:08 health checks OK\n11:09 released lease\nTask\n\u251c\u2500\u2500 Run 1\n\u251c\u2500\u2500 Run 2\n\u251c\u2500\u2500 Run 3\n\u2514\u2500\u2500 Run N\nLa tarea es durable.
\nEl run es una ejecuci\u00f3n.
\nDentro de wetopi-ai-hub:
tools/workledger/\nInterfaz:
\nwork.list(...)\nwork.get(...)\nwork.create(...)\nwork.claim(...)\nwork.release(...)\nwork.update(...)\nwork.comment(...)\nwork.complete(...)\n\nwork.find_conflicts(...)\nwork.acquire(...)\nwork.renew(...)\nwork.release_resources(...)\n\nwork.link(...)\nOperaci\u00f3n sem\u00e1ntica de alto nivel:
\nwork.plan(...)\nCandidatos analizados:
\nLa decisi\u00f3n arquitect\u00f3nica importante es:
\n\n\nel backend es reemplazable.
\n
Los agentes consumen el Wetopi Work Ledger contract, no directamente el API espec\u00edfico del producto.
\nWetopi ya dispone de una memoria propia inspirada conceptualmente en Memory Bank.
\nDebe evolucionar hacia un servicio institucional central.
\nHermes Developer\nHermes SRE\nHermes Marketing\nADK Observers\nADK Specialists\nADK Managers\nfuturos agentes\nEl servicio X usa PostgreSQL.\nosd.4 est\u00e1 actualmente degraded.\nDebe admitir TTL.
\nPara retirar un OSD se sigue este procedimiento.\nADR-034 decidi\u00f3 adoptar X.\nINC-1234 tuvo s\u00edntomas similares.\nNo sobrescribir sin historia.
\nfact version 1\nvalid_until\n\nfact version 2\nvalid_from\nAPI preferida:
\nmemory.propose_fact(...)\nResultado:
\naccept\nmerge\nsupersede\nreject\nneeds_validation\nCada memoria relevante debe poder almacenar:
\nsource:\n actor:\n tool:\n system:\n evidence:\n observed_at:\n\nconfidence:\nvalid_from:\nvalid_until:\nscope:\nAuthoritative live source\n>\nFleet Memory\n>\nLocal agent memory\n>\nLLM inference\ncompany:wetopi\ncrew:platform\ncrew:development\ncrew:marketing\ncustomer:X\nsite:X\nuser:X\nagent:X\nmemory_search()\nmemory_get_fact()\nmemory_propose_fact()\nmemory_update_fact()\nmemory_get_history()\nmemory_forget()\nLa implementaci\u00f3n interna queda desacoplada.
\nCada Hermes tendr\u00e1 dos tipos de memoria.
\nHermes\n\u251c\u2500\u2500 Local Personal Memory\n\u2502 \u251c\u2500\u2500 USER.md\n\u2502 \u2514\u2500\u2500 MEMORY.md\n\u2502\n\u2514\u2500\u2500 Wetopi Fleet Memory\nAlmacena informaci\u00f3n \u00fatil para asistir a ese empleado:
\nNo se comparte autom\u00e1ticamente.
\nMemoria institucional:
\n\n\nLocal memory is permissive. Fleet memory is conservative.
\n
Hermes dispone de un mecanismo de Memory Provider Plugin.
\nLa integraci\u00f3n propuesta:
\nHermes\n\u251c\u2500\u2500 USER.md\n\u251c\u2500\u2500 MEMORY.md\n\u2514\u2500\u2500 Wetopi Memory Provider\n \u2502\n \u25bc\n Wetopi Memory API\nEl provider puede ejecutar:
\nprefetch(query)\nantes del turno.
\nAs\u00ed Hermes recibe autom\u00e1ticamente contexto institucional relevante.
\nSe recomienda:
\nsync_turn() -> no-op hacia Fleet Memory\non_memory_write() -> no-op hacia Fleet Memory\nLas memorias personales permanecen personales.
\nTools:
\nwetopi_memory_search()\nwetopi_memory_get()\nwetopi_memory_propose_fact()\nwetopi_memory_get_history()\nOpenBao es el candidato principal.
\nEl requisito de autofill de navegador no es relevante.
\nLa Web UI es suficiente para humanos.
\nPodr\u00e1n:
\nLos agentes deben usar workload identity.
\nKubernetes ServiceAccount\n \u2193\nJWT\n \u2193\nOpenBao Kubernetes Auth\n \u2193\nRole\n \u2193\nPolicy\n \u2193\nShort-lived token\nNo:
\nLLM -> read secret -> token plaintext\nS\u00ed:
\nLLM\n \u2193\ntool\n \u2193\ntool runtime\n \u2193\nOpenBao\n \u2193\nexternal API\nObjetivo futuro:
\nTodos los accesos relevantes deben quedar auditados.
\nMigrar progresivamente de:
\nMarkdown + BookStack\na:
\nMarkdown + Git + MkDocs Material\nLos agentes no necesitan API de documentaci\u00f3n.
\nPueden utilizar:
\nread\ngrep\nripgrep\nedit\ndiff\ncommit\nbranch\nPR\nMkDocs Material aporta:
\n\nAgent / Human\n \u2193\nGit\n \u2193\nCI\n \u2193\nmkdocs build\n \u2193\nstatic HTML\n \u2193\nweb server\n \u2193\ndocs.wetopi.net\nLa nueva plataforma necesita una estrategia expl\u00edcita de repositorios desde el inicio.
\nEl objetivo es evitar:
\nSe propone separar cinco categor\u00edas.
\nWetopi Git Organization\n\u2502\n\u251c\u2500\u2500 wetopi-ai-hub\n\u2502\n\u251c\u2500\u2500 wetopi-docs\n\u2502\n\u251c\u2500\u2500 wetopi-agents\n\u2502 \u2514\u2500\u2500 o repos por crew/agente si crece\n\u2502\n\u251c\u2500\u2500 wetopi-agent-services\n\u2502\n\u2514\u2500\u2500 wetopi-platform\nLa nomenclatura definitiva puede cambiar, pero las responsabilidades deben mantenerse separadas.
\nwetopi-ai-hubRepositorio central reutilizable por todos los sistemas ag\u00e9nticos.
\nContendr\u00e1:
\nNo contendr\u00e1:
\nwetopi-docsSource of truth de la documentaci\u00f3n institucional y operativa.
\nRenderizado mediante MkDocs Material.
\nwetopi-agentsOpci\u00f3n inicial recomendada para mantener la flota interna mientras el n\u00famero de agentes sea manejable.
\nPuede contener:
\nagents/\n\u251c\u2500\u2500 platform/\n\u251c\u2500\u2500 wordpress/\n\u251c\u2500\u2500 support/\n\u251c\u2500\u2500 development/\n\u2514\u2500\u2500 marketing/\nCada agente tendr\u00e1:
\nagent.yaml / config\ninstructions\nruntime integration\ndeployment\ntests\nLos tools comunes NO se duplican aqu\u00ed.
\nSe consumen desde wetopi-ai-hub.
wetopi-agent-servicesServicios centrales del plano ag\u00e9ntico:
\nmemory-service/\nwork-ledger-adapter/\nhermes-memory-provider/\nagent-identity/\napproval-service/\nfuture shared services/\nPuede comenzar como monorepo y separarse cuando alg\u00fan componente necesite lifecycle independiente.
\nwetopi-platformInfraestructura declarativa Kubernetes.
\nEjemplo:
\nclusters/\nbase/\napps/\nnamespaces/\npolicies/\nnetworking/\nstorage/\nobservability/\nagent-services/\nDebe ser la fuente GitOps de la infraestructura nueva.
\nEste repo ser\u00e1 una pieza estrat\u00e9gica.
\nSu prop\u00f3sito es responder:
\n\n\n\u00bfD\u00f3nde vive cualquier capability reutilizable por m\u00e1s de un agente?
\n
Respuesta:
\nwetopi-ai-hub\nwetopi-ai-hub/\n\u2502\n\u251c\u2500\u2500 README.md\n\u251c\u2500\u2500 pyproject.toml\n\u2502\n\u251c\u2500\u2500 skills/\n\u2502 \u251c\u2500\u2500 kubernetes/\n\u2502 \u251c\u2500\u2500 ceph/\n\u2502 \u251c\u2500\u2500 wordpress/\n\u2502 \u251c\u2500\u2500 incident-response/\n\u2502 \u251c\u2500\u2500 development/\n\u2502 \u251c\u2500\u2500 marketing/\n\u2502 \u2514\u2500\u2500 ...\n\u2502\n\u251c\u2500\u2500 tools/\n\u2502 \u251c\u2500\u2500 kubernetes/\n\u2502 \u251c\u2500\u2500 ceph/\n\u2502 \u251c\u2500\u2500 consul/\n\u2502 \u251c\u2500\u2500 wordpress/\n\u2502 \u251c\u2500\u2500 workledger/\n\u2502 \u251c\u2500\u2500 memory/\n\u2502 \u251c\u2500\u2500 docs/\n\u2502 \u251c\u2500\u2500 github/\n\u2502 \u2514\u2500\u2500 ...\n\u2502\n\u251c\u2500\u2500 components/\n\u2502 \u251c\u2500\u2500 evidence-ledger/\n\u2502 \u251c\u2500\u2500 investigation-loop/\n\u2502 \u251c\u2500\u2500 approval/\n\u2502 \u251c\u2500\u2500 conflict-check/\n\u2502 \u2514\u2500\u2500 ...\n\u2502\n\u251c\u2500\u2500 contracts/\n\u2502 \u251c\u2500\u2500 work-ledger/\n\u2502 \u251c\u2500\u2500 memory/\n\u2502 \u251c\u2500\u2500 actor-identity/\n\u2502 \u251c\u2500\u2500 evidence/\n\u2502 \u2514\u2500\u2500 ...\n\u2502\n\u251c\u2500\u2500 schemas/\n\u2502 \u251c\u2500\u2500 task.schema.json\n\u2502 \u251c\u2500\u2500 fact.schema.json\n\u2502 \u251c\u2500\u2500 actor.schema.json\n\u2502 \u2514\u2500\u2500 ...\n\u2502\n\u251c\u2500\u2500 sdk/\n\u2502 \u251c\u2500\u2500 python/\n\u2502 \u2514\u2500\u2500 ...\n\u2502\n\u251c\u2500\u2500 evals/\n\u2502 \u251c\u2500\u2500 tool-calls/\n\u2502 \u251c\u2500\u2500 memory-retrieval/\n\u2502 \u2514\u2500\u2500 ...\n\u2502\n\u2514\u2500\u2500 tests/\nDebe mantenerse una separaci\u00f3n sem\u00e1ntica.
\nDescribe:
\nEjemplo:
\nskills/ceph/replace-osd.md\nEjecuta una acci\u00f3n concreta.
\nEjemplo:
\ntools/ceph/check_health.py\nUn skill puede utilizar varias tools.
\nLos components son comportamientos reutilizables de nivel superior.
\nEjemplos:
\ninvestigation-loop\nhypothesis-manager\nevidence-ledger\nwork-conflict-guard\nhuman-approval\nmemory-curator\nNo son necesariamente tools individuales.
\nPueden ser:
\nLos sistemas centrales deben exponerse mediante contratos propios.
\nEjemplos:
\nWetopi Work Ledger Contract\nWetopi Memory Contract\nWetopi Actor Identity Contract\nWetopi Evidence Contract\nAs\u00ed se evita que los agentes dependan directamente de:
\nTodo componente compartido debe ser versionable.
\nEjemplo:
\nwetopi-ai-hub==0.8.2\no componentes versionados independientemente cuando sea necesario.
\nLos agentes desplegados deben poder informar:
\nai_hub_version: 0.8.2\npara reproducibilidad.
\nNo se recomienda copiar manualmente tools.
\nOpciones v\u00e1lidas:
\npip install wetopi-ai-hub\no paquetes separados:
\nwetopi-agent-tools\nwetopi-memory-sdk\nwetopi-workledger-sdk\nLa imagen del agente incorpora las capabilities necesarias durante build.
\n\u00datiles en desarrollo local/monorepo, menos deseables como mecanismo de distribuci\u00f3n productivo.
\n\n\nUna tool reutilizada por dos agentes deja de pertenecer al agente y pasa a
\nwetopi-ai-hub.
Se propone inicialmente un repositorio principal:
\nwetopi-docs\nEstructura:
\nwetopi-docs/\n\u251c\u2500\u2500 mkdocs.yml\n\u2514\u2500\u2500 docs/\n \u251c\u2500\u2500 index.md\n \u2502\n \u251c\u2500\u2500 company/\n \u2502\n \u251c\u2500\u2500 architecture/\n \u2502\n \u251c\u2500\u2500 infrastructure/\n \u2502\n \u251c\u2500\u2500 kubernetes/\n \u2502\n \u251c\u2500\u2500 wordpress/\n \u2502\n \u251c\u2500\u2500 procedures/\n \u2502\n \u251c\u2500\u2500 runbooks/\n \u2502\n \u251c\u2500\u2500 agents/\n \u2502\n \u251c\u2500\u2500 security/\n \u2502\n \u251c\u2500\u2500 incidents/\n \u2502\n \u2514\u2500\u2500 adr/\nSi la documentaci\u00f3n crece mucho, podr\u00eda separarse en:
\nwetopi-docs-platform\nwetopi-docs-wordpress\nwetopi-docs-business\npero no hacerlo inicialmente sin necesidad.
\nUn repositorio \u00fanico ofrece:
\nLa documentaci\u00f3n espec\u00edfica de desarrollo de un componente puede quedarse junto al c\u00f3digo.
\nEjemplo:
\nwetopi-agent-services/memory-service/README.md\nMientras que el conocimiento institucional debe ir a:
\nwetopi-docs\nRegla:
\n\n\nREADME explica el componente.
\n
\nwetopi-docsexplica c\u00f3mo Wetopi lo utiliza, opera y gobierna.
Flujo propuesto:
\n wetopi-ai-hub\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502 \u2502\n \u25bc \u25bc \u25bc\n Hermes Developer Hermes SRE ADK Fleet\n \u2502 \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n shared contracts\nQue existan 70 tools en el AI Hub no significa que un agente deba verlas todas.
\nCada agente declara sus capabilities.
\nEjemplo:
\nagent: wordpress-db-specialist\n\ntools:\n - memory.search\n - work.plan\n - work.comment\n - wordpress.inspect_db\n - database.query_readonly\nEl bundle se construye seg\u00fan:
\nruntime\ncrew\nrole\nenvironment\nCada tool reutilizable debe tener:
\nSon tres sistemas diferentes.
\n\u00bfQu\u00e9 estamos haciendo?\n\u00bfQu\u00e9 sabemos?\n\u00bfQu\u00e9 conocimiento hemos decidido formalizar?\nFlujo:
\nincident\n \u2193\nticket\n \u2193\ninvestigation\n \u2193\nvalidated facts\n \u2193\nMemory Plane\n \u2193\nprocedure / ADR / postmortem\n \u2193\nwetopi-docs\nLa identidad debe ser transversal.
\nEjemplo:
\nactor_id = platform-sre-manager\nDebe aparecer en:
\nIdeal:
\nactor_id:\nactor_type:\nruntime:\ncrew:\nrole:\nuser:\nenvironment:\nConvenci\u00f3n compartida:
\ncompany:wetopi\ncrew:platform\ncrew:development\ncrew:marketing\ncustomer:123\nsite:456\ncluster:k3s01\nnamespace:agents\nrepo:wetopi-api\nservice:foo\nPuede utilizarse para:
\nCampos comunes:
\nactor_id\nactor_type\nruntime\ncrew\ntask_id\nrun_id\nresource\ntool\naction\nresult\nduration\ntimestamp\nEjemplo:
\nactor_id: platform-sre-manager\ntask_id: INC-492\nrun_id: run-492-03\ntool: kubernetes.restart\nresource: k8s:deployment:support-manager\nresult: success\nNo registrar:
\nObserver\n \u2193\ndetect\n \u2193\nWork Ledger\n \u2193\nINC-492\n \u2193\nSpecialist claims\n \u2193\nresource lease\n \u2193\ninvestigation\n \u2193\njournal\n \u2193\nfix\n \u2193\nreview\n \u2193\ndone\nHermes SRE\n \u2193\nwork.plan()\n \u2193\nconflict detected\n \u2193\nresource owned by ADK specialist\n \u2193\nHermes waits / comments / coordinates\nHermes\n \u2193\npropose_fact\n \u2193\nMemory Service\n \u2193\nvalidation\n \u2193\nFleet Memory\n \u2193\navailable to ADK + other Hermes\nAgent\n \u2193\ntool call\n \u2193\nTool Runtime\n \u2193\nOpenBao\n \u2193\ncredential\n \u2193\nexternal service\n\nLLM does not see credential\nAgent\n \u2193\nedit Markdown\n \u2193\nbranch / commit / PR\n \u2193\nreview\n \u2193\nmerge\n \u2193\nMkDocs build\n \u2193\nhuman-readable website\nPropuesta conceptual:
\nKubernetes\n\u2502\n\u251c\u2500\u2500 agent-runtime/\n\u2502 \u251c\u2500\u2500 hermes-*\n\u2502 \u2514\u2500\u2500 adk-*\n\u2502\n\u251c\u2500\u2500 agent-services/\n\u2502 \u251c\u2500\u2500 memory-service\n\u2502 \u251c\u2500\u2500 work-ledger\n\u2502 \u251c\u2500\u2500 openbao\n\u2502 \u2514\u2500\u2500 docs\n\u2502\n\u251c\u2500\u2500 shared-data/\n\u2502 \u251c\u2500\u2500 postgres\n\u2502 \u2514\u2500\u2500 object-storage\n\u2502\n\u251c\u2500\u2500 observability/\n\u2502\n\u2514\u2500\u2500 business-workloads/\nLos namespaces definitivos deber\u00e1n dise\u00f1arse considerando:
\nDesplegar OpenBao.
\nImplementar:
\nCrear:
\nwetopi-platform\nwetopi-ai-hub\nwetopi-docs\nwetopi-agents\nwetopi-agent-services\nA\u00f1adir:
\nProductivizar la memoria existente.
\nA\u00f1adir:
\nSeleccionar backend.
\nCrear el adapter/contract propio.
\nImplementar:
\nCrear wetopi-docs.
Desplegar:
\nMigrar progresivamente documentaci\u00f3n \u00fatil existente.
\nConsolidar tools hoy dispersas.
\nPrioridad inicial:
\ntools/memory\ntools/workledger\ntools/kubernetes\ntools/ceph\ntools/wordpress\nDefinir contratos estables.
\nDesplegar un Hermes por empleado seg\u00fan necesidad.
\nCada Hermes tendr\u00e1:
\npersonal local memory\n+\nWetopi Fleet Memory\n+\nWork Ledger\n+\nrole-specific tools\n+\ndocumentation access\nDesplegar observers, specialists y managers.
\nTodos usan:
\nCon los servicios ag\u00e9nticos ya operativos, comenzar la migraci\u00f3n/reconstrucci\u00f3n de los dem\u00e1s workloads de Wetopi.
\nLos agentes participan desde el inicio en:
\nLa nueva Wetopi se reconstruir\u00e1 sobre Kubernetes.
\nLos sistemas ag\u00e9nticos se despliegan antes que el resto de la plataforma.
\nSe combinar\u00e1n:
\nHermes utilizar\u00e1:
\nlocal personal memory\n+\nWetopi Fleet Memory\nLos ADK utilizar\u00e1n principalmente Fleet Memory m\u00e1s state/session cuando corresponda.
\nSe reutilizar\u00e1 y evolucionar\u00e1 el Memory Plane ya creado internamente.
\nEl retrieval global se integrar\u00e1 mediante el mecanismo de provider de Hermes.
\nNo se volcar\u00e1n conversaciones personales autom\u00e1ticamente a Fleet Memory.
\nOpenBao ser\u00e1 el candidato principal como secrets manager central.
\nLas tools consumir\u00e1n secrets sin introducirlos en contexto LLM siempre que sea posible.
\nEl Kanban se considera UI de un sistema m\u00e1s profundo de coordinaci\u00f3n.
\nMuster, Plane o alternativa se utilizar\u00e1n detr\u00e1s de un contrato Wetopi.
\nMarkdown + Git ser\u00e1 el source of truth.
\nSe utilizar\u00e1 como primera opci\u00f3n para renderizado web de documentaci\u00f3n.
\nSe crear\u00e1 wetopi-ai-hub como repositorio com\u00fan de:
Una tool usada por m\u00e1s de un agente debe trasladarse al AI Hub.
\nLa infraestructura Kubernetes debe quedar declarada en Git.
\nRiesgo: agentes almacenan hip\u00f3tesis como hechos.
\nControl:
\nRiesgo: varios agentes modifican el mismo recurso.
\nControl:
\nRiesgo: token termina en prompt/log/trace.
\nControl:
\nRiesgo: cada agente implementa su versi\u00f3n.
\nControl:
\nwetopi-ai-hub;Riesgo: demasiadas tools disponibles para un LLM.
\nControl:
\nRiesgo: los agentes conf\u00edan en docs antiguas.
\nControl:
\nA\u00fan deben decidirse:
\nBackend final del Work Coordination Plane:
\nAuthentication humana central:
\nModelo exacto de namespaces Kubernetes.
\nStorage definitivo de Memory Plane.
\nEstrategia de embeddings/reranking del Memory Plane.
\nConvenci\u00f3n exacta de nombres de scopes.
\nSi wetopi-agents permanece monorepo o se divide cuando crezca.
Packaging definitivo de wetopi-ai-hub.
CI/CD y GitOps concretos.
\nModelo de aprobaci\u00f3n humana para acciones de alto riesgo.
\nAl finalizar la primera etapa, Wetopi deber\u00eda disponer de una base parecida a:
\n WETOPI AI-FIRST FOUNDATION\n\n Humans\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc \u25bc \u25bc\n Hermes Kanban Docs\n \u2502 \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502\n Work Ledger Fleet Memory\n \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n wetopi-ai-hub\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc \u25bc \u25bc\n Hermes Fleet ADK Fleet Future Agents\n \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n OpenBao\n \u2502\n Kubernetes\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc \u25bc \u25bc\n Infra Hosting Business\nLa consecuencia m\u00e1s importante es que Wetopi no tendr\u00e1 simplemente \"algunos agentes\".
\nLa organizaci\u00f3n dispondr\u00e1 de una infraestructura operacional dise\u00f1ada para humanos y agentes simult\u00e1neamente:
\n\u00c9sta ser\u00e1 la base sobre la que reconstruir el resto de Wetopi.
\n" }, { "id": 55, "title": "bootstrap plan", "slug": "bootstrap-plan", "book_id": 8, "chapter_id": 29, "description": null, "created_at": "2026-09-06T13:39:01.000000Z", "updated_at": "2026-09-06T13:39:01.000000Z", "markdown": "> Migrado desde `wetopi-platform/docs@main` (design/bootstrap-plan.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.\n\n# Wetopi Platform Bootstrap Plan\n## Plan de arranque de la nueva plataforma RKE2 y organizaci\u00f3n GitHub\n\n**Fecha:** 28 de agosto de 2026 \n**Estado:** Plan de bootstrap consolidado \n**Organizaci\u00f3n GitHub:** `wetopi-platform` \n**\u00c1mbito:** Creaci\u00f3n de repositorios, bootstrap de infraestructura, RKE2, servicios base y posterior incorporaci\u00f3n de agentes y workloads.\n\n---\n\n# \u00cdndice\n\n1. Resumen ejecutivo\n2. Objetivo del bootstrap\n3. Decisiones ya tomadas\n4. Organizaci\u00f3n GitHub\n5. Repositorios iniciales\n6. Git como source of truth\n7. OpenTofu vs Terraform\n8. Modelo de infraestructura inicial\n9. VM externa de operaciones: `wetopi-ops01`\n10. Control plane RKE2\n11. Platform workers\n12. Distribuci\u00f3n sobre Proxmox\n13. Networking de las VMs\n14. Red Ceph existente\n15. RKE2 y etcd\n16. Bootstrap desde fuera del cluster\n17. Rol inicial de Codex\n18. Cu\u00e1ndo desplegar Hermes\n19. Orden de despliegue de servicios base\n20. Cilium y Hubble\n21. Ceph CSI\n22. GitOps\n23. OpenBao\n24. Documentation Plane\n25. Memory Plane\n26. Work Coordination Plane\n27. AI Hub\n28. Agentes Hermes y ADK\n29. Edge nodes\n30. Hosting workers\n31. Migraci\u00f3n progresiva desde Swarm\n32. Permisos y credenciales\n33. Primeros encargos para Codex\n34. Fases del bootstrap\n35. Checklist de arranque\n36. Resultado esperado\n\n---\n\n# 1. Resumen ejecutivo\n\nLa nueva plataforma de Wetopi se construir\u00e1 desde cero, en paralelo a la infraestructura actual basada en Docker Swarm.\n\nNo se realizar\u00e1 una migraci\u00f3n in-place.\n\nLa nueva base ser\u00e1:\n\n```text\nProxmox\n \u2193\nVMs Ubuntu\n \u2193\nRKE2\n \u2193\nCilium / Hubble\n \u2193\nCeph CSI\n \u2193\nGitOps\n \u2193\nOpenBao\n \u2193\nDocs / Memory / Work Ledger\n \u2193\nHermes + ADK\n \u2193\nEdge\n \u2193\nHosting workloads\n```\n\nEl bootstrap inicial se har\u00e1 **desde fuera de Kubernetes**.\n\nDurante esta primera etapa:\n\n- Codex trabajar\u00e1 desde el laptop del operador;\n- GitHub ser\u00e1 el source of truth;\n- existir\u00e1 una VM externa `wetopi-ops01`;\n- se crear\u00e1n 3 control planes RKE2;\n- se crear\u00e1n 3 platform workers;\n- no se desplegar\u00e1n a\u00fan workloads de customers;\n- no se tocar\u00e1 la red Ceph existente `192.168.96.0/20`;\n- cada VM tendr\u00e1 una \u00fanica interfaz LAN;\n- Hermes se desplegar\u00e1 despu\u00e9s, ya dentro del cluster definitivo.\n\nLa organizaci\u00f3n GitHub ser\u00e1:\n\n```text\nwetopi-platform\n```\n\ny los repositorios iniciales ser\u00e1n:\n\n```text\nwetopi-platform/infrastructure\nwetopi-platform/agent-services\nwetopi-platform/ai-hub\nwetopi-platform/agents\nwetopi-platform/docs\n```\n\n---\n\n# 2. Objetivo del bootstrap\n\nEl objetivo de esta fase no es migrar inmediatamente los ~700 contenedores actuales.\n\nEl objetivo es construir una **foundation nueva, reproducible y segura** sobre la que posteriormente se pueda migrar Wetopi.\n\nEl milestone inicial ser\u00e1:\n\n> disponer de un cluster RKE2 HA vac\u00edo, estable, observable, con storage, secrets, GitOps y servicios ag\u00e9nticos comunes funcionando antes de mover tr\u00e1fico de clientes.\n\n---\n\n# 3. Decisiones ya tomadas\n\n## Plataforma Kubernetes\n\n`RKE2`\n\n## CNI\n\n`Cilium`\n\n## Observabilidad de red\n\n`Hubble`\n\n## Storage\n\n`Ceph existente mediante CSI`\n\n## Secrets\n\n`OpenBao`\n\n## Documentaci\u00f3n\n\n`Markdown + Git + MkDocs Material`\n\n## Infraestructura declarativa\n\n`OpenTofu + Ansible`\n\n## Bootstrap\n\n`Codex desde laptop + wetopi-ops01`\n\n## Agentes\n\n`Hermes + ADK`\n\nLos agentes se desplegar\u00e1n posteriormente dentro del cluster definitivo.\n\n---\n\n# 4. Organizaci\u00f3n GitHub\n\nSe adopta como organizaci\u00f3n:\n\n```text\ngithub.com/wetopi-platform\n```\n\nSe descartan nombres temporales como `wetopi3`, `wetopi-v3` o `wetopi-new` porque describen una migraci\u00f3n, no una plataforma duradera.\n\n`wetopi-platform` se considera adecuado como namespace t\u00e9cnico de largo plazo.\n\n---\n\n# 5. Repositorios iniciales\n\nSe crear\u00e1n inicialmente **cinco repositorios**.\n\n## 5.1 `infrastructure`\n\nResponsabilidad:\n\n- Proxmox;\n- OpenTofu;\n- Ansible;\n- RKE2;\n- networking base;\n- Cilium;\n- Hubble;\n- Ceph CSI;\n- OpenBao deployment;\n- GitOps;\n- observabilidad base;\n- edge;\n- NATGW;\n- bootstrap;\n- disaster recovery.\n\nEstructura propuesta:\n\n```text\ninfrastructure/\n\u251c\u2500\u2500 tofu/\n\u2502 \u2514\u2500\u2500 proxmox/\n\u2502 \u251c\u2500\u2500 modules/\n\u2502 \u2514\u2500\u2500 environments/\n\u2502 \u2514\u2500\u2500 prod/\n\u251c\u2500\u2500 ansible/\n\u2502 \u251c\u2500\u2500 inventories/\n\u2502 \u251c\u2500\u2500 roles/\n\u2502 \u2502 \u251c\u2500\u2500 common/\n\u2502 \u2502 \u251c\u2500\u2500 rke2-server/\n\u2502 \u2502 \u251c\u2500\u2500 rke2-agent/\n\u2502 \u2502 \u251c\u2500\u2500 edge/\n\u2502 \u2502 \u2514\u2500\u2500 natgw/\n\u2502 \u2514\u2500\u2500 playbooks/\n\u251c\u2500\u2500 kubernetes/\n\u2502 \u251c\u2500\u2500 bootstrap/\n\u2502 \u251c\u2500\u2500 cilium/\n\u2502 \u251c\u2500\u2500 hubble/\n\u2502 \u251c\u2500\u2500 ceph-csi/\n\u2502 \u251c\u2500\u2500 openbao/\n\u2502 \u251c\u2500\u2500 gitops/\n\u2502 \u251c\u2500\u2500 namespaces/\n\u2502 \u2514\u2500\u2500 observability/\n\u251c\u2500\u2500 scripts/\n\u2514\u2500\u2500 docs/\n \u251c\u2500\u2500 bootstrap.md\n \u251c\u2500\u2500 networking.md\n \u2514\u2500\u2500 disaster-recovery.md\n```\n\n## 5.2 `agent-services`\n\nServicios compartidos de la capa ag\u00e9ntica:\n\n```text\nmemory-service/\nwork-ledger-adapter/\nhermes-memory-provider/\nidentity/\napproval/\n```\n\nInicialmente se priorizar\u00e1n `memory-service`, `hermes-memory-provider` y `work-ledger-adapter`.\n\n## 5.3 `ai-hub`\n\nRepositorio com\u00fan de:\n\n- skills;\n- tools;\n- components;\n- contracts;\n- schemas;\n- SDKs;\n- evals;\n- tests.\n\nRegla:\n\n> una tool utilizada por m\u00e1s de un agente debe vivir en `ai-hub`.\n\n## 5.4 `agents`\n\nMonorepo inicial de la flota:\n\n```text\nagents/\n\u251c\u2500\u2500 hermes/\n\u2502 \u251c\u2500\u2500 sre/\n\u2502 \u251c\u2500\u2500 developer/\n\u2502 \u2514\u2500\u2500 marketing/\n\u251c\u2500\u2500 adk/\n\u2502 \u251c\u2500\u2500 platform/\n\u2502 \u251c\u2500\u2500 wordpress/\n\u2502 \u2514\u2500\u2500 support/\n\u251c\u2500\u2500 base/\n\u251c\u2500\u2500 deployments/\n\u2514\u2500\u2500 tests/\n```\n\n## 5.5 `docs`\n\nSource of truth documental:\n\n```text\ndocs/\n\u251c\u2500\u2500 mkdocs.yml\n\u2514\u2500\u2500 docs/\n \u251c\u2500\u2500 architecture/\n \u251c\u2500\u2500 infrastructure/\n \u251c\u2500\u2500 kubernetes/\n \u251c\u2500\u2500 networking/\n \u251c\u2500\u2500 wordpress/\n \u251c\u2500\u2500 agents/\n \u251c\u2500\u2500 procedures/\n \u251c\u2500\u2500 runbooks/\n \u251c\u2500\u2500 security/\n \u2514\u2500\u2500 adr/\n```\n\n---\n\n# 6. Git como source of truth\n\nDesde el primer d\u00eda:\n\n```text\nGit\n \u2193\nreview\n \u2193\nautomation\n \u2193\ninfrastructure\n```\n\nLa shell history no ser\u00e1 source of truth.\n\nTodo cambio relevante debe terminar en Git.\n\n---\n\n# 7. OpenTofu vs Terraform\n\nSe selecciona inicialmente `OpenTofu` por:\n\n- licencia open source;\n- gobernanza Linux Foundation;\n- evitar dependencia BSL;\n- compatibilidad con el ecosistema Terraform;\n- idoneidad para un proyecto interno de largo plazo.\n\nTerraform seguir\u00eda siendo t\u00e9cnicamente v\u00e1lido.\n\n---\n\n# 8. Modelo de infraestructura inicial\n\nSe crear\u00e1n inicialmente:\n\n```text\n1 x ops VM\n3 x RKE2 control plane\n3 x platform workers\n```\n\nTotal:\n\n```text\n7 VMs\n```\n\nNo se crear\u00e1n todav\u00eda hosting workers masivos ni edge productivos.\n\n---\n\n# 9. VM externa de operaciones: `wetopi-ops01`\n\nSe crear\u00e1 `wetopi-ops01` fuera del cluster Kubernetes.\n\nObjetivo:\n\n> bootstrap host, bastion administrativo y punto de recuperaci\u00f3n cuando Kubernetes no est\u00e9 disponible.\n\nSizing orientativo:\n\n```text\n4 vCPU\n8 GB RAM\n40-80 GB disk\nUbuntu 24.04\n1 NIC LAN\n```\n\nHerramientas:\n\n```text\ngit\nssh\nansible\nopentofu\nkubectl\nhelm\njq\nyq\nrke2 tooling\n```\n\nNo alojar\u00e1 workloads productivos.\n\nDebe permitir operar si fallan Cilium, GitOps, OpenBao o parte del cluster.\n\n---\n\n# 10. Control plane RKE2\n\nSe desplegar\u00e1n:\n\n```text\nrke-cp01\nrke-cp02\nrke-cp03\n```\n\nCada uno en un host Proxmox f\u00edsico diferente siempre que sea posible.\n\nSizing orientativo:\n\n```text\n8 vCPU\n16 GB RAM\n50-100 GB disk\nUbuntu 24.04\n1 NIC LAN\n```\n\n---\n\n# 11. Platform workers\n\nSe desplegar\u00e1n:\n\n```text\nrke-platform01\nrke-platform02\nrke-platform03\n```\n\nSizing orientativo:\n\n```text\n8-16 vCPU\n16-32 GB RAM\nUbuntu 24.04\n1 NIC LAN\n```\n\nLabel:\n\n```text\nwetopi.io/role=platform\n```\n\n---\n\n# 12. Distribuci\u00f3n sobre Proxmox\n\nEjemplo:\n\n```text\npve01 -> rke-cp01 + rke-platform01\npve02 -> rke-cp02 + rke-platform02\npve03 -> rke-cp03 + rke-platform03\n```\n\nLa distribuci\u00f3n real debe respetar failure domains y recursos disponibles.\n\n---\n\n# 13. Networking de las VMs\n\nCada VM tendr\u00e1 **una \u00fanica interfaz LAN**.\n\n```text\nVM\n \u2514\u2500\u2500 eth0\n \u2514\u2500\u2500 Wetopi LAN / vRack\n```\n\nLos rangos Kubernetes ser\u00e1n independientes:\n\n```text\nVM LAN != Pod CIDR != Service CIDR != Ceph network\n```\n\nLos rangos concretos deben definirse tras inventariar las redes actuales.\n\n---\n\n# 14. Red Ceph existente\n\nExiste una red Ceph:\n\n```text\n192.168.96.0/20\n```\n\nque forma parte de la infraestructura hiperconvergida Proxmox + Ceph.\n\nDecisi\u00f3n:\n\n> NO tocar, renumerar ni redise\u00f1ar esta red durante el bootstrap RKE2.\n\nRKE2 consumir\u00e1 Ceph mediante CSI adapt\u00e1ndose a la infraestructura existente.\n\n---\n\n# 15. RKE2 y etcd\n\nSe utilizar\u00e1 `embedded etcd` sobre los tres server nodes.\n\n```text\ncp01 etcd\ncp02 etcd\ncp03 etcd\n```\n\nQuorum: 2 de 3.\n\nSe priorizar\u00e1 almacenamiento de baja latencia para etcd. Debe evaluarse local NVMe si la alternativa Ceph introduce latencia innecesaria.\n\nLos snapshots de etcd deber\u00e1n almacenarse fuera de los propios nodos y el procedimiento de restore deber\u00e1 probarse.\n\n---\n\n# 16. Bootstrap desde fuera del cluster\n\nModelo:\n\n```text\nLaptop\n \u2502\nCodex\n \u2502\nGitHub\n \u2502\nops01\n \u2502\nOpenTofu + Ansible\n \u2502\nProxmox / RKE2\n```\n\nKubernetes no depender\u00e1 de agentes internos para poder crearse o repararse.\n\n---\n\n# 17. Rol inicial de Codex\n\nDurante el bootstrap Codex act\u00faa como `bootstrap engineer`.\n\nTrabajar\u00e1 contra:\n\n- GitHub;\n- repos;\n- `ops01`;\n- Proxmox con credenciales limitadas;\n- posteriormente Kubernetes.\n\nTodo cambio relevante debe quedar versionado.\n\n---\n\n# 18. Cu\u00e1ndo desplegar Hermes\n\nNo se desplegar\u00e1n varios Hermes temporales antes del cluster.\n\nLos Hermes definitivos se desplegar\u00e1n dentro de RKE2 cuando existan al menos:\n\n```text\nRKE2 estable\nCilium\nHubble\nCeph CSI\nGitOps\nOpenBao\nDocumentation\nMemory Plane\nWork Ledger\n```\n\nDespu\u00e9s:\n\n```text\nHermes SRE\nHermes Developer\nHermes Marketing\n```\n\n---\n\n# 19. Orden de despliegue de servicios base\n\n```text\nRKE2\n \u2193\nCilium\n \u2193\nHubble\n \u2193\nCeph CSI\n \u2193\nGitOps\n \u2193\nOpenBao\n \u2193\nDocumentation\n \u2193\nMemory Plane\n \u2193\nWork Ledger\n \u2193\nAI Hub integration\n \u2193\nHermes\n \u2193\nADK Fleet\n```\n\n---\n\n# 20. Cilium y Hubble\n\nCilium ser\u00e1 el CNI base.\n\nInicialmente:\n\n- networking;\n- NetworkPolicy;\n- Hubble;\n- observabilidad.\n\nNo se activar\u00e1n desde el d\u00eda uno BGP WAN, egress complejo o service mesh.\n\n---\n\n# 21. Ceph CSI\n\nPruebas m\u00ednimas antes de customer workloads:\n\n```text\ncrear PVC\nmontar PVC\nborrar Pod\nrecrear Pod\nvalidar persistencia\nsnapshot si aplica\n```\n\n---\n\n# 22. GitOps\n\nEl cluster evolucionar\u00e1 hacia:\n\n```text\nGit -> reconciler -> Kubernetes\n```\n\nLa herramienta concreta puede decidirse durante el bootstrap.\n\n---\n\n# 23. OpenBao\n\nOpenBao ser\u00e1 el secrets manager central.\n\nConfigurar:\n\n- HA;\n- persistence;\n- Kubernetes Auth;\n- audit;\n- policies;\n- backups.\n\n---\n\n# 24. Documentation Plane\n\nRepo:\n\n```text\nwetopi-platform/docs\n```\n\nRender:\n\n```text\nMkDocs Material\n```\n\nMarkdown ser\u00e1 el source of truth.\n\n---\n\n# 25. Memory Plane\n\nLa memoria institucional propia de Wetopi se productivizar\u00e1 como servicio central para Hermes, ADK y futuros agentes.\n\nDebe soportar:\n\n- factual memory;\n- scopes;\n- provenance;\n- confidence;\n- temporalidad;\n- contradictions;\n- candidate facts.\n\n---\n\n# 26. Work Coordination Plane\n\nSe desplegar\u00e1 un backend Kanban/ticketing desacoplado mediante contrato Wetopi.\n\nResponsabilidades:\n\n- tareas;\n- ownership;\n- resource scopes;\n- leases;\n- intent;\n- journal;\n- coordinaci\u00f3n entre humanos y agentes.\n\n---\n\n# 27. AI Hub\n\nRepo:\n\n```text\nwetopi-platform/ai-hub\n```\n\nPara tools, skills, contracts, components, SDKs, schemas y evals.\n\n---\n\n# 28. Agentes Hermes y ADK\n\nUna vez montada la base, Hermes y ADK participar\u00e1n en:\n\n- inventario;\n- migraci\u00f3n;\n- testing;\n- observabilidad;\n- documentaci\u00f3n;\n- automatizaci\u00f3n;\n- desarrollo;\n- operaci\u00f3n.\n\n---\n\n# 29. Edge nodes\n\nNo forman parte del bootstrap inicial.\n\nDespu\u00e9s se a\u00f1adir\u00e1n:\n\n```text\nrke-edge01\nrke-edge02\nrke-edge03\n...\n```\n\nintegrados con IPVS, VRRP, DSR y Gateway API.\n\n---\n\n# 30. Hosting workers\n\nDespu\u00e9s del edge se crear\u00e1n pocos workers de hosting:\n\n```text\nrke-host01\nrke-host02\nrke-host03\n```\n\nPrimero se migrar\u00e1n aproximadamente 10 sites y se validar\u00e1 la plataforma antes de escalar.\n\n---\n\n# 31. Migraci\u00f3n progresiva desde Swarm\n\nSwarm seguir\u00e1 produciendo durante el bootstrap.\n\nNo habr\u00e1 big-bang.\n\nLa capacidad se trasladar\u00e1 progresivamente hasta retirar Swarm.\n\n---\n\n# 32. Permisos y credenciales\n\nCodex puede tener permisos amplios de GitHub sobre la organizaci\u00f3n, pero:\n\n```text\nGitHub permissions != production credentials\n```\n\nNo entregar inicialmente:\n\n- OVH master credentials;\n- Proxmox root;\n- Ceph admin global;\n- OpenBao root permanente.\n\nCrear una identidad espec\u00edfica de automatizaci\u00f3n para Proxmox con m\u00ednimos permisos necesarios.\n\n---\n\n# 33. Primeros encargos para Codex\n\n## Encargo 1 \u2014 Repos\n\nPreparar los cinco repos con README, branch protection, CODEOWNERS y CI m\u00ednima.\n\n## Encargo 2 \u2014 Inventario Proxmox\n\nInventariar:\n\n- nodos;\n- CPU;\n- RAM;\n- storage;\n- bridges;\n- VLANs;\n- redes;\n- failure domains.\n\nNo inventar direccionamiento.\n\n## Encargo 3 \u2014 Ubuntu template\n\nCrear template Ubuntu 24.04 reproducible con cloud-init.\n\n## Encargo 4 \u2014 `ops01`\n\nCrear `wetopi-ops01` mediante OpenTofu y provisionarla con Ansible.\n\n## Encargo 5 \u2014 Control plane\n\nCrear `rke-cp01`, `rke-cp02`, `rke-cp03`.\n\n## Encargo 6 \u2014 Platform workers\n\nCrear `rke-platform01`, `rke-platform02`, `rke-platform03`.\n\n## Encargo 7 \u2014 RKE2 HA\n\nInstalar RKE2 con embedded etcd y endpoint API estable.\n\n## Encargo 8 \u2014 Failure testing\n\nValidar:\n\n```text\nreboot cp\nloss of one cp\nworker reboot\ndrain\nuncordon\netcd snapshot\nrestore procedure\n```\n\nantes de seguir.\n\n---\n\n# 34. Fases del bootstrap\n\n## Fase 0 \u2014 GitHub\n\nCrear organizaci\u00f3n y repos.\n\n## Fase 1 \u2014 Inventario\n\nProxmox + networking + storage.\n\n## Fase 2 \u2014 IaC\n\nOpenTofu + Ansible.\n\n## Fase 3 \u2014 Ops host\n\n`wetopi-ops01`.\n\n## Fase 4 \u2014 RKE2 HA\n\n3 control planes + 3 platform workers.\n\n## Fase 5 \u2014 Networking\n\nCilium + Hubble.\n\n## Fase 6 \u2014 Storage\n\nCeph CSI.\n\n## Fase 7 \u2014 GitOps\n\nReconciliaci\u00f3n desde Git.\n\n## Fase 8 \u2014 Secrets\n\nOpenBao.\n\n## Fase 9 \u2014 Docs / Memory / Work Ledger\n\nServicios ag\u00e9nticos compartidos.\n\n## Fase 10 \u2014 Hermes + ADK\n\nDespliegue definitivo.\n\n## Fase 11 \u2014 Edge\n\nPrimer tr\u00e1fico HTTP en RKE2.\n\n## Fase 12 \u2014 Hosting\n\nPrimeros WordPress.\n\n## Fase 13 \u2014 Migraci\u00f3n masiva\n\nSwarm -> RKE2.\n\n---\n\n# 35. Checklist de arranque\n\n## GitHub\n\n- [ ] organizaci\u00f3n `wetopi-platform`\n- [ ] repo `infrastructure`\n- [ ] repo `agent-services`\n- [ ] repo `ai-hub`\n- [ ] repo `agents`\n- [ ] repo `docs`\n- [ ] Codex autorizado\n- [ ] branch protection\n- [ ] CODEOWNERS\n\n## Proxmox\n\n- [ ] inventario\n- [ ] bridges\n- [ ] IP ranges\n- [ ] failure domains\n- [ ] Ubuntu template\n- [ ] automation account\n\n## Bootstrap\n\n- [ ] `wetopi-ops01`\n- [ ] `rke-cp01`\n- [ ] `rke-cp02`\n- [ ] `rke-cp03`\n- [ ] `rke-platform01`\n- [ ] `rke-platform02`\n- [ ] `rke-platform03`\n\n## Networking\n\n- [ ] una NIC LAN por VM\n- [ ] evitar overlaps\n- [ ] no modificar `192.168.96.0/20`\n- [ ] definir Pod CIDR\n- [ ] definir Service CIDR\n\n## RKE2\n\n- [ ] HA\n- [ ] embedded etcd\n- [ ] API stable endpoint\n- [ ] snapshots\n- [ ] restore tested\n\n## Platform\n\n- [ ] Cilium\n- [ ] Hubble\n- [ ] Ceph CSI\n- [ ] GitOps\n- [ ] OpenBao\n- [ ] MkDocs\n- [ ] Memory Plane\n- [ ] Work Ledger\n\n## Agents\n\n- [ ] Hermes SRE\n- [ ] Hermes Developer\n- [ ] Hermes Marketing\n- [ ] ADK Fleet\n\n## Migration\n\n- [ ] Edge nodes\n- [ ] Gateway API\n- [ ] first hosting workers\n- [ ] first 10 sites\n- [ ] production validation\n- [ ] progressive migration\n- [ ] Swarm retirement\n\n---\n\n# 36. Resultado esperado\n\nAl finalizar el bootstrap inicial:\n\n```text\n GitHub\n wetopi-platform\n \u2502\n infrastructure\n \u2502\n Codex\n \u2502\n wetopi-ops01\n \u2502\n OpenTofu/Ansible\n \u2502\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 RKE2 HA \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502 \u2502\n 3 Control Cilium 3 Platform\n Planes Hubble Workers\n \u2502 \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n Ceph CSI\n \u2502\n GitOps\n \u2502\n OpenBao\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc \u25bc \u25bc\n Docs Memory Work Ledger\n \u2502 \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n Hermes + ADK\n```\n\nTodav\u00eda sin necesidad de haber movido un solo cliente.\n\nDespu\u00e9s:\n\n```text\nEdge\n \u2193\nfirst hosting workers\n \u2193\nfirst websites\n \u2193\nprogressive migration\n \u2193\nSwarm retirement\n```\n\n---\n\n# Conclusi\u00f3n\n\nLa estrategia de arranque queda fijada as\u00ed:\n\n1. Git primero.\n2. IaC desde el primer d\u00eda.\n3. Bootstrap desde fuera de Kubernetes.\n4. RKE2 greenfield.\n5. Tres control planes.\n6. Tres platform workers.\n7. Una sola NIC LAN por VM.\n8. No tocar Ceph `192.168.96.0/20`.\n9. Cilium/Hubble.\n10. Ceph CSI.\n11. GitOps.\n12. OpenBao.\n13. Docs, Memory y Work Ledger.\n14. Hermes y ADK ya dentro del cluster definitivo.\n15. Edge despu\u00e9s.\n16. Hosting despu\u00e9s.\n17. Migraci\u00f3n progresiva desde Swarm.\n\nLa prioridad es crear una foundation que pueda romperse, reconstruirse y probarse sin afectar a producci\u00f3n, y solo despu\u00e9s empezar a trasladar tr\u00e1fico y workloads reales.", "html": "\n\nMigrado desde
\nwetopi-platform/docs@main(design/bootstrap-plan.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
Fecha: 28 de agosto de 2026
\nEstado: Plan de bootstrap consolidado
\nOrganizaci\u00f3n GitHub: wetopi-platform
\n\u00c1mbito: Creaci\u00f3n de repositorios, bootstrap de infraestructura, RKE2, servicios base y posterior incorporaci\u00f3n de agentes y workloads.
wetopi-ops01La nueva plataforma de Wetopi se construir\u00e1 desde cero, en paralelo a la infraestructura actual basada en Docker Swarm.
\nNo se realizar\u00e1 una migraci\u00f3n in-place.
\nLa nueva base ser\u00e1:
\nProxmox\n \u2193\nVMs Ubuntu\n \u2193\nRKE2\n \u2193\nCilium / Hubble\n \u2193\nCeph CSI\n \u2193\nGitOps\n \u2193\nOpenBao\n \u2193\nDocs / Memory / Work Ledger\n \u2193\nHermes + ADK\n \u2193\nEdge\n \u2193\nHosting workloads\nEl bootstrap inicial se har\u00e1 desde fuera de Kubernetes.
\nDurante esta primera etapa:
\nwetopi-ops01;192.168.96.0/20;La organizaci\u00f3n GitHub ser\u00e1:
\nwetopi-platform\ny los repositorios iniciales ser\u00e1n:
\nwetopi-platform/infrastructure\nwetopi-platform/agent-services\nwetopi-platform/ai-hub\nwetopi-platform/agents\nwetopi-platform/docs\nEl objetivo de esta fase no es migrar inmediatamente los ~700 contenedores actuales.
\nEl objetivo es construir una foundation nueva, reproducible y segura sobre la que posteriormente se pueda migrar Wetopi.
\nEl milestone inicial ser\u00e1:
\n\n\ndisponer de un cluster RKE2 HA vac\u00edo, estable, observable, con storage, secrets, GitOps y servicios ag\u00e9nticos comunes funcionando antes de mover tr\u00e1fico de clientes.
\n
RKE2
Cilium
Hubble
Ceph existente mediante CSI
OpenBao
Markdown + Git + MkDocs Material
OpenTofu + Ansible
Codex desde laptop + wetopi-ops01
Hermes + ADK
Los agentes se desplegar\u00e1n posteriormente dentro del cluster definitivo.
\nSe adopta como organizaci\u00f3n:
\ngithub.com/wetopi-platform\nSe descartan nombres temporales como wetopi3, wetopi-v3 o wetopi-new porque describen una migraci\u00f3n, no una plataforma duradera.
wetopi-platform se considera adecuado como namespace t\u00e9cnico de largo plazo.
Se crear\u00e1n inicialmente cinco repositorios.
\ninfrastructureResponsabilidad:
\nEstructura propuesta:
\ninfrastructure/\n\u251c\u2500\u2500 tofu/\n\u2502 \u2514\u2500\u2500 proxmox/\n\u2502 \u251c\u2500\u2500 modules/\n\u2502 \u2514\u2500\u2500 environments/\n\u2502 \u2514\u2500\u2500 prod/\n\u251c\u2500\u2500 ansible/\n\u2502 \u251c\u2500\u2500 inventories/\n\u2502 \u251c\u2500\u2500 roles/\n\u2502 \u2502 \u251c\u2500\u2500 common/\n\u2502 \u2502 \u251c\u2500\u2500 rke2-server/\n\u2502 \u2502 \u251c\u2500\u2500 rke2-agent/\n\u2502 \u2502 \u251c\u2500\u2500 edge/\n\u2502 \u2502 \u2514\u2500\u2500 natgw/\n\u2502 \u2514\u2500\u2500 playbooks/\n\u251c\u2500\u2500 kubernetes/\n\u2502 \u251c\u2500\u2500 bootstrap/\n\u2502 \u251c\u2500\u2500 cilium/\n\u2502 \u251c\u2500\u2500 hubble/\n\u2502 \u251c\u2500\u2500 ceph-csi/\n\u2502 \u251c\u2500\u2500 openbao/\n\u2502 \u251c\u2500\u2500 gitops/\n\u2502 \u251c\u2500\u2500 namespaces/\n\u2502 \u2514\u2500\u2500 observability/\n\u251c\u2500\u2500 scripts/\n\u2514\u2500\u2500 docs/\n \u251c\u2500\u2500 bootstrap.md\n \u251c\u2500\u2500 networking.md\n \u2514\u2500\u2500 disaster-recovery.md\nagent-servicesServicios compartidos de la capa ag\u00e9ntica:
\nmemory-service/\nwork-ledger-adapter/\nhermes-memory-provider/\nidentity/\napproval/\nInicialmente se priorizar\u00e1n memory-service, hermes-memory-provider y work-ledger-adapter.
ai-hubRepositorio com\u00fan de:
\nRegla:
\n\n\nuna tool utilizada por m\u00e1s de un agente debe vivir en
\nai-hub.
agentsMonorepo inicial de la flota:
\nagents/\n\u251c\u2500\u2500 hermes/\n\u2502 \u251c\u2500\u2500 sre/\n\u2502 \u251c\u2500\u2500 developer/\n\u2502 \u2514\u2500\u2500 marketing/\n\u251c\u2500\u2500 adk/\n\u2502 \u251c\u2500\u2500 platform/\n\u2502 \u251c\u2500\u2500 wordpress/\n\u2502 \u2514\u2500\u2500 support/\n\u251c\u2500\u2500 base/\n\u251c\u2500\u2500 deployments/\n\u2514\u2500\u2500 tests/\ndocsSource of truth documental:
\ndocs/\n\u251c\u2500\u2500 mkdocs.yml\n\u2514\u2500\u2500 docs/\n \u251c\u2500\u2500 architecture/\n \u251c\u2500\u2500 infrastructure/\n \u251c\u2500\u2500 kubernetes/\n \u251c\u2500\u2500 networking/\n \u251c\u2500\u2500 wordpress/\n \u251c\u2500\u2500 agents/\n \u251c\u2500\u2500 procedures/\n \u251c\u2500\u2500 runbooks/\n \u251c\u2500\u2500 security/\n \u2514\u2500\u2500 adr/\nDesde el primer d\u00eda:
\nGit\n \u2193\nreview\n \u2193\nautomation\n \u2193\ninfrastructure\nLa shell history no ser\u00e1 source of truth.
\nTodo cambio relevante debe terminar en Git.
\nSe selecciona inicialmente OpenTofu por:
Terraform seguir\u00eda siendo t\u00e9cnicamente v\u00e1lido.
\nSe crear\u00e1n inicialmente:
\n1 x ops VM\n3 x RKE2 control plane\n3 x platform workers\nTotal:
\n7 VMs\nNo se crear\u00e1n todav\u00eda hosting workers masivos ni edge productivos.
\nwetopi-ops01Se crear\u00e1 wetopi-ops01 fuera del cluster Kubernetes.
Objetivo:
\n\n\nbootstrap host, bastion administrativo y punto de recuperaci\u00f3n cuando Kubernetes no est\u00e9 disponible.
\n
Sizing orientativo:
\n4 vCPU\n8 GB RAM\n40-80 GB disk\nUbuntu 24.04\n1 NIC LAN\nHerramientas:
\ngit\nssh\nansible\nopentofu\nkubectl\nhelm\njq\nyq\nrke2 tooling\nNo alojar\u00e1 workloads productivos.
\nDebe permitir operar si fallan Cilium, GitOps, OpenBao o parte del cluster.
\nSe desplegar\u00e1n:
\nrke-cp01\nrke-cp02\nrke-cp03\nCada uno en un host Proxmox f\u00edsico diferente siempre que sea posible.
\nSizing orientativo:
\n8 vCPU\n16 GB RAM\n50-100 GB disk\nUbuntu 24.04\n1 NIC LAN\nSe desplegar\u00e1n:
\nrke-platform01\nrke-platform02\nrke-platform03\nSizing orientativo:
\n8-16 vCPU\n16-32 GB RAM\nUbuntu 24.04\n1 NIC LAN\nLabel:
\nwetopi.io/role=platform\nEjemplo:
\npve01 -> rke-cp01 + rke-platform01\npve02 -> rke-cp02 + rke-platform02\npve03 -> rke-cp03 + rke-platform03\nLa distribuci\u00f3n real debe respetar failure domains y recursos disponibles.
\nCada VM tendr\u00e1 una \u00fanica interfaz LAN.
\nVM\n \u2514\u2500\u2500 eth0\n \u2514\u2500\u2500 Wetopi LAN / vRack\nLos rangos Kubernetes ser\u00e1n independientes:
\nVM LAN != Pod CIDR != Service CIDR != Ceph network\nLos rangos concretos deben definirse tras inventariar las redes actuales.
\nExiste una red Ceph:
\n192.168.96.0/20\nque forma parte de la infraestructura hiperconvergida Proxmox + Ceph.
\nDecisi\u00f3n:
\n\n\nNO tocar, renumerar ni redise\u00f1ar esta red durante el bootstrap RKE2.
\n
RKE2 consumir\u00e1 Ceph mediante CSI adapt\u00e1ndose a la infraestructura existente.
\nSe utilizar\u00e1 embedded etcd sobre los tres server nodes.
cp01 etcd\ncp02 etcd\ncp03 etcd\nQuorum: 2 de 3.
\nSe priorizar\u00e1 almacenamiento de baja latencia para etcd. Debe evaluarse local NVMe si la alternativa Ceph introduce latencia innecesaria.
\nLos snapshots de etcd deber\u00e1n almacenarse fuera de los propios nodos y el procedimiento de restore deber\u00e1 probarse.
\nModelo:
\nLaptop\n \u2502\nCodex\n \u2502\nGitHub\n \u2502\nops01\n \u2502\nOpenTofu + Ansible\n \u2502\nProxmox / RKE2\nKubernetes no depender\u00e1 de agentes internos para poder crearse o repararse.
\nDurante el bootstrap Codex act\u00faa como bootstrap engineer.
Trabajar\u00e1 contra:
\nops01;Todo cambio relevante debe quedar versionado.
\nNo se desplegar\u00e1n varios Hermes temporales antes del cluster.
\nLos Hermes definitivos se desplegar\u00e1n dentro de RKE2 cuando existan al menos:
\nRKE2 estable\nCilium\nHubble\nCeph CSI\nGitOps\nOpenBao\nDocumentation\nMemory Plane\nWork Ledger\nDespu\u00e9s:
\nHermes SRE\nHermes Developer\nHermes Marketing\nRKE2\n \u2193\nCilium\n \u2193\nHubble\n \u2193\nCeph CSI\n \u2193\nGitOps\n \u2193\nOpenBao\n \u2193\nDocumentation\n \u2193\nMemory Plane\n \u2193\nWork Ledger\n \u2193\nAI Hub integration\n \u2193\nHermes\n \u2193\nADK Fleet\nCilium ser\u00e1 el CNI base.
\nInicialmente:
\nNo se activar\u00e1n desde el d\u00eda uno BGP WAN, egress complejo o service mesh.
\nPruebas m\u00ednimas antes de customer workloads:
\ncrear PVC\nmontar PVC\nborrar Pod\nrecrear Pod\nvalidar persistencia\nsnapshot si aplica\nEl cluster evolucionar\u00e1 hacia:
\nGit -> reconciler -> Kubernetes\nLa herramienta concreta puede decidirse durante el bootstrap.
\nOpenBao ser\u00e1 el secrets manager central.
\nConfigurar:
\nRepo:
\nwetopi-platform/docs\nRender:
\nMkDocs Material\nMarkdown ser\u00e1 el source of truth.
\nLa memoria institucional propia de Wetopi se productivizar\u00e1 como servicio central para Hermes, ADK y futuros agentes.
\nDebe soportar:
\nSe desplegar\u00e1 un backend Kanban/ticketing desacoplado mediante contrato Wetopi.
\nResponsabilidades:
\nRepo:
\nwetopi-platform/ai-hub\nPara tools, skills, contracts, components, SDKs, schemas y evals.
\nUna vez montada la base, Hermes y ADK participar\u00e1n en:
\nNo forman parte del bootstrap inicial.
\nDespu\u00e9s se a\u00f1adir\u00e1n:
\nrke-edge01\nrke-edge02\nrke-edge03\n...\nintegrados con IPVS, VRRP, DSR y Gateway API.
\nDespu\u00e9s del edge se crear\u00e1n pocos workers de hosting:
\nrke-host01\nrke-host02\nrke-host03\nPrimero se migrar\u00e1n aproximadamente 10 sites y se validar\u00e1 la plataforma antes de escalar.
\nSwarm seguir\u00e1 produciendo durante el bootstrap.
\nNo habr\u00e1 big-bang.
\nLa capacidad se trasladar\u00e1 progresivamente hasta retirar Swarm.
\nCodex puede tener permisos amplios de GitHub sobre la organizaci\u00f3n, pero:
\nGitHub permissions != production credentials\nNo entregar inicialmente:
\nCrear una identidad espec\u00edfica de automatizaci\u00f3n para Proxmox con m\u00ednimos permisos necesarios.
\nPreparar los cinco repos con README, branch protection, CODEOWNERS y CI m\u00ednima.
\nInventariar:
\nNo inventar direccionamiento.
\nCrear template Ubuntu 24.04 reproducible con cloud-init.
\nops01Crear wetopi-ops01 mediante OpenTofu y provisionarla con Ansible.
Crear rke-cp01, rke-cp02, rke-cp03.
Crear rke-platform01, rke-platform02, rke-platform03.
Instalar RKE2 con embedded etcd y endpoint API estable.
\nValidar:
\nreboot cp\nloss of one cp\nworker reboot\ndrain\nuncordon\netcd snapshot\nrestore procedure\nantes de seguir.
\nCrear organizaci\u00f3n y repos.
\nProxmox + networking + storage.
\nOpenTofu + Ansible.
\nwetopi-ops01.
3 control planes + 3 platform workers.
\nCilium + Hubble.
\nCeph CSI.
\nReconciliaci\u00f3n desde Git.
\nOpenBao.
\nServicios ag\u00e9nticos compartidos.
\nDespliegue definitivo.
\nPrimer tr\u00e1fico HTTP en RKE2.
\nPrimeros WordPress.
\nSwarm -> RKE2.
\nwetopi-platforminfrastructureagent-servicesai-hubagentsdocswetopi-ops01rke-cp01rke-cp02rke-cp03rke-platform01rke-platform02rke-platform03192.168.96.0/20Al finalizar el bootstrap inicial:
\n GitHub\n wetopi-platform\n \u2502\n infrastructure\n \u2502\n Codex\n \u2502\n wetopi-ops01\n \u2502\n OpenTofu/Ansible\n \u2502\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 RKE2 HA \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502 \u2502\n 3 Control Cilium 3 Platform\n Planes Hubble Workers\n \u2502 \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n Ceph CSI\n \u2502\n GitOps\n \u2502\n OpenBao\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc \u25bc \u25bc\n Docs Memory Work Ledger\n \u2502 \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n Hermes + ADK\nTodav\u00eda sin necesidad de haber movido un solo cliente.
\nDespu\u00e9s:
\nEdge\n \u2193\nfirst hosting workers\n \u2193\nfirst websites\n \u2193\nprogressive migration\n \u2193\nSwarm retirement\nLa estrategia de arranque queda fijada as\u00ed:
\n192.168.96.0/20.La prioridad es crear una foundation que pueda romperse, reconstruirse y probarse sin afectar a producci\u00f3n, y solo despu\u00e9s empezar a trasladar tr\u00e1fico y workloads reales.
\n" }, { "id": 56, "title": "decisions log", "slug": "decisions-log", "book_id": 8, "chapter_id": 29, "description": null, "created_at": "2026-09-06T13:39:02.000000Z", "updated_at": "2026-09-06T13:39:02.000000Z", "markdown": "> Migrado desde `wetopi-platform/docs@main` (design/decisions-log.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.\n\n# Wetopi Platform \u2014 Memoria t\u00e9cnica y ejecutiva de decisiones recientes\n## RKE2, networking, GitOps, Vault y bootstrap para Codex\n\n**Fecha:** 28 de agosto de 2026 \n**Estado:** Decisiones consolidadas para implementaci\u00f3n \n**Organizaci\u00f3n GitHub:** `wetopi-platform` \n**Audiencia principal:** Codex / agentes de implementaci\u00f3n / operadores SRE \n**Objetivo:** preservar el contexto t\u00e9cnico y ejecutivo de las \u00faltimas decisiones para continuar el proyecto sin reabrir debates ya cerrados.\n\n---\n\n# \u00cdndice\n\n1. Resumen ejecutivo\n2. Contexto del proyecto\n3. Decisiones consolidadas\n4. RKE2 y topolog\u00eda inicial\n5. Bootstrap externo y `wetopi-ops01`\n6. Organizaci\u00f3n GitHub y repos\n7. OpenTofu + Ansible\n8. Networking de las VMs\n9. Ceph existente\n10. CIDRs propuestos\n11. Cilium + Hubble\n12. Ingress WAN: IPVS + VRRP + DSR\n13. Edge nodes\n14. Traefik + Gateway API\n15. Sustituci\u00f3n de Consul/Registrator/consul-template\n16. Egress simple con NATGW Ubuntu\n17. Selecci\u00f3n de egress por site/customer\n18. Argo CD\n19. Infisical como Wetopi Vault\n20. SOPS + age como bootstrap\n21. Kubernetes Secrets como materializaci\u00f3n runtime\n22. Modelo de acceso de agentes a secrets\n23. Tool runtimes y aislamiento de credenciales\n24. Hermes personales y coding agents\n25. ServiceAccounts, RBAC y namespaces\n26. Secrets encryption at rest\n27. Orden de bootstrap actualizado\n28. Primeras tareas para Codex\n29. Decisiones a\u00fan abiertas\n30. Criterios de aceptaci\u00f3n\n31. Resumen final para implementaci\u00f3n\n32. PLATFORM-018 \u2014 CNI Cilium desde el d\u00eda uno (sin Canal provisional)\n\n---\n\n# 1. Resumen ejecutivo\n\nWetopi est\u00e1 reconstruyendo su plataforma de Managed WordPress Hosting desde Docker Swarm hacia una nueva base RKE2 sobre su cloud privado Proxmox + Ceph.\n\nLa nueva plataforma debe ser AI-first y operable tanto por humanos como por agentes.\n\nLas decisiones ya cerradas son:\n\n```text\nKubernetes distribution:\nRKE2\n\nCNI:\nCilium\n\nNetwork observability:\nHubble\n\nGitOps:\nArgo CD\n\nVault / secrets manager:\nInfisical self-hosted\n\nBootstrap secret encryption:\nSOPS + age\n\nInfrastructure as Code:\nOpenTofu + Ansible\n\nIngress WAN:\nmantener IPVS + VRRP + DSR inicialmente\n\nProxy edge inicial:\nTraefik\n\nHTTP routing:\nGateway API\n\nEgress:\nNATGW Ubuntu simples con gateway interno estable y WAN IP cambiable\n```\n\nEl cluster inicial se construir\u00e1 greenfield, en paralelo al Docker Swarm actual. No habr\u00e1 big-bang.\n\nLa plataforma se levantar\u00e1 primero, se validar\u00e1, y despu\u00e9s se migrar\u00e1n edge nodes, hosting workers y sites de forma progresiva.\n\n---\n\n# 2. Contexto del proyecto\n\nWetopi es un Managed WordPress Hosting ejecutado actualmente sobre:\n\n```text\nOVH\n \u2193\nvRack\n \u2193\nProxmox\n \u2193\nCeph\n \u2193\nDocker Swarm\n```\n\nEl workload principal son `wepod`, aproximadamente un contenedor por website, con:\n\n```text\ns6\nnginx\nphp-fpm\nmysql\n```\n\nEl tr\u00e1fico WAN actual utiliza:\n\n```text\nVIP\n \u2193\nVRRP / Keepalived\n \u2193\nIPVS\n \u2193\nDSR\n \u2193\nWAN nginx proxies\n \u2193\nSwarm workloads\n```\n\nEl descubrimiento de backends se realiza mediante:\n\n```text\nRegistrator\n \u2193\nConsul\n \u2193\nconsul-template\n \u2193\nnginx.conf\n```\n\nLa nueva plataforma debe preservar las propiedades valiosas de esta arquitectura, especialmente DSR y la distribuci\u00f3n del ancho de banda de salida entre distintos hosts f\u00edsicos de OVH.\n\n---\n\n# 3. Decisiones consolidadas\n\n## PLATFORM-001 \u2014 RKE2\n\nRKE2 ser\u00e1 la distribuci\u00f3n Kubernetes de producci\u00f3n.\n\n## PLATFORM-002 \u2014 Greenfield\n\nEl cluster nuevo se construir\u00e1 en paralelo a Swarm.\n\n## PLATFORM-003 \u2014 3 control planes\n\nEl control plane inicial tendr\u00e1 tres nodos RKE2 server con embedded etcd.\n\n## PLATFORM-004 \u2014 3 platform workers\n\nInicialmente habr\u00e1 tres workers dedicados a servicios de plataforma.\n\n## PLATFORM-005 \u2014 Bootstrap externo\n\nKubernetes se construye desde fuera del cluster.\n\n## PLATFORM-006 \u2014 `wetopi-ops01`\n\nExistir\u00e1 una VM externa de operaciones, bootstrap y recovery.\n\n## PLATFORM-007 \u2014 Cilium\n\nCilium ser\u00e1 el CNI.\n\n## PLATFORM-008 \u2014 Hubble\n\nHubble ser\u00e1 la base de network observability.\n\n## PLATFORM-009 \u2014 Argo CD\n\nArgo CD ser\u00e1 el sistema GitOps.\n\n## PLATFORM-010 \u2014 Infisical\n\nInfisical self-hosted ser\u00e1 el Vault operativo de Wetopi.\n\n## PLATFORM-011 \u2014 SOPS + age\n\nSOPS + age se utilizar\u00e1 para bootstrap secrets, no como Vault principal.\n\n## PLATFORM-012 \u2014 IPVS/DSR\n\nSe preserva IPVS + VRRP + DSR inicialmente.\n\n## PLATFORM-013 \u2014 Traefik\n\nTraefik ser\u00e1 el proxy edge inicial.\n\n## PLATFORM-014 \u2014 Gateway API\n\nEl routing HTTP se expresar\u00e1 con Gateway API.\n\n## PLATFORM-015 \u2014 Egress simple\n\nSe usar\u00e1n NATGW Ubuntu simples con gateway interno estable y WAN IP cambiable.\n\n## PLATFORM-016 \u2014 No Istio inicialmente\n\nNo se desplegar\u00e1 service mesh sin necesidad concreta.\n\n## PLATFORM-017 \u2014 No BGP como requisito inicial\n\nLa plataforma no depender\u00e1 de BGP para arrancar.\n\n---\n\n# 4. RKE2 y topolog\u00eda inicial\n\nSe crear\u00e1n inicialmente:\n\n```text\n1 x wetopi-ops01\n3 x RKE2 control plane\n3 x platform workers\n```\n\nTotal inicial:\n\n```text\n7 VMs\n```\n\nNombres previstos:\n\n```text\nwetopi-ops01\n\nrke-cp01\nrke-cp02\nrke-cp03\n\nrke-platform01\nrke-platform02\nrke-platform03\n```\n\nLos control planes y workers deben distribuirse entre distintos hosts Proxmox.\n\nEjemplo:\n\n```text\npve01 -> rke-cp01 + rke-platform01\npve02 -> rke-cp02 + rke-platform02\npve03 -> rke-cp03 + rke-platform03\n```\n\nLa distribuci\u00f3n real deber\u00e1 validarse con el inventario.\n\n---\n\n# 5. Bootstrap externo y `wetopi-ops01`\n\nLa regla es:\n\n> Kubernetes no debe depender de servicios internos de Kubernetes para poder crearse o repararse.\n\nModelo:\n\n```text\nLaptop\n \u2502\nCodex\n \u2502\nGitHub\n \u2502\nwetopi-ops01\n \u2502\nOpenTofu + Ansible\n \u2502\nProxmox\n \u2502\nRKE2\n```\n\nCodex trabajar\u00e1 inicialmente desde el laptop del operador.\n\n`wetopi-ops01` ser\u00e1 la m\u00e1quina durable de bootstrap y disaster recovery.\n\nDebe tener, como m\u00ednimo:\n\n```text\ngit\nssh\nansible\nopentofu\nkubectl\nhelm\njq\nyq\n```\n\nNo alojar\u00e1 workloads productivos.\n\n---\n\n# 6. Organizaci\u00f3n GitHub y repos\n\nOrganizaci\u00f3n:\n\n```text\nwetopi-platform\n```\n\nRepos iniciales:\n\n```text\nwetopi-platform/infrastructure\nwetopi-platform/agent-services\nwetopi-platform/ai-hub\nwetopi-platform/agents\nwetopi-platform/docs\n```\n\nNo crear repos adicionales sin necesidad real.\n\n---\n\n# 7. OpenTofu + Ansible\n\nSe selecciona:\n\n```text\nOpenTofu\n+\nAnsible\n```\n\nOpenTofu:\n\n- crea VMs;\n- configura recursos Proxmox;\n- modela infraestructura declarativa.\n\nAnsible:\n\n- baseline Ubuntu;\n- sysctl;\n- paquetes;\n- RKE2;\n- configuraci\u00f3n de hosts;\n- operaciones bootstrap.\n\nGit es source of truth.\n\nNo se considera v\u00e1lido dejar cambios permanentes solo en shell history.\n\n---\n\n# 8. Networking de las VMs\n\nCada VM tendr\u00e1 inicialmente:\n\n```text\n1 NIC LAN\n```\n\nModelo:\n\n```text\nVM\n \u2514\u2500\u2500 eth0\n \u2514\u2500\u2500 LAN / vRack\n```\n\nNo se introducir\u00e1n m\u00faltiples interfaces salvo necesidad demostrada.\n\n---\n\n# 9. Ceph existente\n\nLa red Ceph actual es:\n\n```text\n192.168.96.0/20\n```\n\nEs parte de la infraestructura hiperconvergida Proxmox + Ceph.\n\nDecisi\u00f3n:\n\n> NO modificar, renumerar ni redise\u00f1ar esta red.\n\nCeph se consumir\u00e1 desde Kubernetes mediante CSI.\n\nLa conectividad exacta entre RKE2 nodes y Ceph debe comprobarse, no asumirse.\n\n---\n\n# 10. CIDRs propuestos\n\nPropuesta actual:\n\n```text\nNode LAN:\npendiente de inventario real\n\nPod CIDR:\n10.64.0.0/14\n\nService CIDR:\n10.68.0.0/16\n\nCeph:\n192.168.96.0/20\nEXISTENTE / NO TOCAR\n```\n\nLos CIDRs definitivos deben validarse contra todas las redes Wetopi existentes.\n\n---\n\n# 11. Cilium + Hubble\n\nCilium ser\u00e1 el CNI base.\n\nPrimera versi\u00f3n:\n\n```text\nCilium\n+\nVXLAN encapsulation\n+\nHubble\n```\n\nRaz\u00f3n de VXLAN inicial:\n\n- m\u00ednima dependencia de underlay;\n- no requiere que la red f\u00edsica conozca PodCIDRs;\n- simplifica bootstrap;\n- una sola NIC por VM.\n\nNo activar inicialmente:\n\n- BGP WAN;\n- service mesh;\n- egress avanzado;\n- features experimentales innecesarias.\n\nHubble se utilizar\u00e1 para flows y correlaci\u00f3n de tr\u00e1fico con pods/namespaces/sites.\n\n---\n\n# 12. Ingress WAN: IPVS + VRRP + DSR\n\nEl frontend actual se conserva inicialmente.\n\n```text\nInternet\n \u2193\nVIP\n \u2193\nVRRP\n \u2193\nIPVS\n \u2193\nDSR\n \u2193\nRKE2 Edge\n```\n\nMotivo principal:\n\nDSR permite que cada edge node responda directamente a Internet usando el uplink/bandwidth domain del host f\u00edsico Proxmox/OVH donde corre.\n\nEsto permite agregar capacidad de salida entre varios hosts f\u00edsicos.\n\nEsta propiedad no debe perderse durante la migraci\u00f3n.\n\n---\n\n# 13. Edge nodes\n\nLos antiguos WAN proxies evolucionar\u00e1n a RKE2 edge workers.\n\nEjemplo:\n\n```text\nrke-edge-a01\nrke-edge-a02\nrke-edge-b01\nrke-edge-b02\n```\n\nLabels:\n\n```text\nwetopi.io/role=edge\nwetopi.io/wan-zone=a\nwetopi.io/proxmox-host=pve01\nwetopi.io/bandwidth-domain=pve01\n```\n\nTaint:\n\n```text\nwetopi.io/edge=true:NoSchedule\n```\n\nEl proxy se desplegar\u00e1 como DaemonSet cuando tenga sentido mantener una instancia por edge node.\n\n---\n\n# 14. Traefik + Gateway API\n\nSe elige Traefik inicialmente.\n\nRazones:\n\n- sencillo;\n- Kubernetes-native;\n- soporte de Gateway API;\n- TLS;\n- routing din\u00e1mico;\n- m\u00e9tricas;\n- separaci\u00f3n clara respecto a Cilium.\n\nModelo:\n\n```text\nhostname\n \u2193\nHTTPRoute\n \u2193\nService\n \u2193\nEndpointSlice\n \u2193\nPod\n```\n\nEnvoy podr\u00e1 reevaluarse m\u00e1s adelante si aparecen necesidades L7 m\u00e1s avanzadas.\n\n---\n\n# 15. Sustituci\u00f3n de Consul/Registrator/consul-template\n\nAntes:\n\n```text\nDocker\n \u2193\nRegistrator\n \u2193\nConsul\n \u2193\nconsul-template\n \u2193\nnginx.conf\n \u2193\nreload\n```\n\nDespu\u00e9s:\n\n```text\nPod\n \u2193\nService\n \u2193\nEndpointSlice\n \u2193\nGateway controller\n \u2193\ndynamic routing\n```\n\nKubernetes ser\u00e1 el service registry del hosting.\n\n---\n\n# 16. Egress simple con NATGW Ubuntu\n\nNo se construir\u00e1 inicialmente un egress control plane complejo.\n\nModelo base:\n\n```text\ngw100\nLAN: 192.168.88.100\nWAN: PUBLIC_IP_A\n\ngw101\nLAN: 192.168.88.101\nWAN: PUBLIC_IP_B\n\ngw102\nLAN: 192.168.88.102\nWAN: PUBLIC_IP_C\n```\n\nCada gateway:\n\n```text\nUbuntu\n+\nrouting\n+\nnftables/iptables\n+\nSNAT\n```\n\nObjetivo:\n\n- f\u00e1cil de operar;\n- f\u00e1cil de cambiar;\n- f\u00e1cil de observar;\n- f\u00e1cil de separar reputaci\u00f3n.\n\n---\n\n# 17. Selecci\u00f3n de egress por site/customer\n\nModelo l\u00f3gico:\n\n```text\nsite/customer\n \u2193\negress_gateway\n \u2193\ngw100 / gw101 / gw102\n```\n\nEjemplo:\n\n```yaml\nwetopi.io/egress-gateway: gw101\n```\n\nLa implementaci\u00f3n inicial puede usar:\n\n- policy routing;\n- automatizaci\u00f3n ligera;\n- helper/DaemonSet que observe labels y mantenga reglas.\n\nNo construir todav\u00eda un controlador complejo.\n\nFuturo:\n\n```text\nCiliumEgressGatewayPolicy\n```\n\npuede sustituir el mecanismo sin cambiar el modelo l\u00f3gico.\n\nSeparar desde el inicio:\n\n```text\nhosting-general\nhosting-premium\nplatform\nagents/crawlers\n```\n\nLos crawlers/agentes no deben compartir IP de salida con customers.\n\n---\n\n# 18. Argo CD\n\nSe ha decidido usar:\n\n```text\nArgo CD\n```\n\nRazones principales:\n\n- excelente Web UI;\n- estado Synced/OutOfSync;\n- health;\n- diffs;\n- historial;\n- visibilidad para operadores humanos;\n- modelo declarativo;\n- API/CLI;\n- RBAC/SSO/OIDC futuro.\n\nModelo:\n\n```text\nGit\n \u2193\nArgo CD\n \u2193\nKubernetes\n```\n\nNo se utilizar\u00e1 un pipeline CI basado en `kubectl apply` como mecanismo normal de producci\u00f3n.\n\nPendiente decidir el layout exacto:\n\n- App of Apps;\n- ApplicationSet;\n- AppProjects;\n- estructura por cluster/environment.\n\n---\n\n# 19. Infisical como Wetopi Vault\n\nEl Vault operativo de Wetopi ser\u00e1:\n\n```text\nInfisical self-hosted\n```\n\nEl requisito real no es solamente almacenar secrets declarativos.\n\nSe necesita un servicio vivo donde humanos y agentes puedan:\n\n- crear credentials;\n- guardar username/password;\n- guardar API tokens;\n- guardar SSH keys;\n- recuperar secrets;\n- rotarlos;\n- listarlos;\n- actualizarlos;\n- auditarlos;\n- compartirlos con servicios.\n\nEjemplo conceptual:\n\n```text\nvault.put(\n path=\"/platform/proxmox/api\",\n username=\"automation\",\n password=\"...\"\n)\n```\n\nInfisical ser\u00e1 el source of truth operativo de credentials.\n\n---\n\n# 20. SOPS + age como bootstrap\n\nSOPS NO ser\u00e1 el Vault principal.\n\nSe utilizar\u00e1 \u00fanicamente para bootstrap.\n\nModelo:\n\n```text\nGit\n \u2193\nSOPS + age\n \u2193\nbootstrap secrets\n \u2193\nInfisical starts\n```\n\nSOPS protege los secretos necesarios para levantar:\n\n- Infisical;\n- integraci\u00f3n inicial de Argo CD;\n- bootstrap identities;\n- recovery material m\u00ednimo.\n\nUna vez Infisical funciona, los nuevos credentials se guardan en Infisical, no en Git.\n\nDebe existir al menos una recovery key `age` fuera del cluster y fuera de Git.\n\n---\n\n# 21. Kubernetes Secrets como materializaci\u00f3n runtime\n\nModelo:\n\n```text\nInfisical\n \u2502\n \u2502 sync/operator\n \u25bc\nKubernetes Secret\n \u2502\n \u25bc\nDeployment / Pod\n```\n\nInfisical es source of truth.\n\nKubernetes Secret es materializaci\u00f3n runtime.\n\nEsto permite:\n\n- cambiar un secret centralmente;\n- sincronizarlo al cluster;\n- consumirlo con mecanismos nativos de Kubernetes.\n\n---\n\n# 22. Modelo de acceso de agentes a secrets\n\nLa regla principal es:\n\n> el agente debe poder USAR un secret sin necesidad de VER su valor cuando sea posible.\n\nEjemplo:\n\n```text\nHermes\n \u2502\n \u2502 tool call\n \u25bc\nplatform-tools\n \u2502\n \u2502 reads secret from Infisical/K8s Secret\n \u25bc\nProxmox / Cloudflare / OVH\n```\n\nEl LLM no necesita recibir el token.\n\n---\n\n# 23. Tool runtimes y aislamiento de credenciales\n\nNo montar todos los secrets directamente en el proceso LLM.\n\nAgrupar tools por security domain.\n\nEjemplo:\n\n```text\nplatform-tools\n\u251c\u2500\u2500 kubernetes\n\u251c\u2500\u2500 proxmox\n\u251c\u2500\u2500 ceph\n\u2514\u2500\u2500 networking\n\nwordpress-tools\n\u251c\u2500\u2500 wordpress\n\u251c\u2500\u2500 mysql\n\u2514\u2500\u2500 backups\n\nmarketing-tools\n\u251c\u2500\u2500 intercom\n\u251c\u2500\u2500 analytics\n\u2514\u2500\u2500 ...\n```\n\nCada runtime recibe \u00fanicamente los secrets necesarios.\n\nNo crear un sidecar por tool individual.\n\nEn `ai-hub`, las tools deber\u00edan declarar sus requirements.\n\nEjemplo:\n\n```yaml\ntool: proxmox.vm_restart\nruntime: platform-tools\nsecrets:\n - proxmox-api-token\n```\n\nLos agentes deben consumir un contrato propio Wetopi, no acoplarse directamente a Infisical.\n\nAPI conceptual:\n\n```text\nvault.get()\nvault.put()\nvault.update()\nvault.delete()\nvault.list()\nvault.exists()\n```\n\n---\n\n# 24. Hermes personales y coding agents\n\nHermes personales/coding agents necesitar\u00e1n operar la plataforma.\n\nPodr\u00e1n:\n\n- llamar tools privilegiadas;\n- usar credentials;\n- escribir secrets en Vault;\n- recuperar credentials autorizados;\n- operar GitHub;\n- operar Proxmox;\n- operar Kubernetes.\n\nPero:\n\n- no recibir todas las credenciales en env;\n- no montar un vault global;\n- no dar acceso indiscriminado a todos los paths.\n\nEl permiso debe seguir el rol.\n\nEjemplo conceptual:\n\n```text\nHermes SRE\n /platform/*\n /infrastructure/*\n /agents/hermes-sre/*\n\nHermes Developer\n /development/*\n /github/*\n /agents/hermes-developer/*\n\nHermes Marketing\n /marketing/*\n /agents/hermes-marketing/*\n```\n\n---\n\n# 25. ServiceAccounts, RBAC y namespaces\n\nPara operar Kubernetes, los agentes dentro del cluster usar\u00e1n:\n\n```text\nServiceAccount\n+\nRBAC\n```\n\nNo kubeconfig est\u00e1tico ni password.\n\nEjemplo:\n\n```text\nHermes SRE\n -> permisos operacionales amplios\n\nHermes Developer\n -> namespaces dev + deployments\n\nADK observer\n -> readonly\n\nADK specialist\n -> permissions espec\u00edficas\n```\n\nNamespaces orientativos:\n\n```text\nwetopi-platform\nwetopi-edge\nwetopi-hosting\nwetopi-agents-sre\nwetopi-agents-development\nwetopi-agents-marketing\nwetopi-observability\nwetopi-security\n```\n\nLos namespaces forman parte de la frontera de secrets y RBAC.\n\nImportante:\n\nquien puede crear Pods en un namespace puede potencialmente montar Secrets de ese namespace.\n\nPor tanto, permisos de workload creation y secret access deben dise\u00f1arse conjuntamente.\n\n---\n\n# 26. Secrets encryption at rest\n\nSOPS protege Git.\n\nInfisical protege el Vault.\n\nKubernetes Secrets materializados deben protegerse en etcd.\n\nRKE2 deber\u00e1 tener:\n\n```text\nSecrets Encryption at Rest\n```\n\nhabilitado.\n\nModelo:\n\n```text\nInfisical\n \u2193\nKubernetes Secret\n \u2193\nRKE2 API server\n \u2193\nencrypted etcd storage\n```\n\n---\n\n# 27. Orden de bootstrap actualizado\n\nOrden actual recomendado:\n\n```text\n1. GitHub organization + repos\n2. Inventario Proxmox/redes\n3. OpenTofu + Ansible\n4. wetopi-ops01\n5. 3 x RKE2 control plane\n6. 3 x platform workers\n7. RKE2 HA + etcd\n8. Cilium\n9. Hubble\n10. Ceph CSI\n11. Argo CD\n12. SOPS + age bootstrap\n13. Infisical\n14. Kubernetes Secrets integration/operator\n15. Docs\n16. Memory Plane\n17. Work Ledger\n18. AI Hub\n19. Hermes\n20. ADK Fleet\n21. Edge nodes\n22. Gateway API + Traefik\n23. NATGW\n24. first hosting workers\n25. first websites\n26. progressive migration from Swarm\n```\n\n---\n\n# 28. Primeras tareas para Codex\n\n## Task 1 \u2014 Crear repos\n\nCrear:\n\n```text\ninfrastructure\nagent-services\nai-hub\nagents\ndocs\n```\n\nAplicar:\n\n- README;\n- CODEOWNERS;\n- branch protection;\n- CI m\u00ednima;\n- convenciones.\n\n## Task 2 \u2014 Inventario Proxmox\n\nRecoger:\n\n- nodos;\n- CPU;\n- RAM;\n- storage;\n- bridges;\n- VLANs;\n- redes;\n- IP ranges;\n- failure domains;\n- bandwidth domains.\n\nNo crear a\u00fan direccionamiento inventado.\n\n## Task 3 \u2014 Validar redes\n\nConfirmar que:\n\n```text\nPod CIDR 10.64.0.0/14\nService CIDR 10.68.0.0/16\n```\n\nno colisionan.\n\nConfirmar conectividad RKE2 node -> Ceph `192.168.96.0/20`.\n\n## Task 4 \u2014 Ubuntu template\n\nCrear template reproducible Ubuntu 24.04 + cloud-init.\n\n## Task 5 \u2014 `wetopi-ops01`\n\nCrear/provisionar `wetopi-ops01`.\n\n## Task 6 \u2014 Control plane\n\nCrear:\n\n```text\nrke-cp01\nrke-cp02\nrke-cp03\n```\n\n## Task 7 \u2014 Platform workers\n\nCrear:\n\n```text\nrke-platform01\nrke-platform02\nrke-platform03\n```\n\n## Task 8 \u2014 RKE2\n\nConfigurar:\n\n- embedded etcd;\n- HA;\n- API endpoint estable;\n- secrets encryption;\n- snapshots.\n\n## Task 9 \u2014 Cilium/Hubble\n\nDesplegar:\n\n- Cilium VXLAN;\n- Hubble;\n- networking tests.\n\n## Task 10 \u2014 Ceph CSI\n\nValidar:\n\n- PVC create;\n- mount;\n- pod restart;\n- persistence.\n\n## Task 11 \u2014 Argo CD\n\nDesplegar declarativamente.\n\n## Task 12 \u2014 SOPS bootstrap\n\nGenerar:\n\n- age production key;\n- age recovery key;\n- `.sops.yaml`.\n\nGuardar recovery key fuera del cluster y fuera de Git.\n\n## Task 13 \u2014 Infisical\n\nDesplegar self-hosted.\n\nConfigurar:\n\n- persistence;\n- backups;\n- admin bootstrap;\n- projects/environments;\n- machine identities m\u00ednimas;\n- Kubernetes integration.\n\n## Task 14 \u2014 Vault contract\n\nCrear en `ai-hub` una capa estable:\n\n```text\nvault.get()\nvault.put()\nvault.update()\nvault.delete()\nvault.list()\nvault.exists()\n```\n\nLos agentes no deben quedar acoplados a la API concreta de Infisical.\n\n---\n\n# 29. Decisiones a\u00fan abiertas\n\n## 29.1 Node LAN CIDR\n\nPendiente de inventario.\n\n## 29.2 Endpoint HA de RKE2\n\nPropuesta previa:\n\n```text\n2 x HAProxy + Keepalived\n+\nprivate VIP\n```\n\nDebe cerrarse antes del deploy final.\n\n## 29.3 DSR validation\n\nDebe hacerse PoC real con capture.\n\n## 29.4 Traefik deployment exacto\n\nDefinir:\n\n- hostNetwork;\n- DaemonSet;\n- VIP handling;\n- health endpoint.\n\n## 29.5 Egress enforcement exacto\n\nDefinir implementaci\u00f3n inicial de:\n\n```text\nsite label\n \u2193\ngateway selection\n \u2193\npolicy routing\n```\n\n## 29.6 Infisical OSS/licensing\n\nValidar exactamente:\n\n- features necesarias en self-hosted;\n- machine identities;\n- Kubernetes integration;\n- audit;\n- versioning;\n- posibles l\u00edmites comerciales.\n\n## 29.7 GitOps layout\n\nDefinir estructura exacta Argo CD:\n\n- Application;\n- AppProject;\n- App of Apps o ApplicationSet.\n\n## 29.8 OIDC\n\nPendiente.\n\n## 29.9 Backup externo\n\nPendiente destino definitivo de:\n\n- etcd snapshots;\n- Infisical DB;\n- SOPS recovery material;\n- Git.\n\n## 29.10 Observabilidad central\n\nPendiente arquitectura final.\n\n---\n\n# 30. Criterios de aceptaci\u00f3n\n\nAntes de desplegar Hermes o workloads reales:\n\n## Cluster\n\n- [ ] p\u00e9rdida de 1 control plane no interrumpe API;\n- [ ] etcd mantiene quorum;\n- [ ] snapshot etcd funcional;\n- [ ] restore documentado y probado;\n- [ ] workers pueden drain/uncordon.\n\n## Networking\n\n- [ ] pod-to-pod;\n- [ ] pod-to-service;\n- [ ] DNS;\n- [ ] Hubble flows visibles;\n- [ ] no collision CIDRs;\n- [ ] nodes alcanzan Ceph.\n\n## Storage\n\n- [ ] PVC;\n- [ ] delete/recreate pod;\n- [ ] persistencia;\n- [ ] failure test b\u00e1sico.\n\n## GitOps\n\n- [ ] Argo CD reconciles desde Git;\n- [ ] manual drift detectable;\n- [ ] rollback verificable.\n\n## Vault\n\n- [ ] agente/tool puede crear secret;\n- [ ] servicio puede consumir secret;\n- [ ] secret no queda plaintext en Git;\n- [ ] access control validado;\n- [ ] backup de Infisical probado;\n- [ ] recovery bootstrap documentado.\n\n## Secrets\n\n- [ ] SOPS bootstrap funciona;\n- [ ] recovery age key offline;\n- [ ] Kubernetes secrets cifrados at rest;\n- [ ] namespaces/RBAC limitan acceso.\n\n## Agentes\n\n- [ ] Hermes puede usar tool privilegiada sin recibir raw token;\n- [ ] ADK readonly no puede leer secrets ajenos;\n- [ ] coding agent solo accede a vault paths autorizados.\n\n---\n\n# 31. Resumen final para implementaci\u00f3n\n\nLa platform foundation actual queda as\u00ed:\n\n```text\n GitHub\n wetopi-platform\n \u2502\n infrastructure\n \u2502\n Codex\n \u2502\n wetopi-ops01\n \u2502\n OpenTofu + Ansible\n \u2502\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 RKE2 HA \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n Cilium + Hubble\n \u2502\n Ceph CSI\n \u2502\n Argo CD\n \u2502\n SOPS bootstrap\n \u2502\n Infisical\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502 \u2502\n Kubernetes Tool runtimes Humans\n Secrets \u2502 Web UI\n \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502\n Hermes ADK\n```\n\nNetworking:\n\n```text\nINGRESS\n\nInternet\n \u2193\nVRRP\n \u2193\nIPVS\n \u2193\nDSR\n \u2193\nRKE2 Edge\n \u2193\nTraefik\n \u2193\nGateway API\n \u2193\nService\n \u2193\nWebsite\n```\n\n```text\nEGRESS\n\nSite / Customer\n \u2193\negress label/profile\n \u2193\ngw100 / gw101 / gw102\n \u2193\nWAN IP\n \u2193\nInternet\n```\n\nLa intenci\u00f3n global es mantener la primera versi\u00f3n simple:\n\n- no Istio;\n- no BGP obligatorio;\n- no controller avanzado de egress;\n- no OpenBao;\n- no service mesh complejo;\n- no redise\u00f1o simult\u00e1neo de Ceph.\n\nLa plataforma se estabiliza primero.\n\nDespu\u00e9s se despliegan:\n\n```text\nDocs\nMemory Plane\nWork Ledger\nAI Hub\nHermes\nADK\nEdge\nHosting\n```\n\ny finalmente se inicia la migraci\u00f3n progresiva de Swarm a RKE2.\n\n---\n\n# 32. PLATFORM-018 \u2014 CNI Cilium desde el d\u00eda uno (sin Canal provisional)\n\n**Fecha:** 29 de agosto de 2026\n**Estado:** Decidido (operador). Refina PLATFORM-007 (Cilium ser\u00e1 el CNI) y\nPLATFORM-008 (Hubble). Se graba como PLATFORM-018 \u2014 el siguiente identificador\nlibre \u2014 porque PLATFORM-008 ya estaba asignado a Hubble en el registro original.\n\nDecisi\u00f3n operativa para el primer addon del cluster RKE2:\n\n- No habr\u00e1 CNI provisional Canal. Cilium es el CNI desde el primer nodo y\n sustituye al Canal que RKE2 instala por defecto: la configuraci\u00f3n de\n rke2-server debe fijar `cni: none`, y el rol de RKE2 debe mantener\n deshabilitado el Canal incluido de serie.\n- Chart oficial Cilium fijado a la versi\u00f3n `1.20.1`\n (`oci://quay.io/cilium/charts/cilium`, namespace `kube-system`). Las\n versiones de chart no llevan prefijo `v`.\n- Encapsulaci\u00f3n VXLAN (`routingMode: tunnel`, `tunnelProtocol: vxlan`); sin\n BGP y sin native routing inicialmente, seg\u00fan el plan maestro (14.4) y\n PLATFORM-017.\n- IPAM `cluster-pool` con `clusterPoolIPv4PodCIDRList: [10.64.0.0/14]` y\n `clusterPoolIPv4MaskSize: 24`, coincidiendo exactamente con el pod CIDR de\n RKE2. RKE2 es quien posee el pod CIDR y el service CIDR (10.68.0.0/16);\n Cilium no los re-deriva y no toca el CIDR de servicios.\n- `kubeProxyReplacement: false` en v1: kube-proxy est\u00e1ndar convive con Cilium\n VXLAN. La sustituci\u00f3n de kube-proxy queda documentada como opci\u00f3n futura.\n- Hubble habilitado con Hubble Relay (`hubble.enabled: true`,\n `hubble.relay.enabled: true`); `hubble.ui.enabled: false` por defecto,\n activaci\u00f3n opcional documentada.\n- La MTU queda por defecto (auto ~1450 sobre el LAN de nodos MTU 1500).\n- Puertos obligatorios entre nodos: UDP 8472 (VXLAN) y TCP 4244 (Hubble\n Relay).\n\nM\u00e9todo de instalaci\u00f3n: Argo CD no existe todav\u00eda en el primer bootstrap, as\u00ed\nque el addon se instala desde `wetopi-ops01` con Helm durante la fase de\napply (playbook `ansible/playbooks/cilium.yml` en el repo infrastructure).\nEs una instalaci\u00f3n solo de bootstrap: cuando Argo CD exista, la gesti\u00f3n del\naddon pasa a \u00e9l y el playbook se retira. Los valores pinneados viven en\n`kubernetes/cilium/values.yaml`; la lista de verificaci\u00f3n de post-instalaci\u00f3n\nen `kubernetes/cilium/VALIDATION.md`.\n\nCIDRs: el ownership es de RKE2 (pod CIDR 10.64.0.0/14 y service CIDR\n10.68.0.0/16); el pool de Cilium se limita a consumir el rango de pods y\nnunca debe diverger de la configuraci\u00f3n de rke2-server.", "html": "\n\nMigrado desde
\nwetopi-platform/docs@main(design/decisions-log.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
Fecha: 28 de agosto de 2026
\nEstado: Decisiones consolidadas para implementaci\u00f3n
\nOrganizaci\u00f3n GitHub: wetopi-platform
\nAudiencia principal: Codex / agentes de implementaci\u00f3n / operadores SRE
\nObjetivo: preservar el contexto t\u00e9cnico y ejecutivo de las \u00faltimas decisiones para continuar el proyecto sin reabrir debates ya cerrados.
wetopi-ops01Wetopi est\u00e1 reconstruyendo su plataforma de Managed WordPress Hosting desde Docker Swarm hacia una nueva base RKE2 sobre su cloud privado Proxmox + Ceph.
\nLa nueva plataforma debe ser AI-first y operable tanto por humanos como por agentes.
\nLas decisiones ya cerradas son:
\nKubernetes distribution:\nRKE2\n\nCNI:\nCilium\n\nNetwork observability:\nHubble\n\nGitOps:\nArgo CD\n\nVault / secrets manager:\nInfisical self-hosted\n\nBootstrap secret encryption:\nSOPS + age\n\nInfrastructure as Code:\nOpenTofu + Ansible\n\nIngress WAN:\nmantener IPVS + VRRP + DSR inicialmente\n\nProxy edge inicial:\nTraefik\n\nHTTP routing:\nGateway API\n\nEgress:\nNATGW Ubuntu simples con gateway interno estable y WAN IP cambiable\nEl cluster inicial se construir\u00e1 greenfield, en paralelo al Docker Swarm actual. No habr\u00e1 big-bang.
\nLa plataforma se levantar\u00e1 primero, se validar\u00e1, y despu\u00e9s se migrar\u00e1n edge nodes, hosting workers y sites de forma progresiva.
\nWetopi es un Managed WordPress Hosting ejecutado actualmente sobre:
\nOVH\n \u2193\nvRack\n \u2193\nProxmox\n \u2193\nCeph\n \u2193\nDocker Swarm\nEl workload principal son wepod, aproximadamente un contenedor por website, con:
s6\nnginx\nphp-fpm\nmysql\nEl tr\u00e1fico WAN actual utiliza:
\nVIP\n \u2193\nVRRP / Keepalived\n \u2193\nIPVS\n \u2193\nDSR\n \u2193\nWAN nginx proxies\n \u2193\nSwarm workloads\nEl descubrimiento de backends se realiza mediante:
\nRegistrator\n \u2193\nConsul\n \u2193\nconsul-template\n \u2193\nnginx.conf\nLa nueva plataforma debe preservar las propiedades valiosas de esta arquitectura, especialmente DSR y la distribuci\u00f3n del ancho de banda de salida entre distintos hosts f\u00edsicos de OVH.
\nRKE2 ser\u00e1 la distribuci\u00f3n Kubernetes de producci\u00f3n.
\nEl cluster nuevo se construir\u00e1 en paralelo a Swarm.
\nEl control plane inicial tendr\u00e1 tres nodos RKE2 server con embedded etcd.
\nInicialmente habr\u00e1 tres workers dedicados a servicios de plataforma.
\nKubernetes se construye desde fuera del cluster.
\nwetopi-ops01Existir\u00e1 una VM externa de operaciones, bootstrap y recovery.
\nCilium ser\u00e1 el CNI.
\nHubble ser\u00e1 la base de network observability.
\nArgo CD ser\u00e1 el sistema GitOps.
\nInfisical self-hosted ser\u00e1 el Vault operativo de Wetopi.
\nSOPS + age se utilizar\u00e1 para bootstrap secrets, no como Vault principal.
\nSe preserva IPVS + VRRP + DSR inicialmente.
\nTraefik ser\u00e1 el proxy edge inicial.
\nEl routing HTTP se expresar\u00e1 con Gateway API.
\nSe usar\u00e1n NATGW Ubuntu simples con gateway interno estable y WAN IP cambiable.
\nNo se desplegar\u00e1 service mesh sin necesidad concreta.
\nLa plataforma no depender\u00e1 de BGP para arrancar.
\nSe crear\u00e1n inicialmente:
\n1 x wetopi-ops01\n3 x RKE2 control plane\n3 x platform workers\nTotal inicial:
\n7 VMs\nNombres previstos:
\nwetopi-ops01\n\nrke-cp01\nrke-cp02\nrke-cp03\n\nrke-platform01\nrke-platform02\nrke-platform03\nLos control planes y workers deben distribuirse entre distintos hosts Proxmox.
\nEjemplo:
\npve01 -> rke-cp01 + rke-platform01\npve02 -> rke-cp02 + rke-platform02\npve03 -> rke-cp03 + rke-platform03\nLa distribuci\u00f3n real deber\u00e1 validarse con el inventario.
\nwetopi-ops01La regla es:
\n\n\nKubernetes no debe depender de servicios internos de Kubernetes para poder crearse o repararse.
\n
Modelo:
\nLaptop\n \u2502\nCodex\n \u2502\nGitHub\n \u2502\nwetopi-ops01\n \u2502\nOpenTofu + Ansible\n \u2502\nProxmox\n \u2502\nRKE2\nCodex trabajar\u00e1 inicialmente desde el laptop del operador.
\nwetopi-ops01 ser\u00e1 la m\u00e1quina durable de bootstrap y disaster recovery.
Debe tener, como m\u00ednimo:
\ngit\nssh\nansible\nopentofu\nkubectl\nhelm\njq\nyq\nNo alojar\u00e1 workloads productivos.
\nOrganizaci\u00f3n:
\nwetopi-platform\nRepos iniciales:
\nwetopi-platform/infrastructure\nwetopi-platform/agent-services\nwetopi-platform/ai-hub\nwetopi-platform/agents\nwetopi-platform/docs\nNo crear repos adicionales sin necesidad real.
\nSe selecciona:
\nOpenTofu\n+\nAnsible\nOpenTofu:
\nAnsible:
\nGit es source of truth.
\nNo se considera v\u00e1lido dejar cambios permanentes solo en shell history.
\nCada VM tendr\u00e1 inicialmente:
\n1 NIC LAN\nModelo:
\nVM\n \u2514\u2500\u2500 eth0\n \u2514\u2500\u2500 LAN / vRack\nNo se introducir\u00e1n m\u00faltiples interfaces salvo necesidad demostrada.
\nLa red Ceph actual es:
\n192.168.96.0/20\nEs parte de la infraestructura hiperconvergida Proxmox + Ceph.
\nDecisi\u00f3n:
\n\n\nNO modificar, renumerar ni redise\u00f1ar esta red.
\n
Ceph se consumir\u00e1 desde Kubernetes mediante CSI.
\nLa conectividad exacta entre RKE2 nodes y Ceph debe comprobarse, no asumirse.
\nPropuesta actual:
\nNode LAN:\npendiente de inventario real\n\nPod CIDR:\n10.64.0.0/14\n\nService CIDR:\n10.68.0.0/16\n\nCeph:\n192.168.96.0/20\nEXISTENTE / NO TOCAR\nLos CIDRs definitivos deben validarse contra todas las redes Wetopi existentes.
\nCilium ser\u00e1 el CNI base.
\nPrimera versi\u00f3n:
\nCilium\n+\nVXLAN encapsulation\n+\nHubble\nRaz\u00f3n de VXLAN inicial:
\nNo activar inicialmente:
\nHubble se utilizar\u00e1 para flows y correlaci\u00f3n de tr\u00e1fico con pods/namespaces/sites.
\nEl frontend actual se conserva inicialmente.
\nInternet\n \u2193\nVIP\n \u2193\nVRRP\n \u2193\nIPVS\n \u2193\nDSR\n \u2193\nRKE2 Edge\nMotivo principal:
\nDSR permite que cada edge node responda directamente a Internet usando el uplink/bandwidth domain del host f\u00edsico Proxmox/OVH donde corre.
\nEsto permite agregar capacidad de salida entre varios hosts f\u00edsicos.
\nEsta propiedad no debe perderse durante la migraci\u00f3n.
\nLos antiguos WAN proxies evolucionar\u00e1n a RKE2 edge workers.
\nEjemplo:
\nrke-edge-a01\nrke-edge-a02\nrke-edge-b01\nrke-edge-b02\nLabels:
\nwetopi.io/role=edge\nwetopi.io/wan-zone=a\nwetopi.io/proxmox-host=pve01\nwetopi.io/bandwidth-domain=pve01\nTaint:
\nwetopi.io/edge=true:NoSchedule\nEl proxy se desplegar\u00e1 como DaemonSet cuando tenga sentido mantener una instancia por edge node.
\nSe elige Traefik inicialmente.
\nRazones:
\nModelo:
\nhostname\n \u2193\nHTTPRoute\n \u2193\nService\n \u2193\nEndpointSlice\n \u2193\nPod\nEnvoy podr\u00e1 reevaluarse m\u00e1s adelante si aparecen necesidades L7 m\u00e1s avanzadas.
\nAntes:
\nDocker\n \u2193\nRegistrator\n \u2193\nConsul\n \u2193\nconsul-template\n \u2193\nnginx.conf\n \u2193\nreload\nDespu\u00e9s:
\nPod\n \u2193\nService\n \u2193\nEndpointSlice\n \u2193\nGateway controller\n \u2193\ndynamic routing\nKubernetes ser\u00e1 el service registry del hosting.
\nNo se construir\u00e1 inicialmente un egress control plane complejo.
\nModelo base:
\ngw100\nLAN: 192.168.88.100\nWAN: PUBLIC_IP_A\n\ngw101\nLAN: 192.168.88.101\nWAN: PUBLIC_IP_B\n\ngw102\nLAN: 192.168.88.102\nWAN: PUBLIC_IP_C\nCada gateway:
\nUbuntu\n+\nrouting\n+\nnftables/iptables\n+\nSNAT\nObjetivo:
\nModelo l\u00f3gico:
\nsite/customer\n \u2193\negress_gateway\n \u2193\ngw100 / gw101 / gw102\nEjemplo:
\nwetopi.io/egress-gateway: gw101\nLa implementaci\u00f3n inicial puede usar:
\nNo construir todav\u00eda un controlador complejo.
\nFuturo:
\nCiliumEgressGatewayPolicy\npuede sustituir el mecanismo sin cambiar el modelo l\u00f3gico.
\nSeparar desde el inicio:
\nhosting-general\nhosting-premium\nplatform\nagents/crawlers\nLos crawlers/agentes no deben compartir IP de salida con customers.
\nSe ha decidido usar:
\nArgo CD\nRazones principales:
\nModelo:
\nGit\n \u2193\nArgo CD\n \u2193\nKubernetes\nNo se utilizar\u00e1 un pipeline CI basado en kubectl apply como mecanismo normal de producci\u00f3n.
Pendiente decidir el layout exacto:
\nEl Vault operativo de Wetopi ser\u00e1:
\nInfisical self-hosted\nEl requisito real no es solamente almacenar secrets declarativos.
\nSe necesita un servicio vivo donde humanos y agentes puedan:
\nEjemplo conceptual:
\nvault.put(\n path=\"/platform/proxmox/api\",\n username=\"automation\",\n password=\"...\"\n)\nInfisical ser\u00e1 el source of truth operativo de credentials.
\nSOPS NO ser\u00e1 el Vault principal.
\nSe utilizar\u00e1 \u00fanicamente para bootstrap.
\nModelo:
\nGit\n \u2193\nSOPS + age\n \u2193\nbootstrap secrets\n \u2193\nInfisical starts\nSOPS protege los secretos necesarios para levantar:
\nUna vez Infisical funciona, los nuevos credentials se guardan en Infisical, no en Git.
\nDebe existir al menos una recovery key age fuera del cluster y fuera de Git.
Modelo:
\nInfisical\n \u2502\n \u2502 sync/operator\n \u25bc\nKubernetes Secret\n \u2502\n \u25bc\nDeployment / Pod\nInfisical es source of truth.
\nKubernetes Secret es materializaci\u00f3n runtime.
\nEsto permite:
\nLa regla principal es:
\n\n\nel agente debe poder USAR un secret sin necesidad de VER su valor cuando sea posible.
\n
Ejemplo:
\nHermes\n \u2502\n \u2502 tool call\n \u25bc\nplatform-tools\n \u2502\n \u2502 reads secret from Infisical/K8s Secret\n \u25bc\nProxmox / Cloudflare / OVH\nEl LLM no necesita recibir el token.
\nNo montar todos los secrets directamente en el proceso LLM.
\nAgrupar tools por security domain.
\nEjemplo:
\nplatform-tools\n\u251c\u2500\u2500 kubernetes\n\u251c\u2500\u2500 proxmox\n\u251c\u2500\u2500 ceph\n\u2514\u2500\u2500 networking\n\nwordpress-tools\n\u251c\u2500\u2500 wordpress\n\u251c\u2500\u2500 mysql\n\u2514\u2500\u2500 backups\n\nmarketing-tools\n\u251c\u2500\u2500 intercom\n\u251c\u2500\u2500 analytics\n\u2514\u2500\u2500 ...\nCada runtime recibe \u00fanicamente los secrets necesarios.
\nNo crear un sidecar por tool individual.
\nEn ai-hub, las tools deber\u00edan declarar sus requirements.
Ejemplo:
\ntool: proxmox.vm_restart\nruntime: platform-tools\nsecrets:\n - proxmox-api-token\nLos agentes deben consumir un contrato propio Wetopi, no acoplarse directamente a Infisical.
\nAPI conceptual:
\nvault.get()\nvault.put()\nvault.update()\nvault.delete()\nvault.list()\nvault.exists()\nHermes personales/coding agents necesitar\u00e1n operar la plataforma.
\nPodr\u00e1n:
\nPero:
\nEl permiso debe seguir el rol.
\nEjemplo conceptual:
\nHermes SRE\n /platform/*\n /infrastructure/*\n /agents/hermes-sre/*\n\nHermes Developer\n /development/*\n /github/*\n /agents/hermes-developer/*\n\nHermes Marketing\n /marketing/*\n /agents/hermes-marketing/*\nPara operar Kubernetes, los agentes dentro del cluster usar\u00e1n:
\nServiceAccount\n+\nRBAC\nNo kubeconfig est\u00e1tico ni password.
\nEjemplo:
\nHermes SRE\n -> permisos operacionales amplios\n\nHermes Developer\n -> namespaces dev + deployments\n\nADK observer\n -> readonly\n\nADK specialist\n -> permissions espec\u00edficas\nNamespaces orientativos:
\nwetopi-platform\nwetopi-edge\nwetopi-hosting\nwetopi-agents-sre\nwetopi-agents-development\nwetopi-agents-marketing\nwetopi-observability\nwetopi-security\nLos namespaces forman parte de la frontera de secrets y RBAC.
\nImportante:
\nquien puede crear Pods en un namespace puede potencialmente montar Secrets de ese namespace.
\nPor tanto, permisos de workload creation y secret access deben dise\u00f1arse conjuntamente.
\nSOPS protege Git.
\nInfisical protege el Vault.
\nKubernetes Secrets materializados deben protegerse en etcd.
\nRKE2 deber\u00e1 tener:
\nSecrets Encryption at Rest\nhabilitado.
\nModelo:
\nInfisical\n \u2193\nKubernetes Secret\n \u2193\nRKE2 API server\n \u2193\nencrypted etcd storage\nOrden actual recomendado:
\n1. GitHub organization + repos\n2. Inventario Proxmox/redes\n3. OpenTofu + Ansible\n4. wetopi-ops01\n5. 3 x RKE2 control plane\n6. 3 x platform workers\n7. RKE2 HA + etcd\n8. Cilium\n9. Hubble\n10. Ceph CSI\n11. Argo CD\n12. SOPS + age bootstrap\n13. Infisical\n14. Kubernetes Secrets integration/operator\n15. Docs\n16. Memory Plane\n17. Work Ledger\n18. AI Hub\n19. Hermes\n20. ADK Fleet\n21. Edge nodes\n22. Gateway API + Traefik\n23. NATGW\n24. first hosting workers\n25. first websites\n26. progressive migration from Swarm\nCrear:
\ninfrastructure\nagent-services\nai-hub\nagents\ndocs\nAplicar:
\nRecoger:
\nNo crear a\u00fan direccionamiento inventado.
\nConfirmar que:
\nPod CIDR 10.64.0.0/14\nService CIDR 10.68.0.0/16\nno colisionan.
\nConfirmar conectividad RKE2 node -> Ceph 192.168.96.0/20.
Crear template reproducible Ubuntu 24.04 + cloud-init.
\nwetopi-ops01Crear/provisionar wetopi-ops01.
Crear:
\nrke-cp01\nrke-cp02\nrke-cp03\nCrear:
\nrke-platform01\nrke-platform02\nrke-platform03\nConfigurar:
\nDesplegar:
\nValidar:
\nDesplegar declarativamente.
\nGenerar:
\n.sops.yaml.Guardar recovery key fuera del cluster y fuera de Git.
\nDesplegar self-hosted.
\nConfigurar:
\nCrear en ai-hub una capa estable:
vault.get()\nvault.put()\nvault.update()\nvault.delete()\nvault.list()\nvault.exists()\nLos agentes no deben quedar acoplados a la API concreta de Infisical.
\nPendiente de inventario.
\nPropuesta previa:
\n2 x HAProxy + Keepalived\n+\nprivate VIP\nDebe cerrarse antes del deploy final.
\nDebe hacerse PoC real con capture.
\nDefinir:
\nDefinir implementaci\u00f3n inicial de:
\nsite label\n \u2193\ngateway selection\n \u2193\npolicy routing\nValidar exactamente:
\nDefinir estructura exacta Argo CD:
\nPendiente.
\nPendiente destino definitivo de:
\nPendiente arquitectura final.
\nAntes de desplegar Hermes o workloads reales:
\nLa platform foundation actual queda as\u00ed:
\n GitHub\n wetopi-platform\n \u2502\n infrastructure\n \u2502\n Codex\n \u2502\n wetopi-ops01\n \u2502\n OpenTofu + Ansible\n \u2502\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 RKE2 HA \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n Cilium + Hubble\n \u2502\n Ceph CSI\n \u2502\n Argo CD\n \u2502\n SOPS bootstrap\n \u2502\n Infisical\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502 \u2502\n Kubernetes Tool runtimes Humans\n Secrets \u2502 Web UI\n \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502\n Hermes ADK\nNetworking:
\nINGRESS\n\nInternet\n \u2193\nVRRP\n \u2193\nIPVS\n \u2193\nDSR\n \u2193\nRKE2 Edge\n \u2193\nTraefik\n \u2193\nGateway API\n \u2193\nService\n \u2193\nWebsite\nEGRESS\n\nSite / Customer\n \u2193\negress label/profile\n \u2193\ngw100 / gw101 / gw102\n \u2193\nWAN IP\n \u2193\nInternet\nLa intenci\u00f3n global es mantener la primera versi\u00f3n simple:
\nLa plataforma se estabiliza primero.
\nDespu\u00e9s se despliegan:
\nDocs\nMemory Plane\nWork Ledger\nAI Hub\nHermes\nADK\nEdge\nHosting\ny finalmente se inicia la migraci\u00f3n progresiva de Swarm a RKE2.
\nFecha: 29 de agosto de 2026\nEstado: Decidido (operador). Refina PLATFORM-007 (Cilium ser\u00e1 el CNI) y\nPLATFORM-008 (Hubble). Se graba como PLATFORM-018 \u2014 el siguiente identificador\nlibre \u2014 porque PLATFORM-008 ya estaba asignado a Hubble en el registro original.
\nDecisi\u00f3n operativa para el primer addon del cluster RKE2:
\ncni: none, y el rol de RKE2 debe mantener\ndeshabilitado el Canal incluido de serie.1.20.1\n(oci://quay.io/cilium/charts/cilium, namespace kube-system). Las\nversiones de chart no llevan prefijo v.routingMode: tunnel, tunnelProtocol: vxlan); sin\nBGP y sin native routing inicialmente, seg\u00fan el plan maestro (14.4) y\nPLATFORM-017.cluster-pool con clusterPoolIPv4PodCIDRList: [10.64.0.0/14] y\nclusterPoolIPv4MaskSize: 24, coincidiendo exactamente con el pod CIDR de\nRKE2. RKE2 es quien posee el pod CIDR y el service CIDR (10.68.0.0/16);\nCilium no los re-deriva y no toca el CIDR de servicios.kubeProxyReplacement: false en v1: kube-proxy est\u00e1ndar convive con Cilium\nVXLAN. La sustituci\u00f3n de kube-proxy queda documentada como opci\u00f3n futura.hubble.enabled: true,\nhubble.relay.enabled: true); hubble.ui.enabled: false por defecto,\nactivaci\u00f3n opcional documentada.M\u00e9todo de instalaci\u00f3n: Argo CD no existe todav\u00eda en el primer bootstrap, as\u00ed\nque el addon se instala desde wetopi-ops01 con Helm durante la fase de\napply (playbook ansible/playbooks/cilium.yml en el repo infrastructure).\nEs una instalaci\u00f3n solo de bootstrap: cuando Argo CD exista, la gesti\u00f3n del\naddon pasa a \u00e9l y el playbook se retira. Los valores pinneados viven en\nkubernetes/cilium/values.yaml; la lista de verificaci\u00f3n de post-instalaci\u00f3n\nen kubernetes/cilium/VALIDATION.md.
CIDRs: el ownership es de RKE2 (pod CIDR 10.64.0.0/14 y service CIDR\n10.68.0.0/16); el pool de Cilium se limita a consumir el rango de pods y\nnunca debe diverger de la configuraci\u00f3n de rke2-server.
\n" }, { "id": 57, "title": "master architecture plan", "slug": "master-architecture-plan", "book_id": 8, "chapter_id": 29, "description": null, "created_at": "2026-09-06T13:39:02.000000Z", "updated_at": "2026-09-06T13:39:02.000000Z", "markdown": "> Migrado desde `wetopi-platform/docs@main` (design/master-architecture-plan.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.\n\n# Wetopi Platform \u2014 Master Architecture, Operating Model & Implementation Plan\n## Memoria ejecutiva, arquitectura t\u00e9cnica general y plan de construcci\u00f3n\n\n**Fecha:** 28 de agosto de 2026 \n**Estado:** Documento maestro para revisi\u00f3n e implementaci\u00f3n \n**Organizaci\u00f3n GitHub:** `wetopi-platform` \n**Audiencia:** Platform Engineering, SRE, DevOps, Development, Security, Network Engineering y agentes Codex/Hermes/ADK \n**Prop\u00f3sito:** servir como documento principal para explicar, revisar, construir y operar la nueva plataforma Wetopi.\n\n---\n\n# 0. C\u00f3mo usar este documento\n\nEste documento consolida y sustituye como referencia principal a las memorias parciales creadas durante el dise\u00f1o inicial:\n\n- `wetopi-ai-first-kubernetes-project-memory.md`\n- `wetopi-platform-bootstrap-plan.md`\n- `wetopi-rke2-network-hosting-architecture.md`\n- `wetopi-platform-decisions-argocd-infisical-rke2.md`\n\nCuando exista una contradicci\u00f3n con documentos anteriores, **prevalece este documento**, especialmente en decisiones posteriores como:\n\n- **Argo CD** como GitOps.\n- **Infisical self-hosted** como Wetopi Vault.\n- **SOPS + age** \u00fanicamente como bootstrap/recovery de secretos declarativos.\n- **OpenBao queda fuera de v1**.\n\nNo todas las decisiones est\u00e1n cerradas. Las cuestiones a\u00fan abiertas se marcan expl\u00edcitamente como `OPEN` y **no deben ser inventadas por Codex ni por un ingeniero durante la implementaci\u00f3n**.\n\n---\n\n# 1. Resumen ejecutivo\n\nWetopi va a reconstruir su plataforma de Managed WordPress Hosting sobre una nueva foundation basada en **RKE2**, desplegada en su cloud privado sobre **Proxmox + Ceph + OVH vRack**.\n\nLa migraci\u00f3n ser\u00e1 **greenfield y progresiva**. Docker Swarm seguir\u00e1 dando servicio mientras se construye, prueba y estabiliza la nueva plataforma.\n\nLa arquitectura inicial queda resumida en:\n\n```text\nProxmox + Ceph + OVH vRack\n \u2502\n \u25bc\n RKE2\n \u2502\n Cilium + Hubble\n \u2502\n Ceph CSI\n \u2502\n Argo CD\n \u2502\n SOPS bootstrap\n \u2502\n Infisical\n \u2502\n Docs / Memory / Work Coordination\n \u2502\n Hermes + ADK + Codex workflows\n \u2502\n Edge RKE2 + Traefik + Gateway API\n \u2502\n Hosting workers + WordPress sites\n```\n\nLa entrada WAN conservar\u00e1 inicialmente la arquitectura probada de Wetopi:\n\n```text\nInternet\n \u2193\nVRRP VIP\n \u2193\nIPVS\n \u2193\nDSR\n \u2193\nRKE2 Edge nodes\n \u2193\nTraefik\n \u2193\nGateway API\n \u2193\nServices / EndpointSlices\n \u2193\nWordPress workloads\n```\n\nLa salida ser\u00e1 deliberadamente sencilla:\n\n```text\nsite/customer/workload\n \u2193\nselected internal gateway\n \u2193\n192.168.88.100 / .101 / .102 / ...\n \u2193\nUbuntu NATGW\n \u2193\nselected WAN public IP\n \u2193\nInternet\n```\n\nLa plataforma ser\u00e1 **AI-first**, pero no se permitir\u00e1 que la autonom\u00eda de los agentes degrade la operabilidad o seguridad. Git, Vault, Work Ledger, Memory Plane, documentaci\u00f3n y tools compartidas ser\u00e1n servicios de primera clase tanto para humanos como para agentes.\n\n---\n\n# 2. Objetivos de negocio y plataforma\n\n## 2.1 Objetivos de negocio\n\nLa nueva plataforma debe permitir a Wetopi:\n\n- migrar desde Docker Swarm sin big-bang;\n- operar cientos y posteriormente miles de workloads WordPress;\n- mantener la eficiencia y capacidad WAN actual;\n- reducir configuraci\u00f3n artesanal y componentes legacy;\n- ofrecer mejor aislamiento por cliente;\n- permitir IPs de salida dedicadas o f\u00e1cilmente rotables;\n- mejorar observabilidad y seguridad;\n- incorporar automatizaci\u00f3n y agentes de forma nativa;\n- mantener un modelo operativo entendible por ingenieros humanos.\n\n## 2.2 Objetivos t\u00e9cnicos\n\nLa plataforma debe proporcionar:\n\n- Kubernetes HA;\n- IaC reproducible;\n- GitOps declarativo;\n- CNI moderno;\n- observabilidad de flujos;\n- CSI para Ceph;\n- Vault operativo con API;\n- secrets materializados en Kubernetes;\n- documentaci\u00f3n Markdown versionada;\n- memoria institucional com\u00fan;\n- coordinaci\u00f3n de trabajo entre agentes y humanos;\n- trazabilidad;\n- separaci\u00f3n clara de security domains;\n- recuperaci\u00f3n desde p\u00e9rdida parcial o total del cluster.\n\n---\n\n# 3. Principios de dise\u00f1o\n\n## 3.1 Greenfield antes que migraci\u00f3n in-place\n\nLa nueva plataforma se construye en paralelo al Swarm existente.\n\n```text\nPRODUCCI\u00d3N ACTUAL NUEVA PLATAFORMA\nDocker Swarm RKE2 greenfield\n \u2502 \u2502\n \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 coexistencia \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\nNo se reutilizan nodos de producci\u00f3n hasta que RKE2 haya demostrado estabilidad.\n\n## 3.2 Git es source of truth del estado declarativo\n\n```text\nGit\n \u2193\nreview\n \u2193\nArgo CD / IaC\n \u2193\nreal infrastructure\n```\n\nLa shell history no es source of truth.\n\n## 3.3 La fuente viva gana a la memoria\n\nPara infraestructura real:\n\n```text\nAPI autoritativa / sistema vivo\n>\nFleet Memory\n>\nDocs hist\u00f3ricas\n>\nLLM inference\n```\n\n## 3.4 Simplicidad antes que sofisticaci\u00f3n\n\nNo desplegar v1 con:\n\n- Istio;\n- BGP WAN obligatorio;\n- service mesh global;\n- Cilium egress controller complejo;\n- Suricata inline global;\n- WAF distribuido avanzado;\n- PKI din\u00e1mica compleja;\n- automatizaci\u00f3n de reputaci\u00f3n completa.\n\nLa arquitectura debe permitir a\u00f1adirlo despu\u00e9s.\n\n## 3.5 Human-compatible, agent-native\n\nTodos los servicios importantes deben disponer de:\n\n- API/CLI/Git para agentes;\n- UI cuando aporte valor a humanos;\n- identidad y permisos;\n- auditabilidad.\n\n## 3.6 Least privilege\n\nCada workload, tool runtime y agente recibe \u00fanicamente los permisos necesarios.\n\n## 3.7 Secrets usados, no necesariamente vistos\n\nCuando sea posible:\n\n```text\nLLM\n \u2193 tool call\nTool Runtime\n \u2193 secret\nExternal API\n```\n\nEl modelo no necesita recibir el valor del secret.\n\n---\n\n# 4. Arquitectura general de componentes\n\n```mermaid\nflowchart TB\n Internet((Internet))\n OVH[OVH / vRack]\n PVE[Proxmox Cluster]\n Ceph[(Ceph)]\n\n subgraph RKE2[Wetopi RKE2 Production]\n CP[3x RKE2 Control Plane\\nembedded etcd]\n Platform[Platform Workers]\n Edge[Edge Workers]\n Hosting[Hosting Workers]\n Agents[Agent Workers / Platform Workers]\n\n Cilium[Cilium CNI]\n Hubble[Hubble]\n Argo[Argo CD]\n Vault[Infisical Vault]\n Docs[MkDocs / Docs]\n Memory[Memory Plane]\n Work[Work Coordination Plane]\n Hermes[Hermes Fleet]\n ADK[ADK Fleet]\n Traefik[Traefik / Gateway API]\n CSI[Ceph CSI]\n end\n\n Internet --> OVH\n OVH --> PVE\n PVE --> RKE2\n PVE --- Ceph\n CSI --> Ceph\n\n CP --- Cilium\n Platform --- Cilium\n Edge --- Cilium\n Hosting --- Cilium\n Agents --- Cilium\n Cilium --> Hubble\n\n Argo --> Platform\n Vault --> Platform\n Docs --> Hermes\n Memory --> Hermes\n Memory --> ADK\n Work --> Hermes\n Work --> ADK\n Hermes --> Agents\n ADK --> Agents\n Traefik --> Hosting\n```\n\n---\n\n# 5. Operating Model: todos los \u201cOps\u201d de Wetopi\n\nLa plataforma se divide en dominios operativos. Cada dominio tiene un prop\u00f3sito, source of truth y responsabilidades.\n\n| \u00c1rea | Responsabilidad principal | Source of truth principal |\n|---|---|---|\n| BootstrapOps | crear/reparar la foundation | Git + `wetopi-ops01` |\n| InfraOps | VMs, Proxmox, RKE2 nodes | OpenTofu + Ansible |\n| GitOps | estado declarativo Kubernetes | Git + Argo CD |\n| NetOps | CNI, routing, CIDR, policies | Git + APIs de red |\n| EdgeOps | IPVS/DSR, proxies WAN, Gateway API | Git + IPVS state |\n| EgressOps | NATGW, rutas, WAN IPs | Git/config + gateway state |\n| StorageOps | Ceph CSI/PVC/storage | Ceph + Kubernetes |\n| VaultOps / SecretOps | credentials y secrets | Infisical |\n| SecOps | RBAC, policies, security controls | Git + audit logs |\n| ObservabilityOps | m\u00e9tricas, logs, traces, flows | observability stack |\n| DevOps | CI/build/release/app lifecycle | Git + CI + Argo CD |\n| AgentOps | runtimes Hermes/ADK y capability bundles | `agents` + `ai-hub` |\n| MemoryOps | memoria institucional | Memory Plane |\n| WorkOps | coordinaci\u00f3n, tickets, leases | Work Ledger |\n| DocOps | docs/runbooks/ADR | Markdown Git |\n| BackupOps | backups y restore | backup repositories |\n| RecoveryOps | cluster/Vault/etcd disaster recovery | runbooks + backups |\n| HostingOps | WordPress/customer workloads | Kubernetes + business control plane |\n\n---\n\n# 6. BootstrapOps\n\n## 6.1 Prop\u00f3sito\n\nBootstrapOps existe para que la plataforma pueda construirse y repararse **sin depender de la propia plataforma**.\n\n## 6.2 Componentes\n\n```mermaid\nflowchart LR\n Laptop[Engineer Laptop\\nCodex]\n GH[GitHub\\nwetopi-platform]\n Ops[wetopi-ops01]\n Tofu[OpenTofu]\n Ansible[Ansible]\n PVE[Proxmox API]\n RKE[RKE2]\n\n Laptop --> GH\n Laptop --> Ops\n GH --> Ops\n Ops --> Tofu\n Ops --> Ansible\n Tofu --> PVE\n Ansible --> RKE\n```\n\n## 6.3 `wetopi-ops01`\n\nVM externa al cluster Kubernetes.\n\nResponsabilidades:\n\n- bootstrap;\n- bastion operativo;\n- OpenTofu;\n- Ansible;\n- kubectl/helm;\n- herramientas de recovery;\n- acceso a backups;\n- reparaci\u00f3n cuando GitOps/Cilium/Vault est\u00e9n ca\u00eddos.\n\nNo ejecuta workloads productivos.\n\nSizing inicial orientativo:\n\n```text\n4 vCPU\n8 GB RAM\n40\u201380 GB disk\nUbuntu 24.04\n1 NIC LAN\n```\n\n---\n\n# 7. InfraOps\n\n## 7.1 RKE2 inicial\n\n```text\n3 x control-plane\n3 x platform worker\n1 x ops VM externa\n```\n\nNombres:\n\n```text\nwetopi-ops01\n\nrke-cp01\nrke-cp02\nrke-cp03\n\nrke-platform01\nrke-platform02\nrke-platform03\n```\n\n## 7.2 Distribuci\u00f3n f\u00edsica\n\nIdeal:\n\n```text\npve01 -> rke-cp01 + rke-platform01\npve02 -> rke-cp02 + rke-platform02\npve03 -> rke-cp03 + rke-platform03\n```\n\nDebe respetarse failure domain.\n\n## 7.3 Control plane\n\nRKE2 con:\n\n- embedded etcd;\n- quorum 2/3;\n- API endpoint estable;\n- snapshots;\n- restore probado;\n- sin customer workloads.\n\nSizing orientativo:\n\n```text\n8 vCPU\n16 GB RAM\n50\u2013100 GB disk\n```\n\nEl almacenamiento de etcd debe priorizar baja latencia.\n\n## 7.4 IaC\n\n```text\nOpenTofu -> Proxmox resources / VMs\nAnsible -> OS / RKE2 / host configuration\nArgo CD -> Kubernetes workloads\n```\n\nNo mezclar responsabilidades.\n\n---\n\n# 8. GitHub y repositorios\n\nOrganizaci\u00f3n:\n\n```text\nwetopi-platform\n```\n\nRepositorios iniciales:\n\n```text\nwetopi-platform/\n\u251c\u2500\u2500 infrastructure\n\u251c\u2500\u2500 agent-services\n\u251c\u2500\u2500 ai-hub\n\u251c\u2500\u2500 agents\n\u2514\u2500\u2500 docs\n```\n\n## 8.1 `infrastructure`\n\n```text\ninfrastructure/\n\u251c\u2500\u2500 tofu/\n\u2502 \u2514\u2500\u2500 proxmox/\n\u251c\u2500\u2500 ansible/\n\u2502 \u251c\u2500\u2500 inventories/\n\u2502 \u251c\u2500\u2500 roles/\n\u2502 \u2514\u2500\u2500 playbooks/\n\u251c\u2500\u2500 clusters/\n\u2502 \u2514\u2500\u2500 production/\n\u251c\u2500\u2500 apps/\n\u251c\u2500\u2500 platform/\n\u251c\u2500\u2500 networking/\n\u251c\u2500\u2500 storage/\n\u251c\u2500\u2500 security/\n\u251c\u2500\u2500 observability/\n\u251c\u2500\u2500 bootstrap/\n\u2514\u2500\u2500 docs/\n```\n\n## 8.2 `agent-services`\n\nServicios compartidos:\n\n- Memory Plane;\n- Work Ledger adapter;\n- Hermes Memory Provider;\n- identidad/approval cuando sea necesario.\n\n## 8.3 `ai-hub`\n\n```text\nskills/\ntools/\ncomponents/\ncontracts/\nschemas/\nsdk/\nevals/\ntests/\n```\n\nRegla:\n\n> Una tool utilizada por m\u00e1s de un agente deja de pertenecer al agente y pasa a `ai-hub`.\n\n## 8.4 `agents`\n\nMonorepo inicial de:\n\n- Hermes;\n- ADK;\n- configs;\n- capability bundles;\n- deployments;\n- tests.\n\n## 8.5 `docs`\n\nMarkdown + MkDocs Material.\n\n---\n\n# 9. GitOps \u2014 Argo CD\n\n## 9.1 Decisi\u00f3n\n\nArgo CD ser\u00e1 el sistema GitOps de Wetopi.\n\n## 9.2 Modelo\n\n```mermaid\nflowchart LR\n Dev[Human / Agent]\n Git[GitHub]\n Argo[Argo CD]\n K8s[RKE2 API]\n Apps[Applications]\n\n Dev -->|PR / merge| Git\n Git --> Argo\n Argo --> K8s\n K8s --> Apps\n```\n\n## 9.3 Principios\n\n- CI no har\u00e1 `kubectl apply` en producci\u00f3n como mecanismo normal.\n- Git describe estado deseado.\n- Argo CD reconcilia.\n- Drift manual debe ser visible.\n- Aplicaciones deben agruparse por `AppProject`/dominio.\n- Human UI se considera una ventaja operativa importante.\n\n## 9.4 Layout GitOps\n\n**OPEN:** decidir entre:\n\n- App-of-Apps;\n- ApplicationSet;\n- combinaci\u00f3n por environments/domains.\n\nNo bloquear bootstrap del cluster por esta decisi\u00f3n, pero cerrarla antes de escalar apps.\n\n---\n\n# 10. SecretOps / VaultOps\n\n## 10.1 Decisi\u00f3n\n\nEl Vault operativo de Wetopi ser\u00e1:\n\n```text\nInfisical self-hosted\n```\n\nOpenBao no forma parte de v1.\n\n## 10.2 Requisito real\n\nEl Vault debe permitir que un agente o humano pueda:\n\n```text\ncreate\nread\nupdate\nrotate\ndelete\nlist\n```\n\ncredentials como:\n\n- username/password;\n- API token;\n- SSH key;\n- deploy token;\n- client secret;\n- customer credential.\n\nEjemplo conceptual:\n\n```text\n/platform/proxmox/api\n/customers/837/wp-admin\n/development/github/token\n/agents/hermes-sre/private\n```\n\n## 10.3 Source of truth\n\n```text\nInfisical = source of truth operativo\nKubernetes Secret = materializaci\u00f3n runtime\nSOPS = bootstrap/recovery declarativo\n```\n\n## 10.4 Arquitectura\n\n```mermaid\nflowchart TB\n Humans[Humans\\nWeb UI / CLI]\n Agents[Hermes / ADK / Codex]\n Tools[Tool runtimes]\n Vault[Infisical]\n Operator[Infisical Kubernetes Integration]\n KSec[Kubernetes Secrets]\n Pods[Applications / Pods]\n\n Humans --> Vault\n Agents -->|vault contract| Vault\n Agents --> Tools\n Tools --> Vault\n Vault --> Operator\n Operator --> KSec\n KSec --> Pods\n```\n\n## 10.5 Wetopi Vault Contract\n\nLos agentes no deben quedar fuertemente acoplados a la API concreta de Infisical.\n\n`ai-hub` debe exponer:\n\n```text\nvault.get()\nvault.put()\nvault.update()\nvault.delete()\nvault.list()\nvault.exists()\n```\n\n## 10.6 Infisical validation\n\n**OPEN / REQUIRED BEFORE PRODUCTION:** verificar:\n\n- features self-hosted requeridas;\n- modelo de licensing;\n- machine identities;\n- Kubernetes integration;\n- audit;\n- versioning;\n- backup/restore;\n- l\u00edmites que puedan afectar a una flota de agentes grande.\n\n---\n\n# 11. SOPS + age\n\nSOPS no ser\u00e1 el Vault.\n\nSu responsabilidad es resolver el problema inevitable de bootstrap:\n\n> \u00bfD\u00f3nde guardamos los secretos necesarios para arrancar el sistema que guarda secretos?\n\nModelo:\n\n```mermaid\nflowchart LR\n Git[Git]\n SOPS[SOPS encrypted files]\n Age[age keys]\n Bootstrap[Bootstrap process]\n Vault[Infisical]\n\n Git --> SOPS\n Age --> Bootstrap\n SOPS --> Bootstrap\n Bootstrap --> Vault\n```\n\nSe utilizar\u00e1n al menos:\n\n- clave `age` de bootstrap/automation;\n- clave `age` de recovery offline.\n\nNunca almacenar claves privadas `age` en Git.\n\n---\n\n# 12. Kubernetes Secrets\n\nUna vez Infisical est\u00e1 operativo:\n\n```text\nInfisical\n \u2193\nKubernetes integration/operator\n \u2193\nKubernetes Secret\n \u2193\nPod/container\n```\n\nRKE2 debe tener **Secrets Encryption at Rest** habilitado para proteger la copia persistida en etcd.\n\n---\n\n# 13. Agent Secret Model\n\n## 13.1 Regla principal\n\nUn agente puede necesitar permiso para **usar** una credencial sin tener permiso para **leer su contenido**.\n\n## 13.2 Tool runtime model\n\n```mermaid\nsequenceDiagram\n participant LLM as Hermes / ADK\n participant Tool as Platform Tool Runtime\n participant Vault as Infisical\n participant API as External API\n\n LLM->>Tool: proxmox.restart_vm(vm=123)\n Tool->>Vault: get /platform/proxmox/api\n Vault-->>Tool: credential\n Tool->>API: authenticated request\n API-->>Tool: result\n Tool-->>LLM: sanitized result\n```\n\n## 13.3 Capability domains\n\nNo crear un sidecar por tool.\n\nAgrupar por dominio:\n\n```text\nplatform-tools\n\u251c\u2500\u2500 kubernetes\n\u251c\u2500\u2500 proxmox\n\u251c\u2500\u2500 ceph\n\u2514\u2500\u2500 networking\n\nwordpress-tools\n\u251c\u2500\u2500 wordpress\n\u251c\u2500\u2500 database\n\u2514\u2500\u2500 backups\n\nmarketing-tools\n\u251c\u2500\u2500 intercom\n\u2514\u2500\u2500 analytics\n```\n\n## 13.4 Coding agents\n\nHermes/Codex de desarrollo o SRE pueden necesitar acceso m\u00e1s amplio.\n\nDebe otorgarse por paths/roles, no con acceso global al Vault.\n\n---\n\n# 14. NetOps\n\n## 14.1 NICs\n\nCada VM tendr\u00e1 inicialmente una sola NIC LAN:\n\n```text\nVM\n \u2514\u2500\u2500 eth0\n \u2514\u2500\u2500 LAN / OVH vRack\n```\n\n## 14.2 Ceph network\n\nRed existente:\n\n```text\n192.168.96.0/20\n```\n\nEstado:\n\n```text\nEXISTING\nDO NOT MODIFY\n```\n\nEst\u00e1 hiperconvergida con Proxmox/Ceph.\n\nNo renumerar ni redise\u00f1ar como parte de RKE2.\n\n## 14.3 CIDRs propuestos\n\n```text\nNode LAN: OPEN \u2014 pendiente inventario\nPod CIDR: 10.64.0.0/14\nService CIDR: 10.68.0.0/16\nCeph: 192.168.96.0/20 existente\n```\n\nLos rangos Pod/Service deben verificarse contra todo el direccionamiento existente antes de aplicarse.\n\n## 14.4 Cilium\n\nCilium ser\u00e1 CNI.\n\nBaseline inicial:\n\n```text\nCilium\nVXLAN encapsulation\nHubble enabled\n```\n\nNo exigir native routing/BGP inicialmente.\n\n## 14.5 Hubble\n\nObjetivos:\n\n- flow visibility;\n- pod identity;\n- namespace identity;\n- source/destination;\n- policy verdicts;\n- futuras correlaciones con site/customer.\n\n---\n\n# 15. StorageOps\n\nCeph seguir\u00e1 siendo storage principal.\n\nRKE2 consumir\u00e1 Ceph mediante CSI.\n\n```mermaid\nflowchart LR\n Pod[Pod]\n PVC[PVC]\n CSI[Ceph CSI]\n Node[RKE2 Node]\n Ceph[(Ceph Cluster)]\n\n Pod --> PVC\n PVC --> CSI\n CSI --> Node\n Node --> Ceph\n```\n\nTests obligatorios antes de hosting:\n\n- crear PVC;\n- mount;\n- pod delete/recreate;\n- persistencia;\n- failure b\u00e1sico;\n- snapshot si se adopta;\n- rendimiento aceptable.\n\n---\n\n# 16. Endpoint HA de la API RKE2\n\nRKE2 necesita un endpoint estable:\n\n```text\nrke2-api.\n\nMigrado desde
\nwetopi-platform/docs@main(design/master-architecture-plan.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
Fecha: 28 de agosto de 2026
\nEstado: Documento maestro para revisi\u00f3n e implementaci\u00f3n
\nOrganizaci\u00f3n GitHub: wetopi-platform
\nAudiencia: Platform Engineering, SRE, DevOps, Development, Security, Network Engineering y agentes Codex/Hermes/ADK
\nProp\u00f3sito: servir como documento principal para explicar, revisar, construir y operar la nueva plataforma Wetopi.
Este documento consolida y sustituye como referencia principal a las memorias parciales creadas durante el dise\u00f1o inicial:
\nwetopi-ai-first-kubernetes-project-memory.mdwetopi-platform-bootstrap-plan.mdwetopi-rke2-network-hosting-architecture.mdwetopi-platform-decisions-argocd-infisical-rke2.mdCuando exista una contradicci\u00f3n con documentos anteriores, prevalece este documento, especialmente en decisiones posteriores como:
\nNo todas las decisiones est\u00e1n cerradas. Las cuestiones a\u00fan abiertas se marcan expl\u00edcitamente como OPEN y no deben ser inventadas por Codex ni por un ingeniero durante la implementaci\u00f3n.
Wetopi va a reconstruir su plataforma de Managed WordPress Hosting sobre una nueva foundation basada en RKE2, desplegada en su cloud privado sobre Proxmox + Ceph + OVH vRack.
\nLa migraci\u00f3n ser\u00e1 greenfield y progresiva. Docker Swarm seguir\u00e1 dando servicio mientras se construye, prueba y estabiliza la nueva plataforma.
\nLa arquitectura inicial queda resumida en:
\nProxmox + Ceph + OVH vRack\n \u2502\n \u25bc\n RKE2\n \u2502\n Cilium + Hubble\n \u2502\n Ceph CSI\n \u2502\n Argo CD\n \u2502\n SOPS bootstrap\n \u2502\n Infisical\n \u2502\n Docs / Memory / Work Coordination\n \u2502\n Hermes + ADK + Codex workflows\n \u2502\n Edge RKE2 + Traefik + Gateway API\n \u2502\n Hosting workers + WordPress sites\nLa entrada WAN conservar\u00e1 inicialmente la arquitectura probada de Wetopi:
\nInternet\n \u2193\nVRRP VIP\n \u2193\nIPVS\n \u2193\nDSR\n \u2193\nRKE2 Edge nodes\n \u2193\nTraefik\n \u2193\nGateway API\n \u2193\nServices / EndpointSlices\n \u2193\nWordPress workloads\nLa salida ser\u00e1 deliberadamente sencilla:
\nsite/customer/workload\n \u2193\nselected internal gateway\n \u2193\n192.168.88.100 / .101 / .102 / ...\n \u2193\nUbuntu NATGW\n \u2193\nselected WAN public IP\n \u2193\nInternet\nLa plataforma ser\u00e1 AI-first, pero no se permitir\u00e1 que la autonom\u00eda de los agentes degrade la operabilidad o seguridad. Git, Vault, Work Ledger, Memory Plane, documentaci\u00f3n y tools compartidas ser\u00e1n servicios de primera clase tanto para humanos como para agentes.
\nLa nueva plataforma debe permitir a Wetopi:
\nLa plataforma debe proporcionar:
\nLa nueva plataforma se construye en paralelo al Swarm existente.
\nPRODUCCI\u00d3N ACTUAL NUEVA PLATAFORMA\nDocker Swarm RKE2 greenfield\n \u2502 \u2502\n \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 coexistencia \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\nNo se reutilizan nodos de producci\u00f3n hasta que RKE2 haya demostrado estabilidad.
\nGit\n \u2193\nreview\n \u2193\nArgo CD / IaC\n \u2193\nreal infrastructure\nLa shell history no es source of truth.
\nPara infraestructura real:
\nAPI autoritativa / sistema vivo\n>\nFleet Memory\n>\nDocs hist\u00f3ricas\n>\nLLM inference\nNo desplegar v1 con:
\nLa arquitectura debe permitir a\u00f1adirlo despu\u00e9s.
\nTodos los servicios importantes deben disponer de:
\nCada workload, tool runtime y agente recibe \u00fanicamente los permisos necesarios.
\nCuando sea posible:
\nLLM\n \u2193 tool call\nTool Runtime\n \u2193 secret\nExternal API\nEl modelo no necesita recibir el valor del secret.
\nflowchart TB\n Internet((Internet))\n OVH[OVH / vRack]\n PVE[Proxmox Cluster]\n Ceph[(Ceph)]\n\n subgraph RKE2[Wetopi RKE2 Production]\n CP[3x RKE2 Control Plane\\nembedded etcd]\n Platform[Platform Workers]\n Edge[Edge Workers]\n Hosting[Hosting Workers]\n Agents[Agent Workers / Platform Workers]\n\n Cilium[Cilium CNI]\n Hubble[Hubble]\n Argo[Argo CD]\n Vault[Infisical Vault]\n Docs[MkDocs / Docs]\n Memory[Memory Plane]\n Work[Work Coordination Plane]\n Hermes[Hermes Fleet]\n ADK[ADK Fleet]\n Traefik[Traefik / Gateway API]\n CSI[Ceph CSI]\n end\n\n Internet --> OVH\n OVH --> PVE\n PVE --> RKE2\n PVE --- Ceph\n CSI --> Ceph\n\n CP --- Cilium\n Platform --- Cilium\n Edge --- Cilium\n Hosting --- Cilium\n Agents --- Cilium\n Cilium --> Hubble\n\n Argo --> Platform\n Vault --> Platform\n Docs --> Hermes\n Memory --> Hermes\n Memory --> ADK\n Work --> Hermes\n Work --> ADK\n Hermes --> Agents\n ADK --> Agents\n Traefik --> Hosting\nLa plataforma se divide en dominios operativos. Cada dominio tiene un prop\u00f3sito, source of truth y responsabilidades.
\n| \u00c1rea | \nResponsabilidad principal | \nSource of truth principal | \n
|---|---|---|
| BootstrapOps | \ncrear/reparar la foundation | \nGit + wetopi-ops01 | \n
| InfraOps | \nVMs, Proxmox, RKE2 nodes | \nOpenTofu + Ansible | \n
| GitOps | \nestado declarativo Kubernetes | \nGit + Argo CD | \n
| NetOps | \nCNI, routing, CIDR, policies | \nGit + APIs de red | \n
| EdgeOps | \nIPVS/DSR, proxies WAN, Gateway API | \nGit + IPVS state | \n
| EgressOps | \nNATGW, rutas, WAN IPs | \nGit/config + gateway state | \n
| StorageOps | \nCeph CSI/PVC/storage | \nCeph + Kubernetes | \n
| VaultOps / SecretOps | \ncredentials y secrets | \nInfisical | \n
| SecOps | \nRBAC, policies, security controls | \nGit + audit logs | \n
| ObservabilityOps | \nm\u00e9tricas, logs, traces, flows | \nobservability stack | \n
| DevOps | \nCI/build/release/app lifecycle | \nGit + CI + Argo CD | \n
| AgentOps | \nruntimes Hermes/ADK y capability bundles | \nagents + ai-hub | \n
| MemoryOps | \nmemoria institucional | \nMemory Plane | \n
| WorkOps | \ncoordinaci\u00f3n, tickets, leases | \nWork Ledger | \n
| DocOps | \ndocs/runbooks/ADR | \nMarkdown Git | \n
| BackupOps | \nbackups y restore | \nbackup repositories | \n
| RecoveryOps | \ncluster/Vault/etcd disaster recovery | \nrunbooks + backups | \n
| HostingOps | \nWordPress/customer workloads | \nKubernetes + business control plane | \n
BootstrapOps existe para que la plataforma pueda construirse y repararse sin depender de la propia plataforma.
\nflowchart LR\n Laptop[Engineer Laptop\\nCodex]\n GH[GitHub\\nwetopi-platform]\n Ops[wetopi-ops01]\n Tofu[OpenTofu]\n Ansible[Ansible]\n PVE[Proxmox API]\n RKE[RKE2]\n\n Laptop --> GH\n Laptop --> Ops\n GH --> Ops\n Ops --> Tofu\n Ops --> Ansible\n Tofu --> PVE\n Ansible --> RKE\nwetopi-ops01VM externa al cluster Kubernetes.
\nResponsabilidades:
\nNo ejecuta workloads productivos.
\nSizing inicial orientativo:
\n4 vCPU\n8 GB RAM\n40\u201380 GB disk\nUbuntu 24.04\n1 NIC LAN\n3 x control-plane\n3 x platform worker\n1 x ops VM externa\nNombres:
\nwetopi-ops01\n\nrke-cp01\nrke-cp02\nrke-cp03\n\nrke-platform01\nrke-platform02\nrke-platform03\nIdeal:
\npve01 -> rke-cp01 + rke-platform01\npve02 -> rke-cp02 + rke-platform02\npve03 -> rke-cp03 + rke-platform03\nDebe respetarse failure domain.
\nRKE2 con:
\nSizing orientativo:
\n8 vCPU\n16 GB RAM\n50\u2013100 GB disk\nEl almacenamiento de etcd debe priorizar baja latencia.
\nOpenTofu -> Proxmox resources / VMs\nAnsible -> OS / RKE2 / host configuration\nArgo CD -> Kubernetes workloads\nNo mezclar responsabilidades.
\nOrganizaci\u00f3n:
\nwetopi-platform\nRepositorios iniciales:
\nwetopi-platform/\n\u251c\u2500\u2500 infrastructure\n\u251c\u2500\u2500 agent-services\n\u251c\u2500\u2500 ai-hub\n\u251c\u2500\u2500 agents\n\u2514\u2500\u2500 docs\ninfrastructureinfrastructure/\n\u251c\u2500\u2500 tofu/\n\u2502 \u2514\u2500\u2500 proxmox/\n\u251c\u2500\u2500 ansible/\n\u2502 \u251c\u2500\u2500 inventories/\n\u2502 \u251c\u2500\u2500 roles/\n\u2502 \u2514\u2500\u2500 playbooks/\n\u251c\u2500\u2500 clusters/\n\u2502 \u2514\u2500\u2500 production/\n\u251c\u2500\u2500 apps/\n\u251c\u2500\u2500 platform/\n\u251c\u2500\u2500 networking/\n\u251c\u2500\u2500 storage/\n\u251c\u2500\u2500 security/\n\u251c\u2500\u2500 observability/\n\u251c\u2500\u2500 bootstrap/\n\u2514\u2500\u2500 docs/\nagent-servicesServicios compartidos:
\nai-hubskills/\ntools/\ncomponents/\ncontracts/\nschemas/\nsdk/\nevals/\ntests/\nRegla:
\n\n\nUna tool utilizada por m\u00e1s de un agente deja de pertenecer al agente y pasa a
\nai-hub.
agentsMonorepo inicial de:
\ndocsMarkdown + MkDocs Material.
\nArgo CD ser\u00e1 el sistema GitOps de Wetopi.
\nflowchart LR\n Dev[Human / Agent]\n Git[GitHub]\n Argo[Argo CD]\n K8s[RKE2 API]\n Apps[Applications]\n\n Dev -->|PR / merge| Git\n Git --> Argo\n Argo --> K8s\n K8s --> Apps\nkubectl apply en producci\u00f3n como mecanismo normal.AppProject/dominio.OPEN: decidir entre:
\nNo bloquear bootstrap del cluster por esta decisi\u00f3n, pero cerrarla antes de escalar apps.
\nEl Vault operativo de Wetopi ser\u00e1:
\nInfisical self-hosted\nOpenBao no forma parte de v1.
\nEl Vault debe permitir que un agente o humano pueda:
\ncreate\nread\nupdate\nrotate\ndelete\nlist\ncredentials como:
\nEjemplo conceptual:
\n/platform/proxmox/api\n/customers/837/wp-admin\n/development/github/token\n/agents/hermes-sre/private\nInfisical = source of truth operativo\nKubernetes Secret = materializaci\u00f3n runtime\nSOPS = bootstrap/recovery declarativo\nflowchart TB\n Humans[Humans\\nWeb UI / CLI]\n Agents[Hermes / ADK / Codex]\n Tools[Tool runtimes]\n Vault[Infisical]\n Operator[Infisical Kubernetes Integration]\n KSec[Kubernetes Secrets]\n Pods[Applications / Pods]\n\n Humans --> Vault\n Agents -->|vault contract| Vault\n Agents --> Tools\n Tools --> Vault\n Vault --> Operator\n Operator --> KSec\n KSec --> Pods\nLos agentes no deben quedar fuertemente acoplados a la API concreta de Infisical.
\nai-hub debe exponer:
vault.get()\nvault.put()\nvault.update()\nvault.delete()\nvault.list()\nvault.exists()\nOPEN / REQUIRED BEFORE PRODUCTION: verificar:
\nSOPS no ser\u00e1 el Vault.
\nSu responsabilidad es resolver el problema inevitable de bootstrap:
\n\n\n\u00bfD\u00f3nde guardamos los secretos necesarios para arrancar el sistema que guarda secretos?
\n
Modelo:
\nflowchart LR\n Git[Git]\n SOPS[SOPS encrypted files]\n Age[age keys]\n Bootstrap[Bootstrap process]\n Vault[Infisical]\n\n Git --> SOPS\n Age --> Bootstrap\n SOPS --> Bootstrap\n Bootstrap --> Vault\nSe utilizar\u00e1n al menos:
\nage de bootstrap/automation;age de recovery offline.Nunca almacenar claves privadas age en Git.
Una vez Infisical est\u00e1 operativo:
\nInfisical\n \u2193\nKubernetes integration/operator\n \u2193\nKubernetes Secret\n \u2193\nPod/container\nRKE2 debe tener Secrets Encryption at Rest habilitado para proteger la copia persistida en etcd.
\nUn agente puede necesitar permiso para usar una credencial sin tener permiso para leer su contenido.
\nsequenceDiagram\n participant LLM as Hermes / ADK\n participant Tool as Platform Tool Runtime\n participant Vault as Infisical\n participant API as External API\n\n LLM->>Tool: proxmox.restart_vm(vm=123)\n Tool->>Vault: get /platform/proxmox/api\n Vault-->>Tool: credential\n Tool->>API: authenticated request\n API-->>Tool: result\n Tool-->>LLM: sanitized result\nNo crear un sidecar por tool.
\nAgrupar por dominio:
\nplatform-tools\n\u251c\u2500\u2500 kubernetes\n\u251c\u2500\u2500 proxmox\n\u251c\u2500\u2500 ceph\n\u2514\u2500\u2500 networking\n\nwordpress-tools\n\u251c\u2500\u2500 wordpress\n\u251c\u2500\u2500 database\n\u2514\u2500\u2500 backups\n\nmarketing-tools\n\u251c\u2500\u2500 intercom\n\u2514\u2500\u2500 analytics\nHermes/Codex de desarrollo o SRE pueden necesitar acceso m\u00e1s amplio.
\nDebe otorgarse por paths/roles, no con acceso global al Vault.
\nCada VM tendr\u00e1 inicialmente una sola NIC LAN:
\nVM\n \u2514\u2500\u2500 eth0\n \u2514\u2500\u2500 LAN / OVH vRack\nRed existente:
\n192.168.96.0/20\nEstado:
\nEXISTING\nDO NOT MODIFY\nEst\u00e1 hiperconvergida con Proxmox/Ceph.
\nNo renumerar ni redise\u00f1ar como parte de RKE2.
\nNode LAN: OPEN \u2014 pendiente inventario\nPod CIDR: 10.64.0.0/14\nService CIDR: 10.68.0.0/16\nCeph: 192.168.96.0/20 existente\nLos rangos Pod/Service deben verificarse contra todo el direccionamiento existente antes de aplicarse.
\nCilium ser\u00e1 CNI.
\nBaseline inicial:
\nCilium\nVXLAN encapsulation\nHubble enabled\nNo exigir native routing/BGP inicialmente.
\nObjetivos:
\nCeph seguir\u00e1 siendo storage principal.
\nRKE2 consumir\u00e1 Ceph mediante CSI.
\nflowchart LR\n Pod[Pod]\n PVC[PVC]\n CSI[Ceph CSI]\n Node[RKE2 Node]\n Ceph[(Ceph Cluster)]\n\n Pod --> PVC\n PVC --> CSI\n CSI --> Node\n Node --> Ceph\nTests obligatorios antes de hosting:
\nRKE2 necesita un endpoint estable:
\nrke2-api.<internal-domain>\n \u2193\n VIP\n \u2193\ncp01 / cp02 / cp03\nPropuesta actual:
\n2 x HAProxy + Keepalived\nprivate VIP\nEstado:
\nOPEN \u2014 debe cerrarse antes de instalar el cluster definitivo.
\nLa propuesta se considera conservadora y compatible con el conocimiento operativo existente, pero todav\u00eda no est\u00e1 aprobada como decisi\u00f3n final.
\nConservar inicialmente:
\nVRRP\n+\nIPVS\n+\nDSR\nLos WAN nodes est\u00e1n distribuidos sobre varios hosts Proxmox, y cada host f\u00edsico dispone de capacidad de salida OVH propia.
\nDSR permite:
\nrequest:\nclient -> IPVS -> edge\n\nresponse:\nedge -> client DIRECT\nPor tanto se preserva la agregaci\u00f3n de bandwidth domains.
\nflowchart TB\n Client[Internet Client]\n VIP[Public VIP / VRRP]\n LVS[IPVS Directors]\n Edge1[RKE2 Edge 1\\nTraefik]\n Edge2[RKE2 Edge 2\\nTraefik]\n Edge3[RKE2 Edge 3\\nTraefik]\n Gateway[Gateway API]\n Service[Kubernetes Service]\n Site[WordPress Pod]\n\n Client --> VIP\n VIP --> LVS\n LVS -->|DSR| Edge1\n LVS -->|DSR| Edge2\n LVS -->|DSR| Edge3\n Edge1 --> Gateway\n Edge2 --> Gateway\n Edge3 --> Gateway\n Gateway --> Service\n Service --> Site\n\n Edge1 -. direct response .-> Client\n Edge2 -. direct response .-> Client\n Edge3 -. direct response .-> Client\nRequired before production:
\nTraefik ser\u00e1 el proxy inicial.
\nResponsabilidades:
\nEstado de detalles:
\nOPEN: definir deployment exacto:
\nDaemonSet;hostNetwork s\u00ed/no seg\u00fan PoC DSR;Gateway API ser\u00e1 el modelo declarativo L7.
\nDomain\n \u2193\nHTTPRoute\n \u2193\nService\n \u2193\nEndpointSlice\n \u2193\nPod\nEsto elimina del path Kubernetes:
\nEgress v1 debe ser simple.
\nNo se construye inicialmente un controlador sofisticado de reputaci\u00f3n/IPAM.
\nEjemplo:
\ngw100\nLAN: 192.168.88.100\nWAN: PUBLIC_IP_A\n\ngw101\nLAN: 192.168.88.101\nWAN: PUBLIC_IP_B\n\ngw102\nLAN: 192.168.88.102\nWAN: PUBLIC_IP_C\nVM Ubuntu con:
\nsite/customer/workload\n \u2193\negress profile / label\n \u2193\ngw100 / gw101 / gw102\nEjemplo l\u00f3gico:
\nwetopi.io/egress-gateway: gw101\nEstado:
\nOPEN.
\nBaseline propuesto:
\nEvoluci\u00f3n posible:
\nCilium Egress Gateway\nsin cambiar el contrato l\u00f3gico.
\nSeparar al menos:
\nhosting-general\nhosting-premium\nplatform\nagents-crawlers\nLos crawlers nunca deben compartir egress WAN IP con hosting general.
\nLa observabilidad futura debe combinar:
\nKubernetes metrics\napplication logs\nHubble flows\nTraefik access logs\nNATGW logs\nKubernetes audit\nInfisical audit\nArgo CD events\nagent/tool audit\nPosteriormente:
\nSuricata\nWAF events\nDNS telemetry\nreputation signals\nUna observaci\u00f3n de red deber\u00eda poder acabar enriquecida con:
\ncustomer_id\nsite_id\npod\nnamespace\nnode\nproxmox_host\negress_gateway\negress_public_ip\ndestination\nprotocol\nverdict\nSecurity foundation v1:
\nNo asumir que cannot get Secret equivale a no poder acceder a secretos: un usuario con capacidad de crear Pods en un namespace puede potencialmente montar Secrets de ese namespace.
Por tanto RBAC de Pods y Vault/Secrets debe dise\u00f1arse conjuntamente.
\nPropuesta inicial orientativa:
\nwetopi-platform\nwetopi-edge\nwetopi-hosting\nwetopi-agents-sre\nwetopi-agents-development\nwetopi-agents-marketing\nwetopi-observability\nwetopi-security\nEstado:
\nOPEN: modelo definitivo de namespaces y AppProjects/RBAC.
\nDevOps cubre lifecycle de software, no infraestructura f\u00edsica.
\nPipeline conceptual:
\nflowchart LR\n Code[Application Code]\n CI[CI / Tests / Build]\n Registry[Container Registry]\n Config[GitOps Repo]\n Argo[Argo CD]\n RKE[RKE2]\n\n Code --> CI\n CI --> Registry\n CI -->|version bump / PR| Config\n Config --> Argo\n Argo --> RKE\nPrincipios:
\nActor\n\u251c\u2500\u2500 Human\n\u251c\u2500\u2500 Hermes\n\u251c\u2500\u2500 ADK Agent\n\u2514\u2500\u2500 future runtime\nCampos de identidad:
\nactor_id\nactor_type\nruntime\ncrew\nrole\nuser_id\nenvironment\nHermes personales:
\nCada Hermes tendr\u00e1:
\nlocal personal memory\n+\nFleet Memory\n+\nWork Ledger\n+\nrole-specific tools\n+\nVault permissions\nADK fleet:
\nTodos consumen servicios compartidos.
\nMemory Plane es memoria institucional, no source of truth de infraestructura viva.
\nClases:
\nDebe soportar:
\nHermes combina:
\nLocal Memory\n+\nWetopi Fleet Memory\nWork Coordination Plane evita conflictos entre agentes/humanos.
\nNo es solo un Kanban.
\nDebe manejar:
\nContrato m\u00ednimo:
\nwork.list()\nwork.get()\nwork.create()\nwork.claim()\nwork.release()\nwork.update()\nwork.comment()\nwork.complete()\nwork.find_conflicts()\nwork.acquire()\nwork.renew()\nwork.release_resources()\nBackend a\u00fan desacoplado.
\nDocumentaci\u00f3n:
\nMarkdown\n+\nGit\n+\nMkDocs Material\nRoles:
\nREADME del repo\n= c\u00f3mo funciona el componente\n\nwetopi-platform/docs\n= c\u00f3mo Wetopi lo dise\u00f1a, opera y gobierna\nFlujo:
\nincident / project\n \u2193\nWork Ledger\n \u2193\nvalidated knowledge\n \u2193\nMemory\n \u2193\nADR / runbook / architecture doc\n \u2193\nDocs repo\nDebe existir recuperaci\u00f3n fuera del cluster.
\nObjetos a proteger:
\nGit repos\netcd snapshots\nInfisical database/state\nSOPS/age recovery material\nCeph configuration/data policies\nArgo configuration (also in Git)\ncritical application databases\nOPEN: destino externo definitivo de backups.
\nPrincipio:
\n\n\nun backup no se considera v\u00e1lido hasta haber probado un restore.
\n
No refactorizar todo wepod a la vez.
Inicialmente:
\nDocker wepod\n \u2193\nPod / StatefulSet\nmanteniendo cuando convenga:
\ns6\nnginx\nphp-fpm\nmysql\nLa separaci\u00f3n de DB/frontend puede llegar despu\u00e9s.
\nSe crean despu\u00e9s de estabilizar Platform y Edge.
\nrke-host01\nrke-host02\nrke-host03\n...\nPrimera cohorte:
\n~10 sites\nDespu\u00e9s escalar progresivamente.
\nWordPressSite CRDPosible modelo futuro:
\napiVersion: wetopi.io/v1\nkind: WordPressSite\nmetadata:\n name: site-12345\nspec:\n domains:\n - example.com\n network:\n egressGateway: gw100\n storage:\n size: 20Gi\nNo es requisito de v1.
\nflowchart TB\n subgraph Control[CONTROL PLANES]\n Git[GitHub]\n Tofu[OpenTofu]\n Ansible[Ansible]\n Argo[Argo CD]\n Vault[Infisical]\n Work[Work Ledger]\n Memory[Memory Plane]\n Docs[Docs]\n end\n\n subgraph K8s[KUBERNETES CONTROL]\n API[RKE2 API]\n ETCD[(etcd)]\n Cilium[Cilium control]\n end\n\n subgraph Data[DATA PLANES]\n IPVS[IPVS / DSR]\n Edge[Traefik Edge]\n Pods[Hosting Pods]\n NATGW[NAT Gateways]\n Ceph[(Ceph)]\n end\n\n Git --> Tofu\n Git --> Ansible\n Git --> Argo\n Tofu --> API\n Ansible --> API\n Argo --> API\n API --> ETCD\n API --> Cilium\n Vault --> Pods\n IPVS --> Edge\n Edge --> Pods\n Pods --> NATGW\n Pods --> Ceph\nflowchart TD\n Git[GitHub org + repos]\n Inventory[Proxmox / Network Inventory]\n IaC[OpenTofu + Ansible]\n Ops[wetopi-ops01]\n VMs[3 CP + 3 Platform VMs]\n RKE[RKE2 HA]\n Cilium[Cilium]\n Hubble[Hubble]\n CSI[Ceph CSI]\n Argo[Argo CD]\n SOPS[SOPS + age bootstrap]\n Vault[Infisical]\n Docs[Docs]\n Memory[Memory Plane]\n Work[Work Ledger]\n AI[AI Hub]\n Agents[Hermes + ADK]\n Edge[Edge workers + Traefik]\n Egress[NATGW]\n Hosting[Hosting workers]\n Sites[First sites]\n\n Git --> Inventory\n Inventory --> IaC\n IaC --> Ops\n Ops --> VMs\n VMs --> RKE\n RKE --> Cilium\n Cilium --> Hubble\n Cilium --> CSI\n CSI --> Argo\n Argo --> SOPS\n SOPS --> Vault\n Vault --> Docs\n Vault --> Memory\n Vault --> Work\n Docs --> AI\n Memory --> Agents\n Work --> Agents\n AI --> Agents\n Agents --> Edge\n Edge --> Egress\n Egress --> Hosting\n Hosting --> Sites\nDeliverables:
\nwetopi-platform;Inventariar:
\nNo crear ranges nuevos hasta completar este inventario.
\nCrear/provisionar wetopi-ops01.
Crear:
\nrke-cp01\nrke-cp02\nrke-cp03\nCerrar previamente endpoint HA.
\nCrear:
\nrke-platform01\nrke-platform02\nrke-platform03\nvault.* contract.Deploy definitive agents after shared services are healthy.
\nIncrease cohorts while shrinking Swarm.
\nflowchart LR\n Swarm[Docker Swarm\\nCurrent Production]\n New[RKE2 Platform]\n Pilot[10 Sites]\n Batch1[50 Sites]\n Batch2[100+ Sites]\n Final[Swarm Retirement]\n\n Swarm --> New\n New --> Pilot\n Pilot --> Batch1\n Batch1 --> Batch2\n Batch2 --> Final\nPrincipio:
\nRKE2 failure during early phases\n!=\ncustomer outage\n| ID | \nDecisi\u00f3n | \n
|---|---|
| ARC-001 | \nRKE2 | \n
| ARC-002 | \n3 CP + embedded etcd | \n
| ARC-003 | \ngreenfield paralelo a Swarm | \n
| ARC-004 | \nwetopi-ops01 externo | \n
| ARC-005 | \nOpenTofu + Ansible | \n
| ARC-006 | \nCilium | \n
| ARC-007 | \nHubble | \n
| ARC-008 | \nCeph existente + CSI | \n
| ARC-009 | \nArgo CD | \n
| ARC-010 | \nInfisical self-hosted Vault | \n
| ARC-011 | \nSOPS + age solo bootstrap/recovery | \n
| ARC-012 | \nIPVS + VRRP + DSR inicial | \n
| ARC-013 | \nTraefik inicial | \n
| ARC-014 | \nGateway API | \n
| ARC-015 | \nNATGW Ubuntu simples | \n
| ARC-016 | \nno Istio inicialmente | \n
| ARC-017 | \nno BGP obligatorio inicialmente | \n
| ARC-018 | \nagents/crawlers separados de customer egress | \n
| ARC-019 | \nMarkdown + MkDocs | \n
| ARC-020 | \nAI Hub compartido | \n
Estas decisiones deben ser revisadas por Platform Engineering antes de implementaci\u00f3n o antes de su fase correspondiente.
\n| Tema | \nEstado | \nCu\u00e1ndo debe cerrarse | \n
|---|---|---|
| Node LAN CIDR | \nOPEN | \nantes de crear VMs RKE2 | \n
Pod CIDR 10.64.0.0/14 | \nProposed | \nantes de RKE2 install | \n
Service CIDR 10.68.0.0/16 | \nProposed | \nantes de RKE2 install | \n
| API HA: HAProxy+Keepalived | \nProposed | \nantes de CP bootstrap | \n
| Traefik hostNetwork/DSR details | \nOPEN | \nantes de Edge PoC | \n
| Egress enforcement implementation | \nOPEN | \nantes de Egress PoC | \n
| Infisical licensing/feature validation | \nOPEN | \nantes de prod Vault | \n
| Argo app layout | \nOPEN | \nantes de app scale-out | \n
| OIDC provider | \nOPEN | \nantes de broad human access | \n
| Namespace model final | \nOPEN | \nantes de production RBAC | \n
| External backup destination | \nOPEN | \nantes de production data | \n
| Central observability stack | \nOPEN | \nantes de migration scale | \n
| Work Ledger backend | \nOPEN | \nantes de multi-agent operations | \n
Riesgo: desplegar demasiado antes de tener workloads.
\nMitigaci\u00f3n: v1 simple, roadmap posterior.
\nRiesgo: degradar capacidad WAN agregada.
\nMitigaci\u00f3n: PoC + packet capture + no retirar IPVS hasta equivalencia demostrada.
\nRiesgo: colisi\u00f3n con redes legacy/OVH/Ceph.
\nMitigaci\u00f3n: inventory first.
\nRiesgo: tokens en prompts/logs.
\nMitigaci\u00f3n: tool runtimes + Infisical + least privilege.
\nRiesgo: features cr\u00edticas fuera de self-hosted assumptions.
\nMitigaci\u00f3n: validar Infisical y usar Wetopi vault.* contract.
Riesgo: un merge despliega cambio err\u00f3neo global.
\nMitigaci\u00f3n: AppProjects, environments, PR review, health gates.
\nRiesgo: Kubernetes storage depende del cluster Ceph hiperconvergido.
\nMitigaci\u00f3n: no modificar Ceph en bootstrap, test de CSI y recovery independientes.
\nRiesgo: dos agentes modifican recurso simult\u00e1neamente.
\nMitigaci\u00f3n: Work Ledger + leases + resource scopes.
\nai-hub;Para presentar este documento a otros ingenieros, la primera sesi\u00f3n deber\u00eda revisar en este orden:
\nNo intentar resolver Work Ledger, WAF, BGP, CRDs y service mesh en esa primera reuni\u00f3n.
\nCrear/inicializar los cinco repos en wetopi-platform.
Dar acceso a Codex a la organizaci\u00f3n GitHub con permisos suficientes para gestionar repos, PRs y workflows.
\nCrear un inventario automatizado/read-only de Proxmox y networking.
\nProducir un network-inventory.md con todos los rangos actuales antes de fijar Node/Pod/Service CIDRs.
Cerrar el dise\u00f1o del endpoint API HA.
\nCrear plantilla Ubuntu 24.04 y wetopi-ops01.
Implementar OpenTofu/Ansible de CP/workers sin desplegar todav\u00eda servicios de negocio.
\nHacer acceptance test de RKE2 antes de Cilium/Ceph/Argo.
\nflowchart TB\n People[Engineers / Operators / Employees]\n Git[GitHub wetopi-platform]\n Codex[Codex]\n Hermes[Hermes]\n ADK[ADK Fleet]\n\n subgraph OpsPlanes[Operational Planes]\n IaC[InfraOps\\nOpenTofu + Ansible]\n GitOps[GitOps\\nArgo CD]\n VaultOps[VaultOps\\nInfisical]\n NetOps[NetOps\\nCilium / IPVS / NATGW]\n ObsOps[ObservabilityOps\\nHubble + logs/metrics]\n MemoryOps[MemoryOps]\n WorkOps[WorkOps]\n DocsOps[DocOps]\n end\n\n subgraph Infra[Private Cloud]\n PVE[Proxmox]\n Ceph[(Ceph)]\n RKE[RKE2]\n Edge[Edge]\n Host[Hosting]\n end\n\n People --> Git\n People --> VaultOps\n People --> DocsOps\n Codex --> Git\n Hermes --> WorkOps\n Hermes --> MemoryOps\n Hermes --> VaultOps\n ADK --> WorkOps\n ADK --> MemoryOps\n ADK --> VaultOps\n\n Git --> IaC\n Git --> GitOps\n IaC --> PVE\n PVE --> RKE\n Ceph --> RKE\n GitOps --> RKE\n VaultOps --> RKE\n NetOps --> RKE\n ObsOps --> RKE\n RKE --> Edge\n RKE --> Host\nLa nueva Wetopi no debe construirse como \u201cun cluster Kubernetes con algunos agentes\u201d.
\nDebe construirse como una plataforma operativa integrada donde:
\nGit = desired state\nOpenTofu = virtual infrastructure provisioning\nAnsible = host bootstrap/configuration\nRKE2 = orchestration platform\nCilium = cluster network foundation\nHubble = network visibility\nCeph = persistent storage\nArgo CD = continuous declarative reconciliation\nInfisical = operational Vault\nSOPS + age = Vault/bootstrap recovery bridge\nIPVS + DSR = WAN ingress foundation v1\nTraefik = HTTP/TLS edge v1\nNATGW = simple controlled egress v1\nMemory Plane = institutional knowledge\nWork Ledger = coordination\nAI Hub = shared capabilities\nHermes/ADK = operational agents\nDocs = institutional documentation\nEl orden de implementaci\u00f3n debe preservar siempre dos propiedades:
\n\u00c9sta es la base t\u00e9cnica y organizativa que debe revisarse con Platform Engineering y Development antes de iniciar la construcci\u00f3n del cluster definitivo.
\n" }, { "id": 58, "title": "network hosting architecture", "slug": "network-hosting-architecture", "book_id": 8, "chapter_id": 29, "description": null, "created_at": "2026-09-06T13:39:03.000000Z", "updated_at": "2026-09-06T13:39:03.000000Z", "markdown": "> Migrado desde `wetopi-platform/docs@main` (design/network-hosting-architecture.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.\n\n# Wetopi RKE2 Network & Hosting Architecture\n## Memoria ejecutiva y t\u00e9cnica para la migraci\u00f3n desde Docker Swarm\n\n**Fecha:** 28 de agosto de 2026 \n**Estado:** Dise\u00f1o base consolidado \n**\u00c1mbito:** Red, ingress, egress, hosting WordPress y transici\u00f3n desde Docker Swarm hacia RKE2 sobre Proxmox + Ceph.\n\n---\n\n# \u00cdndice\n\n1. Resumen ejecutivo\n2. Contexto actual de Wetopi\n3. Objetivos del nuevo dise\u00f1o\n4. Principios de arquitectura\n5. Decisi\u00f3n de plataforma: RKE2\n6. Arquitectura global objetivo\n7. Control plane RKE2\n8. Workers y dominios de infraestructura\n9. Cilium como CNI base\n10. Hubble como observabilidad de red\n11. Ingress WAN: conservar IPVS + VRRP + DSR\n12. Edge nodes RKE2\n13. Proxy HTTP/TLS y Gateway API\n14. Sustituci\u00f3n de Consul, Registrator y consul-template\n15. Modelo inicial de WordPress workloads\n16. Egress: dise\u00f1o simple de NAT Gateways\n17. Asignaci\u00f3n de clientes a gateways de salida\n18. Rotaci\u00f3n y cambio de IPs de salida\n19. Separaci\u00f3n de reputaci\u00f3n de IPs\n20. Ingress IP dedicada por cliente\n21. Integraci\u00f3n con OVH vRack\n22. Proxmox y agregaci\u00f3n de ancho de banda\n23. Storage: Ceph\n24. Observabilidad y seguridad de red\n25. Suricata y WAF futuros\n26. Qu\u00e9 NO vamos a desplegar inicialmente\n27. Modelo de alta disponibilidad\n28. Mantenimiento y draining de edge nodes\n29. Modelo de namespaces y roles de nodos\n30. Flujos de tr\u00e1fico de ejemplo\n31. Migraci\u00f3n progresiva desde Docker Swarm\n32. Fases de implementaci\u00f3n\n33. Decisiones consolidadas\n34. Riesgos principales y mitigaciones\n35. Evoluci\u00f3n futura\n36. Arquitectura objetivo resumida\n\n---\n\n# 1. Resumen ejecutivo\n\nWetopi es un Managed WordPress Hosting ejecutado sobre un cloud privado basado en Proxmox + Ceph, con Docker Swarm como scheduler actual y del orden de 700 contenedores.\n\nLa arquitectura WAN existente tiene dos propiedades valiosas que se preservar\u00e1n inicialmente:\n\n- balanceo L3/L4 mediante IPVS;\n- VIPs con VRRP/Keepalived;\n- Direct Server Return (DSR);\n- proxies WAN repartidos entre varios hosts Proxmox.\n\nDSR no solo proporciona eficiencia y HA: permite que las respuestas salgan directamente por distintos nodos WAN y, por tanto, aprovechar la capacidad de salida de m\u00faltiples hosts f\u00edsicos de OVH sin hacer retornar todo el tr\u00e1fico por un \u00fanico balanceador.\n\nLa nueva plataforma se construir\u00e1 sobre:\n\n- RKE2;\n- Cilium;\n- Hubble;\n- Kubernetes Services;\n- EndpointSlices;\n- Gateway API;\n- edge proxies Kubernetes-native;\n- Ceph mediante CSI.\n\nNo se intentar\u00e1 transformar toda la red a la vez.\n\nLa primera generaci\u00f3n conservar\u00e1 IPVS + VRRP + DSR en la capa WAN y sustituir\u00e1, detr\u00e1s de ella:\n\n- Docker Swarm;\n- Registrator;\n- Consul como service registry de websites;\n- consul-template;\n- generaci\u00f3n din\u00e1mica de nginx.conf.\n\nEl egress tambi\u00e9n se mantendr\u00e1 deliberadamente simple.\n\nSe desplegar\u00e1n varios NAT Gateways Ubuntu con IP interna estable y una IP p\u00fablica asociada:\n\n```text\n192.168.88.100 -> WAN IP A\n192.168.88.101 -> WAN IP B\n192.168.88.102 -> WAN IP C\n```\n\nDiferentes clientes, sites o clases de workload podr\u00e1n usar gateways distintos.\n\nEsto permite:\n\n- aislar reputaci\u00f3n;\n- separar crawlers/agentes de customers;\n- rotar WAN IPs;\n- mover clientes entre gateways;\n- ofrecer IP de salida dedicada;\n- mantener una soluci\u00f3n f\u00e1cil de operar.\n\nCilium se adopta como foundation de networking y observabilidad, pero no se requiere utilizar desde el primer d\u00eda BGP, Egress Gateway avanzado, CRDs propios o automatizaci\u00f3n compleja.\n\nLa filosof\u00eda es:\n\n> construir una base moderna, observable y extensible sin sobrecomplicar la primera versi\u00f3n.\n\n---\n\n# 2. Contexto actual de Wetopi\n\nLa plataforma actual se ejecuta sobre:\n\n```text\nOVH\n \u2193\nProxmox\n \u2193\nCeph\n \u2193\nDocker Swarm\n```\n\nEl workload principal son websites WordPress.\n\nCada website se representa aproximadamente mediante un `wepod`, que contiene actualmente:\n\n```text\ns6\n\u251c\u2500\u2500 nginx\n\u251c\u2500\u2500 php-fpm\n\u2514\u2500\u2500 mysql\n```\n\nLa entrada WAN actual es aproximadamente:\n\n```text\nInternet\n \u2193\nPublic VIP\n \u2193\nVRRP / Keepalived\n \u2193\nIPVS directors\n \u2193\nDSR\n \u2193\nWAN Nginx proxies\n \u2193\nDocker workloads\n```\n\nEl descubrimiento y configuraci\u00f3n se realiza mediante:\n\n```text\nDocker\n \u2193\nRegistrator\n \u2193\nConsul\n \u2193\nconsul-template\n \u2193\nnginx.conf\n \u2193\nreload\n```\n\nEste modelo ha sido funcional, pero introduce piezas que Kubernetes puede sustituir de forma nativa.\n\n---\n\n# 3. Objetivos del nuevo dise\u00f1o\n\nLa nueva red debe:\n\n1. Migrar Docker Swarm a RKE2.\n2. Mantener Proxmox + Ceph.\n3. Preservar IPVS/DSR inicialmente.\n4. Preservar la agregaci\u00f3n distribuida de egress.\n5. Eliminar Registrator.\n6. Eliminar Consul como registry de websites.\n7. Eliminar consul-template del path HTTP.\n8. Utilizar Service + EndpointSlice.\n9. Adoptar Gateway API.\n10. Utilizar un CNI moderno.\n11. Observar tr\u00e1fico por workload.\n12. Poder separar clientes por egress gateway.\n13. Poder cambiar f\u00e1cilmente una IP de salida.\n14. Permitir IP dedicada de salida.\n15. Permitir ingress dedicado en el futuro.\n16. Dejar preparada la red para WAF/IDS.\n17. Mantener simplicidad operacional.\n18. Permitir migraci\u00f3n progresiva.\n19. No a\u00f1adir service mesh sin necesidad.\n20. No depender de BGP para el bootstrap.\n\n---\n\n# 4. Principios de arquitectura\n\n## 4.1 Conservar lo que resuelve un problema real\n\nIPVS + VRRP + DSR seguir\u00e1 siendo inicialmente la capa L3/L4 WAN.\n\n## 4.2 Kubernetes sustituye el service discovery artesanal\n\nKubernetes ya proporciona:\n\n- Services;\n- EndpointSlices;\n- DNS;\n- readiness;\n- labels;\n- selectors.\n\nNo se recrear\u00e1 Consul dentro de Kubernetes para registrar websites.\n\n## 4.3 Capas separadas\n\n```text\nL3/L4 WAN -> IPVS / DSR\nHTTP/TLS -> Gateway / proxy\nCluster networking -> Cilium\nFlow visibility -> Hubble\nDeep IDS futuro -> Suricata\nHTTP security -> WAF futuro\n```\n\n## 4.4 Egress simple\n\nPrimera versi\u00f3n:\n\n```text\nUbuntu gateway\n+\nrouting\n+\nnftables/iptables\n+\nWAN IP\n```\n\n## 4.5 Evoluci\u00f3n progresiva\n\nLa base debe permitir a\u00f1adir capacidades avanzadas despu\u00e9s, no obligar a desplegarlas ya.\n\n---\n\n# 5. Decisi\u00f3n de plataforma: RKE2\n\nSe selecciona RKE2 como distribuci\u00f3n Kubernetes principal.\n\nRazones:\n\n- Kubernetes conformante;\n- control plane cercano a upstream;\n- etcd embebido;\n- static pods;\n- containerd;\n- HA sencilla;\n- soporte s\u00f3lido de CNI;\n- lifecycle m\u00e1s f\u00e1cil que kubeadm;\n- adecuada para plataforma productiva de largo plazo.\n\nRancher UI no es requisito.\n\n---\n\n# 6. Arquitectura global objetivo\n\n```text\n INTERNET\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502\n VIP A VIP B\n \u2502 \u2502\n IPVS IPVS\n \u2502 \u2502\n DSR DSR\n \u2502 \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 RKE2 EDGE A \u2502 \u2502 RKE2 EDGE B \u2502\n \u2502 proxy HTTP/TLS \u2502 \u2502 proxy HTTP/TLS \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n Gateway API\n \u2502\n Service / EndpointSlice\n \u2502\n Pods\n \u2502\n Ceph\n\nEGRESS:\n\nPods / Sites / Agents\n \u2502\n \u25bc\ninternal gateway selection\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc \u25bc \u25bc\n.100 .101 .102\n GW A GW B GW C\n \u2502 \u2502 \u2502\nWAN A WAN B WAN C\n \u2502 \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500 Internet \u2500\u2518\n```\n\n---\n\n# 7. Control plane RKE2\n\nBaseline:\n\n```text\n3 x RKE2 server nodes\n+\nembedded etcd\n```\n\nModelo:\n\n```text\nAPI stable endpoint / VIP\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc \u25bc \u25bc\ncp01 cp02 cp03\netcd etcd etcd\n```\n\nLos control planes deben quedar separados de los workloads de customers.\n\nSe prioriza:\n\n- NVMe;\n- baja latencia;\n- backups de etcd;\n- restore probado;\n- endpoint API estable.\n\n---\n\n# 8. Workers y dominios de infraestructura\n\nSe definir\u00e1n roles expl\u00edcitos:\n\n```text\nwetopi.io/role=edge\nwetopi.io/role=hosting\nwetopi.io/role=agents\nwetopi.io/role=database\nwetopi.io/role=infra\n```\n\nAdem\u00e1s, cuando sea \u00fatil:\n\n```text\nwetopi.io/proxmox-host=pve01\nwetopi.io/bandwidth-domain=pve01\nwetopi.io/wan-zone=a\n```\n\nEl host Proxmox no es solo un failure domain: tambi\u00e9n puede ser un bandwidth domain.\n\n---\n\n# 9. Cilium como CNI base\n\nSe selecciona Cilium como CNI.\n\nObjetivos iniciales:\n\n- red de pods;\n- NetworkPolicy;\n- observabilidad;\n- Hubble;\n- integraci\u00f3n con Kubernetes;\n- base futura para capacidades avanzadas.\n\nNo es requisito inicial:\n\n- sustituir IPVS;\n- BGP WAN;\n- gestionar todas las IPs p\u00fablicas;\n- controlar todo egress mediante CRDs.\n\n---\n\n# 10. Hubble como observabilidad de red\n\nHubble ser\u00e1 la primera capa de visibilidad de flows.\n\nPermitir\u00e1 responder preguntas como:\n\n- qu\u00e9 pod abri\u00f3 una conexi\u00f3n;\n- qu\u00e9 site est\u00e1 haciendo egress;\n- qu\u00e9 namespace habla con qu\u00e9 destino;\n- qu\u00e9 tr\u00e1fico fue bloqueado;\n- qu\u00e9 workload est\u00e1 aumentando conexiones.\n\nEl objetivo a medio plazo es correlacionar:\n\n```text\npod\nnamespace\nsite\ncustomer\nnode\ngateway\negress IP\ndestination\n```\n\n---\n\n# 11. Ingress WAN: conservar IPVS + VRRP + DSR\n\nSe conserva el patr\u00f3n:\n\n```text\nInternet\n \u2193\nVIP\n \u2193\nVRRP\n \u2193\nIPVS\n \u2193\nDSR\n \u2193\nedge node\n```\n\n## Por qu\u00e9\n\nCon DSR:\n\n```text\nrequest:\nclient -> IPVS -> edge\n\nresponse:\nedge -> client\n```\n\nLa respuesta no vuelve por el director.\n\nEsto preserva la capacidad de salida distribuida entre hosts f\u00edsicos OVH.\n\n---\n\n# 12. Edge nodes RKE2\n\nLos antiguos WAN proxies pasar\u00e1n progresivamente a edge workers RKE2.\n\nEjemplo:\n\n```text\nedge-a01\nedge-a02\nedge-b01\nedge-b02\n```\n\nCada edge VM debe estar distribuida deliberadamente entre hosts Proxmox distintos.\n\nLabels ejemplo:\n\n```text\nwetopi.io/role=edge\nwetopi.io/wan-zone=a\nwetopi.io/proxmox-host=pve01\n```\n\nTaint recomendado:\n\n```text\nwetopi.io/edge=true:NoSchedule\n```\n\n---\n\n# 13. Proxy HTTP/TLS y Gateway API\n\nEl edge seguir\u00e1 terminando TLS.\n\nCandidatos:\n\n- Traefik;\n- Envoy.\n\nLa arquitectura no debe depender todav\u00eda de la elecci\u00f3n final.\n\nResponsabilidades:\n\n- TLS;\n- routing HTTP;\n- forwarding de IP real;\n- headers;\n- m\u00e9tricas;\n- readiness;\n- futura integraci\u00f3n WAF.\n\nRouting:\n\n```text\nhostname\n \u2193\nHTTPRoute\n \u2193\nService\n \u2193\nEndpointSlice\n \u2193\nPod\n```\n\n---\n\n# 14. Sustituci\u00f3n de Consul, Registrator y consul-template\n\nANTES:\n\n```text\ncontainer\n \u2193\nRegistrator\n \u2193\nConsul\n \u2193\nconsul-template\n \u2193\nnginx.conf\n \u2193\nreload\n```\n\nNUEVO:\n\n```text\nPod\n \u2193\nService selector\n \u2193\nEndpointSlice\n \u2193\nGateway controller\n \u2193\ndynamic route\n```\n\nConsul puede seguir existiendo para otros usos si hubiera necesidad, pero deja de ser necesario como service registry del hosting web.\n\n---\n\n# 15. Modelo inicial de WordPress workloads\n\nNo se recomienda refactorizar simult\u00e1neamente todo el `wepod`.\n\nPrimera migraci\u00f3n:\n\n```text\nDocker wepod\n \u2193\nKubernetes Pod / StatefulSet\n```\n\nmanteniendo inicialmente:\n\n```text\ns6\nnginx\nphp-fpm\nmysql\n```\n\ncuando sea pr\u00e1ctico.\n\nLa separaci\u00f3n posterior de database/frontend debe tratarse como proyecto independiente.\n\n---\n\n# 16. Egress: dise\u00f1o simple de NAT Gateways\n\nPrimera versi\u00f3n:\n\n```text\ngw100\n LAN: 192.168.88.100\n WAN: PUBLIC_IP_A\n\ngw101\n LAN: 192.168.88.101\n WAN: PUBLIC_IP_B\n\ngw102\n LAN: 192.168.88.102\n WAN: PUBLIC_IP_C\n```\n\nCada gateway ser\u00e1 una VM Ubuntu sencilla.\n\nFunciones:\n\n- routing;\n- forwarding;\n- SNAT/MASQUERADE;\n- logging;\n- health;\n- m\u00e9tricas b\u00e1sicas.\n\nNo necesita initially un controller Kubernetes.\n\n---\n\n# 17. Asignaci\u00f3n de clientes a gateways de salida\n\nModelo:\n\n```text\nsite/customer\n \u2193\negress class\n \u2193\ngateway interno\n```\n\nEjemplo:\n\n```text\nsite-1001 -> 192.168.88.100\nsite-1002 -> 192.168.88.100\nsite-2001 -> 192.168.88.101\ncrawler -> 192.168.88.102\n```\n\nLa implementaci\u00f3n puede empezar con routing/policy routing y evolucionar despu\u00e9s.\n\nLo importante es que sea:\n\n- determinista;\n- observable;\n- f\u00e1cil de cambiar.\n\n---\n\n# 18. Rotaci\u00f3n y cambio de IPs de salida\n\nCaso 1:\n\n```text\ngw100:\nPUBLIC_IP_A\n \u2193\nPUBLIC_IP_B\n```\n\nCaso 2:\n\n```text\ncustomer/site\ngw100\n \u2193\ngw101\n```\n\nLos workloads no deben necesitar conocer la WAN IP.\n\nSolo la pol\u00edtica/ruta de salida.\n\n---\n\n# 19. Separaci\u00f3n de reputaci\u00f3n de IPs\n\nDesde el principio se recomienda separar:\n\n```text\nhosting-general\nhosting-premium\nplatform\nagents/crawlers\n```\n\nEjemplo:\n\n```text\n192.168.88.100\n -> shared hosting\n -> 51.x.x.10\n\n192.168.88.101\n -> premium/dedicated pool\n -> 51.x.x.11\n\n192.168.88.102\n -> agents/crawlers\n -> 51.x.x.12\n```\n\nLos crawlers no deben compartir IP con customer hosting.\n\n---\n\n# 20. Ingress IP dedicada por cliente\n\nNo es prioridad inicial.\n\nEl sistema debe permitir en el futuro:\n\n```text\ncustomer premium\n \u2193\ndedicated public VIP\n \u2193\nIPVS / DSR\n \u2193\nedge\n \u2193\nHTTPRoute\n```\n\nNo requiere redise\u00f1ar el backend Kubernetes.\n\n---\n\n# 21. Integraci\u00f3n con OVH vRack\n\nvRack seguir\u00e1 siendo la red privada base.\n\nEl dise\u00f1o inicial no depende de BGP.\n\nLos bloques Additional IP podr\u00e1n utilizarse para:\n\n- VIPs;\n- IPs de NATGW;\n- futuras IPs dedicadas.\n\nBGP Service de OVH podr\u00e1 evaluarse cuando sea suficientemente maduro para producci\u00f3n.\n\n---\n\n# 22. Proxmox y agregaci\u00f3n de ancho de banda\n\nLa distribuci\u00f3n de edge nodes debe respetar la topolog\u00eda f\u00edsica.\n\nEjemplo:\n\n```text\npve01 -> edge01\npve02 -> edge02\npve03 -> edge03\n```\n\nEvitar:\n\n```text\npve01 -> edge01 + edge02 + edge03\n```\n\ncuando el objetivo es sumar failure/bandwidth domains.\n\n---\n\n# 23. Storage: Ceph\n\nCeph seguir\u00e1 siendo el storage principal.\n\nRKE2 consumir\u00e1 storage mediante CSI.\n\nObjetivos:\n\n- PVC nativo;\n- StorageClasses;\n- snapshots donde sea \u00fatil;\n- evitar dependencias de Docker volumes espec\u00edficos.\n\n---\n\n# 24. Observabilidad y seguridad de red\n\nPrimera capa:\n\n```text\nCilium\n \u2193\nHubble\n```\n\nM\u00e1s adelante:\n\n```text\nHubble flows\n+\nedge HTTP logs\n+\nNATGW logs\n+\nSuricata\n+\nWAF\n```\n\nTodos podr\u00e1n converger en la plataforma de observabilidad.\n\n---\n\n# 25. Suricata y WAF futuros\n\nNo son requisito de bootstrap.\n\n## Suricata\n\n\u00datil para:\n\n- IDS;\n- scans;\n- anomal\u00edas;\n- C2;\n- DNS;\n- metadata de protocolos.\n\nPosibles puntos:\n\n- edge;\n- egress gateway;\n- TAP/mirror.\n\n## WAF\n\nDebe operar despu\u00e9s de TLS termination para ver HTTP.\n\n```text\nInternet\n \u2193\nTLS termination\n \u2193\nWAF\n \u2193\nbackend\n```\n\n---\n\n# 26. Qu\u00e9 NO vamos a desplegar inicialmente\n\nNo se considera necesario de salida:\n\n- Istio;\n- service mesh global;\n- BGP WAN;\n- Cilium BGP externo;\n- controlador propio de egress;\n- IP reputation controller;\n- IPAM p\u00fablico din\u00e1mico;\n- Suricata inline global;\n- WAF sofisticado distribuido;\n- CRDs propios para toda la red.\n\nLa ausencia de estas piezas es deliberada.\n\n---\n\n# 27. Modelo de alta disponibilidad\n\n## Control plane\n\n```text\n3 x RKE2 server\n```\n\n## Ingress\n\n```text\nVRRP\n+\nIPVS\n+\nvarios edge nodes\n+\nDSR\n```\n\n## Egress\n\nVarios gateways:\n\n```text\ngw100\ngw101\ngw102\n```\n\nLa HA de una IP dedicada concreta podr\u00e1 implementarse cuando exista necesidad.\n\nNo es necesario resolver desde el primer d\u00eda todos los casos premium.\n\n---\n\n# 28. Mantenimiento y draining de edge nodes\n\nProcedimiento recomendado:\n\n```text\n1. sacar edge de IPVS\n2. dejar de enviar nuevas conexiones\n3. esperar conexiones existentes\n4. kubectl drain\n5. mantenimiento\n6. kubectl uncordon\n7. validar proxy\n8. reinsertar en IPVS\n```\n\nDespu\u00e9s podr\u00e1 automatizarse.\n\n---\n\n# 29. Modelo de namespaces y roles de nodos\n\nNamespaces orientativos:\n\n```text\nwetopi-edge\nwetopi-hosting\nwetopi-agents\nwetopi-platform\nwetopi-observability\nwetopi-security\n```\n\nRoles de nodo:\n\n```text\ncontrol-plane\nedge\nhosting\nagents\ninfra\n```\n\nSe utilizar\u00e1n:\n\n- labels;\n- taints;\n- affinity;\n- topology spread.\n\n---\n\n# 30. Flujos de tr\u00e1fico de ejemplo\n\n## Entrada HTTP\n\n```text\nClient\n \u2193\nVIP\n \u2193\nIPVS\n \u2193 DSR\nedge02\n \u2193\nTLS\n \u2193\nHTTPRoute\n \u2193\nService\n \u2193\nsite pod\n```\n\nRespuesta:\n\n```text\nedge02\n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25ba Client\n```\n\n## Salida shared\n\n```text\nsite-1001\n \u2193\n192.168.88.100\n \u2193\nSNAT\n \u2193\n51.x.x.10\n \u2193\nInternet\n```\n\n## Salida premium\n\n```text\nsite-2001\n \u2193\n192.168.88.101\n \u2193\nSNAT\n \u2193\n51.x.x.11\n```\n\n## Crawler\n\n```text\nagent-crawler\n \u2193\n192.168.88.102\n \u2193\n51.x.x.12\n```\n\n## Rotaci\u00f3n\n\n```text\ncustomer X\n \u2193\ngw100\n```\n\npasa a:\n\n```text\ncustomer X\n \u2193\ngw101\n```\n\nsin mover el workload.\n\n---\n\n# 31. Migraci\u00f3n progresiva desde Docker Swarm\n\nNo big-bang.\n\nDurante la transici\u00f3n:\n\n```text\nIPVS\n \u251c\u2500\u2500 old WAN -> Swarm\n \u2514\u2500\u2500 new edge -> RKE2\n```\n\npueden coexistir.\n\nMigraci\u00f3n:\n\n```text\nsite 1\nsite 2\nsite 3\n...\n```\n\nde forma progresiva.\n\nIdealmente la VIP p\u00fablica permanece y el routing decide el backend.\n\n---\n\n# 32. Fases de implementaci\u00f3n\n\n## Fase 0 \u2014 Inventario\n\nDocumentar:\n\n- VIPs;\n- IPVS;\n- VRRP;\n- VLANs;\n- vRack;\n- Proxmox;\n- WAN bandwidth domains;\n- NATGW;\n- Ceph.\n\n## Fase 1 \u2014 RKE2\n\n- 3 control planes;\n- workers;\n- embedded etcd;\n- API VIP;\n- GitOps;\n- Ceph CSI.\n\n## Fase 2 \u2014 Cilium/Hubble\n\n- Cilium;\n- Hubble;\n- primeras NetworkPolicies;\n- flow visibility.\n\n## Fase 3 \u2014 Edge\n\n- edge workers;\n- proxy DaemonSet;\n- DSR;\n- health checks;\n- integraci\u00f3n con IPVS.\n\n## Fase 4 \u2014 Gateway API\n\n- Gateway;\n- HTTPRoute;\n- Services;\n- EndpointSlices.\n\n## Fase 5 \u2014 Primeros sites\n\nValidar:\n\n- TLS;\n- IP cliente;\n- headers;\n- storage;\n- health;\n- performance;\n- egress.\n\n## Fase 6 \u2014 NATGW simples\n\nDesplegar:\n\n```text\n192.168.88.100\n192.168.88.101\n192.168.88.102\n```\n\ncon WAN IPs separadas.\n\n## Fase 7 \u2014 Migraci\u00f3n masiva\n\nMover progresivamente el grueso de los workloads.\n\n## Fase 8 \u2014 Observabilidad avanzada\n\nEnriquecimiento de flows y dashboards.\n\n## Fase 9 \u2014 Seguridad avanzada opcional\n\n- Suricata;\n- WAF;\n- egress automation;\n- IP reputation;\n- dedicated IP automation.\n\n---\n\n# 33. Decisiones consolidadas\n\n## NET-001\nRKE2 ser\u00e1 la distribuci\u00f3n Kubernetes.\n\n## NET-002\nCilium ser\u00e1 el CNI inicial.\n\n## NET-003\nHubble ser\u00e1 la base de network observability.\n\n## NET-004\nIPVS + VRRP + DSR se mantienen inicialmente.\n\n## NET-005\nLos edge nodes seguir\u00e1n repartidos por hosts f\u00edsicos Proxmox.\n\n## NET-006\nRegistrator deja de ser necesario.\n\n## NET-007\nConsul deja de ser el registry de websites.\n\n## NET-008\nconsul-template deja de generar la configuraci\u00f3n HTTP de RKE2.\n\n## NET-009\nGateway API ser\u00e1 la abstracci\u00f3n de routing HTTP.\n\n## NET-010\nTraefik o Envoy actuar\u00e1n como proxy edge.\n\n## NET-011\nEl egress inicial usar\u00e1 NATGW Ubuntu simples.\n\n## NET-012\nCada NATGW tendr\u00e1 una IP interna estable.\n\nEjemplo:\n\n```text\n192.168.88.100\n192.168.88.101\n192.168.88.102\n```\n\n## NET-013\nLa WAN IP asociada a un gateway debe ser f\u00e1cil de cambiar.\n\n## NET-014\nAgents/crawlers no compartir\u00e1n IP de salida con customers.\n\n## NET-015\nSe podr\u00e1 asignar un cliente a un gateway espec\u00edfico.\n\n## NET-016\nNo se desplegar\u00e1 Istio inicialmente.\n\n## NET-017\nEl bootstrap no depender\u00e1 de BGP.\n\n## NET-018\nNo se construir\u00e1 inicialmente un egress control plane complejo si gateways est\u00e1ticos resuelven el problema.\n\n---\n\n# 34. Riesgos principales y mitigaciones\n\n## Cambio simult\u00e1neo excesivo\n\nMitigaci\u00f3n:\n\n- migraci\u00f3n por capas;\n- conservar IPVS;\n- conservar inicialmente wepod.\n\n## P\u00e9rdida accidental de DSR\n\nMitigaci\u00f3n:\n\n- edge nodes dedicados;\n- pruebas expl\u00edcitas de dataplane.\n\n## Mala distribuci\u00f3n f\u00edsica\n\nMitigaci\u00f3n:\n\n- labels de Proxmox host;\n- anti-affinity;\n- topology awareness.\n\n## NATGW como punto de fallo\n\nMitigaci\u00f3n:\n\n- varios gateways;\n- reasignaci\u00f3n sencilla;\n- HA premium cuando se requiera.\n\n## Reputaci\u00f3n compartida\n\nMitigaci\u00f3n:\n\n- separar pools desde el principio.\n\n## Overengineering\n\nMitigaci\u00f3n:\n\n- no desplegar capacidades avanzadas sin necesidad operacional concreta.\n\n---\n\n# 35. Evoluci\u00f3n futura\n\nUna vez estable la base se podr\u00e1 evaluar:\n\n## Cilium Egress Gateway\n\nPara seleccionar egress por labels de Kubernetes.\n\n## Egress pools\n\n```text\nshared\npremium\ncrawler\ndedicated\n```\n\n## Reputation monitoring\n\nMedici\u00f3n y rotaci\u00f3n de IPs.\n\n## Suricata\n\nIDS/IPS de red.\n\n## WAF\n\nInspecci\u00f3n HTTP tras TLS.\n\n## BGP\n\nCuando OVH ofrezca una opci\u00f3n madura y productiva.\n\n## Eliminaci\u00f3n de IPVS\n\nSolo si una alternativa BGP/ECMP conserva:\n\n- HA;\n- DSR/equivalente;\n- capacidad;\n- simplicidad.\n\n## WordPressSite CRD\n\nEn una fase posterior cada website podr\u00eda representarse como:\n\n```yaml\napiVersion: wetopi.io/v1\nkind: WordPressSite\n```\n\ny generar autom\u00e1ticamente:\n\n- workload;\n- Service;\n- HTTPRoute;\n- PVC;\n- monitoring;\n- policies.\n\nNo es requisito de la primera migraci\u00f3n.\n\n---\n\n# 36. Arquitectura objetivo resumida\n\n```text\n OVH / vRack\n \u2502\n Proxmox + Ceph\n \u2502\n RKE2\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502 \u2502\n Control Plane Edge Hosting\n \u2502 \u2502 \u2502\n \u2502 Cilium \u2502\n \u2502 Hubble \u2502\n \u2502 \u2502 \u2502\n \u2502 Gateway API \u2502\n \u2502 \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n Websites\n```\n\nINGRESS:\n\n```text\nInternet\n \u2193\nVIP / VRRP\n \u2193\nIPVS\n \u2193 DSR\nRKE2 Edge\n \u2193\nTLS / HTTP\n \u2193\nGateway API\n \u2193\nWebsite\n```\n\nEGRESS:\n\n```text\nWorkload\n \u2193\nselected internal GW\n\n \u251c\u2500\u2500 192.168.88.100 -> WAN IP A\n \u251c\u2500\u2500 192.168.88.101 -> WAN IP B\n \u2514\u2500\u2500 192.168.88.102 -> WAN IP C\n\n \u2193\nInternet\n```\n\nOBSERVABILITY:\n\n```text\nCilium\n \u2193\nHubble\n \u2193\nflows\n \u2193\nfuture security analytics\n```\n\nFUTURO:\n\n```text\nSuricata\nWAF\nreputation monitoring\nautomated egress\nBGP\ndedicated IP automation\n```\n\n---\n\n# Conclusi\u00f3n\n\nEl dise\u00f1o inicial de la nueva Wetopi debe optimizar robustez, visibilidad y capacidad de evoluci\u00f3n, no complejidad.\n\nLa base acordada es:\n\n```text\nRKE2\n+\nCilium/Hubble\n+\nIPVS/VRRP/DSR existente\n+\nGateway API\n+\nedge proxies Kubernetes-native\n+\nServices/EndpointSlices\n+\nNATGW Ubuntu simples\n```\n\nEste modelo conserva las propiedades valiosas de la plataforma actual:\n\n- DSR;\n- retorno directo;\n- capacidad WAN distribuida;\n- IPs movibles;\n- separaci\u00f3n de egress;\n\ny elimina progresivamente las piezas que Kubernetes puede resolver mejor:\n\n- Docker Swarm;\n- Registrator;\n- Consul como service registry web;\n- consul-template;\n- regeneraci\u00f3n de nginx.conf.\n\nSobre esta foundation se podr\u00e1n a\u00f1adir m\u00e1s adelante Cilium Egress Gateway, Suricata, WAF, reputaci\u00f3n autom\u00e1tica, BGP e IPs dedicadas gestionadas sin condicionar ni complicar el primer despliegue.", "html": "\n\nMigrado desde
\nwetopi-platform/docs@main(design/network-hosting-architecture.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
Fecha: 28 de agosto de 2026
\nEstado: Dise\u00f1o base consolidado
\n\u00c1mbito: Red, ingress, egress, hosting WordPress y transici\u00f3n desde Docker Swarm hacia RKE2 sobre Proxmox + Ceph.
Wetopi es un Managed WordPress Hosting ejecutado sobre un cloud privado basado en Proxmox + Ceph, con Docker Swarm como scheduler actual y del orden de 700 contenedores.
\nLa arquitectura WAN existente tiene dos propiedades valiosas que se preservar\u00e1n inicialmente:
\nDSR no solo proporciona eficiencia y HA: permite que las respuestas salgan directamente por distintos nodos WAN y, por tanto, aprovechar la capacidad de salida de m\u00faltiples hosts f\u00edsicos de OVH sin hacer retornar todo el tr\u00e1fico por un \u00fanico balanceador.
\nLa nueva plataforma se construir\u00e1 sobre:
\nNo se intentar\u00e1 transformar toda la red a la vez.
\nLa primera generaci\u00f3n conservar\u00e1 IPVS + VRRP + DSR en la capa WAN y sustituir\u00e1, detr\u00e1s de ella:
\nEl egress tambi\u00e9n se mantendr\u00e1 deliberadamente simple.
\nSe desplegar\u00e1n varios NAT Gateways Ubuntu con IP interna estable y una IP p\u00fablica asociada:
\n192.168.88.100 -> WAN IP A\n192.168.88.101 -> WAN IP B\n192.168.88.102 -> WAN IP C\nDiferentes clientes, sites o clases de workload podr\u00e1n usar gateways distintos.
\nEsto permite:
\nCilium se adopta como foundation de networking y observabilidad, pero no se requiere utilizar desde el primer d\u00eda BGP, Egress Gateway avanzado, CRDs propios o automatizaci\u00f3n compleja.
\nLa filosof\u00eda es:
\n\n\nconstruir una base moderna, observable y extensible sin sobrecomplicar la primera versi\u00f3n.
\n
La plataforma actual se ejecuta sobre:
\nOVH\n \u2193\nProxmox\n \u2193\nCeph\n \u2193\nDocker Swarm\nEl workload principal son websites WordPress.
\nCada website se representa aproximadamente mediante un wepod, que contiene actualmente:
s6\n\u251c\u2500\u2500 nginx\n\u251c\u2500\u2500 php-fpm\n\u2514\u2500\u2500 mysql\nLa entrada WAN actual es aproximadamente:
\nInternet\n \u2193\nPublic VIP\n \u2193\nVRRP / Keepalived\n \u2193\nIPVS directors\n \u2193\nDSR\n \u2193\nWAN Nginx proxies\n \u2193\nDocker workloads\nEl descubrimiento y configuraci\u00f3n se realiza mediante:
\nDocker\n \u2193\nRegistrator\n \u2193\nConsul\n \u2193\nconsul-template\n \u2193\nnginx.conf\n \u2193\nreload\nEste modelo ha sido funcional, pero introduce piezas que Kubernetes puede sustituir de forma nativa.
\nLa nueva red debe:
\nIPVS + VRRP + DSR seguir\u00e1 siendo inicialmente la capa L3/L4 WAN.
\nKubernetes ya proporciona:
\nNo se recrear\u00e1 Consul dentro de Kubernetes para registrar websites.
\nL3/L4 WAN -> IPVS / DSR\nHTTP/TLS -> Gateway / proxy\nCluster networking -> Cilium\nFlow visibility -> Hubble\nDeep IDS futuro -> Suricata\nHTTP security -> WAF futuro\nPrimera versi\u00f3n:
\nUbuntu gateway\n+\nrouting\n+\nnftables/iptables\n+\nWAN IP\nLa base debe permitir a\u00f1adir capacidades avanzadas despu\u00e9s, no obligar a desplegarlas ya.
\nSe selecciona RKE2 como distribuci\u00f3n Kubernetes principal.
\nRazones:
\nRancher UI no es requisito.
\n INTERNET\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502\n VIP A VIP B\n \u2502 \u2502\n IPVS IPVS\n \u2502 \u2502\n DSR DSR\n \u2502 \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 RKE2 EDGE A \u2502 \u2502 RKE2 EDGE B \u2502\n \u2502 proxy HTTP/TLS \u2502 \u2502 proxy HTTP/TLS \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n Gateway API\n \u2502\n Service / EndpointSlice\n \u2502\n Pods\n \u2502\n Ceph\n\nEGRESS:\n\nPods / Sites / Agents\n \u2502\n \u25bc\ninternal gateway selection\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc \u25bc \u25bc\n.100 .101 .102\n GW A GW B GW C\n \u2502 \u2502 \u2502\nWAN A WAN B WAN C\n \u2502 \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500 Internet \u2500\u2518\nBaseline:
\n3 x RKE2 server nodes\n+\nembedded etcd\nModelo:
\nAPI stable endpoint / VIP\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc \u25bc \u25bc\ncp01 cp02 cp03\netcd etcd etcd\nLos control planes deben quedar separados de los workloads de customers.
\nSe prioriza:
\nSe definir\u00e1n roles expl\u00edcitos:
\nwetopi.io/role=edge\nwetopi.io/role=hosting\nwetopi.io/role=agents\nwetopi.io/role=database\nwetopi.io/role=infra\nAdem\u00e1s, cuando sea \u00fatil:
\nwetopi.io/proxmox-host=pve01\nwetopi.io/bandwidth-domain=pve01\nwetopi.io/wan-zone=a\nEl host Proxmox no es solo un failure domain: tambi\u00e9n puede ser un bandwidth domain.
\nSe selecciona Cilium como CNI.
\nObjetivos iniciales:
\nNo es requisito inicial:
\nHubble ser\u00e1 la primera capa de visibilidad de flows.
\nPermitir\u00e1 responder preguntas como:
\nEl objetivo a medio plazo es correlacionar:
\npod\nnamespace\nsite\ncustomer\nnode\ngateway\negress IP\ndestination\nSe conserva el patr\u00f3n:
\nInternet\n \u2193\nVIP\n \u2193\nVRRP\n \u2193\nIPVS\n \u2193\nDSR\n \u2193\nedge node\nCon DSR:
\nrequest:\nclient -> IPVS -> edge\n\nresponse:\nedge -> client\nLa respuesta no vuelve por el director.
\nEsto preserva la capacidad de salida distribuida entre hosts f\u00edsicos OVH.
\nLos antiguos WAN proxies pasar\u00e1n progresivamente a edge workers RKE2.
\nEjemplo:
\nedge-a01\nedge-a02\nedge-b01\nedge-b02\nCada edge VM debe estar distribuida deliberadamente entre hosts Proxmox distintos.
\nLabels ejemplo:
\nwetopi.io/role=edge\nwetopi.io/wan-zone=a\nwetopi.io/proxmox-host=pve01\nTaint recomendado:
\nwetopi.io/edge=true:NoSchedule\nEl edge seguir\u00e1 terminando TLS.
\nCandidatos:
\nLa arquitectura no debe depender todav\u00eda de la elecci\u00f3n final.
\nResponsabilidades:
\nRouting:
\nhostname\n \u2193\nHTTPRoute\n \u2193\nService\n \u2193\nEndpointSlice\n \u2193\nPod\nANTES:
\ncontainer\n \u2193\nRegistrator\n \u2193\nConsul\n \u2193\nconsul-template\n \u2193\nnginx.conf\n \u2193\nreload\nNUEVO:
\nPod\n \u2193\nService selector\n \u2193\nEndpointSlice\n \u2193\nGateway controller\n \u2193\ndynamic route\nConsul puede seguir existiendo para otros usos si hubiera necesidad, pero deja de ser necesario como service registry del hosting web.
\nNo se recomienda refactorizar simult\u00e1neamente todo el wepod.
Primera migraci\u00f3n:
\nDocker wepod\n \u2193\nKubernetes Pod / StatefulSet\nmanteniendo inicialmente:
\ns6\nnginx\nphp-fpm\nmysql\ncuando sea pr\u00e1ctico.
\nLa separaci\u00f3n posterior de database/frontend debe tratarse como proyecto independiente.
\nPrimera versi\u00f3n:
\ngw100\n LAN: 192.168.88.100\n WAN: PUBLIC_IP_A\n\ngw101\n LAN: 192.168.88.101\n WAN: PUBLIC_IP_B\n\ngw102\n LAN: 192.168.88.102\n WAN: PUBLIC_IP_C\nCada gateway ser\u00e1 una VM Ubuntu sencilla.
\nFunciones:
\nNo necesita initially un controller Kubernetes.
\nModelo:
\nsite/customer\n \u2193\negress class\n \u2193\ngateway interno\nEjemplo:
\nsite-1001 -> 192.168.88.100\nsite-1002 -> 192.168.88.100\nsite-2001 -> 192.168.88.101\ncrawler -> 192.168.88.102\nLa implementaci\u00f3n puede empezar con routing/policy routing y evolucionar despu\u00e9s.
\nLo importante es que sea:
\nCaso 1:
\ngw100:\nPUBLIC_IP_A\n \u2193\nPUBLIC_IP_B\nCaso 2:
\ncustomer/site\ngw100\n \u2193\ngw101\nLos workloads no deben necesitar conocer la WAN IP.
\nSolo la pol\u00edtica/ruta de salida.
\nDesde el principio se recomienda separar:
\nhosting-general\nhosting-premium\nplatform\nagents/crawlers\nEjemplo:
\n192.168.88.100\n -> shared hosting\n -> 51.x.x.10\n\n192.168.88.101\n -> premium/dedicated pool\n -> 51.x.x.11\n\n192.168.88.102\n -> agents/crawlers\n -> 51.x.x.12\nLos crawlers no deben compartir IP con customer hosting.
\nNo es prioridad inicial.
\nEl sistema debe permitir en el futuro:
\ncustomer premium\n \u2193\ndedicated public VIP\n \u2193\nIPVS / DSR\n \u2193\nedge\n \u2193\nHTTPRoute\nNo requiere redise\u00f1ar el backend Kubernetes.
\nvRack seguir\u00e1 siendo la red privada base.
\nEl dise\u00f1o inicial no depende de BGP.
\nLos bloques Additional IP podr\u00e1n utilizarse para:
\nBGP Service de OVH podr\u00e1 evaluarse cuando sea suficientemente maduro para producci\u00f3n.
\nLa distribuci\u00f3n de edge nodes debe respetar la topolog\u00eda f\u00edsica.
\nEjemplo:
\npve01 -> edge01\npve02 -> edge02\npve03 -> edge03\nEvitar:
\npve01 -> edge01 + edge02 + edge03\ncuando el objetivo es sumar failure/bandwidth domains.
\nCeph seguir\u00e1 siendo el storage principal.
\nRKE2 consumir\u00e1 storage mediante CSI.
\nObjetivos:
\nPrimera capa:
\nCilium\n \u2193\nHubble\nM\u00e1s adelante:
\nHubble flows\n+\nedge HTTP logs\n+\nNATGW logs\n+\nSuricata\n+\nWAF\nTodos podr\u00e1n converger en la plataforma de observabilidad.
\nNo son requisito de bootstrap.
\n\u00datil para:
\nPosibles puntos:
\nDebe operar despu\u00e9s de TLS termination para ver HTTP.
\nInternet\n \u2193\nTLS termination\n \u2193\nWAF\n \u2193\nbackend\nNo se considera necesario de salida:
\nLa ausencia de estas piezas es deliberada.
\n3 x RKE2 server\nVRRP\n+\nIPVS\n+\nvarios edge nodes\n+\nDSR\nVarios gateways:
\ngw100\ngw101\ngw102\nLa HA de una IP dedicada concreta podr\u00e1 implementarse cuando exista necesidad.
\nNo es necesario resolver desde el primer d\u00eda todos los casos premium.
\nProcedimiento recomendado:
\n1. sacar edge de IPVS\n2. dejar de enviar nuevas conexiones\n3. esperar conexiones existentes\n4. kubectl drain\n5. mantenimiento\n6. kubectl uncordon\n7. validar proxy\n8. reinsertar en IPVS\nDespu\u00e9s podr\u00e1 automatizarse.
\nNamespaces orientativos:
\nwetopi-edge\nwetopi-hosting\nwetopi-agents\nwetopi-platform\nwetopi-observability\nwetopi-security\nRoles de nodo:
\ncontrol-plane\nedge\nhosting\nagents\ninfra\nSe utilizar\u00e1n:
\nClient\n \u2193\nVIP\n \u2193\nIPVS\n \u2193 DSR\nedge02\n \u2193\nTLS\n \u2193\nHTTPRoute\n \u2193\nService\n \u2193\nsite pod\nRespuesta:
\nedge02\n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25ba Client\nsite-1001\n \u2193\n192.168.88.100\n \u2193\nSNAT\n \u2193\n51.x.x.10\n \u2193\nInternet\nsite-2001\n \u2193\n192.168.88.101\n \u2193\nSNAT\n \u2193\n51.x.x.11\nagent-crawler\n \u2193\n192.168.88.102\n \u2193\n51.x.x.12\ncustomer X\n \u2193\ngw100\npasa a:
\ncustomer X\n \u2193\ngw101\nsin mover el workload.
\nNo big-bang.
\nDurante la transici\u00f3n:
\nIPVS\n \u251c\u2500\u2500 old WAN -> Swarm\n \u2514\u2500\u2500 new edge -> RKE2\npueden coexistir.
\nMigraci\u00f3n:
\nsite 1\nsite 2\nsite 3\n...\nde forma progresiva.
\nIdealmente la VIP p\u00fablica permanece y el routing decide el backend.
\nDocumentar:
\nValidar:
\nDesplegar:
\n192.168.88.100\n192.168.88.101\n192.168.88.102\ncon WAN IPs separadas.
\nMover progresivamente el grueso de los workloads.
\nEnriquecimiento de flows y dashboards.
\nRKE2 ser\u00e1 la distribuci\u00f3n Kubernetes.
\nCilium ser\u00e1 el CNI inicial.
\nHubble ser\u00e1 la base de network observability.
\nIPVS + VRRP + DSR se mantienen inicialmente.
\nLos edge nodes seguir\u00e1n repartidos por hosts f\u00edsicos Proxmox.
\nRegistrator deja de ser necesario.
\nConsul deja de ser el registry de websites.
\nconsul-template deja de generar la configuraci\u00f3n HTTP de RKE2.
\nGateway API ser\u00e1 la abstracci\u00f3n de routing HTTP.
\nTraefik o Envoy actuar\u00e1n como proxy edge.
\nEl egress inicial usar\u00e1 NATGW Ubuntu simples.
\nCada NATGW tendr\u00e1 una IP interna estable.
\nEjemplo:
\n192.168.88.100\n192.168.88.101\n192.168.88.102\nLa WAN IP asociada a un gateway debe ser f\u00e1cil de cambiar.
\nAgents/crawlers no compartir\u00e1n IP de salida con customers.
\nSe podr\u00e1 asignar un cliente a un gateway espec\u00edfico.
\nNo se desplegar\u00e1 Istio inicialmente.
\nEl bootstrap no depender\u00e1 de BGP.
\nNo se construir\u00e1 inicialmente un egress control plane complejo si gateways est\u00e1ticos resuelven el problema.
\nMitigaci\u00f3n:
\nMitigaci\u00f3n:
\nMitigaci\u00f3n:
\nMitigaci\u00f3n:
\nMitigaci\u00f3n:
\nMitigaci\u00f3n:
\nUna vez estable la base se podr\u00e1 evaluar:
\nPara seleccionar egress por labels de Kubernetes.
\nshared\npremium\ncrawler\ndedicated\nMedici\u00f3n y rotaci\u00f3n de IPs.
\nIDS/IPS de red.
\nInspecci\u00f3n HTTP tras TLS.
\nCuando OVH ofrezca una opci\u00f3n madura y productiva.
\nSolo si una alternativa BGP/ECMP conserva:
\nEn una fase posterior cada website podr\u00eda representarse como:
\napiVersion: wetopi.io/v1\nkind: WordPressSite\ny generar autom\u00e1ticamente:
\nNo es requisito de la primera migraci\u00f3n.
\n OVH / vRack\n \u2502\n Proxmox + Ceph\n \u2502\n RKE2\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502 \u2502\n Control Plane Edge Hosting\n \u2502 \u2502 \u2502\n \u2502 Cilium \u2502\n \u2502 Hubble \u2502\n \u2502 \u2502 \u2502\n \u2502 Gateway API \u2502\n \u2502 \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n Websites\nINGRESS:
\nInternet\n \u2193\nVIP / VRRP\n \u2193\nIPVS\n \u2193 DSR\nRKE2 Edge\n \u2193\nTLS / HTTP\n \u2193\nGateway API\n \u2193\nWebsite\nEGRESS:
\nWorkload\n \u2193\nselected internal GW\n\n \u251c\u2500\u2500 192.168.88.100 -> WAN IP A\n \u251c\u2500\u2500 192.168.88.101 -> WAN IP B\n \u2514\u2500\u2500 192.168.88.102 -> WAN IP C\n\n \u2193\nInternet\nOBSERVABILITY:
\nCilium\n \u2193\nHubble\n \u2193\nflows\n \u2193\nfuture security analytics\nFUTURO:
\nSuricata\nWAF\nreputation monitoring\nautomated egress\nBGP\ndedicated IP automation\nEl dise\u00f1o inicial de la nueva Wetopi debe optimizar robustez, visibilidad y capacidad de evoluci\u00f3n, no complejidad.
\nLa base acordada es:
\nRKE2\n+\nCilium/Hubble\n+\nIPVS/VRRP/DSR existente\n+\nGateway API\n+\nedge proxies Kubernetes-native\n+\nServices/EndpointSlices\n+\nNATGW Ubuntu simples\nEste modelo conserva las propiedades valiosas de la plataforma actual:
\ny elimina progresivamente las piezas que Kubernetes puede resolver mejor:
\nSobre esta foundation se podr\u00e1n a\u00f1adir m\u00e1s adelante Cilium Egress Gateway, Suricata, WAF, reputaci\u00f3n autom\u00e1tica, BGP e IPs dedicadas gestionadas sin condicionar ni complicar el primer despliegue.
\n" }, { "id": 59, "title": "RUN-TOFU-STATE-LOSS \u2014 reimport del estado prod", "slug": "run-tofu-state-loss-reimport-del-estado-prod", "book_id": 6, "chapter_id": 22, "description": null, "created_at": "2026-09-06T13:39:03.000000Z", "updated_at": "2026-09-06T21:45:55.000000Z", "markdown": "> Migrado desde `wetopi-platform/docs@main` (operations/runbooks/tofu-prod-state-loss-and-reimport.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.\n\n OpenTofu Proxmox PROD state loss and re-import recovery\n\n## Purpose\n\nRecover full OpenTofu control of the 8 Wetopi Proxmox PROD VMs after the\ntfstate is lost, WITHOUT touching the hypervisors: the import path is\nread-only against Proxmox (GET API calls + `tofu import` only \u2014 never an\napply), and the acceptance gate proves the recovered state matches live\nreality exactly.\n\nThe single executed instance of this runbook is task t_690e108f\n(wetopi-senior, 2026-08-30): 28 resources imported, `tofu plan` = No changes\n(0 add / 0 change / 0 destroy), verified twice. Evidence paths and the\npatch scripts it produced are listed under Evidence.\n\n## Trigger / symptoms\n\n- `tofu plan` for `tofu/proxmox/environments/prod` wants to CREATE all 28\n resources (empty or absent `terraform.tfstate`), or errors with\n \"state data does not exist\" / no state backend configured.\n- Nobody can answer \"which tfstate describes PROD?\" \u2014 the controller\n location is unknown or its state file is missing/corrupt.\n\n## Impact\n\nWithout state, IaC has no claim on the live infrastructure: any speculative\n`tofu apply` would try to create duplicate VMs, template, network devices\nand storage. The RKE2 apply runbook (PROC-RKE2-APPLY) precondition \"render\nansible inventory from tofu output\" is blocked. Blast radius is contained as\nlong as no apply runs: Proxmox itself is unaffected.\n\n## Preconditions\n\n- SSH key `/opt/data/.ssh/bigardo` with access to the ops01 bastion\n (`ubuntu@92.222.245.91`) \u2014 the secrets source.\n- `tofu` on PATH (controller: `/usr/local/bin/tofu`).\n- NO other operator is concurrently running tofu against PROD (single-writer\n discipline; the state is local, not locked by a remote backend).\n- Read access to the infrastructure repo at current `main`.\n\n### Phase-1 tofu controller (PERMANENT location)\n\nThe phase-1 controller is this harness VM \u2014 NOT ops01 (operator decision\n2026-08-30). State root:\n\n`/opt/data/kanban/workspaces/t_690e108f/infrastructure/tofu/proxmox/environments/prod/`\n\nController artifacts (all mode 0600, all gitignored via `.gitignore`):\n\n| Artifact | Role |\n|---|---|\n| `terraform.tfstate` | recovered state \u2014 28 instances |\n| `terraform.tfvars` | regenerated from the live Proxmox API (never commit; `*.tfvars` is gitignored) |\n| provider env vars | session-only, pulled over SSH from ops01 (below) \u2014 never written to harness disk |\n\n## Concurrency model (phase 1)\n\nThe phase-1 backend is the LOCAL state file `terraform.tfstate` \u2014 it has NO\nlocking of any kind. Two concurrent tofu operations (even two `plan` runs)\ncan corrupt it. Therefore **only ONE tofu operation runs at a time**,\nperiod.\n\nEnforcement is BY CONVENTION, not by a technical gate:\n\n- All tofu-touching tasks are routed through the `wetopi-ops` lane.\n- Serialization is intended to be enforced by the dispatcher via\n `max_in_progress_per_profile = 1` for tofu-touching tasks; the CURRENT\n configured value in this harness is `3` (not yet 1), so today the ACTUAL\n enforcement is that the **lead dispatches tofu tasks one-at-a-time by\n design** \u2014 never two tofu-touching tasks `running` simultaneously.\n- Future hard gate: a per-board lock (only one tofu task claimable at a\n time) \u2014 DEFERRED. Until it exists, the one-at-a-time dispatch convention\n above is load-bearing.\n\nDeferred backend transition:\n\n> Remote backend (SOPS-encrypted or Infisical-backed) arrives with the\n> secrets phase \u2014 until then phase-1 local state with the serialization\n> convention above.\n\n## Safety checks\n\n1. Confirm you are in the phase-1 controller directory and the state file\n is present, mode 0600, and NOT tracked by git\n (`git check-ignore terraform.tfstate` \u2192 ignored).\n2. Confirm no credentials are on disk in the controller tree: tokens are\n sourced over SSH per session only (see Procedure step 2).\n3. Read-only recon of the live Proxmox API (GET only) to know what will be\n imported before importing: `/nodes`, `/nodes/\n\nMigrado desde
\nwetopi-platform/docs@main(operations/runbooks/tofu-prod-state-loss-and-reimport.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
OpenTofu Proxmox PROD state loss and re-import recovery
\nRecover full OpenTofu control of the 8 Wetopi Proxmox PROD VMs after the\ntfstate is lost, WITHOUT touching the hypervisors: the import path is\nread-only against Proxmox (GET API calls + tofu import only \u2014 never an\napply), and the acceptance gate proves the recovered state matches live\nreality exactly.
The single executed instance of this runbook is task t_690e108f\n(wetopi-senior, 2026-08-30): 28 resources imported, tofu plan = No changes\n(0 add / 0 change / 0 destroy), verified twice. Evidence paths and the\npatch scripts it produced are listed under Evidence.
tofu plan for tofu/proxmox/environments/prod wants to CREATE all 28\nresources (empty or absent terraform.tfstate), or errors with\n\"state data does not exist\" / no state backend configured.Without state, IaC has no claim on the live infrastructure: any speculative\ntofu apply would try to create duplicate VMs, template, network devices\nand storage. The RKE2 apply runbook (PROC-RKE2-APPLY) precondition \"render\nansible inventory from tofu output\" is blocked. Blast radius is contained as\nlong as no apply runs: Proxmox itself is unaffected.
/opt/data/.ssh/bigardo with access to the ops01 bastion\n(ubuntu@92.222.245.91) \u2014 the secrets source.tofu on PATH (controller: /usr/local/bin/tofu).main.The phase-1 controller is this harness VM \u2014 NOT ops01 (operator decision\n2026-08-30). State root:
\n/opt/data/kanban/workspaces/t_690e108f/infrastructure/tofu/proxmox/environments/prod/
Controller artifacts (all mode 0600, all gitignored via .gitignore):
| Artifact | \nRole | \n
|---|---|
terraform.tfstate | \nrecovered state \u2014 28 instances | \n
terraform.tfvars | \nregenerated from the live Proxmox API (never commit; *.tfvars is gitignored) | \n
| provider env vars | \nsession-only, pulled over SSH from ops01 (below) \u2014 never written to harness disk | \n
The phase-1 backend is the LOCAL state file terraform.tfstate \u2014 it has NO\nlocking of any kind. Two concurrent tofu operations (even two plan runs)\ncan corrupt it. Therefore only ONE tofu operation runs at a time,\nperiod.
Enforcement is BY CONVENTION, not by a technical gate:
\nwetopi-ops lane.max_in_progress_per_profile = 1 for tofu-touching tasks; the CURRENT\nconfigured value in this harness is 3 (not yet 1), so today the ACTUAL\nenforcement is that the lead dispatches tofu tasks one-at-a-time by\ndesign \u2014 never two tofu-touching tasks running simultaneously.Deferred backend transition:
\n\n\nRemote backend (SOPS-encrypted or Infisical-backed) arrives with the\nsecrets phase \u2014 until then phase-1 local state with the serialization\nconvention above.
\n
git check-ignore terraform.tfstate \u2192 ignored)./nodes, /nodes/<n>/qemu,\n/nodes/<n>/qemu/<id>/config, /nodes/<n>/network, /storage,\n/nodes/<n>/storage/<ds>/content.Clone infrastructure at current main and enter the environment:\ngit clone https://github.com/wetopi-platform/infrastructure,\ncd tofu/proxmox/environments/prod.
Source provider tokens (session-only, never on harness disk) and export:
\neval \"$(ssh -i /opt/data/.ssh/bigardo ubuntu@92.222.245.91 \\\n 'grep -E \"^PROXMOX\" /home/ubuntu/secrets/.env.proxmox; \\\n grep -E \"CLOUDFLARE\" /home/ubuntu/secrets/.cloudflare-wetopi-net.env')\"\nexport PROXMOX_VE_ENDPOINT PROXMOX_VE_API_TOKEN PROXMOX_VE_INSECURE=true CLOUDFLARE_API_TOKEN\nPROXMOX_VE_INSECURE=true is REQUIRED when running from the harness VM\n(untrusted PVE certificate). The ops01 copy of the secret says false\nbut ops01 trusts the cert \u2014 on the harness it must be true.
Regenerate terraform.tfvars from the LIVE API data (shape:\nterraform.tfvars.example; every VM field was verifiable 1:1 from\n/nodes/<n>/qemu/<id>/config). Mode 0600.
tofu init, then import the live resources (read-only vs Proxmox) with\nthe bpg/proxmox 0.111.1 import ID formats:
| Resource class | \nImport ID format | \nExample | \n
|---|---|---|
VM (module.vm) | \n<node>/<vmid> | \nns3249468/8001 | \n
network vlan/bridge (proxmox_virtual_environment_network) | \n<node>:<iface> | \nns3235424:vlan_rke | \n
| download file (ISO) | \n<node>/<datastore>:<content>/<file> | \nns3243421/local:iso/ubuntu-24.04-20260826-server-cloudimg-amd64.img | \n
| firewall options | \nvm/<node>/<vmid> | \nvm/ns3249468/8001 | \n
storage (proxmox_storage_directory) | \n<storage-id> | \nlocal | \n
The full 28-resource inventory with per-import return codes is in\nEvidence. Declared-but-not-applied resources are NOT\nimported: edge VMs 8041/8042 (F1 build-only, create_vms=false) and the\nCloudflare records edge-test/docs-new (F3/F5 pending \u2014\ncloudflare_dns_records unset in tfvars).
Exact resource addresses (the 28 imports, verbatim from the executed\nrun \u2014 tofu import <ADDR> <IMPORT-ID>):
Address derivation (module source, quoted):
\nmodule \"vm\" \u2014 tofu/proxmox/environments/prod/main.tf lines 93-97:\nmodule \"vm\" { for_each = { for name, vm in var.vms : name => vm if try(var.create_vms[name], false) } \u2026 source = \"../../modules/vm\" };\nthe child resource is\ntofu/proxmox/modules/vm/main.tf line 1:\nresource \"proxmox_virtual_environment_vm\" \"this\" \u2192 per-VM address\nmodule.vm[\"<vm-name>\"].proxmox_virtual_environment_vm.this.module \"platform_network\" \u2014 main.tf lines 1-3:\nmodule \"platform_network\" { for_each = var.manage_platform_network ? var.proxmox_nodes : {} \u2026 source = \"../../modules/node-vlan\" }.proxmox_virtual_environment_firewall_options.bastion \u2014\ntofu/proxmox/environments/prod/bastion.tf lines 9-10:\nresource \"proxmox_virtual_environment_firewall_options\" \"bastion\" { for_each = local.bastion_vms.var.vms (the 8 live VM names); node placement\n(ns*) was read live from the Proxmox API during the run.Per-VM imports (8) \u2014 one per live VM, key = name in var.vms:
tofu import 'module.vm[\"wetopi-ops01\"].proxmox_virtual_environment_vm.this' ns3249468/8001\ntofu import 'module.vm[\"rke-cp01\"].proxmox_virtual_environment_vm.this' ns3243421/8011\ntofu import 'module.vm[\"rke-cp02\"].proxmox_virtual_environment_vm.this' ns3226939/8012\ntofu import 'module.vm[\"rke-cp03\"].proxmox_virtual_environment_vm.this' ns3214108/8013\ntofu import 'module.vm[\"rke-platform01\"].proxmox_virtual_environment_vm.this' ns31826929/8021\ntofu import 'module.vm[\"rke-platform02\"].proxmox_virtual_environment_vm.this' ns3226931/8022\ntofu import 'module.vm[\"rke-platform03\"].proxmox_virtual_environment_vm.this' ns31826925/8023\ntofu import 'module.vm[\"wetopi-natgw01\"].proxmox_virtual_environment_vm.this' ns3243421/8031\nNon-VM imports (20) \u2014 template, ISO download, firewall options, storage,\nand the 16 per-node network devices:
\ntofu import 'proxmox_virtual_environment_vm.ubuntu_noble_template[0]' ns3243421/9000\ntofu import 'proxmox_download_file.ubuntu_noble[0]' ns3243421/local:iso/ubuntu-24.04-20260826-server-cloudimg-amd64.img\ntofu import 'proxmox_virtual_environment_firewall_options.bastion[\"wetopi-ops01\"]' vm/ns3249468/8001\ntofu import 'proxmox_storage_directory.local' local\ntofu import 'module.platform_network[\"ns31826925\"].proxmox_network_linux_bridge.platform' ns31826925:vmbr4\ntofu import 'module.platform_network[\"ns31826925\"].proxmox_network_linux_vlan.platform' ns31826925:vlan_rke\ntofu import 'module.platform_network[\"ns31826929\"].proxmox_network_linux_bridge.platform' ns31826929:vmbr4\ntofu import 'module.platform_network[\"ns31826929\"].proxmox_network_linux_vlan.platform' ns31826929:vlan_rke\ntofu import 'module.platform_network[\"ns3214108\"].proxmox_network_linux_bridge.platform' ns3214108:vmbr4\ntofu import 'module.platform_network[\"ns3214108\"].proxmox_network_linux_vlan.platform' ns3214108:vlan_rke\ntofu import 'module.platform_network[\"ns3226931\"].proxmox_network_linux_bridge.platform' ns3226931:vmbr4\ntofu import 'module.platform_network[\"ns3226931\"].proxmox_network_linux_vlan.platform' ns3226931:vlan_rke\ntofu import 'module.platform_network[\"ns3226939\"].proxmox_network_linux_bridge.platform' ns3226939:vmbr4\ntofu import 'module.platform_network[\"ns3226939\"].proxmox_network_linux_vlan.platform' ns3226939:vlan_rke\ntofu import 'module.platform_network[\"ns3235424\"].proxmox_network_linux_bridge.platform' ns3235424:vmbr4\ntofu import 'module.platform_network[\"ns3235424\"].proxmox_network_linux_vlan.platform' ns3235424:vlan_rke\ntofu import 'module.platform_network[\"ns3243421\"].proxmox_network_linux_bridge.platform' ns3243421:vmbr4\ntofu import 'module.platform_network[\"ns3243421\"].proxmox_network_linux_vlan.platform' ns3243421:vlan_rke\ntofu import 'module.platform_network[\"ns3249468\"].proxmox_network_linux_bridge.platform' ns3249468:vmbr4\ntofu import 'module.platform_network[\"ns3249468\"].proxmox_network_linux_vlan.platform' ns3249468:vlan_rke\nThese 28 addresses are exactly the output of tofu state list after the\nrun (Evidence: state-list.txt), each with an rc=0 import receipt\n(Evidence: import-log.txt).
Patch the three known bpg 0.111.1 import gaps BEFORE gating. Each gap\nmakes the plan diverge from the pinned HCL and must be patched by\nediting the LOCAL state JSON only (no Proxmox mutation):
\nproxmox_download_file: import leaves url, checksum,\nchecksum_algorithm, verify, overwrite, upload_timeout NULL\n(read() only refreshes size from the datastore listing) \u2192 the plan\nwants to REPLACE the file AND cascade-replace template VM 9000 via its\nchained file_id. Patch the pinned HCL values into the state.disk[0].file_id: imports as \"\" \u2192 patch to the resolved\nvolid local:iso/ubuntu-24.04-20260826-server-cloudimg-amd64.img\n(NOT the resource address \u2014 the address form also forces replacement).stop_on_destroy imports false (HCL says true) and\nagent.wait_for_ip imports as an empty block (HCL: disabled=true)\n\u2192 patch all 8 VMs.\nAfter patching: bump serial, keep lineage.Acceptance gate \u2014 full tofu plan (no -target): MUST be\nNo changes (0 add / 0 change / 0 destroy). Verify twice.
tofu plan output verbatim shows No changes. with the counts\nline empty (0/0/0). ANY destroy line = STOP (see Escalation).tofu state list returns the expected 28 instances.tofu plan re-run immediately after the gate reproduces \"No changes\"\n(drift between the two runs means someone else is writing to Proxmox).terraform.tfstate.bak*); restore from the newest backup if a patch\nstep goes wrong.proxmox_download_file cascading into template\nVM 9000 \u2192 you skipped caveat 1 of step 5; patch the URL/checksum fields\ninto state, do NOT let the plan run as an apply.stop_on_destroy, agent.wait_for_ip)\n\u2192 caveat 3 not applied to every VM.x509: certificate signed by unknown authority against the Proxmox API\n\u2192 PROXMOX_VE_INSECURE=true missing (harness requirement, step 2).:iface form\n(step 4 table), and VLAN/bridge must exist on that node (check recon).tofu version must satisfy\nrequired_version = \">= 1.6.0\" (quoted from\ntofu/proxmox/environments/prod/versions.tf line 2). The recovery\nwas executed with tofu 1.12.6.= 0.111.1 \u2014 quoted from\ntofu/proxmox/environments/prod/versions.tf line 7\n(version = \"= 0.111.1\") and confirmed by\ntofu/proxmox/environments/prod/.terraform.lock.hcl lines 4-6\n(provider \"registry.opentofu.org/bpg/proxmox\" { version = \"0.111.1\", constraints = \"0.111.1\" }).\nEither assertion failing \u2192 STOP; the import ID formats and caveats are\nversion-specific and must be re-verified before use.Executed run: task t_690e108f (wetopi-senior, 2026-08-30), records in the\nKanban Memory Plane. Artifacts:
| Path | \nContent | \n
|---|---|
Per-import receipts: harness workspace \u2026/t_690e108f/evidence/import-log.txt | \n28 import commands + rc=0 receipts | \n
\u2026/t_690e108f/evidence/plan-final.txt | \nacceptance gate: tofu plan = No changes (verified twice) | \n
\u2026/t_690e108f/evidence/state-list.txt | \ntofu state list \u2014 the 28 imported instances | \n
\u2026/t_690e108f/patch-state.py, patch-state2.py | \nthe state-patch scripts that closed the step-5 caveats | \n
\u2026/t_690e108f/recon.py | \nreusable GET-only recon script (precondition check 3) | \n
(the \u2026 prefix is /opt/data/kanban/workspaces/t_690e108f \u2014 the phase-1\ncontroller workspace; all artifacts also attached to the t_690e108f board\ncard.)
\n\nMigrado desde
\nwetopi-platform/docs@main(platform/rke2-apply-runbook.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
This is the single reviewed procedure a human follows to bring the RKE2 cluster up, with validation gates between phases. Every phase ends in an explicit STOP gate: do not proceed past a failing gate. Each gate result is recorded in the apply log with a date.
\n\n\nStatus (2026-08-30, post-Phase-D update): the live apply has RESUMED and executed through Phase D. Executed 2026-08-29/30, fully green: B (cp01 bootstrap, infra PR #6 fix d553643), C (cp02+cp03 quorum, PR #7), F6-1..3 (CP-only acceptance: quorum recorded, one-CP failure drill via VIP failover, etcd snapshot + real restore), E (Cilium 1.20.1 + Hubble, PR #8), and D (all 3 platform workers joined via
\nrke2-workers.ymlat infra PR #9 f841ae7 \u2014 serial gates green, per-workerkubectl wait+ exact-label asserts passed incl. the/var/lib/rancher/rke2/agent/kubelet.kubeconfigdata-dir gate; live-verified on rke-platform01). Cluster state: 6/6 nodes Ready, CP taints enforced (node-role.kubernetes.io/control-plane:NoSchedule), workers labelledwetopi.io/role=platform, kube-system pods Running, in-cluster DNS resolving (10.68.0.0/16), scheduling test landed on a platform worker, CPs workload-free (rke2-workers-validate.ymlrack green \u2014 details its runbook section describes for D2 remain next). Next in executable order: F (Ceph, blocked on operator-held facts, t_80ca13af); G (Argo CD) has since APPLIED with recorded deviations (see the post-Phase-G update below and the Phase G section). Earlier halt notice (retained for history): the live apply (task t_a88bb36b) was HALTED BEFORE PHASE B and FROZEN per operator directive #38 \u2014 the 2026-08-29 23:17-23:26 UTC attempt failed atrke2-server : Create the RKE2 server configurationwithDestination directory /etc/rancher/rke2 does not existon rke-cp01 (neither rke2 role created the config dir); fix was infrastructure PR #6 (merged main d553643). cp01 carried keepalived (active+enabled) + RKE2 binaries from that pre-halt attempt \u2014 roles proved idempotent (RKE2 install guarded bycreates:) when B/C re-ran clean on 2026-08-30 (tasks t_b0eebfac, t_58bd66bb, t_f7099d9d, t_b3841920). Verified against merged infrastructuremain@ f841ae7 (PRs #1-#9) and docsmain@ 7cc387a.
Phase order (the authorized executable order \u2014 stated IDENTICALLY in this banner, the F6 heading + STOP rule, the Phase W heading, the D2 handoff, the DoD mapping, and the provenance section): B \u2192 C \u2192 STOP \u2192 F6-1..3 \u2192 E \u2192 D \u2192 D2 \u2192 W \u2192 F \u2192 G. Concretely: A (VIP) is folded into the B/C gates (no standalone A apply) \u2192 B (cp01 bootstrap + VIP bring-up) \u2192 C (quorum cp02+cp03 incl. VIP failover) \u2192 STOP (mandatory pause after the Phase C quorum gate: before any F6-2/F6-3 server stop, every F6 STOP prerequisite below must be PASS and dated in the apply log; historically this was the frozen-apply boundary of the 2026-08-29 halt \u2014 that freeze is superseded history, see the retained halt notice in the status banner above) \u2192 Fase-6 acceptance, CP-only portion: F6-1 (quorum recorded), F6-2 (one-CP failure drill), F6-3 (etcd snapshot + REAL restore test) \u2014 these three execute on the CP-only cluster (3 nodes, no workers) BEFORE any addon, per master plan \u00a733 (Fase 6 before Fase 7), and are gated by the hard F6 STOP prerequisites (external verified snapshot copy, SHA-256 match, quorum evidence, explicit human authorization) \u2192 E (Cilium install + basic install validation ONLY) \u2192 D (workers join, registration + exact-label gates only) \u2192 D2 (full post-Cilium cluster validation on the FINAL 6-node topology, incl. the full Cilium acceptance) \u2192 W = Phase W \u2014 the worker lifecycle drill (drain + reboot; separately named; former F6-4/F6-5) \u2014 worker drain/reboot happens HERE, after D2, NEVER before addons \u2192 F (Ceph, PR-only, default-off) \u2192 G (Argo CD). Known tension with master plan \u00a733, resolved EXPLICITLY: \u00a733 orders Fase 5 (workers) and Fase 6 (which includes node drain/reboot) BEFORE Fase 7 (Cilium), but ADR-0005 requires Cilium as the IMMEDIATE first addon with NO pre-Cilium worker-Ready/DNS gates \u2014 with cni: none, workers can never pass Ready/drain/reboot gates before Cilium exists. Resolution: the CP-only gates of Fase 6 (quorum, CP failure, snapshot + real restore) run before the first addon, and the worker-lifecycle drill is the separately-named Phase W landing after D2 on the final 6-node topology; the DoD maps drain/reboot to that POST-D2 evidence, never to pre-addon acceptance. Do NOT count Phase W as completed pre-addon acceptance. With cni: none at the RKE2 level, no node can be Ready before Cilium exists: health before E = /readyz + etcd member counts ONLY; nodes NotReady is EXPECTED before E and there are NO worker-Ready/CoreDNS/DNS expectations before Cilium. The FULL Cilium acceptance (connectivity test, Hubble flows, DNS, NetworkPolicy, CiliumNode per node) runs in D2 on the final 6-node topology, not in E.
Work through this checklist in order. Each item has its concrete verification.
\nADRs accepted \u2014 docs/decisions/ADR-0004-rke2-api-endpoint.md and docs/decisions/ADR-0005-cilium-first-addon.md merged in docs main (merge commit 5740404).
Implementation PRs merged to infrastructure main \u2014 verified at d553643 (current main, 2026-08-30):
feature/cilium-hubble-addon: ansible/playbooks/cilium.yml, kubernetes/cilium/* \u2014 merged as e283e24)feature/rke2-server-role: rke2-server + keepalived roles, ansible/playbooks/rke2-cp.yml, scripts/gen-rke2-token.sh \u2014 merged as b15a087)feature/rke2-agent-role: rke2-agent role, ansible/playbooks/rke2-workers.yml, ansible/playbooks/rke2-workers-validate.yml \u2014 merged as 8d836e8)fix/rke2-config-dir-create, head 3ff2ee8 \u2014 merged as d553643): creates /etc/rancher/rke2 (root:root 0750) before templating in BOTH rke2 roles. This is the fix for the config-directory defect that halted the 2026-08-29 apply attempt (Destination directory /etc/rancher/rke2 does not exist at rke2-server : Create the RKE2 server configuration; byte-identical defect in rke2-agent). Without it, phases B and D fail at config templating.STEP 0 gate: git -C infrastructure log --oneline -1 on the checkout used for apply must show d553643 or a verified descendant \u2014 8d836e8 (pre-PR#6) is NO LONGER sufficient: resuming phases B/C against a checkout without the config-dir fix reproduces the halt. Verify the fix is present: git -C infrastructure show d553643 --stat mentions both rke2-server and rke2-agent tasks. git grep secrets-encryption still returns nothing on d553643 (the explicit hardening line is still absent \u2014 see item 8 for the current measured-evidence gate rule).
Branch protection \u2014 main is the reviewed line; all changes land via PR with wetopi-approver review; never commit directly to main.
Token \u2014 on the controller (the infrastructure checkout that will run the playbooks \u2014 the evidenced controller is the WORKSTATION, not ops01; see item 9), run scripts/gen-rke2-token.sh (umask 077, openssl rand -hex 16, 32-char hex). It writes ansible/files/rke2/node-token, mode 0600, gitignored.\nVerify: stat -c '%a' ansible/files/rke2/node-token \u2192 600. NEVER commit it.\nBoth rke2-cp.yml and rke2-workers.yml pre-flight plays hard-fail if the file is missing or not 0600.
Inventory \u2014 ./scripts/render-ansible-inventory.sh regenerates ansible/inventories/prod/hosts.json from the tofu ansible_inventory output (mode 600, gitignored); the generated hostvars set ansible_ssh_common_args = \"-o ProxyJump=ubuntu@92.222.245.91\" for every non-bastion VM \u2014 that ProxyJump form is the inventory's internal contract consumed by Ansible. Bastion is wetopi-ops01 = 92.222.245.91 (WAN) / 10.72.0.10, SSH account ubuntu (bastion SSH policy allows only ubuntu; the workstation keeps all private keys; ops01 is a ProxyJump target only). Human reachability check from the workstation uses the documented double-key form:\nssh -i /opt/data/.ssh/bigardo -o ProxyCommand=\"ssh -i /opt/data/.ssh/bigardo -W %h:%p ubuntu@92.222.245.91\" <user>@<host>, or plain ping 10.72.0.21 from ops01.
Proxmox snapshots (rollback plane) \u2014 before Phase B, snapshot every VM that phases B-G will touch, from the Proxmox node hosting it (standard Proxmox CLI qm snapshot <vmid> pre-rke2-apply):
| VM | \nvm_id | \nProxmox node | \nIP | \n
|---|---|---|---|
| wetopi-ops01 | \n8001 | \nns3249468 | \n10.72.0.10 | \n
| rke-cp01 | \n8011 | \nns3243421 | \n10.72.0.21 | \n
| rke-cp02 | \n8012 | \nns3226939 | \n10.72.0.22 | \n
| rke-cp03 | \n8013 | \nns3214108 | \n10.72.0.23 | \n
| rke-platform01 | \n8021 | \nns31826929 | \n10.72.0.41 | \n
| rke-platform02 | \n8022 | \nns3226931 | \n10.72.0.42 | \n
| rke-platform03 | \n8023 | \nns31826925 | \n10.72.0.43 | \n
| wetopi-natgw01 | \n8031 | \nns3243421 | \n10.72.0.2 | \n
Rollback = qm rollback <vmid> pre-rke2-apply (VM must be stopped). These are the Proxmox-level safety net; the per-phase rollbacks below are the first line.
Cluster facts (from ADR-0004) \u2014 API VIP 10.72.0.20 (vmbr4/VLAN 4 node LAN 10.72.0.0/23, never Internet); server URL https://10.72.0.20:9345; API https://10.72.0.20:6443; tls-san = 10.72.0.20/.21/.22/.23 (IP-only, no DNS SAN); cluster-cidr 10.64.0.0/14; service-cidr 10.68.0.0/16; CNI none at RKE2 level (Cilium is the first addon, ADR-0005).
At-rest encryption gate \u2014 ONE current statement (reconciled 2026-08-30, Phase G live evidence, card t_e0cace4b). The earlier \"honest gap\" assumption (that a missing secrets-encryption line in the config template means Secrets sit PLAINTEXT in etcd) is superseded: it was wrong for this RKE2 build. Measured fact: RKE2 v1.36.3+rke2r1 enables AES-CBC at-rest encryption BY DEFAULT \u2014 the config template renders no secrets-encryption line, yet rke2 secrets-encrypt status measured on ALL THREE CPs shows Encryption Status: Enabled / Current Rotation Stage: start / Server Encryption Hashes: All hashes match / Active * AES-CBC aescbckey, and raw etcd probes (approved kubectl-exec etcdctl form, below) of argocd/argocd-initial-admin-secret (316B) and argocd/repo-infra-https (685B) returned k8s:enc:aescbc:v1:aescbckey:<opaque ciphertext> \u2014 NOT plaintext.\nCURRENT GATE RULE (this is the operative rule; it supersedes every older wording elsewhere in this runbook, including any residual plaintext framing): current protection is determined by MEASURED EVIDENCE, not by config-template contents \u2014 for any Secret at any gate, the measured checks below. What the explicit secrets-encryption: true line is: a SEPARATE, tracked, future reviewed hardening/rotation prerequisite (explicit declarative intent in the rke2-server role template, needed BEFORE any future key rotation). It is NOT an unlock condition for any gate while the measured checks pass \u2014 a build with default-on encryption and no explicit line is PROTECTED, not blocked.\nMeasured-protection checks (the form used at any [encryption-gate]):
rke2 secrets-encrypt status on all three CPs, each showing:\nEncryption Status: Enabled\nCurrent Rotation Stage: start\nServer Encryption Hashes: All hashes match\n* AES-CBC aescbckey... (the * marks the active key);/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n kube-system exec etcd-rke-cp01 -- env ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt --key=/var/lib/rancher/rke2/server/tls/etcd/client.key get /registry/secrets/<namespace>/<NAME>\nk8s:enc:aescbc ciphertext \u2014 opaque), NOT the plaintext returned by kubectl get secret <name> -n <namespace> -o jsonpath='{.data.<KEY>}' | base64 -d. STOP: plaintext visible \u2192 protection NOT confirmed \u2192 the [encryption-gate] in question stays STOPped.\nStatus of this cluster (2026-08-30, Phase G): measured PROTECTED \u2014 (i) verified on all 3 CPs, (ii) verified by raw etcd ciphertext probes on both argocd Secrets. The future hardening/rotation prerequisite (reviewed secrets-encryption: true merged to infrastructure main, then sequential restart cp01\u2192cp02\u2192cp03 one at a time with each returning Ready before the next, re-verifying (i) and (ii)) is NOT yet merged (git grep secrets-encryption on infrastructure main still returns empty) \u2014 it must land via its own reviewed PR BEFORE any future encryption-key rotation; it does not block F2/G while the measured checks pass.\nFuture shape: a reviewed secrets-encryption: true setting in /etc/rancher/rke2/config.yaml (rendered by the rke2-server role), merged to infrastructure main. NOTE: there is no manually-managed key file \u2014 RKE2's documented behavior (rancher/rke2-docs, docs/security/secrets_encryption.md) is that once secrets encryption is enabled, RKE2 automatically generates an AES-CBC key and the encryption-provider config file at /var/lib/rancher/rke2/server/cred/encryption-config.json and passes it to the kube-apiserver as --encryption-provider-config. The config file is node-local state auto-generated and managed by RKE2 itself \u2014 never commit it to Git.Controller and execution environment \u2014 define and verify BEFORE the first phase command. Direct evidence from the 2026-08-29 apply (t_a88bb36b): wetopi-ops01 had NO Ansible venv, NO infrastructure checkout, and NO kubeconfig; the playbooks ran from the WORKSTATION, which holds the SSH private keys (/opt/data/.ssh/bigardo). ops01 is a ProxyJump target only (item 5): it holds no private keys and cannot SSH to the LAN VMs by itself. This runbook therefore fixes the execution context explicitly instead of assuming it:
rke2-cp.yml, rke2-workers.yml, rke2-workers-validate.yml run from the controller and reach the VMs over SSH through the bastion). Verify on the controller BEFORE phases B/C (record each check with a date):\ngit -C infrastructure log --oneline -1 shows d553643 or a descendant, and git -C infrastructure status --porcelain is empty;requirements-dev.txt (pins ansible-core==2.21.3): python3 -m venv .venv then .venv/bin/pip install -r requirements-dev.txt; verify .venv/bin/ansible-playbook --version reports ansible-core 2.21.3;./scripts/render-ansible-inventory.sh regenerates ansible/inventories/prod/hosts.json (item 5) \u2014 or use the synthesized gitignored inventory shape the 2026-08-29 apply verified against outputs.tf; EITHER form must include the ungrouped localhost host with ansible_connection: local, ansible_become: false (the roles' delegate_to: localhost token checks fail on a sudo-less controller without it \u2014 evidenced 2026-08-29);stat -c '%a' ansible/files/rke2/node-token \u2192 600;sudo -v succeeds there).connection: local Helm op of cilium.yml, the cilium/hubble CLI acceptance in D2, and the F2/G commands; NONE of this existed at apply time):\ngit clone https://github.com/wetopi-platform/infrastructure.git ~/wetopi-infrastructure then git checkout main and verify the head;python3 -m venv .venv && .venv/bin/pip install -r requirements-dev.txt (Phase E invokes .venv/bin/ansible-playbook there);ansible/inventories/prod/hosts.json is gitignored \u2014 either render it on ops01 (requires the tofu state) or copy the controller's verified copy over (same relay as the kubeconfig below); record which was done;helm version --short (v3.x; the chart reference was validated with Helm 3.19), kubectl version --client, and the cilium + hubble CLIs (GitHub releases cilium/cilium-cli and cilium/hubble; verify with cilium version and hubble version; record exact versions \u2014 the cilium CLI must be compatible with the pinned chart 1.20.1);sudo cat /etc/rancher/rke2/rke2.yaml > /tmp/rke2-admin-kubeconfig && chmod 600 /tmp/rke2-admin-kubeconfig;scp -i /opt/data/.ssh/bigardo -o ProxyCommand=\"ssh -i /opt/data/.ssh/bigardo -W %h:%p ubuntu@92.222.245.91\" ubuntu@10.72.0.21:/tmp/rke2-admin-kubeconfig /tmp/rke2-admin-kubeconfig;scp -i /opt/data/.ssh/bigardo /tmp/rke2-admin-kubeconfig ubuntu@92.222.245.91:~/secrets/rke2-admin-kubeconfig;/tmp copies on cp01 and the workstation;sed -i 's|https://127.0.0.1:6443|https://10.72.0.20:6443|' ~/secrets/rke2-admin-kubeconfig && chmod 600 ~/secrets/rke2-admin-kubeconfig (the RKE2 admin kubeconfig points at 127.0.0.1 by default; the VIP is in tls-san per ADR-0004, so the apiserver cert validates), then verify: KUBECONFIG=~/secrets/rke2-admin-kubeconfig kubectl get nodes lists the 3 CPs. NEVER commit the kubeconfig \u2014 it is cluster-admin credentials./var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml ... \u2014 plain kubectl is NOT in PATH on CP nodes; only the RKE2-bundled binary is (same form the playbooks use, e.g. the rke2_kubectl fact in rke2-workers-validate.yml);kubectl ... with KUBECONFIG pointing at the transferred admin kubeconfig (ops01 provisioning above).Keepalived is NOT a standalone phase: it is the first role of both RKE2 plays inside ansible/playbooks/rke2-cp.yml (role order per CP: keepalived, then rke2-server). Play 1 pre-flights all CPs (asserts VIP+addresses in tls-san, CIDRs defined, VIP == 10.72.0.20, token file 0600). The bootstrap play (play 2, hosts rke-cp01, serial: 1, any_errors_fatal: true) renders /etc/rancher/rke2/config.yaml via the rke2-server template (token; server: only on non-bootstrap nodes; tls-san list; cluster-cidr; service-cidr; cni: none).
Keepalived design facts (from ansible/roles/keepalived/): interface eth0; VIP 10.72.0.20/23; VRID 51; unicast peers = the other two CPs; priorities cp01=150 (initial MASTER), cp02=100, cp03=50; advert_int 1; health-check script /etc/keepalived/check_rke2_api.sh probes the LOCAL API only: curl -sf --cacert /var/lib/rancher/rke2/server/tls/server-ca.crt --cert .../client-kube-apiserver.crt --key .../client-kube-apiserver.key https://127.0.0.1:6443/readyz plus a TCP probe of 127.0.0.1:9345; interval 2, fall 3, rise 2.
Consequence (fail-safe by design): a CP only holds the VIP while its own local API is healthy \u2014 before cp01's API is up there is deliberately NO VIP holder, so ping 10.72.0.20 failing before Phase B completes is EXPECTED, not a bug. There is NO standalone Phase-A apply; VIP acceptance is folded into the B/C gates below.
Command (from the CONTROLLER \u2014 the workstation per \u00a70 item 9, in its infrastructure checkout):
\nANSIBLE_CONFIG=ansible/ansible.cfg \\\n .venv/bin/ansible-playbook -i ansible/inventories/prod/hosts.json \\\n ansible/playbooks/rke2-cp.yml\nThe playbook is one invocation for B AND C \u2014 play 1 pre-flights all CPs, play 2 bootstraps cp01, play 3 joins cp02/cp03. Phase B is complete when play 2's built-in gates pass: wait_for 127.0.0.1:6443 (600s timeout), /readyz via the VIP (Ansible uri with client certs, retries 30 \u00d7 10s, expects status 200), and etcd member list == 1 (see gate B below).
curl -k https://10.72.0.20:6443/readyz \u2192 ok (API via VIP works; VIP now held by cp01)systemctl is-active keepalived on cp01 ONLY (Phase B applies cp01; the all-three-CP keepalived check belongs to gate C, where cp02/cp03 are applied) \u2192 activeip -4 addr show eth0 on each CP: exactly one node lists 10.72.0.20/23 (the current owner; cp01 when healthy)ping -c 3 10.72.0.20 from wetopi-ops01 \u2192 0% loss/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n kube-system exec etcd-rke-cp01 -- env ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt --key=/var/lib/rancher/rke2/server/tls/etcd/client.key member list\ncni: none, the CP-local form (\u00a70 item 9) /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes on cp01 shows cp01 NotReady \u2014 EXPECTED. CP health at this stage = /readyz via VIP + etcd member count only.STOP: any playbook play fails (any_errors_fatal aborts the whole run \u2014 do not re-run blindly; read the failure first); /readyz via VIP fails; etcd member count != 1; keepalived not active on cp01.
Same playbook, play 3 (hosts rke-cp02:rke-cp03, serial: 1, roles keepalived + rke2-server, joining via server: https://10.72.0.20:9345; per-node expected etcd member counts: after cp02 = 2, after cp03 = 3). It continues automatically once Phase B gates pass.
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n kube-system exec etcd-rke-cp01 -- env ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt --key=/var/lib/rancher/rke2/server/tls/etcd/client.key member list\ncurl -k https://10.72.0.20:6443/readyz \u2192 oksystemctl is-active keepalived on all three CPs (the all-three-CP keepalived check lives HERE, in Phase C where cp02/cp03 are applied) \u2192 active/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes (CP-local form, on cp01) \u2192 rke-cp01..03 all NotReady \u2014 EXPECTED (pre-CNI); do NOT treat this as a failureip -4 addr show eth0 on each CP) run sudo systemctl stop keepalived; expect the VIP to move to the highest-priority healthy peer (cp02) within roughly 10 s (advert_int 1 \u00d7 fall 3, plus check rise 2 \u00d7 interval 2 on the receiver); verify from ops01: ping -c 3 10.72.0.20 still 0% loss and curl -k https://10.72.0.20:6443/readyz still ok; then sudo systemctl start keepalived on cp01 \u2014 cp01 (priority 150) reclaims the VIP by preemption; confirm with ip -4 addr again.sudo systemctl stop rke2-server; then from ops01 curl -k https://10.72.0.20:6443/readyz must STILL return ok (2 members keep quorum; API served via VIP on remaining CPs) and the etcd member list (kubectl-exec form, from cp01) still lists 3 members; restart with sudo systemctl start rke2-server. STOP the whole run if the API via VIP fails during this test (that is a quorum/VIP defect, not a node defect)./var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml describe node rke-cp0X | grep -i taint \u2014 record the taint state on all three CPs in the apply log. The no-platform-workloads-on-CPs guarantee is enforced by the workers playbook play 2, which applies node-role.kubernetes.io/control-plane=true:NoSchedule on all three CPs (idempotency decided from the API pre-state, not kubectl stdout), so ordinary pods can never schedule onto a CP; system components tolerate the taint by upstream default. Do not proceed if /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n kube-system get pods -o wide (same form) shows any platform-labeled pod scheduled on a CP.STOP: member count != 3; /readyz fails; keepalived not active on all three CPs; API via VIP fails with one CP stopped or during VIP failover; any platform workload found on a CP.
\nstarted.These are GATES, not suggestions. F6-3's restore test stops ALL THREE servers \u2014 full cluster API downtime \u2014 and replaces the etcd peer databases; F6-2 stops one server. Until every prerequisite below is PASS, this phase halts after the read-only F6-1 and NO systemctl stop rke2-server may be issued.
curl -k https://10.72.0.20:6443/readyz \u2192 ok; record both outputs.rke2 etcd-snapshot save --name pre-addons-fase6\nrke2 etcd-snapshot ls\nsave reports the snapshot file written under /var/lib/rancher/rke2/server/db/snapshots/ (on-demand snapshots default to on-demand-<node>-<unix-ts>; --name sets the base name to pre-addons-fase6); ls lists the new snapshot with a size and timestamp. Record both outputs. Then copy the snapshot file OFF cp01 to external storage \u2014 e.g. wetopi-ops01 ~/rke2-snapshots/ or the workstation's disk \u2014 using the relay documented in \u00a70 item 9 (on cp01 place it under /tmp readable by ubuntu; pull it with the workstation's ProxyCommand form; ops01 cannot pull from cp01 itself because it holds no private keys). The external copy MUST live outside /var/lib/rancher/rke2/server/db AND outside cp01's failure domain: NOT on cp01, NOT on any CP node, NOT inside the live db tree, and preferably not on cp01's Proxmox node (ns3243421) either \u2014 wetopi-ops01 (ns3249468) or the workstation satisfies this. cp01 keeps the copy the reset consumes; the external copy is the recovery plane if the reset damages cp01's db.sha256sum /var/lib/rancher/rke2/server/db/snapshots/<snapshot-file> on cp01, and \u2014 independently, ON the host holding the external copy (not via a shell relayed through cp01) \u2014 sha256sum <external-path>/<snapshot-file>. The two digests MUST be identical; record both in the apply log. Mismatch \u2192 re-copy and re-verify before anything else.systemctl stop rke2-server of F6-3. No recorded authorization line = gate NOT passed.STOP (prerequisites): any P1-P4 item not PASS \u2192 do NOT run F6-2/F6-3; the cluster stays up (read-only F6-1 is the only permitted test); escalate to the lead.
\nsudo systemctl stop rke2-server on cp02, then curl -k https://10.72.0.20:6443/readyz from ops01, then sudo systemctl start rke2-server on cp02./readyz returns ok throughout (2 members keep quorum); API served via VIP on remaining CPs; after restart, member list still 3.\n\nThis is a REAL, EXECUTED acceptance test \u2014 not a documented-only procedure. Master plan \u00a735.1 requires \"restore probado\"; a snapshot-save-only check does NOT satisfy it. The restore test must be executed and recorded HERE, before Phase E (Cilium). If the restore test cannot be executed, the DoD marks it UNMET and the runbook mandates STOP before all subsequent phases (E onward). The snapshot itself was saved and externally checksum-verified in the F6 STOP prerequisites P2/P3 above; this test consumes the cp01-local copy.
\n
The snapshot file is the one saved and externally verified in prerequisites P2/P3 above (cp01-local path /var/lib/rancher/rke2/server/db/snapshots/<file>; the verified external copy is the recovery plane \u2014 if the cp01-local file is damaged, re-copy from the external copy and re-verify its SHA-256 before use). EXECUTE the restore procedure below as the acceptance test and record the evidence (expected-output lines, member list 3/3, /readyz, and the post-restore journal checks in step 6).
rke2 server --cluster-reset (multi-server, 3 CPs)\n\nCRITICAL: there is NO
\nrke2 etcd-snapshot restoresubcommand. Verified from the authoritative RKE2 docs (rancher/rke2-docs, docs/datastore/backup_restore.md): theetcd-snapshotsubcommands aredelete/ls(list,l) /prune/saveONLY. Restore is the server cluster-reset flow.
Snapshot on N1 = cp01; peers N2/N3 = cp02/cp03:
\nsystemctl stop rke2-server on cp01, cp02, cp03.rke2 server --cluster-reset --cluster-reset-restore-path=<PATH-TO-SNAPSHOT> (path = the snapshot file recorded by rke2 etcd-snapshot ls, under /var/lib/rancher/rke2/server/db/snapshots/).\nExpected output on success:\nManaged etcd cluster membership has been reset, restart without --cluster-reset flag now.\nBackup and delete ${datadir}/server/db on each peer etcd server and rejoin the nodes.\nsystemctl start rke2-server.rm -rf /var/lib/rancher/rke2/server/db/ then systemctl start rke2-server \u2014 each peer rejoins the restored cluster by re-downloading the db.curl -k https://10.72.0.20:6443/readyz \u2192 ok; the CP-local form /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes (on cp01) shows 3/3 CPs registered.sudo journalctl -u rke2-server --since \"-30 min\" --no-pager | grep -E 'level=(error|fatal)' || echo CLEAN\nlevel=error/level=fatal line mentioning corruption, corrupt, checksum, panic, or db/data-dir damage \u2014 OR a recurring failed to start persisting more than ~2 minutes after that node's start \u2014 means the restore test is FAILED: apply the STOP-and-escalate rule below; do NOT retry. Transient retry/election/timeout lines during peer rejoin that stop once quorum re-forms are acceptable. Required to PASS: CLEAN output (or only transient lines that cease) on all three CPs.Mechanics (from the RKE2 docs): the current etcd db files are moved to ${data-dir}/server/db/etcd-old-$TIMESTAMP/; the snapshot checksum is verified; all etcd members except the restore node are removed from the cluster, and peers rejoin by re-downloading the db.
Reset-flag safety mechanism (rancher/rke2-docs, docs/datastore/backup_restore.md; semantics verified 2026-08-30 against the k3s source vendored by the pinned rke2_version: v1.36.3+rke2r1 \u2014 k3s commit 4ca29faac89e, pkg/cluster/managed.go + pkg/etcd/etcd.go): when RKE2 resets the cluster it creates an empty file at /var/lib/rancher/rke2/server/db/reset-flag to prevent accidental repeated cluster resets; the flag is deleted only when RKE2 starts normally (a normal start of the managed database clears it). Version-exact consequences: (a) the flag is written after the snapshot-restore step succeeds \u2014 an early failure (bad snapshot path, download/checksum/restore error) happens with NO flag present; a late failure (post-reset etcd startup) happens WITH the flag present; (b) the flag blocks only a bare rke2 server --cluster-reset (no restore path) \u2014 the pinned version does NOT consult the flag when --cluster-reset-restore-path is set (source comment: \"If we're restoring from a snapshot, don't check the reset-flag - just reset and restore\"); (c) the pre-restore db is moved aside to db/etcd-old-<timestamp>/ by the restore step, so a failed attempt may leave the db dir absent or partially restored.
/readyz is not OK: do NOT blindly retry \u2014 the failure state is flag- and db-dependent per (a)-(c) above, and all CPs are stopped. Escalate to the lead with the full output; the cluster stays STOPped. No addons or workloads until recovery is complete and quorum is re-verified.systemctl start rke2-server \u2014 the documented flag-clearing mechanism. It starts from whatever db state the failed attempt left: if the snapshot-restore step succeeded and only the startup failed, the node comes up as the restored single member; if the restore step itself failed (db moved aside, restore incomplete), the node may fail to start or come up uninitialized. Record the actual observed state (flag present? db/etcd-old-* present? member list, /readyz) and STOP again if anything is unexpected. A successful normal start is the precondition for considering any second reset attempt.rke2 server --cluster-reset --cluster-reset-restore-path=<PATH> on cp01 \u2192 normal start cp01 \u2192 peers rm -rf /var/lib/rancher/rke2/server/db/ + start). Version fact for the record: the flag check applies only to a bare --cluster-reset; the pinned version does not consult the flag when a restore path is set. This runbook never performs a bare reset \u2014 do not remove reset-flag manually; if it is ever encountered blocking a start, that is itself a STOP-and-escalate condition.qm rollback <vmid> pre-rke2-apply applied to ALL CPs together \u2014 the snapshot predates Phase B, so this returns the cluster to the pre-B state and phases B/C must be re-run.STOP (F6 overall): any F6 STOP prerequisite P1-P4 not PASS (no server may be stopped); any F6-1/F6-2/F6-3 gate failing \u2014 the cluster is not accepted; fix before Phase E (these three are the pre-addon CP-only gates and are the ONLY Fase-6 gates blocking E). The worker-lifecycle drill is the separately-named Phase W of the same executable order (B \u2192 C \u2192 STOP \u2192 F6-1..3 \u2192 E \u2192 D \u2192 D2 \u2192 W \u2192 F \u2192 G); it runs after D2 and its gates are defined there. The Fase-6 acceptance ledger is COMPLETE only when F6-1..3 AND Phase W's W-1/W-2 all have dated PASS entries; W passes POST-D2 by design (ADR-0005 \u2014 see banner), never pre-addon.
\n\n\nRuns as the immediate first addon after the CP-only Fase-6 gates (F6-1..3) \u2014 and BEFORE workers join: with Cilium already present, newly joined workers can reach Ready as soon as their Cilium agent starts. (The worker-lifecycle drill \u2014 drain/reboot \u2014 is the separately-named Phase W and executes later in the executable order, AFTER D2; it does not block E.) This runbook splits Cilium work in two: Phase E = install + basic install validation ONLY; the full Cilium acceptance (connectivity test, Hubble flows, DNS, NetworkPolicy, one CiliumNode per node) runs in Phase D2 on the FINAL 6-node topology (see D2).
\n
Playbook ansible/playbooks/cilium.yml (infrastructure PR #2, merged as e283e24): runs from the operations host (ops01) with connection: local so the Helm op executes exactly once; chart pinned oci://quay.io/cilium/charts/cilium --version 1.20.1 (full OCI reference; the --repo oci:// form is NOT valid); values kubernetes/cilium/values.yaml (VXLAN tunnel, kubeProxyReplacement false, Hubble enabled).
Gate semantics are FAIL-CLOSED: every kubectl probe uses --ignore-not-found and requires rc 0 \u2014 any rc != 0 (API error, connection refused, auth/TLS failure) ABORTS the run; absence of a resource is proven ONLY by rc 0 with EMPTY stdout. An error is never \"absent\": it is a STOP.
Ordering gates (fail-closed, per ADR-0005): the play refuses to run if a CNI is already deployed (Canal would fight Cilium), requires kube-proxy present (RKE2 with cni: none provides it), and aborts unless the Cilium agent is absent (first-install check).
Command (from ops01 \u2014 AFTER the \u00a70 item 9 ops01 provisioning is verified: checkout at d553643 or a verified descendant, Ansible venv, inventory, helm, and the current transferred kubeconfig in KUBECONFIG; this playbook targets the operations group with connection: local, so the Helm op runs on ops01 itself):
ANSIBLE_CONFIG=ansible/ansible.cfg \\\n .venv/bin/ansible-playbook -i ansible/inventories/prod/hosts.json \\\n ansible/playbooks/cilium.yml\nFirewall prerequisites (node LAN must allow): UDP 8472 (VXLAN) between all nodes; TCP 4244 (Hubble Relay flow scraping). MTU stays Cilium default (~1450) on the MTU-1500 LAN.
\nBasic install validation on the CP-only (3-node) cluster, from ops01 with KUBECONFIG set to the transferred admin kubeconfig (\u00a70 item 9):
cilium status --wait # agents Ready on the 3 CPs; KVStore/Conntrack/ControllerStatus Ok, Hubble Ok/Enabled\nkubectl -n kube-system get pods -l k8s-app=cilium # cilium-ds Running+Ready on all 3 CPs (one per CP)\nkubectl -n kube-system get pods -l io.cilium/app=operator # cilium-operator Running+Ready\nThe following are deliberately NOT Phase E gates (they are D2 gates on the final 6-node topology): cilium connectivity test, Hubble flow observation, DNS resolution, NetworkPolicy enforcement, CiliumNode per-node completeness.
Also note (VALIDATION.md \u00a77): pods that existed before Cilium must be restarted once to fall under the eBPF datapath (in this greenfield order the only such pods are the control-plane static/system pods; kube-proxy static pods and CoreDNS are handled by rollout/restart once nodes go Ready).
\nRollback: helm uninstall cilium --namespace kube-system \u2014 documented in VALIDATION.md \u00a78 with the warning that the cluster has NO other CNI after this (cni: none means RKE2 will not reinstall Canal), so only do this with a replacement plan.
STOP: any fail-closed probe rc != 0; cilium-ds pod not Running+Ready on each of the 3 CPs; cilium-operator not Running+Ready.
\nPlaybook ansible/playbooks/rke2-workers.yml (infrastructure PR #5, merged as 8d836e8): play 1 pre-flights rke_platform (asserts rke2_server_url == https://10.72.0.20:9345, CIDRs inherited from shared group_vars, token file exists 0600 \u2014 no_log), play 2 (hosts rke-cp01) applies/verifies the control-plane NoSchedule taint node-role.kubernetes.io/control-plane=true:NoSchedule on all 3 CPs (idempotency decided from API pre-state), play 3 joins rke-platform01..03 (serial: 1, any_errors_fatal: true, rke2-agent role; config: token from the gitignored 0600 file, server: https://10.72.0.20:9345, node-label: [wetopi.io/role=platform], no taints).
Command (from the CONTROLLER \u2014 the workstation per \u00a70 item 9):
\nANSIBLE_CONFIG=ansible/ansible.cfg \\\n .venv/bin/ansible-playbook -i ansible/inventories/prod/hosts.json \\\n ansible/playbooks/rke2-workers.yml\nPer-worker join gate (inside the serial join play): for each worker, the gate is control-plane-OBSERVED registration + exact label \u2014 kubectl get node <name> -o json delegated to rke-cp01 (retries 60 \u00d7 10s), then assert metadata.name == inventory_hostname and labels['wetopi.io/role'] == 'platform'. A failed gate aborts before the next worker joins. Phase D gate = registration + label ONLY.
Ready state after join (Cilium-before-workers): Cilium was installed in Phase E, so a newly joined worker SHOULD reach Ready once its Cilium agent (the cilium-ds DaemonSet pod) starts on it \u2014 NotReady is a TRANSIENT join state, NOT indefinitely expected. Ready/pod/DNS/scheduling validation is deferred to rke2-workers-validate.yml (Phase D2), which must prove: 6/6 nodes Ready, a cilium-ds pod per node, and a CiliumNode object per node, plus the full Cilium acceptance. A worker that stays NotReady indefinitely is a FAILURE to investigate (Cilium agent attachment issue), never \"expected\".
After the join play, the playbook reports the result; workers are expected to transition to Ready as their Cilium agents come up. Full validation runs next in D2.
\nPlaybook ansible/playbooks/rke2-workers-validate.yml (PR #5, merged as 8d836e8; hosts rke-cp01, become: true, any_errors_fatal: true, read-only queries + two short-lived uniquely-named test pods, always deleted). Run from the CONTROLLER (\u00a70 item 9 \u2014 the play reaches cp01 over SSH) AFTER Cilium (Phase E) and after the workers have joined (Phase D). This phase ALSO carries the full Cilium acceptance (moved from former Phase E) on the final 6-node topology:
ANSIBLE_CONFIG=ansible/ansible.cfg \\\n .venv/bin/ansible-playbook -i ansible/inventories/prod/hosts.json \\\n ansible/playbooks/rke2-workers-validate.yml\nBuilt-in gates (manual equivalents \u2014 run ON cp01 using the CP-local kubectl form, \u00a70 item 9; define once for the shell: RKE2_KUBECTL=\"/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml\"):
$RKE2_KUBECTL get nodes # 6 nodes, ALL Ready (3 CP + 3 platform)\n$RKE2_KUBECTL get nodes -l wetopi.io/role=platform # exactly 3, each carrying the exact label\n$RKE2_KUBECTL -n kube-system get pods -l k8s-app=kube-dns # Running+Ready\n$RKE2_KUBECTL -n kube-system get pods -l k8s-app=kube-dns-autoscaler # Running+Ready\n$RKE2_KUBECTL -n kube-system get pods -l app=rke2-metrics-server # Running+Ready\n$RKE2_KUBECTL -n kube-system get pods -l k8s-app=cilium # Running+Ready, one per node\n$RKE2_KUBECTL -n kube-system get pods -l io.cilium/app=operator # Running+Ready\n$RKE2_KUBECTL -n kube-system get pods -l k8s-app=hubble-relay # Running+Ready\n$RKE2_KUBECTL -n kube-system get pods -l component=kube-proxy # Running+Ready, one per node\n$RKE2_KUBECTL run rke2-dns-check-<suffix> --restart=Never \\\n --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 \\\n --command -- sleep 300\n$RKE2_KUBECTL wait --for=condition=Ready pod/rke2-dns-check-<suffix> --timeout=300s\n$RKE2_KUBECTL exec rke2-dns-check-<suffix> -- nslookup kubernetes.default.svc.cluster.local\n$RKE2_KUBECTL run rke2-sched-check-<suffix> --restart=Never \\\n --image=registry.k8s.io/e2e-test-images/busybox:1.36.1-1 \\\n --overrides='{\"spec\":{\"nodeSelector\":{\"wetopi.io/role\":\"platform\"}}}' \\\n --command -- sleep 300\nExpected success:
\nwetopi.io/role=platform (count == 3);k8s-app=kube-dns and autoscaler k8s-app=kube-dns-autoscaler min 1 each; metrics-server app=rke2-metrics-server min 1; Cilium agent k8s-app=cilium one per node; cilium-operator io.cilium/app=operator min 2; hubble-relay k8s-app=hubble-relay min 1; kube-proxy component=kube-proxy one per node);kubernetes.default.svc.cluster.local to answer address(es) in 10.68.0.0/16 \u2014 the playbook parses the ANSWER addresses (the Address lines AFTER the Name: kubernetes.default... line) into a CSV string and requires every entry to match the octet-bounded regex ^10\\.68\\.\\d{1,3}\\.\\d{1,3}$; the resolver's own Server: 10.68.0.10 line appears BEFORE the Name line and must never satisfy the gate;rke_platform (placement on a CP = FAIL \u2014 investigate taints/labels).KUBECONFIG set to the transferred admin kubeconfig; the kubectl lines in THIS block are the ops01 remote form, not CP-local)cilium status --wait # all agents ready, KVStore/Conntrack/ControllerStatus Ok, Hubble Ok/Enabled\ncilium connectivity test # all tests pass, incl. cross-node pod-to-pod, pod-to-service, north-south\nhubble status # flows counters increasing, sane uptime\nhubble observe --since 5m --from-cluster # real flows with identities and VERDICTs, not empty\nkubectl run dns-utils --image=busybox:1.36 --restart=Never -- sleep 60\nkubectl wait --for=condition=Ready pod/dns-utils --timeout=120s\nkubectl exec dns-utils -- nslookup kubernetes.default # resolves into 10.68.0.0/16\nkubectl delete pod dns-utils\nkubectl get ciliumnodes.cilium.io # one CiliumNode object per node (6/6); podCIDR /24s inside 10.64.0.0/14\nNetworkPolicy sample (VALIDATION.md \u00a75):
\nkubectl create namespace cilium-validation\nkubectl -n cilium-validation run web --image=nginx:1.27 --restart=Never\nkubectl -n cilium-validation expose pod web --port 80\nkubectl apply -f kubernetes/cilium/test-default-deny.yaml # default-deny Ingress in that namespace\nkubectl -n cilium-validation run probe --image=busybox:1.36 --restart=Never \\\n -- wget -qO- --timeout=5 http://web # MUST fail (dropped)\nhubble observe --namespace cilium-validation --since 5m # shows VERDICT Dropped\nkubectl delete -f kubernetes/cilium/test-default-deny.yaml\nkubectl delete namespace cilium-validation\nkubectl -n kube-system get pods -l k8s-app=cilium -o wide \u2192 exactly 6 Running+Ready pods, one on each of the 6 nodes (3 CP + 3 platform).kubectl get ciliumnodes.cilium.io \u2192 exactly 6, one per node, each podCIDR inside 10.64.0.0/14.STOP (D2 overall): any node not Ready after the retry window; any selector returning pods that are not all Running+Ready (or missing entirely); DNS gate failing (rc != 0, zero parsed answers, or any answer outside 10.68.0.0/16); sched-check not placed on a platform-labeled node; cilium connectivity test failures; hubble observe empty; NetworkPolicy probe NOT dropped; any podCIDR outside 10.64.0.0/14; fewer than 6 CiliumNode objects or missing cilium-ds pod on any node.
\nThe executable order (identical to the banner, the F6 heading + STOP rule, the Phase W heading, the DoD mapping, and the provenance section) is B \u2192 C \u2192 STOP \u2192 F6-1..3 \u2192 E \u2192 D \u2192 D2 \u2192 W \u2192 F \u2192 G: the CP-only Fase-6 gates (F6-1 quorum, F6-2 one-CP failure, F6-3 snapshot + REAL restore test) sat BEFORE this phase; Phase W \u2014 the worker lifecycle drill (drain + reboot) \u2014 runs NEXT, after these D2 gates pass, on the final 6-node topology. Do NOT count Phase W as completed pre-addon acceptance.
\n/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml drain rke-platform01 --ignore-daemonsets --delete-emptydir-data\n/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml uncordon rke-platform01\nSchedulingDisabled during the drain; after uncordon the node returns to Ready/Schedulable.sudo reboot on rke-platform02 (via the documented SSH form, \u00a70 item 5), then on cp01 (CP-local form):\n/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get node rke-platform02 -w # wait for Ready\nwetopi.io/role=platform still present; no NotReady residue after ~5 min.STOP (W overall): either W-1/W-2 gate failing \u2014 fix before Phase F. With W-1/W-2 recorded PASS (post-D2) alongside F6-1..3 (pre-addon), the Fase-6 acceptance ledger is COMPLETE; the DoD maps drain/reboot recovery to this POST-D2 evidence, never to pre-addon acceptance.
\nSource artifacts (all on merged main): ansible/roles/nat_gateway (vars natgw_ceph_routing_enabled: false, natgw_ceph_address: \"\" \u2014 default-off until gates), kubernetes/ceph-csi/ (chart ceph-csi-rbd pinned 3.17.1, StorageClass ceph-rbd, secret.example.yaml), ansible/playbooks/ceph-csi.yml, kubernetes/ceph-csi/VALIDATION.md, kubernetes/ceph-csi/README.md.
Phase F has TWO independent sub-phases; the CSI part does not block the cluster (RKE2 installs with no storage class).
\nThe third NIC for the Ceph leg lives in tofu/proxmox/environments/prod/terraform.tfvars.example: a commented-out block at the end of the wetopi-natgw01 nic list with the ADR-0004 comment above it (VLAN 2 / vmbr2, address from 192.168.96.0/20; guest-side config via the nat_gateway role's natgw_ceph_* vars, NOT cloud-init). F1 requires it enabled via a reviewed PR, not an apply-time edit:
# {\n# bridge = \"vmbr2\"\n# ipv4_address = \"192.168.96.<allocated>/20\"\n# firewall = false\n# }\n{ bridge = \"vmbr2\", ipv4_address = \"192.168.96.<allocated>/20\", firewall = false }.tofu -chdir=tofu/proxmox/environments/prod plan\ntofu apply = HUMAN gate \u2014 a human explicitly reviews the plan and runs the apply; record evidence (plan summary + apply output) in the apply log.STOP (F0): plan shows any destroy; plan touches any other VM/resource; apply fails.
\nADR-0004 shape: route, not NAT; third Nic on wetopi-natgw01, VLAN 2, into Ceph LAN 192.168.96.0/20.
\nPreconditions (human, from VALIDATION.md \u00a71 \u2014 each gate verified and recorded BEFORE flipping the vars):
\n| Gate | \nCheck | \n
|---|---|
| (a) VLAN 2 address allocation | \npick a free address in 192.168.96.0/20 \u2014 check live allocations (ip -4 route, Ceph-side ceph mon dump) | \n
| (b) Ceph-side return-route owner | \nwho installs ip route 10.72.0.0/23 via <natgw VLAN 2 addr> on Ceph hosts (not managed by us \u2014 confirm with Ceph owner, record mechanism) | \n
| (c) firewall path both directions | \nnft list chain inet wetopi_filter forward shows the two ceph_net accept rules; nft list table ip wetopi_nat still scopes NAT to oifname WAN only | \n
| (d) rp_filter=1 behavior | \nsysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.default.rp_filter; asymmetric drops = missing gate b | \n
| (e) MTU 1500 on VLAN 2 | \nfrom a CP: ping -c 3 -M do -s 1472 <natgw-ceph-addr> passes, -s 1473 fails | \n
| (f) no CIDR overlap | \n10.64.0.0/14 + 10.68.0.0/16 vs 192.168.96.0/20 + 10.72.0.0/23 (ip -4 route review; ip route get <ceph-host> takes the gateway) | \n
Apply: via reviewed PR, not an apply-time edit \u2014 open a PR setting natgw_ceph_routing_enabled: true + natgw_ceph_address: <chosen> in ansible/inventories/prod/group_vars/all.yml, merge it, then check out a clean main and re-run the baseline playbook (idempotent; its first play applies nat_gateway to wetopi-natgw01):
ANSIBLE_CONFIG=ansible/ansible.cfg \\\n .venv/bin/ansible-playbook -i ansible/inventories/prod/hosts.json \\\n ansible/playbooks/bootstrap.yml\nGateway checks (on wetopi-natgw01, VALIDATION.md \u00a72): ip -4 route shows 192.168.96.0/20 dev eth2 ... src <natgw_ceph_address>; ip route get 192.168.96.10 via eth2; ip route get 10.72.0.21 still via eth0 src 10.72.0.2. Ceph-side (prerequisite, VALIDATION.md \u00a73): ip route get 10.72.0.21 on a Ceph host must go via the natgw VLAN 2 address.
CP-node connectivity (from 10.72.0.21, VALIDATION.md \u00a74):
\nping -c 3 192.168.96.10 # Ceph monitor; 0% loss\ntraceroute -n 192.168.96.10 # one hop 10.72.0.1 (gateway), then the mon host\nnc -vz -w 3 192.168.96.10 3300 # msgr2 open\nnc -vz -w 3 192.168.96.10 6789 # msgr1 open\ntimeout 3 bash -c 'echo > /dev/tcp/192.168.96.10/3300' && echo OK # fallback if nc absent\nHuman prerequisites (README OPEN list): dedicated Ceph pool (suggested kubernetes); dedicated Ceph user + caps (e.g. client.kubernetes-csi, never admin); live clusterID (fsid) + monitor addresses replacing the placeholders in kubernetes/ceph-csi/values.yaml + storageclass.yaml \u2014 via reviewed PR, then merge and a clean main checkout (placeholders are tracked files; never edit them at apply time); csi-rbd-secret in kube-system created BY HAND from that user's keyring per secret.example.yaml (key material never enters Git; Vault flow later).
STOP [encryption-gate] before creating csi-rbd-secret: the Ceph key stored in a Kubernetes Secret must be confirmed PROTECTED at rest by the measured-protection checks of \u00a70 item 8 (the operative gate rule \u2014 current protection is determined by measured evidence, not by config-template contents; this build's default-on AES-CBC state is the baseline, re-verify, do not assume). Before creating the Secret: rke2 secrets-encrypt status on all three CPs showing Encryption Status: Enabled / Current Rotation Stage: start / Server Encryption Hashes: All hashes match and an Active * AES-CBC aescbckey... key (\u00a70 item 8 check (i)). Then create csi-rbd-secret per the prerequisites above and immediately run check (ii) on it: etcdctl ... get /registry/secrets/kube-system/csi-rbd-secret via the approved kubectl-exec 17-arg form (\u00a70 item 8) showing k8s:enc:aescbc ciphertext, NOT the plaintext from kubectl get secret csi-rbd-secret -n kube-system -o jsonpath='{.data.<KEY>}' | base64 -d. STOP if either check fails or shows plaintext \u2014 do not run the csi playbook. (The explicit reviewed secrets-encryption: true line is the separately tracked future hardening/rotation prerequisite per \u00a70 item 8 \u2014 it is not an unlock condition here while the measured checks pass.) Then:
ANSIBLE_CONFIG=ansible/ansible.cfg \\\n .venv/bin/ansible-playbook -i ansible/inventories/prod/hosts.json \\\n ansible/playbooks/ceph-csi.yml\n(Run from the controller or ops01 \u2014 the play targets rke_control_plane with connection: local + run_once: true, so helm/kubectl talk to the cluster API from wherever ansible runs; kubeconfig resolution: -e kubeconfig= \u2192 $KUBECONFIG \u2192 /etc/rancher/rke2/rke2.yaml \u2014 on a non-CP host the transferred kubeconfig must be in KUBECONFIG, \u00a70 item 9.)
The playbook fails early on placeholders or missing secret (by design); it targets rke_control_plane with connection: local, run_once: true, kubeconfig resolution: -e kubeconfig= \u2192 $KUBECONFIG \u2192 /etc/rancher/rke2/rke2.yaml; helm upgrade --install ceph-csi-rbd ceph-csi-rbd --repo https://ceph.github.io/csi-charts --version 3.17.1 -n kube-system --values kubernetes/ceph-csi/values.yaml; then applies storageclass.yaml.
PVC acceptance (VALIDATION.md \u00a75, maps master plan \u00a735.3):
\nkubectl -n kube-system get pods -l app=ceph-csi-rbd # all Running\nkubectl get storageclass ceph-rbd # provisioner rbd.csi.ceph.com, allowVolumeExpansion true\nkubectl apply -f - <<'EOF'\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n name: ceph-rbd-test\nspec:\n accessModes: [\"ReadWriteOnce\"]\n storageClassName: ceph-rbd\n resources:\n requests:\n storage: 1Gi\nEOF\nkubectl get pvc ceph-rbd-test -w # Bound within ~1 min\n# (b) mount+write: kubectl run ceph-rbd-test --rm -it --restart=Never --image=nginx:alpine \\\n# --overrides='{\"spec\":{\"containers\":[{\"name\":\"ceph-rbd-test\",\"image\":\"nginx:alpine\",\"command\":[\"sh\"],\"stdin\":true,\"tty\":true,\"volumeMounts\":[{\"name\":\"data\",\"mountPath\":\"/mnt/data\"}]}],\"volumes\":[{\"name\":\"data\",\"persistentVolumeClaim\":{\"claimName\":\"ceph-rbd-test\"}}]}}'\n# inside: date > /mnt/data/marker.txt && cat /mnt/data/marker.txt && exit\n# (c) persistence: delete the pod, re-run the SAME kubectl run, marker still readable with the same timestamp\n# (d) failure test: kubectl delete pvc ceph-rbd-test; on a Ceph host: rbd ls kubernetes \u2192 no csi-vol- image for the PVC\n# (e) performance: no CSI errors/timeouts in kubectl -n kube-system logs deploy/ceph-csi-rbd-provisioner (and node plugin logs) across (a)-(d); provisioning within ~1 min\nRollback F: routing \u2014 revert natgw_ceph_routing_enabled: false (and clear the address) via reviewed PR and re-run bootstrap.yml (the role removes the leg; the default-off state is the reviewed baseline). CSI \u2014 inverse of the install: helm uninstall ceph-csi-rbd -n kube-system, kubectl delete -f kubernetes/ceph-csi/storageclass.yaml, delete test PVCs first; keep csi-rbd-secret out of Git always. (This rollback line is runbook-authored inverse of the playbook install; the parent README documents no uninstall procedure.)
STOP (F0): plan shows destroys or extra resource churn.\nSTOP (F1): missing connected route, wrong next-hop, ping loss, TCP 3300/6789 closed, MTU probe failing.\nSTOP (F2): [encryption-gate] measured-protection checks failing (\u00a70 item 8); PVC Pending (describe it; check provisioner logs \u2014 usual causes: placeholder clusterID/monitors, wrong userKey, routing regression), any acceptance step (b)-(e) failing.
\nSource: infrastructure PR #1 (MERGED, main): kubernetes/gitops/ + docs/gitops.md. Pinned Argo CD v3.4.8, non-HA, vendored manifest, namespace argocd. Run from ops01 with a clean checkout of infrastructure main at d553643 or a verified descendant (\u00a70 item 9 ops01 provisioning; every kubectl below is the ops01 remote form with KUBECONFIG set to the transferred admin kubeconfig). docs/gitops.md (infrastructure repo) is the authoritative step-by-step procedure and has been reconciled to the deployed state \u2014 its steps 1-19 now match what was applied: HTTPS repository Secret, split/server-side CRD apply, current encryption status, drift deferral. The commands below are the same procedure restated; where wording differs, follow docs/gitops.md.
Install (docs/gitops.md steps 1-6); SSH to ops01 (the bastion itself, 92.222.245.91) is DIRECT \u2014 no ProxyCommand (reserve the ProxyCommand form for LAN targets behind ops01):
\nssh -i /opt/data/.ssh/bigardo ubuntu@92.222.245.91\ncd ~/wetopi-infrastructure\ngit fetch origin && git checkout main && git pull --ff-only\nsha256sum kubernetes/gitops/bootstrap/argocd-install-v3.4.8/install.yaml # optional integrity check\nkubectl apply -f kubernetes/gitops/bootstrap/argocd-install-v3.4.8/00-namespace.yaml\n# The pinned install.yaml is a 59-document manifest. The CRD\n# applicationsets.argoproj.io (1385776 bytes raw; ~341KB serialized) exceeds\n# kubectl's 262144-byte client-side last-applied-configuration annotation\n# cap, so a broad client-side apply of the whole manifest FAILS on that\n# object (\"metadata.annotations: Too long\"). The other large CRD,\n# applications.argoproj.io, serializes to ~191KB and stays under the cap \u2014\n# only applicationsets needs special handling. REQUIRED METHOD (validated\n# for both first install and re-apply; no CRD-presence branch, no extra\n# tools \u2014 plain awk): split the CRD out, apply it SERVER-SIDE (idempotent\n# whether the CRD is absent or already present), apply the remaining 58\n# objects client-side. Same form for installs, re-applies after upgrades,\n# and the Phase G rollback path.\nawk '\n function flush() {\n if (buf == \"\") { return }\n if (want) { print buf > \"/tmp/argocd-install-appsets-crd.yaml\" }\n else {\n if (n++) print \"---\" > \"/tmp/argocd-install-rest.yaml\"\n print buf > \"/tmp/argocd-install-rest.yaml\"\n }\n buf = \"\"; want = 0; kind = 0\n }\n /^---$/ { flush(); next }\n /^kind: CustomResourceDefinition$/ { kind = 1 }\n /^ name: applicationsets\\.argoproj\\.io$/ { if (kind) want = 1 }\n { buf = buf $0 \"\\n\" }\n END { flush() }\n' kubernetes/gitops/bootstrap/argocd-install-v3.4.8/install.yaml\n# sanity check before applying (expected counts on the pinned manifest):\ngrep -c '^apiVersion:' /tmp/argocd-install-appsets-crd.yaml # 1\ngrep -c '^apiVersion:' /tmp/argocd-install-rest.yaml # 58\nkubectl apply --server-side -f /tmp/argocd-install-appsets-crd.yaml\nkubectl apply -n argocd -f /tmp/argocd-install-rest.yaml\nrm /tmp/argocd-install-appsets-crd.yaml /tmp/argocd-install-rest.yaml\nkubectl apply -f kubernetes/gitops/bootstrap/argocd-install-v3.4.8/01-cmd-params.yaml\nValidation (steps 7-12): kubectl -n argocd get pods (application-controller StatefulSet 1 replica by pin, applicationset-controller, dex (idle; SSO OPEN), redis, repo-server, server \u2014 all Running/Ready); port-forward kubectl -n argocd port-forward svc/argocd-server 9090:443 --address=127.0.0.1; install CLI curl -sSLf -o /tmp/argocd https://github.com/argoproj/argo-cd/releases/download/v3.4.8/argocd-linux-amd64 + chmod +x + sudo mv /usr/local/bin + argocd version --short (expect v3.4.8 client+server); initial password kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d; echo (never commit); argocd login 127.0.0.1:9090 --username admin --insecure. UI from workstation via:\nssh -i /opt/data/.ssh/bigardo -L 8443:127.0.0.1:9090 ubuntu@92.222.245.91 (direct to the bastion, NO ProxyCommand \u2014 ops01 IS the bastion; the ProxyCommand form stays reserved for LAN targets behind ops01).
[encryption-gate] on the repo credential (steps 13-17) \u2014 ORDERED GATE PROCEDURE (status-check BEFORE, ciphertext-verify AFTER, cleanup STOP on failure). The Argo CD repo credential is stored as a Kubernetes Secret, and \u2014 key semantic \u2014 creating the labeled repository Secret IS the registration: Argo CD auto-discovers any Secret carrying the label argocd.argoproj.io/secret-type=repository, so a create-without-registering sequence does not exist. The gate therefore does not \"hold registration until after the check\"; it verifies the at-rest protection of the Secret that already exists, per the measured-protection checks of \u00a70 item 8 (the operative gate rule: current protection = measured evidence, not config-template contents). Execute in this exact order:
rke2 secrets-encrypt status on all three CPs, each showing Encryption Status: Enabled / Current Rotation Stage: start / Server Encryption Hashes: All hashes match / Active * AES-CBC aescbckey key (\u00a70 item 8 check (i)). Fails \u2192 STOP here, before any credential material enters the cluster; re-verify before continuing.argocd/repo-infra-https exactly per the \"Repo credential\" block below (protected read -rs token input; stdin-only kubectl -n argocd apply -f - heredoc; label argocd.argoproj.io/secret-type=repository; fields type: git, url, username, password). Creation auto-registers the credential (Argo CD label auto-discovery).get /registry/secrets/argocd/repo-infra-https must return k8s:enc:aescbc ciphertext, NOT the plaintext/base64 of the token.kubectl -n argocd delete secret repo-infra-https (this also removes the auto-registered credential), unset ARGOCD_REPO_TOKEN, then STOP \u2014 do not proceed to the root Application \u2014 and revoke/rotate the token on GitHub as appropriate (a credential that sat readable in etcd must be treated as exposed). Do not re-attempt from step 2 until the at-rest encryption fault is found, fixed, and re-measured (checks (i) and (ii) green again).argocd repo list shows https://github.com/wetopi-platform/infrastructure.git CONNECTION STATUS Successful \u2014 and only then proceed to the root Application (steps 18-19).This gate does NOT block the Argo CD install itself (steps 1-12 above), only the private-repo credential.\nAPPLY-TIME VERDICT (2026-08-30, card t_e0cace4b): gate PASSED on measured evidence, in exactly the ordered sequence above \u2014 check (i) verified on all 3 CPs BEFORE the credential Secret existed; the labeled Secret argocd/repo-infra-https was then created, which auto-registered the credential via Argo CD's label auto-discovery; raw-etcd check (ii) immediately probed the created Secret and returned k8s:enc:aescbc:v1:aescbckey: ciphertext (685B total, NOT plaintext; argocd/argocd-initial-admin-secret also probed: 316B ciphertext); only after that verification did execution proceed to the repo-list gate (argocd repo list \u2192 Successful) and the root Application. The cleanup STOP path was not needed.
Repo credential (steps 13-17; private repo \u2192 mandatory before first sync). APPLY-TIME DEVIATION (lead decision, card t_e0cace4b) \u2014 the deploy-key path is SUPERSEDED: GitHub org policy DISABLES deploy keys on wetopi-platform/infrastructure (API returns HTTP 422 \"Deploy keys are disabled for this repository\", org-level setting), so the deployed mechanism is an HTTPS read credential \u2014 the token-based repository Secret below, auto-discovered by Argo CD. The exact step-by-step procedure (steps 13-16, secret-safe form) lives in docs/gitops.md in the infrastructure repo, \"Private repository credential (apply-phase, before first sync)\" \u2014 read token from GitHub into a shell variable via read -rs (protected input, never typed into a command or history); create/update Secret argocd/repo-infra-https in the argocd namespace by kubectl -n argocd apply -f - heredoc referencing ${ARGOCD_REPO_TOKEN} (the manifest exists only on stdin \u2014 never in Git, never a file on disk) with the required label argocd.argoproj.io/secret-type=repository and required stringData fields type: git, url: https://github.com/wetopi-platform/infrastructure.git, username: box-daxter, password: <the variable>; unset ARGOCD_REPO_TOKEN after; rotation = re-run the same block, cleanup = kubectl -n argocd delete secret repo-infra-https + revoke the token on GitHub. Do NOT instead use inline argocd repo add --password ... \u2014 an inline add was tried at apply time and registered an empty credential (STATUS Failed); the labeled Secret is the only registration. When Infisical lands, this credential migrates to the vault contract.
GATE (step 17, mandatory): argocd repo list shows https://github.com/wetopi-platform/infrastructure.git CONNECTION STATUS Successful. Do not proceed to the root Application until this passes. PASSED at apply time (t_e0cace4b) via the Secret auto-discovery above (one recorded correction: an initial inline-password argocd repo add registered an empty credential, STATUS Failed \u2014 entry removed; the labeled Secret is now the only registration).
First app sync (steps 18-19):
\nkubectl apply -f kubernetes/gitops/app.project.yaml -f kubernetes/gitops/app.root.yaml\nargocd app list # one row: root, project=platform, in-cluster, Synced/Healthy\nargocd app get root # STATUS Synced, HEALTH Healthy, source infrastructure main @ kubernetes/platform\nargocd app sync root # no-op sync (target path holds placeholders only)\nargocd app wait root --health --sync --timeout 120\nkubectl get ns platform # created by the sync via CreateNamespace=true\nDrift check (documented expectations): Git is desired state; manual kubectl edit/apply/delete inside the cluster is drift and shows OutOfSync; root runs automated.prune=true, selfHeal=true so drift self-heals within the reconcile interval and every correction lands in argocd app history root. Drift procedure to record: make a manual change in the platform namespace, confirm argocd app get root shows OutOfSync then self-heals back, confirm the correction in argocd app history root; preferred rollback is always a Git revert (Argo reconciles it) \u2014 argocd app rollback <id> exists for emergencies but must be followed by a Git fix or selfHeal re-applies Git state. Note the Argo CD control plane itself is apply-only from pinned manifests (procedural exclusion, re-apply install.yaml after reviewed upgrades).\nAPPLY-TIME DEVIATION \u2014 drift check DEFERRED (card t_e0cace4b): structurally untestable at bootstrap. kubernetes/platform/ holds a README placeholder only (0 tracked resources), so the Application has no desired state to diverge from: a probe ConfigMap created manually in the platform ns kept Synced/Healthy (Argo cannot flag an object outside its tree; OutOfSync/selfHeal/prune have nothing to act on). Probe deleted; cluster left clean. The automated.prune=true, selfHeal=true manifest evidence IS factual (verified in the app JSON spec). The check is trivially testable once \u22651 real workload exists under kubernetes/platform/ (e.g. the first app-of-apps child of a later phase) \u2014 re-run the drift procedure above then, and test drift by EDITING or DELETING one of the Git-managed objects in the cluster (never by creating another unmanaged object outside the Application's tree, which is invisible to Argo by design).
Apply-time deviations (2026-08-30, card t_e0cace4b \u2014 reconciled into this runbook and the repo):
\napplicationsets.argoproj.io CRD (1385776 bytes raw; ~341KB serialized) exceeds kubectl client-side apply's 262144-byte last-applied-configuration annotation cap \u2014 a broad client-side apply of the whole install manifest fails with metadata.annotations: Too long. Required apply method: awk-split that single CRD out of the pinned manifest and kubectl apply --server-side -f it (recorded serverside-applied; idempotent on first install and re-apply \u2014 no CRD-presence branch needed); the remaining 58 objects stay client-side apply per this runbook. Any re-apply of the pinned manifest must use the same split form.kubernetes/gitops/app.project.yaml and app.root.yaml were applied with repoURL/sourceRepos git@github.com:wetopi-platform/infrastructure.git \u2192 https://github.com/wetopi-platform/infrastructure.git (deploy keys disabled at org level); credential = Secret argocd/repo-infra-https (see \"Repo credential\" above). The infrastructure repo is reconciled to the applied state by PR (infrastructure) \u2014 repoURL lines updated to HTTPS; the app yamls now match what is deployed.Execution evidence (t_e0cace4b, 2026-08-30): Argo CD v3.4.8 installed (ops01 checkout at main f841ae7, PIN SHA-256 e223122b\u2026 matched docs/gitops.md); 7/7 pods Running/Ready; CLI v3.4.8 client+server match; encryption gate PASSED on measured AES-CBC evidence in the ordered sequence (check (i) pre-credential, check (ii) ciphertext on the created repo-infra-https Secret, cleanup path not needed); repo credential Successful; argocd app list \u2192 one row root, project platform, in-cluster, https://github.com/wetopi-platform/infrastructure.git @ kubernetes/platform @ main, SYNCPOLICY Auto-Prune; first sync Succeeded (no-op, placeholders only), root Synced to main (b610209) Healthy; ns platform Active; history ID 0.
STOP: pods not all Running; [encryption-gate] measured-protection checks failing per the ordered procedure (check (i) failing at step 1, or check (ii) failing/showing plaintext at step 3 \u2192 cleanup STOP: delete argocd/repo-infra-https, unset ARGOCD_REPO_TOKEN, revoke/rotate the token, stop \u2014 \u00a70 item 8 and the ordered gate procedure above); argocd repo list not Successful; root app not Synced+Healthy after argocd app wait. (Drift-check STOP superseded by the deferral above: no drift gate applies while kubernetes/platform/ tracks 0 resources \u2014 re-test when the first workload lands.)
On ANY phase failure \u2014 stop, do not proceed; read the playbook/task failure output first (all plays run any_errors_fatal); the run is serial, so a failed phase leaves the cluster in a known partial state. Order of escalation: (1) fix forward and re-run the phase playbooks (roles are idempotent, serial-safe); (2) per-node rollback (below); (3) etcd snapshot restore via the cluster-reset flow (systemctl stop rke2-server on all CPs \u2192 rke2 server --cluster-reset --cluster-reset-restore-path=<PATH-TO-SNAPSHOT> on cp01 \u2192 start cp01 \u2192 rm -rf /var/lib/rancher/rke2/server/db/ + start on cp02/cp03 \u2192 re-verify quorum + /readyz \u2014 see Phase F6-3, INCLUDING its reset-failure rule: a failed reset is an immediate STOP-and-escalate condition, NOT a retry \u2014 reset-flag blocks repeated resets; there is NO rke2 etcd-snapshot restore subcommand \u2014 and the F6 STOP prerequisites P1-P4 gate ANY cluster-reset restore, including from this rollback path) for cluster-state corruption; (4) Proxmox qm rollback <vmid> pre-rke2-apply (last resort \u2014 cluster-level state like etcd membership makes node-level rollback of ONE CP risky once quorum exists: prefer etcd member removal for a node you are rebuilding rather than snapshot rollback of a single CP mid-quorum; snapshot rollback is safest applied to ALL CPs together or before Phase B).
Per-phase specifics (from parent docs; the per-phase STOP blocks above remain the authoritative inline version):
\nsystemctl stop rke2-server; per node serially remove /etc/rancher/rke2, /var/lib/rancher/rke2, /var/lib/kubelet; keepalived may stay (role idempotent) or be removed; re-run the playbook from scratch after regenerating the token; do NOT manually delete etcd members unless removing a node permanently \u2014 then remove the member with the same kubectl-exec etcdctl form (Phase B gate) with member remove <ID>. Rebuild-a-node criterion: node whose config/state is unrecoverable (repeated join failures, corrupt etcd data dir) \u2014 snapshot-rollback or recreate via tofu, then re-run the CP join play.systemctl stop rke2-agent, remove /etc/rancher/rke2, /var/lib/rancher/rke2, /var/lib/kubelet, then kubectl delete node rke-platform0X; re-run the workers playbook after fixing the root cause (idempotent).helm uninstall cilium --namespace kube-system (documented; no other CNI behind it \u2014 replacement plan required).bootstrap.yml; CSI inverse (uninstall chart, delete StorageClass) as authored above.applicationsets.argoproj.io CRD out, kubectl apply --server-side -f the CRD file, kubectl apply -n argocd -f the remaining-objects file; a broad client-side apply of the whole install.yaml fails on that CRD by design), re-register the repo credential (the argocd/repo-infra-https labeled Secret per docs/gitops.md steps 13-16), re-sync; app-level rollback = Git revert (argocd app history root records syncs). Full cluster loss = rebuild from the tofu/ansible phases, then this runbook top to bottom.| Plan item | \nConcrete check in this runbook | \n
|---|---|
| \u00a735.1 p\u00e9rdida de un CP no rompe API | \nPhase C one-CP-outage test + Phase F6-2 (systemctl stop rke2-server on cp02, curl -k https://10.72.0.20:6443/readyz \u2192 ok) | \n
| \u00a735.1 etcd quorum correcto | \nkubectl-exec etcdctl member list == 3 members (Phase B/C gates + F6-1) | \n
| \u00a735.1 nodes reproducibles desde IaC | \nphases B-E run purely from tofu-generated inventory + ansible roles (re-run of the same playbooks reproduces the state; snapshot rollback to pre-rke2 then re-run = full reproduction test) | \n
| \u00a735.1 no customer workloads en CP | \nworkers playbook play 2 taint enforcement (node-role.kubernetes.io/control-plane=true:NoSchedule on all CPs) + Phase C taint record + get pods -o wide scan + D2 scheduling gate placement | \n
| \u00a735.1 snapshot funciona / restore probado | \nF6 STOP prerequisite P2 (rke2 etcd-snapshot save --name pre-addons-fase6 + rke2 etcd-snapshot ls, external copy outside cp01's failure domain) + P3 (SHA-256 match on cp01 and the external copy), then F6-3: the EXECUTED cluster-reset restore test (systemctl stop rke2-server all CPs \u2192 rke2 server --cluster-reset --cluster-reset-restore-path=<PATH-TO-SNAPSHOT> on cp01 \u2192 start cp01 \u2192 rm -rf /var/lib/rancher/rke2/server/db/ + start on cp02/cp03 \u2192 re-verify quorum 3/3 + /readyz + the post-restore journal checks on all three CPs). DoD maps \"restore probado\" to THIS executed-test evidence \u2014 snapshot-save-only is NOT restore probado. If the restore test cannot be executed, this DoD item is UNMET and the runbook mandates STOP before all subsequent phases (E onward). | \n
| \u00a735.2 Pod-to-Pod + Pod-to-Service | \ncilium connectivity test (Phase D2, final 6-node topology) | \n
| \u00a735.2 DNS | \nPhase D2 DNS gate (answer parsed into 10.68.0.0/16) + Phase D2 dns-utils nslookup (final topology) | \n
| \u00a735.2 Hubble visibility | \nhubble status + hubble observe (Phase D2, final 6-node topology \u2014 real flows + VERDICTs) | \n
| \u00a735.2 no CIDR collision | \ngate (f) ip -4 route review + kubectl get ciliumnodes.cilium.io podCIDR \u2282 10.64.0.0/14 (Phase D2) | \n
| \u00a735.2 Ceph reachable desde nodes | \nPhase F1 connectivity block (ping/traceroute/nc 3300+6789) + PVC Bound | \n
| \u00a735.2 DSR demostrado por packet capture | \nNOT covered by phases 0-G (edge/DSR is later work per the master plan); recorded as an explicit gap | \n
| \u00a735.3 Storage: PVC create/persistence/pod recreation/failure test | \nPhase F2 steps (a)-(d) | \n
| \u00a735.3 Storage: performance | \nPhase F2 step (e) | \n
| \u00a735.4 GitOps: Argo reconcilia desde Git | \nfirst sync block | \n
| \u00a735.4 GitOps: drift detectable | \ndrift procedure | \n
| \u00a735.4 GitOps: Git conserva estado deseado | \nselfHeal + Git-revert rollback rule | \n
| \u00a735.4 GitOps: rollback probado | \nGit revert is exercised only in a real incident \u2014 recorded as \"procedure documented, not yet exercised\" | \n
| \u00a735.1/\u00a735.2 node drain + reboot recovery | \nPhase W (W-1 drain, W-2 reboot + Ready return; former F6-4/F6-5) \u2014 the separately-named post-D2 phase, executed on the final 6-node topology in the executable order B \u2192 C \u2192 STOP \u2192 F6-1..3 \u2192 E \u2192 D \u2192 D2 \u2192 W \u2192 F \u2192 G. The DoD maps drain/reboot to this POST-D2 evidence, NOT to pre-addon acceptance: \u00a733 places these Fase-6 items before Fase 7 (Cilium), but ADR-0005 requires Cilium as the immediate first addon with no pre-Cilium worker-Ready gates \u2014 with cni: none the drain/reboot gates are unexecutable before Cilium exists (see the banner for the \u00a733/ADR-0005 tension and its explicit resolution). F6-1..3 (quorum, CP failure, snapshot + REAL restore test) remain the PRE-addon CP-only evidence. | \n
Runbook complete when every phase gate above has a dated PASS entry in the apply log.
\nrke2-server : Create the RKE2 server configuration with Destination directory /etc/rancher/rke2 does not exist on rke-cp01 \u2014 neither rke2 role created the config dir (byte-identical defect in rke2-agent). Infrastructure PR #6 (fix/rke2-config-dir-create, head 3ff2ee8, merged main d553643, 2026-08-29 23:56 UTC) is the fix: config-dir creation (root:root 0750) before templating in both roles. Residual pre-halt state on cp01 (keepalived installed+active, RKE2 binaries present, /etc/rancher absent, rke2-server inactive) was absorbed cleanly by the resumed run \u2014 the roles are idempotent (RKE2 install guarded by creates:). The former resume gates (infrastructure checkout at d553643 or verified descendant per the \u00a70 STEP 0 gate, harness V2 validation, explicit lead unblock) were all satisfied on 2026-08-30 before resume; they remain historical prerequisites of that resume, not open conditions.main @ d553643 (PR #6 merge \u2014 config-dir fix for both RKE2 roles; prior PRs #1 8f3e0e8, #2 e283e24, #3 76782dc, #4 b15a087, #5 8d836e8 all merged \u2014 no open-PR references remain), docs main @ 5740404 (ADR-0005 and platform runbooks in main).cni: none, workers cannot pass Ready (and therefore drain/reboot) gates before Cilium exists; the resolution adopted by this runbook is that Fase-6's CP-only gates (F6-1..3) run before the first addon per \u00a733, and the worker-lifecycle drill is the separately-named Phase W landing after D2 on the final 6-node topology; the DoD maps drain/reboot to that post-D2 evidence, never to pre-addon acceptance. Phase W is not counted as completed pre-addon acceptance.ansible/playbooks/{rke2-cp,rke2-workers,rke2-workers-validate,cilium,ceph-csi,bootstrap}.yml, ansible/roles/{keepalived,rke2-server,rke2-agent,nat_gateway}/, kubernetes/{cilium,ceph-csi,gitops}/, docs/gitops.md, scripts/{gen-rke2-token.sh,render-ansible-inventory.sh}, tofu/proxmox/environments/prod/terraform.tfvars.example.cilium.yml runs its Helm op on the operations host with connection: local. CP-local kubectl invocations use the RKE2-bundled binary path /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml, matching the playbooks' own rke2_kubectl form.secrets-encrypt status Enabled on all 3 CPs; raw etcd probes of both argocd Secrets returned k8s:enc:aescbc ciphertext), NOT by config-template contents. The explicit secrets-encryption: true role-template line is still absent from merged main (git grep secrets-encryption \u2192 empty) \u2014 it is the separately tracked future hardening/rotation prerequisite (must merge via its own reviewed PR before any key rotation), not an unlock condition for F2/G while the measured checks pass. F2's csi-rbd-secret gate (when Phase F executes) re-verifies the same measured checks on that Secret.docs/design/master-architecture-plan.md \u00a733/\u00a735 (Fase 6 = acceptance; CP-only portion BEFORE Fase 7 Cilium; worker-lifecycle portion after D2 per ADR-0005 \u2014 see the executable-order bullet above); docs/decisions/ADR-0004, docs/decisions/ADR-0005.docs/datastore/backup_restore.md (etcd-snapshot subcommands = delete/ls/prune/save ONLY; restore = rke2 server --cluster-reset --cluster-reset-restore-path=<PATH> multi-server flow; reset creates /var/lib/rancher/rke2/server/db/reset-flag preventing repeated resets in succession, deleted only by a normal start \u2014 verified 2026-08-30 against the main branch of rancher/rke2-docs). Flag lifecycle additionally verified 2026-08-30 against the k3s source vendored by the pinned rke2_version: v1.36.3+rke2r1 (k3s commit 4ca29faac89e via rke2 go.mod; pkg/cluster/managed.go: the flag check is bypassed when --cluster-reset-restore-path is set and the flag blocks only a bare reset; pkg/etcd/etcd.go Reset(): the flag is written only after the snapshot-restore step succeeds).docs/security/secrets_encryption.md (auto-generated AES-CBC key + /var/lib/rancher/rke2/server/cred/encryption-config.json; rke2 secrets-encrypt status output shape: Encryption Status / Current Rotation Stage / Server Encryption Hashes / key table).\n\nMigrado desde
\nwetopi-platform/docs@main(project/taking-over-the-platform.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
PROC-PLATFORM-TAKEOVER: Taking over the platform
\nThe platform map is\nARCH-PLATFORM-TOPOLOGY \u2014 read it\nfirst. This guide is the procedure: access, truth sources, reading order,\nday-1 checks, the operations you will run most often, where to escalate, and\nwhat is never allowed. It links out for everything it does not need to\nduplicate.
\nUse this when you are newly taking over operations of the platform \u2014 on a\nshift, a rotation, or an emergency handover. It is deliberately a composed\nguide of steps (the one allowed exception to the one-concept-per-pill rule);\nevery fact it does not restate lives in a linked pill or runbook.
\nNever search for a credential's value in this repo, the infrastructure\nrepo, a ticket or a chat \u2014 only its location is documented (DocOps \u00a78).
\n| Access | \nWho | \nCredential location (never the value) | \nNotes | \n
|---|---|---|---|
| SSH to the fleet | \noperator holding the bigardo key | \nthe workstation's ~/.ssh/bigardo (key pair issued out-of-band; the public side is pushed to VMs by the ssh_keys role) | \nops01 (ubuntu@92.222.245.91) is the bastion; all LAN VMs are reached through it. ops01 itself holds no private keys \u2014 it cannot SSH to the LAN by itself. | \n
| Vault (human UI) | \nthe taking-over operator | \nnative Infisical account (username + password + TOTP) \u2014 the account is provisioned by the current operator at handover time; no value is documented | \nhttps://vault.wetopi.net \u2192 project wetopi-harness. The UI's own login+TOTP is the gate (no basic-auth). | \n
| Vault (machine identities / Universal Auth) | \nagent profiles / automation only | \nclient credentials live under project wetopi-harness in the Vault itself | \nprivate-network-only: http://vault.vault.svc.cluster.local \u2014 never the public route, which fail-closes by design (ADR-0009). | \n
kubectl on the cluster | \noperator via ops01 | \nkubeconfig fetched at apply-time from rke-cp01 (the ops_kubectl role; nothing persisted in Git) | \nsee ARCH-PLATFORM-RKE2 for the API VIP. | \n
| OpenTofu (PROD) | \nsingle writer, by explicit task | \n.env.proxmox on the phase-1 controller (0600); Cloudflare token on ops01 at /home/ubuntu/secrets/.cloudflare-wetopi-net.env (0600) | \npulled over SSH per session, never written to disk in the repo. State: terraform.tfstate 0600, never in Git (ARCH-INFRA-PROVIDERS). | \n
| Proxmox API | \nthe same .env.proxmox | \nProxmox VE HTTPS API token (env var) | \nno SSH to hypervisors \u2014 API only. | \n
| Cloudflare DNS | \nthe Cloudflare token above | \nZone DNS Edit + Zone Read (both required) | \nrecords are tofu-managed \u2014 do not edit in the dashboard. | \n
Handover checklist for credentials: at the handover meeting the outgoing\noperator must (1) verify you can reach ops01 and one LAN node with your\nbigardo key, (2) hand you a working Vault human account (or provision one\nyou can then rotate), (3) confirm the single-writer tofu lock is with you.\nIf any of these fail, stop and escalate \u2014 do not improvise credentials.
kubectl (via ops01), Proxmox API, Cloudflare API,\nCeph. \"What IS happening.\"infrastructure\nrepo (IaC: tofu/, ansible/, kubernetes/). A merge is a deploy.docs/). The map:\nARCH-PLATFORM-TOPOLOGY.terraform.tfstate (0600, phase-1 controller) is the\nOpenTofu claim on PROD. Lost state \u2192\nRUN-TOFU-STATE-LOSS,\nnever a speculative apply.All from ops01 (kubeconfig in ~/.kube/config or fetched per the\nops_kubectl model \u2014 never copied into Git):
kubectl get nodes -o wide \u2014 expect 8/8 Ready\n(cp01/02/03, platform01-03, wan01/02; IPs per\nARCH-PLATFORM-RKE2).kubectl -n argocd get applications \u2014 expect\nroot, cert-manager, docs-site, argo-edge Healthy+Synced; vault\nSynced \u2014 its Degraded health is the documented false positive of\nthe fail-closed deny rules (verify pods 3/3 Running in vault ns and\ncheck SVC-VAULT \"estado honesto\" before\ntreating it as a fault).kubectl -n kube-system get gateway,httproute \u2014 Gateway\nwetopi-edge Accepted+Programmed, listeners 8000/8443;\nkubectl get certificate -A \u2014 argo-tls, docs-tls, vault-tls\n(and edge-test-tls) Ready. Public probe: curl -sI https://docs.wetopi.net and https://argo.wetopi.net return 200/302\nwith a valid cert chain.docs/argo/vault/edge-test .wetopi.net \u2192 79.137.114.193/.194 (grey-cloud, TTL 300) \u2014 tofu-owned;\nverify with dig + the zone, don't trust the dashboard screenshot.tofu plan on PROD should be no-change; a plan\nwanting to CREATE everything means lost state \u2192 the state-loss runbook.blocked cards \u2014\nunderstand what is mid-flight before touching anything.tofu/proxmox/environments/prod/ tfvars (0600) \u2192 PR in the\ninfrastructure repo \u2192 merge is the deploy. The provider needs the\nCloudflare token env (ops01, 0600). Records are\nprevent_destroy-annotated \u2014 deleting one is a recorded decision.kubernetes/<app>/ in\nthe infrastructure repo \u2192 PR \u2192 review (blast-radius: a merge deploys)\n\u2192 merge; Argo CD syncs (prune + selfHeal). No kubectl apply of\nmanifests from a workstation; no CI kubectl apply pipeline exists by\ndesign (ARCH-PLATFORM-GITOPS).vault.wetopi.net,\nproject wetopi-harness): rotate in Infisical, let the\nInfisical Operator sync to the cluster; SOPS+age\nis bootstrap/recovery only, not the daily vault (ADR-0009). The vault\ndatastore backup/restore contract (pg_dump + age, rehearsed) is in the\nADR AC8 \u2014 rehearse before calling it production.kubectl get/describe/logs via ops01.\nRead-only cluster reads are always allowed; anything mutating is a card\nwith a gate.| Situation | \nRunbook / owner | \n
|---|---|
| Cluster node down, CP quorum loss, DR | \nRKE2 apply & restore runbook | \n
terraform.tfstate lost / empty plan wants to create everything | \nRUN-TOFU-STATE-LOSS | \n
| A service is unhealthy | \nits service fiche (deploy path + dependencies + honest state) | \n
| Something that matches no runbook | \nfile a kanban card for the right lane (ops/senior/lead); the board is the queue of truth | \n
These are non-negotiable; violations are incidents, not \"creative\ntroubleshooting\":
\ntofu apply / kubectl apply from a card or on a hunch \u2014\nmutations run only when the task body explicitly authorizes that\nspecific apply with its gates; the board is the gate.You have successfully taken over when: day-1 checks (d) all pass and you\ncan name, for any node in the\ntopology map, its pill, its\ncredential location, and the runbook that applies to its failure mode.
\n" }, { "id": 62, "title": "SVC-ARGOCD \u2014 GitOps plane", "slug": "svc-argocd-gitops-plane", "book_id": 3, "chapter_id": 21, "description": null, "created_at": "2026-09-06T13:39:04.000000Z", "updated_at": "2026-09-06T21:47:08.000000Z", "markdown": "> **Mirror de lectura** \u00b7 Mirror de Git: `docs/services/argocd.md`. Git es el SSOT.\n\n> test banner", "html": "\n\nMirror de lectura \u00b7 Mirror de Git:
\ndocs/services/argocd.md. Git es el SSOT.
\n\n" }, { "id": 63, "title": "SVC-CEPH-CSI \u2014 storage driver", "slug": "svc-ceph-csi-storage-driver", "book_id": 3, "chapter_id": 21, "description": null, "created_at": "2026-09-06T13:39:05.000000Z", "updated_at": "2026-09-06T21:45:54.000000Z", "markdown": "> Migrado desde `wetopi-platform/docs@main` (services/ceph-csi.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.\n\n Ceph CSI (RBD) \u2014 cluster storage\n\n## What is it?\n\nThe Ceph CSI driver gives the RKE2 cluster RBD-backed `PersistentVolumeClaim`s\nfrom the wetopi Ceph cluster (192.168.96.0/20). It is the first real storage\nclass on the platform: stateful workloads (the Vault datastores today,\nstateful services later) persist on Ceph instead of node-local disks.\n\n## How is it mounted?\n\n- **Chart:** `ceph-csi-rbd` **v3.17.1** (repository\n `https://ceph.github.io/csi-charts`), image\n `quay.io/cephcsi/cephcsi:v3.17.1` (chart-pinned), installed in\n `kube-system` via `ansible/playbooks/ceph-csi.yml` (helm).\n `provisioner.replicaCount: 2` (controller Deployment); the node plugin is\n a DaemonSet (one pod per node \u2014 chart default).\n- **StorageClass:** `ceph-rbd` (`kubernetes/ceph-csi/storageclass.yaml`) \u2014\n provisioner `rbd.csi.ceph.com`, pool `kubernetes` (DEDICATED pool to be\n created on the Ceph side \u2014 never reuse existing pools),\n `imageFeatures: layering`, `reclaimPolicy: Delete`,\n `allowVolumeExpansion: true`, fstype ext4.\n- **Secret (reference, never value):** `csi-rbd-secret` in `kube-system` \u2014\n a dedicated Ceph user (e.g. `client.kubernetes-csi`, NOT admin) with\n `userID`/`userKey`; shape in `secret.example.yaml`, values created\n manually at apply time (long term: from the Vault flow, ADR-0009).\n- **Values placeholders:** `clusterID` (Ceph fsid) and `monitors`\n (host:3300 msgr2) are `test banner
\n
\n\nMigrado desde
\nwetopi-platform/docs@main(services/ceph-csi.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
Ceph CSI (RBD) \u2014 cluster storage
\nThe Ceph CSI driver gives the RKE2 cluster RBD-backed PersistentVolumeClaims\nfrom the wetopi Ceph cluster (192.168.96.0/20). It is the first real storage\nclass on the platform: stateful workloads (the Vault datastores today,\nstateful services later) persist on Ceph instead of node-local disks.
ceph-csi-rbd v3.17.1 (repository\nhttps://ceph.github.io/csi-charts), image\nquay.io/cephcsi/cephcsi:v3.17.1 (chart-pinned), installed in\nkube-system via ansible/playbooks/ceph-csi.yml (helm).\nprovisioner.replicaCount: 2 (controller Deployment); the node plugin is\na DaemonSet (one pod per node \u2014 chart default).ceph-rbd (kubernetes/ceph-csi/storageclass.yaml) \u2014\nprovisioner rbd.csi.ceph.com, pool kubernetes (DEDICATED pool to be\ncreated on the Ceph side \u2014 never reuse existing pools),\nimageFeatures: layering, reclaimPolicy: Delete,\nallowVolumeExpansion: true, fstype ext4.csi-rbd-secret in kube-system \u2014\na dedicated Ceph user (e.g. client.kubernetes-csi, NOT admin) with\nuserID/userKey; shape in secret.example.yaml, values created\nmanually at apply time (long term: from the Vault flow, ADR-0009).clusterID (Ceph fsid) and monitors\n(host:3300 msgr2) are <REPLACE_ME> in Git \u2014 filled at apply time from\nlive read-only lookups (ceph fsid, ceph mon dump).flowchart TD\n subgraph CEPH[\"Ceph cluster (192.168.96.0/20)\"]\n MON[\"monitors (msgr2 :3300)\"]\n OSDS[\"OSDs / pool 'kubernetes'\"]\n end\n subgraph K8S[\"cluster wetopi3\"]\n subgraph KSYS[\"kube-system\"]\n PROV[\"csi-rbd-provisioner x2 (Deployment)\"]\n NODE[\"csi-rbd-node (DaemonSet, 1 pod/node)\"]\n SC[\"StorageClass ceph-rbd\"]\n SEC[\"Secret csi-rbd-secret (dedicated Ceph user keyring)\"]\n end\n PVC[\"PVCs (e.g. vault-postgres-data 5Gi, vault-redis-data 1Gi)\"]\n end\n NET[\"storage NIC VLAN 2 / vmbr2 \u2014 192.168.98.10/.11/.12 /25 (workers only)\"]\n PROV --> SC\n PVC --> SC\n PROV -->|\"RBD ops\"| OSDS\n NODE -->|\"mounts RBD images\"| OSDS\n PROV & NODE --> SEC\n K8S --> NET --> MON\nBootstrap-time, human-gated (Phase F of the apply runbook): the Ansible\nplaybook ceph-csi.yml installs the chart + applies the StorageClass and\nfails early while placeholders or the Secret are missing. Prerequisites\n(all human, per kubernetes/ceph-csi/README.md): (a) worker storage-NIC\naddresses DONE by the operator 2026-08-30 (192.168.98.10/.11/.12, /25 \u2014\nNIC add + firewall runs as a reviewed mutating apply, Phase W pattern,\nserial drain/reboot per worker); (b) Ceph pool kubernetes + dedicated\nuser + caps; (c) live fsid/monitors read; (d) csi-rbd-secret created.\nMigration to Argo CD ownership (values/StorageClass under Git sync) is a\nfuture reviewed change \u2014 today the playbook is the installer.
Depends on: the wetopi Ceph cluster itself (monitors + a dedicated\nkubernetes pool); the storage-native network path \u2014 third NIC on VLAN 2\n/ vmbr2 on the three platform workers with no gateway and no return\nroutes (ADR-0007 supersedes the ADR-0004 gateway-routing hypothesis;\nroute-not-NAT, Ceph subnet never renumbered); a dedicated Ceph user keyring\n(operator input, card t_8eee26e4); Cilium for pod networking.
Depended on by: every PVC on the platform \u2014 concretely the Vault\ndatastores today (vault-postgres-data, vault-redis-data,\nStorageClass ceph-rbd, SVC-VAULT); any future stateful\nworkload.
kubectl -n kube-system get pods | grep csi-rbd\n(provisioners 2/2, node plugin Ready on each worker);\nkubectl get sc ceph-rbd (PROVISIONER shown); a scratch PVC binding test\n(create small PVC \u2192 Bound \u2192 delete) is the end-to-end proof.ContainerCreating waiting on\nconfigmaps ceph-config / ceph-csi-encryption-kms-config, the driver's\nconfiguration objects are missing \u2014 the live symptom recorded 2026-08-31\n(inputs operator, card t_8eee26e4); do not work around by hand-creating\nmanifests outside the reviewed flow.PENDING \u2014 not serving PVCs. The chart is installed in kube-system\nbut the pods are in ContainerCreating waiting on the operator-input\nconfigmaps (ceph-config / ceph-csi-encryption-kms-config \u2014 inputs\noperator, card t_8eee26e4); clusterID/monitors and the\ncsi-rbd-secret are still placeholders pending the Ceph-side inputs.\nThe StorageClass ceph-rbd object exists in Git as the target shape but\nis NOT yet usable: no PVC should be created against it until the pods are\nRunning and a scratch-PVC bind test passes. The Vault datastores depend on\nexactly this \u2014 that dependency is why the Vault is also PENDING.
wetopi-harness): long-term\nhome of the csi-rbd-secret values (ADR-0009 \u2014 \"these values come from\nthe Vault when it exists\"); no Ceph secret material in this repo today.\n\nMigrado desde
\nwetopi-platform/docs@main(services/cert-manager.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
cert-manager (ACME / Let's Encrypt)
\ncert-manager is the cluster's ACME client: it issues and automatically\nrenews the TLS certificates that the edge Gateway terminates on. DNS-01\nchallenges are answered through the Cloudflare API, so issuance is\nport-mapping-immune (no open 80/8080 HTTP-01 surface on the edge). It is\nthe service that turned the F2 bootstrap self-signed cert into real LE\ncertificates for every public hostname.
\ncert-manager (Git-declared namespace.yaml, single\ncreator \u2014 sync runs with CreateNamespace=false).cert-manager v1.21.1 from\noci://quay.io/jetstack/charts (OCI is the source of truth; the legacy\nindex lags). appVersion v1.21.1 \u2014 controller/webhook/cainjector/acmesolver\nare one released set; no per-image tag overrides. Cluster requirement\nkubeVersion >= 1.22.0-0 satisfied by RKE2 v1.36.3 (K8s 1.36).cert-manager): source 1 = the pinned chart (values from\nkubernetes/cert-manager/values.yaml via $values; crds.enabled: true\nso the CRDs are part of the release and owned/pruned by Argo); source 2 =\nthis repo at kubernetes/cert-manager rendering EXACTLY the Namespace,\nboth ClusterIssuers and the staging Certificate (include/exclude filter\ndocumented in platform/cert-manager.yaml). The Application is anchored\nin the root path (kubernetes/platform/cert-manager.yaml) and carries the\ncascading-deletion finalizer \u2014 a git revert of the anchoring merge rolls\nthe full F3 deployment back as a foreground cascade.letsencrypt-staging and letsencrypt-prod\n(ClusterIssuers, DNS-01 Cloudflare solver). Both reference the SAME\noperator Secret cloudflare-dns-api-token (key api-token) in\nnamespace cert-manager \u2014 BY NAME ONLY; the token itself is operator\nmaterial (0600 env file on ops01, shared with OpenTofu's cloudflare\nprovider), never in Git.kube-system \u2014 the Gateway's namespace, so the\nlistener certificateRefs are same-namespace, no ReferenceGrant):\nedge-test-tls (staging, the F3 acceptance object), argo-tls,\ndocs-tls, vault-tls (all LE PROD, single-name, no wildcards).flowchart TD\n CF[\"Cloudflare API (wetopi.net zone: DNS-Edit + Zone-Read)\"]\n subgraph CM[\"namespace cert-manager\"]\n CTL[\"cert-manager controller v1.21.1 (+ webhook, cainjector)\"]\n ISP[\"ClusterIssuer letsencrypt-prod\"]\n ISS[\"ClusterIssuer letsencrypt-staging\"]\n TOK[\"Secret cloudflare-dns-api-token (operator)\"]\n end\n subgraph KSYS[\"kube-system (Gateway namespace)\"]\n CE1[\"Certificate edge-test-tls (staging)\"]\n CA[\"Certificate argo-tls (PROD)\"]\n CD[\"Certificate docs-tls (PROD)\"]\n CV[\"Certificate vault-tls (PROD)\"]\n GW[\"Gateway wetopi-edge https listener certificateRefs\"]\n end\n LE[\"Let's Encrypt ACME (staging + prod)\"]\n CF -->|\"_acme-challenge TXT\"| CTL\n CTL -->|ACME orders| LE\n ISP --> CTL\n ISS --> CTL\n TOK --> ISP\n TOK --> ISS\n CE1 --> CA & CD & CV\n CA & CD & CV --> GW\nGitOps: Application cert-manager (anchored in\nkubernetes/platform/cert-manager.yaml, synced by the root app) \u2014\nprune + selfHeal. One-time bootstrap from ops01 (runbook:\nkubernetes/cert-manager/README.md): apply namespace.yaml by hand \u2192\ncreate the cloudflare-dns-api-token Secret from the ops01 0600 env file\n(kubectl create secret ... --from-literal, shape-only verification) \u2192\nthe root app creates the Application and syncs. Prod cutover gate:\nnothing may reference letsencrypt-prod until the F3 staging gate\n(edge-test certificate issued + full chain verified) is signed off \u2014 LE\nprod rate limits burn real budget. Upgrades = a reviewed PR changing\ntargetRevision + values.yaml together.
Depends on: the Cloudflare API token (Zone - DNS - Edit AND Zone - Zone
\nwetopi.net) \u2014 operator-managed, ops01-only canonical file; the\nedge Gateway as the certificate consumer\n(SVC-TRAEFIK-EDGE); DNS records for the issued names\n(tofu-owned, so the _acme-challenge zone exists); the root Argo\nApplication (SVC-ARGOCD) for lifecycle.Depended on by: every public hostname on the edge \u2014 argo-tls\n(argo.wetopi.net), docs-tls (docs.wetopi.net), vault-tls\n(vault.wetopi.net), edge-test-tls (F3 acceptance). Without renewals, the\nwhole https listener degrades at 90-day boundaries.
kubectl -n cert-manager get pods (controller/webhook/\ncainjector Ready); kubectl -n kube-system get certificates \u2014 all four\nReady=True; controller events show ACME order \u2192 challenge \u2192 issue\nchain. Expiry: kubectl get certificate <name> -o jsonpath='{.status.notAfter}'.certificateRef entry from the\nGateway (git revert); the Secret stays and keeps renewing unless the\nCertificate is reverted too.kubernetes/cert-manager/README.md (ownership,\ntoken provisioning, prod cutover gate).DEPLOYED. Live since the F3 apply: controller/webhook/cainjector\nRunning, both issuers configured. Certificates live-verified 2026-08-31 \u2014\n4/4 Ready=True: edge-test-tls (staging), argo-tls, docs-tls,\nvault-tls (LE PROD). Renewals run automatically (renewBefore default\n30 days for 90-day ACME certs).
wetopi-harness): no\ncertificate material lives in the Vault; the ACME account keys\n(letsencrypt-*-account-key Secrets) and the Cloudflare token are\noperator-managed (ops01 + cert-manager namespace Secrets) \u2014 migration\ninto the Vault is part of ADR-0009's long-term picture, not yet done.\n\nMigrado desde
\nwetopi-platform/docs@main(services/docs-site.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
Docs site (docs.wetopi.net)
\nThe public (operator-authenticated) home of Wetopi's institutional\ntechnical knowledge \u2014 this documentation. It is a static MkDocs build of\nwetopi-platform/docs packed into an nginx image and served behind the\nedge; it is also the platform's first end-to-end Git \u2192 CI \u2192 image \u2192\nreviewed promotion \u2192 GitOps deployment chain (\"la \u00faltima build NUNCA se\nauto-despliega\").
docs-site (Git-declared, Argo-created:\nCreateNamespace=true).docs-site: 2 replicas (the edge v1 node side is already\ndocumented as reduced availability \u2014 the workload must not add a\nsingle-pod failure mode). Image\nghcr.io/wetopi-platform/docs-site@sha256:5c67cab5\u202603b23\n(digest-pinned promotion, PR #23 = docs main 7e0d0050, CI run\n33349505431) \u2014 no mutable tags anywhere. Non-root uid 101 (the image's\nnginx user), caps ALL dropped, read-only rootfs, one writable emptyDir at\n/var/cache/nginx, imagePullSecrets: ghcr-pull-secret (pod-scoped; the\nGHCR package is PRIVATE \u2014 anonymous pull returns 401, live-verified).docs-site:80 (ClusterIP) \u2192 targetPort 8080.docs-site (ns docs-site) \u2192 Gateway wetopi-edge\nhttps listener (cross-namespace, from: All), with Basic-Auth\nMiddleware docs-basic-auth FIRST (htpasswd, realm wetopi-docs);\nhttp://docs.wetopi.net 308-redirects to HTTPS (Gateway web\nlistener). TLS: Certificate docs-tls (LE PROD DNS-01, kube-system) on\nthe Gateway listener.docs-site-basic-auth\n(htpasswd \u2014 operator-created on the cluster, MUST exist before the first\nsync; shape secret.example.yaml) and ghcr-pull-secret (GHCR pull\ncredential \u2014 shape secret.ghcr-pull.example.yaml).flowchart TD\n REPO[\"GitHub wetopi-platform/docs (main + PRs)\"]\n CI[\"docs CI: mkdocs build --strict -> nginx image -> ghcr.io/.../docs-site:sha-<commit>\"]\n PROMO[\"human-reviewed PR in infrastructure: pin @sha256:<digest>\"]\n subgraph K8S[\"cluster wetopi3\"]\n ARGODEPS[\"Application docs-site (anchored in kubernetes/platform/, prune+selfHeal)\"]\n subgraph NS[\"namespace docs-site\"]\n DEP[\"Deployment docs-site x2 (nginx uid 101, digest-pinned)\"]\n SVC[\"Service docs-site:80\"]\n MW[\"Middleware docs-basic-auth (htpasswd)\"]\n RT[\"HTTPRoute docs-site -> Gateway \u00a7https\"]\n end\n GW[\"Gateway wetopi-edge (kube-system) + cert docs-tls\"]\n end\n USER[\"operator: https://docs.wetopi.net (basic auth 401 realm wetopi-docs, INTENCIONAL)\"]\n REPO --> CI --> PROMO --> ARGODEPS --> DEP\n DEP --> SVC\n USER --> GW --> RT --> MW --> SVC\nPromotion-only chain (no auto-deploy of the latest build):
\n.github/workflows/docs-build.yml) runs\nmkdocs build --strict on main pushes AND PRs, then packs site/ into\nthe nginx image and pushes ghcr.io/wetopi-platform/docs-site:sha-<commit>\n(main pushes only; GITHUB_TOKEN, packages: write; CI NEVER touches\nthe cluster).infrastructure swaps the digest in\nkubernetes/docs-site/deployment.yaml to the next reviewed\n@sha256:<digest> (digest, not tag \u2014 a digest cannot be re-pushed).docs-site\n(anchored in kubernetes/platform/docs-site.yaml; include-only filter\nover the six workload manifests; prune + selfHeal; cascading-deletion\nfinalizer) deploys exactly the reviewed digest. One-time bootstrap from\nops01 (runbook: kubernetes/docs-site/README.md): create\ndocs-site-basic-auth + ghcr-pull-secret BEFORE the first sync.Depends on: the docs repo CI + GHCR (private package \u2192 pull Secret);\nthe edge Gateway + its docs-tls certificate (SVC-TRAEFIK-EDGE,\nSVC-CERT-MANAGER); DNS A records docs.wetopi.net \u2192\nedge IPs .193/.194 (specific A records beat the *.wetopi.net wildcard \u2192\n.76 in resolution order); the root Argo Application\n(SVC-ARGOCD) for lifecycle.
Depended on by: humans and agents retrieving institutional knowledge;\nthe docs runtime rebuild (OBJ-DOCSRUNTIME-001) is migrating the build to a\nGitHub-webhook-driven flow \u2014 the promotion chain and this deployment shape\nstay.
\nhttps://docs.wetopi.net \u2014 HTTP 401 with realm\nwetopi-docs WITHOUT credentials is intentional (ADR-0006: the v1\nwindow publishes operator-only services behind basic auth; this is NOT a\nbroken site). With credentials: MkDocs site + search.kubectl -n docs-site get pods (2/2 Ready),\nHTTPRoute docs-site Accepted=True with parent resolved to\nkube-system/wetopi-edge; edge probe curl -I https://docs.wetopi.net\n\u2192 401 (auth challenge = healthy), 200 with credentials.kubernetes/docs-site/README.md (infra repo)\n\u2014 build, verify the pushed digest, review the pin PR.DEPLOYED. The first real end-to-end chain is live: promoted digest\n(sha256:5c67cab5\u202603b23 = docs main 7e0d0050) deployed by Argo,\n2/2 pods Ready, docs.wetopi.net serving the LE PROD chain after the DNS\ncutover (APPLY t_dde33585 AC5), basic auth active by design. In\nflight: OBJ-DOCSRUNTIME-001 (docs runtime rebuild via GitHub webhook,\ninfra PR #28 \u2014 pinning decision pending) will change HOW the image is\nbuilt/served; the current digest-pinned promotion chain above remains the\ndeployed reality until that lands.
wetopi-harness): the\ndocs-site-basic-auth htpasswd and the ghcr-pull-secret are\noperator-managed cluster Secrets today; the Vault (ADR-0009) is their\nlong-term home \u2014 no values in this repo.\n\nMigrado desde
\nwetopi-platform/docs@main(services/litellm.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
LiteLLM inference proxy (ns litellm)
\nLiteLLM is the platform's LLM inference proxy. Clients call an\nOpenAI-compatible endpoint at https://infer.wetopi.net and the proxy\nforwards every request to a single upstream: https://llm.wetopi.net/v1\n(same API key as the infer01 deployment). The proxy runs LiteLLM in\nfull mode \u2014 with a Postgres backend for virtual keys, budgets and usage\ntracking \u2014 not the simple no-DB mode.
Topology: client \u2192 edge Gateway (infer.wetopi.net) \u2192 ns litellm (LiteLLM) \u2192 llm.wetopi.net/v1.
Access model (operator directive 2026-09-01, twin of the vault route \u2014\nADR-0014): the edge\ngate is a source-IP allowlist (ipAllowList middleware, range\n51.254.106.87/32 = operator VPN egress). Only sources inside that range\nreach LiteLLM; everything else is rejected at the edge with 403 and never\nreaches the app. Behind that network gate sits LiteLLM's own auth: a\nmaster key for administration and virtual keys (per-client) for API\ntraffic. There is no HTTP Basic Auth anywhere on the route.
All objects are Git-owned in the platform-litellm repo (per-service\nrepo; desired state is the source of truth \u2014 this page only explains it).
litellm \u2014 dedicated; holds both the proxy and its\nPostgres (verified topology decision).litellm \u2014 ghcr.io/berriai/litellm:v1.99.0 (tag-pinned,\nnever a floating tag), 1 replica, strategy Recreate. Requests\n250m CPU / 512Mi memory, limit 2Gi memory (sized after the T12\ngo-live OOMKill at 1Gi). Config via CONFIG_FILE_PATH=/app/config/config.yaml\n(ConfigMap litellm-config). Secrets resolved at runtime from env\n(os.environ/\u2026) sourced by the litellm-auth Secret \u2014 no secret value\nis in this file. Health endpoints /health/liveness +\n/health/readiness (both unauthenticated) drive the startup/liveness/\nreadiness probes.postgres \u2014 postgres:16, 1 replica, strategy Recreate\n(RWO single-writer). Limits 1Gi / requests 256Mi. Probes are\npg_isready -U litellm -d litellm (auth-gated check, so the probe verifies\nwhat the app actually needs). Data on PVC postgres-data 10Gi,\nStorageClass ceph-rbd \u2192 depends on Ceph CSI.litellm (ClusterIP :4000) and postgres (ClusterIP\n:5432). In-cluster address: litellm.litellm.svc.cluster.local:4000.litellm (ns litellm) \u2192 Gateway\nkube-system/wetopi-edge https listener; host infer.wetopi.net; a\nsingle PathPrefix / rule carrying the allowlist-middleware-litellm\nExtensionRef filter and the single backendRef litellm:4000. TLS\nterminates at the Gateway (the route is TLS-free); Certificate\nkube-system/infer-tls (LE PROD, DNS-01) backs the https listener via an\ninfer-tls certificateRef.allowlist-middleware-litellm (ns litellm) \u2014 Traefik\nipAllowList, sourceRange: 51.254.106.87/32. It ships in the SAME\ndirectory as the HTTPRoute (Law 4 \u2014 a desynced ExtensionRef would\nfail-closed with 500 while Argo shows Healthy).services/litellm/litellm \u2014 LITELLM_MASTER_KEY, LITELLM_SALT_KEY,\nLLM_WETOPI_API_KEYservices/litellm/postgres \u2014 POSTGRES_LITELLM_PASSWORDservices/external/llm-wetopi \u2014 LLM_WETOPI_API_KEY (upstream key),\nLLM_WETOPI_URIlitellm-auth (4 keys, incl. DATABASE_URL built in\nmemory from the Postgres password) and postgres-auth (3 keys) are\ncreated at apply-time only from Infisical. Only\nsecret.example.yaml (placeholders) lives in Git.flowchart TD\n OP[\"allowlisted client (51.254.106.87/32 \u2014 operator VPN egress)\"]\n OTH[\"any other source\"]\n subgraph CL[\"cluster wetopi3\"]\n subgraph NSL[\"namespace litellm\"]\n APP[\"Deployment litellm \u2014 LiteLLM v1.99.0 (full mode)\"]\n PG[\"PostgreSQL 16 \u2014 PVC 10Gi (ceph-rbd)\"]\n SVC[\"Service litellm :4000\"]\n MW[\"Middleware allowlist-middleware-litellm (ipAllowList)\"]\n end\n GW[\"Gateway wetopi-edge (kube-system) + cert infer-tls (LE PROD)\"]\n SC[\"StorageClass ceph-rbd (Ceph CSI)\"]\n end\n UP[\"upstream llm.wetopi.net/v1 (OpenAI-compatible)\"]\n OP -->|\"https://infer.wetopi.net\"| GW -->|\"http (PathPrefix /) + ipAllowList\"| SVC\n OTH -.->|\"rejected at edge: 403\"| GW\n SVC --> APP\n APP -->|\"database_url / virtual keys / usage\"| PG\n APP -->|\"forward (model_name wetopi-llm)\"| UP\n PG --> SC\nGitOps, repo-per-service. Two Argo CD anchor Applications in\nplatform-litellm/kubernetes/argo-anchor/ sync the repo's manifests under\nthe root app:
litellm \u2014 sources kubernetes/litellm/, include-only glob\n{configmap,deployment,service,httproute,middleware-allowlist} (the\nsecret.example.yaml is excluded on purpose \u2014 applying the placeholder\nwould clobber the real apply-time litellm-auth Secret; Argo would stay\ngreen, i.e. silent).postgres \u2014 sources kubernetes/postgres/, include-only glob\n{namespace,pvc,deployment,service} (same example-secret exclusion).Bootstrap sync policy is prune: false, selfHeal: false (RC4/Law 3 \u2014\nmanual sync during bootstrap, no automatic prune). Both apps are\nmanually synced via\nkubectl -n argocd patch application <app> --type merge -p '{\"operation\":{\"sync\":{\"revisions\":[\"main\"]}}}' (order: postgres then\nlitellm). The first full apply of the whole service (namespace, k8s\nSecrets from Infisical, the 9 manifests, AppProject + repo credential\nadmission, Argo anchors, the infer-tls Certificate wait, the Gateway\ndelta, and readiness verification) is performed by the apply-block script\nexecuted at T12 go-live \u2014 see Estado honesto for the run that went green.
Depends on: Ceph CSI / StorageClass ceph-rbd for the Postgres PVC\n(SVC-CEPH-CSI); the edge Gateway wetopi-edge + the\nnftables 443\u21928443 redirect (SVC-TRAEFIK-EDGE);\ncert-manager for infer-tls (SVC-CERT-MANAGER); DNS A\nrecords infer.wetopi.net \u2192 edge IPs (tofu-owned, Cloudflare); Infisical\n(SVC-VAULT) for the apply-time Secrets; and the single upstream\nllm.wetopi.net/v1 for actual inference.
Depended on by: anything on the platform that needs a controlled,\nmetered OpenAI-compatible LLM endpoint (virtual keys per consumer, budgets\nand usage tracked in Postgres).
\nkubectl is run on a control-plane node (no local kubectl); every command\nbelow is via ssh rke-cp01 sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml.
kubectl -n litellm get pods (both 1/1 Running, restarts\nlow), kubectl -n argocd get applications litellm postgres\n(Synced/Healthy), HTTPRoute Accepted+ResolvedRefs, Certificate\ninfer-tls Ready. In-cluster probe:\nhttp://litellm.litellm.svc.cluster.local:4000/health/liveness \u2192 200.LITELLM_MASTER_KEY) is an admin credential \u2014 it authenticates to the\nproxy and to the key-management endpoints; it is not meant to be handed\nout. Virtual keys are per-client API keys created by the proxy:\nPOST /key/generate (with the master key) returns a virtual key;\nPOST /key/info and POST /key/delete list and revoke them. Clients call\nthe API with a virtual key (Authorization: Bearer <virtual key>),\nnever with the master key.kubectl -n litellm rollout restart deployment/litellm (single\nreplica + Recreate, so it is a brief stop-the-world). Track with\nkubectl -n litellm rollout status deployment/litellm.kubectl -n litellm logs -l app.kubernetes.io/name=litellm\n(add --tail=200 -f / -p for previous container). Postgres:\nkubectl -n litellm logs -l app=postgres.POST /key/generate on :4000 with the master key as\nAuthorization: Bearer <master key> and a JSON body such as\n{\"key\": \"client-<name>\", \"max_budget\": \"50\", \"models\": [\"wetopi-llm\"]}.\nThe response carries the new virtual key \u2014 distribute it to the client\nand store only its identity, not the value, in this repo.model_list in\nkubernetes/litellm/configmap.yaml (a new model_name + litellm_params\nwith the upstream api_base and an os.environ/<VAR> key), add the new\nsecret key to the litellm-auth Secret from Infisical (apply-time),\nand re-sync (manual Argo sync). Keep the single-valid-backendRef\ninvariant on the HTTPRoute.litellm-auth / postgres-auth and\nrollout restart). DATABASE_URL embeds the Postgres password./key/delete then /key/generate) \u2014 no cluster mutation, no\ndeployment roll.LLM_WETOPI_API_KEY in Infisical, refresh the\nlitellm-auth Secret and roll litellm.LITELLM_SALT_KEY): used to hash stored virtual keys \u2014\nchanging it invalidates every stored virtual key. Rotate only as a\ndeliberate break-the-keys operation, coordinated with the clients.GO-LIVE 2026-09-02 (UTC). The proxy is deployed and serving the public\ncontract; the only unclosed item is the end-to-end upstream leg (network\nroute), not the proxy or the apply.
\nGo-live: executed 2026-09-02. The authoritative run of the apply-block\n(T12) finished with all phases PASS (exit 0) after the sizing fix was\nmerged to platform-litellm main @ f9d5afe41 (PR #6 \u2014 litellm\nrequests 512Mi / limits 2Gi, which resolved the OOMKill at 1Gi seen in\nthe earlier go-live attempt). The service has been live at that revision\nsince; re-verified live 2026-09-02 07:01Z.
Live state (re-verified 2026-09-02 07:01Z): pods litellm-\u2026 and\npostgres-\u2026 1/1 Running, restarts=0 (no OOM recurrence); Argo apps\nlitellm + postgres Synced / Healthy @ f9d5afe41; HTTPRoute\nlitellm Accepted=True + ResolvedRefs=True; Certificate infer-tls\nReady=True (\"Certificate is up to date and has not expired\").
Public contract probes (evidenced at go-live):
\n| Check | \nResult | \n
|---|---|
DNS infer.wetopi.net | \nPASS \u2192 79.137.114.193 + 79.137.114.194 (system + @1.1.1.1 + @9.9.9.9); wildcard *.wetopi.net untouched | \n
| TLS cert | \nPASS \u2014 CN=infer.wetopi.net, issuer Let's Encrypt (YR1), valid 2026-09-02 \u2192 2026-12-01, Verify=0; both edge IPs serve it | \n
In-cluster /health/liveness | \nPASS \u2192 200 (\"I'm alive!\") | \n
In-cluster /health/readiness | \nPASS \u2192 200 | \n
| Public edge, non-allowlisted source | \n403 (by design) \u2014 ops01 92.222.245.91 and workstation 92.222.245.89 are outside 51.254.106.87/32, so /health/liveness and POST /v1 return 403 at the edge (TLS terminates before the 403) | \n
/v1/ without Bearer (in-cluster) | \n401 (app auth) | \n
/v1/models with master key (in-cluster) | \n200 \u2192 model wetopi-llm (master key authenticates) | \n
e2e POST /v1/chat/completions with master key | \n500 \u2014 BLOCKED-UPSTREAM (network) \u2014 see below | \n
Known limitations (honest):
\nsocket.create_connection((\"llm.wetopi.net\", 443)) \u2192\nOSError [Errno 113] No route to host; POST /v1/chat/completions \u2192\n500 OpenAIException - Connection error. The master key authenticates\nfine (/v1/models \u2192 200), so this is the cluster \u2192 llm.wetopi.net:443\nroute/egress, an external-network item outside the apply/proxy. The same\nkey + model is LIVE at infer01.hermeshaven.cloud:8000 (control: 200).\nTracked as a follow-up for the operator/infra (route/egress to\nllm.wetopi.net). The 200 happy path for a real completion is not yet\nevidenced until that route exists.200 from an in-range 51.254.106.87/32\nsource) was not measurable from the available infra (workstation/ops01\negress is outside that range; no in-range source was available). The\nnegative probe (out-of-range \u2192 403) is evidenced from two sources and\nis the correct negative. The 200/401 app contracts were verified\nin-cluster instead.prune:false, selfHeal:false bootstrap policy);\nafter a main change the apps must be synced manually (order\npostgres \u2192 litellm). An initial OutOfSync is only the\nlast-applied kubectl annotation \u2014 specs == git, resolved by the manual\nsync.wetopi-harness \u2192 services/litellm/litellm,\nservices/litellm/postgres, services/external/llm-wetopi.\n\nMigrado desde
\nwetopi-platform/docs@main(services/observability-suite.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
Observability suite (ns observability)
\nThe platform's full observability stack: one Git repo\n(platform-observability), one Argo anchor, one namespace\n(observability), and three pipelines \u2014
/var/log/pods (filelog + k8s_attributes enrichment) and ships OTLP/gRPC\nto Data Prepper, which indexes into OpenSearch (14-day ISM retention on\nlogs-*).Presentation: Grafana (metrics + logs + traces in one UI) and\ncerebro.wetopi.net (OpenSearch Dashboards \u2014 the log search entry point).
\nGo-live: 2026-09-05 (F7.2 apply gate + F7.3 public-contract probes,\nall green \u2014 see Estado honesto).
\nFour hostnames, all served by the WAN edge (Gateway kube-system/wetopi-edge,\nRKE2 Traefik, https listener) and all gated by ONE shared source-IP\nallowlist middleware before the app's own auth \u2014 no HTTP Basic Auth at the\nedge (same posture as litellm / vault / docs routes,\nADR-0014):
| Hostname | \nBackend | \nApp-level auth behind the IP gate | \n
|---|---|---|
grafana.wetopi.net | \ngrafana:3000 | \nGrafana admin login (Infisical services/observability/grafana) | \n
prometheus.wetopi.net | \nprometheus:9090 | \nnone \u2014 the IP gate IS the auth (as argo/vault) | \n
alertmanager.wetopi.net | \nalertmanager:9093 | \nnone \u2014 the IP gate IS the auth | \n
cerebro.wetopi.net | \nopensearch-dashboards:5601 | \nOpenSearch Dashboards login with the DFM credential (Infisical services/observability/opensearch-dashboards) | \n
51.254.106.87/32 (ops01 / operator VPN egress) +\n92.222.245.89/32 (agents01 egress). Any other source gets 403 at the\nedge and never reaches a backend.kube-system/grafana-tls, prometheus-tls, alertmanager-tls,\ncerebro-tls \u2014 declared in the infrastructure repo\n(kubernetes/cert-manager/certificate-{grafana,prometheus,alertmanager,cerebro}.yaml,\nmerged 2026-09-04, infra PR #44) and wired into the Gateway's https\nlistener at the F1.5 edge apply (operator-gated)..193/.194, Cloudflare, TTL 300) in\ninfrastructure (tofu; merged 2026-09-04, infra PR #43; apply gate F1.6).All objects Git-owned in platform-observability\n(kubernetes/observability/<component>/), ns observability.\nDesired state is the source of truth \u2014 this page only explains it.\nEvery image is tag-pinned (never a floating tag):
| Component | \nImage (pinned) | \nShape | \nPVC (ceph-rbd, RWO) | \n
|---|---|---|---|
| Prometheus | \nprom/prometheus:v3.9.1 | \nDeployment, Recreate; 1Gi/2Gi | \nprometheus 20Gi (retention 15d in-server) | \n
| Alertmanager | \nquay.io/prometheus/alertmanager:v0.34.0 | \nDeployment, 256Mi/512Mi | \ndata-alertmanager-0 2Gi | \n
| node-exporter | \nquay.io/prometheus/node-exporter:v1.12.1 | \nDaemonSet, 64Mi/128Mi | \n\u2014 | \n
| kube-state-metrics | \nregistry.k8s.io/kube-state-metrics/kube-state-metrics:v2.9.2 | \nDeployment, 128Mi/256Mi | \n\u2014 | \n
| Tempo | \ngrafana/tempo:3.0.3 | \nStatefulSet, 1Gi/2Gi | \ntempo-data 10Gi (14d retention) | \n
| Grafana | \ngrafana/grafana:13.0.8 | \nDeployment, 512Mi/1Gi | \ngrafana-data 2Gi | \n
| OTel Collector | \notel/opentelemetry-collector-contrib:0.160.0 | \nDaemonSet (1/node), 256Mi/512Mi | \n\u2014 | \n
| Data Prepper | \nopensearchproject/data-prepper:2.16.0 | \nDeployment, 512Mi/1Gi | \n\u2014 | \n
| OpenSearch | \nopensearchproject/opensearch:3.8.0 | \nStatefulSet (single node, raw manifests), 2Gi/3Gi (heap 1Gi) | \nopensearch-data 50Gi | \n
| OpenSearch Dashboards | \nopensearchproject/opensearch-dashboards:3.8.0 | \nDeployment, 512Mi/1Gi | \n\u2014 | \n
StorageClass: ceph-rbd only (there is no local-path on RKE2); every\nPVC pins it explicitly. No NodePort anywhere \u2014 all Services are ClusterIP;\npublic exposure goes through the edge only.
Notable v1 postures (documented in-manifest, not duplicated here):
\nOPENSEARCH_INITIAL_ADMIN_PASSWORD (Secret opensearch), and a\nbootstrap-security.sh (in-pod, apply-time) that creates the app users\n(OSD dfm, data-prepper) with minimal roles. :9200 is HTTPS + auth;\nprobes are tcpSocket accordingly. ISM policy logs-retention-14d +\ncomposable template on logs-* applied out-of-band\n(scripts/ism-apply.sh).render generates the runtime pipeline at startup from\nthe two data-prepper-creds keys (single Secret consumer).grafana-opensearch-datasource@2.34.3 (pinned) so the OpenSearch\ndatasource can load.kubernetes/observability/edge/ carries the four HTTPRoutes\n(grafana / prometheus / alertmanager / cerebro, ns observability)\nand the single shared Middleware allowlist-middleware-observability\n(ipAllowList, the two ranges above). Law 4 (PR #36 lesson): the middleware\nships in the SAME directory/PR as its routes \u2014 a desynced ExtensionRef would\nfail-closed with 500 while Argo shows Healthy. TLS terminates at the\nGateway; the routes are TLS-free.
flowchart TD\n subgraph EDGE[\"WAN edge (Gateway wetopi-edge, LE TLS)\"]\n R[\"4 HTTPRoutes + Middleware ipAllowList (51.254.106.87/32, 92.222.245.89/32)\"]\n end\n subgraph NS[\"namespace observability\"]\n subgraph M[\"metrics\"]\n EX[\"node-exporter DS + kube-state-metrics + apps /metrics\"]\n PR[\"Prometheus v3.9.1 (PVC 20Gi, ret 15d)\"]\n AM[\"Alertmanager v0.34.0 (UI-only v1)\"]\n end\n subgraph L[\"logs\"]\n OC[\"OTel Collector contrib 0.160.0 (DaemonSet) \u2014 filelog + k8s_attributes\"]\n DP[\"Data Prepper 2.16.0 (render-at-startup)\"]\n OS[\"OpenSearch 3.8.0 (PVC 50Gi, ISM 14d)\"]\n OSD[\"OSD 3.8.0 \u2014 cerebro.wetopi.net\"]\n end\n subgraph T[\"traces\"]\n TMP[\"Tempo 3.0.3 (PVC 10Gi, ret 14d)\"]\n end\n GR[\"Grafana 13.0.8 \u2014 grafana.wetopi.net (datasources: prometheus, alertmanager, tempo, opensearch)\"]\n end\n OP[\"allowlisted clients (.87 / .89)\"]\n APPS[\"apps exporting OTLP (litellm = first real traces producer)\"]\n OP -->|HTTPS| R -->|403 for all other sources| EDGE\n R --> GR & PR & AM & OSD\n EX --> PR -->|rules| AM\n OC -->|OTLP/gRPC :2189| DP -->|sink| OS\n APPS -->|OTLP 4317/4318| OC -->|otlp/traces| TMP\n OS -->|datasource| GR\n TMP --> GR\n PR --> GR\nprometheus\n(default, http://prometheus:9090), alertmanager\n(http://alertmanager:9093), tempo (http://tempo:3200), opensearch\n(https://opensearch.observability.svc:9200, type\ngrafana-opensearch-datasource, Basic-Auth with the DFM credential).k8s-overview,\nprometheus-self, litellm, argocd, opensearch-health (F6.3) +\nedge-traefik, alertmanager-fireboard (F6.4).grafana-datasources\nConfigMap in Git is a TEMPLATE OF RECORD carrying the Argo skip marker\n+argocd:skip-file-rendering; scripts/apply-observability.sh renders it\nout-of-band at apply time (envsubst of ${OSD_DFM_USERNAME} /\n${OSD_DFM_PASSWORD} from Infisical) because Grafana's provisioner does\nnot expand ${...} \u2014 a synced placeholder CM would give a 401 datasource.\nBeing outside the Argo tracked set, selfHeal can neither create the\nplaceholder nor revert the rendered CM.Operator decision for v1: no external alert channels. Alertmanager's\ndefault receiver is the no-op null receiver (no email, webhook or pager);\nAlertmanager only stores and displays alerts in its UI. The v1 rule set\n(prometheus-rules-configmap.yaml) covers node health (NodeNotReady,\nNodeFilesystemAlmostFull, NodeClockSkew), pod health\n(PodCrashLooping, PodNotReady, ContainerOOMKilled), storage\n(PVCAlmostFull) and self-monitoring (PrometheusTargetDown), each with a\nseverity label (warning|critical). Adding a real receiver later = appending\nto receivers: and pointing a route sub-tree at it \u2014 no image or PVC\nchange. (Backlog v1.1: Telegram/email channels.)
Project harness, env prod. Four out-of-band k8s Secrets, created at\napply time by scripts/apply-observability.sh from Infisical (the single\nsource of truth for the contract is its secret_contract()):
| k8s Secret (ns observability) | \nInfisical path | \nKeys | \nConsumed by | \n
|---|---|---|---|
opensearch | \nservices/observability/opensearch | \nOPENSEARCH_ADMIN_PASSWORD, OPENSEARCH_ADMIN_HASH | \nOpenSearch STS (admin seed), bootstrap-security, ism-apply | \n
osd-dfm | \nservices/observability/opensearch-dashboards | \nOSD_ADMIN_USER, OSD_ADMIN_PASSWORD | \nOSD Deployment; rendered into the grafana-datasources CM at apply | \n
grafana-admin | \nservices/observability/grafana | \nGF_ADMIN_USER, GF_ADMIN_PASSWORD | \nGrafana Deployment (admin login) | \n
data-prepper-creds | \nservices/observability/data-prepper | \nDATA_PREPPER_OPENSEARCH_PASSWORD, OPENSEARCH_ENDPOINTS (non-secret) | \nData Prepper initContainer render | \n
Only secret.example.yaml (placeholders) lives in Git \u2014 root\nkubernetes/observability/secret.example.yaml (consolidated shape) plus the\nper-component examples. These are EXCLUDED from the Argo include glob and\nfrom any dir-wide apply (litellm-gate RC-2: an example Secret inside an\nArgo-synced tree clobbers the real one while Argo stays green \u2014 the 2026-09-04\nincident that fixed the exclude glob to a single brace glob).
GitOps with two layers:
\nplatform-gitops/apps/observability.yaml: Application\nobservability (project platform) syncs\nplatform-observability:main path kubernetes/observability, directory\nmode recurse: true with\nexclude: '{*example*,*.json,README.md}' (single brace glob \u2014 Argo\nexclude takes ONE glob), automated: prune: true, selfHeal: true,\nCreateNamespace=false (the namespace is created by the apply script /\nnamespace.yaml first), PruneLast=true (the anchor was hardened in\nF7.4). The anchor argocd/application.yaml inside\nplatform-observability is a legacy placeholder \u2014 the live anchor is the\nplatform-gitops one.scripts/apply-observability.sh\n(OPERATOR-GATED; the worker never runs it). It mirrors the proven\napply-litellm.sh pattern and runs on the harness host with no local\nkubectl \u2014 every cluster op goes through\nssh rke-cp01 sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml. Phases: preflight \u2192 Infisical fetch \u2192\ncreate the 4 Secrets \u2192 file-by-file apply in a fixed order (never\ndir-wide; never touches *example* or *.json) \u2192 OpenSearch security\nbootstrap + ISM policy (in-pod, out-of-band) \u2192 WAIT gates (all pods\nReady; certs verified) \u2192 in-cluster probes + summary. --allow-existing\nmakes re-runs idempotent (re-fetch from Infisical and overwrite the\nSecrets). scripts/ism-apply.sh (separately re-runnable) loads the\n14-day ISM policy + logs-* template with the OpenSearch 3.8.0 schema\nfixes live-verified at F7.2.Depends on: Ceph CSI / StorageClass ceph-rbd for all five PVCs\n(SVC-CEPH-CSI); the edge Gateway wetopi-edge + nftables\n443\u21928443 redirect (SVC-TRAEFIK-EDGE); cert-manager for\nthe four obs certificates (SVC-CERT-MANAGER); DNS A\nrecords in infrastructure (tofu-owned, Cloudflare); Infisical\n(SVC-VAULT) for the four apply-time Secrets; the scrape\ntargets across the cluster (kubernetes-SD) and OTLP producers for\nlogs/traces (litellm is the first real traces producer \u2014\nSVC-LITELLM).
Depended on by: every platform operation that needs metrics, log search\nor traces \u2014 incident response, OOM sizing decisions (litellm/vault lessons),\nE2E acceptance gates (F8.2 logs / F8.3 traces / F8.4 metrics).
\nkubectl is run on the control plane (no local kubectl); every command is via\nssh rke-cp01 sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml.
kubectl -n observability get pods (all 1/1, restarts low,\nno OOMKilled in lastState); kubectl -n argocd get application observability (Synced/Healthy); the four public probes (DNS \u2192 2 IPs,\nTLS LE CN, 403 from a non-allowlisted source, 200/302 from\n.89/.87).kubectl -n observability logs -l app.kubernetes.io/name=<component>;\nfor node-level filelog ingestion check the collector DaemonSet pods\n(group-0 read on /var/log/pods is what makes pod logs readable).kubernetes/observability/grafana/ (see the repo README \"Adding a\ndashboard\" section), PR it; the wetopi-obs file provider picks it up\nfrom its mounted path on the next Grafana start (or the provider rescan).\nCM and mount changes are coupled (Law 4 \u2014 ship both in the same PR).grafana-datasources-configmap.yaml \u2014 but remember that file is the\nout-of-band template of record: a datasource needing credentials must\nkeep the ${VAR} placeholder, gain a matching render step in\napply-observability.sh, and a new Infisical key; plain http datasources\nare safe to edit directly (still: re-run the apply render so the live CM\nmatches).scripts/apply-observability.sh --allow-existing (re-fetches and\noverwrites the Secrets), then roll the consuming workloads\n(kubectl -n observability rollout restart deployment/<name> /\nstatefulset/opensearch only when the OpenSearch admin password changes).\nThe OpenSearch admin password is seeded at first boot only\n(OPENSEARCH_INITIAL_ADMIN_PASSWORD) \u2014 changing it later requires a\nsecurity plugin user change, not a PVC wipe.opensearch/bootstrap-security.sh\n(users + roles) and scripts/ism-apply.sh (retention) are the two\nout-of-band steps that re-run after a fresh OpenSearch data volume; both\ncarry the live-verified OpenSearch 3.8.0 schema fixes (policy body wrapped\nas {\"policy\": {...}}, no enabled field in ism_template).The full go-live apply is scripts/apply-observability.sh \u2014 see its header\nfor the phase list and the Wetopi laws baked in (explicit file list,\nnever dir-wide; Secrets out-of-band; name must match secretKeyRef; no\nNodePort; set -e). Operator steps:
harness, env prod) at the four vault paths\nabove with the listed keys.VAULT_API_CLIENTID=*** VAULT_API_KEY=*** \\ scripts/apply-observability.sh\n(idempotent re-runs: add --allow-existing).Preconditions already in place at go-live: the four LE certificates\nReady, the Gateway delta applied (F1.5), DNS A-records live (F1.6),\nthe Argo anchor merged (F2.3/F7.4).
\nplatform-edge \u2014 the\nhttps listener falls back to the remaining certs; the four *-tls\nSecrets stay and cert-manager keeps renewing them. (Invariant R1: a\ncertRef whose Secret is missing takes down the ENTIRE https listener,\nso edge changes are always fail-safe ordered and individually reversible.)prune: true \u2014 revert the anchor merge or argocd app delete observability, then kubectl delete ns observability manually (the\nnamespace is not Argo-created). PVCs carry the Delete policy, so\nmetrics/logs/traces data is lost on rollback \u2014 accepted for v1,\neverything is re-derivable (scrapes and logs re-ingest; retention is\nby config, not by recreating PVCs \u2014 R6: never delete PVCs from a PR).prevent_destroy\ndeliberately).observability except the shared\nhttps listener (mitigated by R1).GO-LIVE 2026-09-05 (UTC). The suite is deployed, serving, and\nend-to-end operational.
\napply-observability.sh executed; all pods\nRunning, Argo observability Synced/Healthy..193/.194), TLS LE certs \u00d74 (correct CN per host), positive 200\nfrom 92.222.245.89 (agents01 egress) via both edge IPs, negative\n403 from 92.222.245.91 (ops01, outside the allowlist) on all four\nhostnames, in-cluster probes on all four backends, pods at 0 restarts.503 at the\nOpenSearch Dashboards system-index step \u2014 root cause: the OSD config\nindex fell under the demo-installer's system-index guard. Fix: OSD\nconfig index pointed to observability-osd-config\n(platform-observability PR #38 + hotfix commits; opensearchDashboards.index\nschema-verified from the 3.8.0 image) plus the dfm role PUT to its\nfinal live-verified shape (commit bea78ae).harness (env prod) \u2192 services/observability/{opensearch, opensearch-dashboards, grafana, data-prepper}.\n\nMigrado desde
\nwetopi-platform/docs@main(services/traefik-edge.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
Traefik edge (WAN edge v1)
\nThe edge is how the platform is reached from the internet. Per ADR-0006\n(WAN exception for a bounded v1 window), the two edge nodes expose the\nRKE2-bundled Traefik directly on their WAN IPs \u2014 no L4 load-balancing\nlayer (VRRP+IPVS+DSR is deferred until an external-client service lands).\nTraefik runs in hostNetwork mode with the Gateway API provider enabled, so\nplain Gateway/HTTPRoute objects are the ingress programming surface.
rke2-traefik 40.1.009 (RKE2 addon), configured in place by the\nHelmChartConfig rke2-traefik in kube-system\n(kubernetes/edge/helmchartconfig-rke2-traefik-edge.yaml \u2014 the exact\nlive-verified valuesContent at F2 merge, apply t_64e49873, run 239):\nhostNetwork: true, entryPoints web=:8000 / websecure=:8443\n(hostPort = containerPort, chart contract), providers.kubernetesGateway\nenabled, nodeSelector + toleration pinning the DaemonSet to\nwetopi.io/role=edge nodes only (zero Traefik pods on CPs/platform\nworkers).edge_lb Ansible role nftables policy\nREDIRECTs WAN+LAN 80\u21928000 and 443\u21928443 in the edge host netns. The\nGateway listeners deliberately sit on 8000/8443 \u2014 declaring 80/443 there\nyields PortUnavailable and an un-Accepted Gateway (the live failure\nthat shaped F2 lead decision A).GatewayClass wetopi-edge\n(controllerName: traefik.io/gateway-controller) and Gateway kube-system/wetopi-edge \u2014 listeners web (8000, allowedRoutes from: Same) and https (8443, TLS Terminate, from: All).certificateRefs (same-namespace Secrets):\nwetopi-edge-default-cert (90-day bootstrap self-signed, F2), argo-tls,\ndocs-tls, vault-tls (LE PROD, issued by cert-manager \u2014 adding a ref\nwhose Secret is missing takes down the WHOLE listener; fail-safe apply\norder documented per entry in gateway-wetopi-edge.yaml).docs-site (ns docs-site),\nargo + argo-http-redirect (kube-system co-location, argo Basic-Auth\nMiddleware), vault (ns vault, path-filtered deny-first), and the F3\nacceptance edge-test-tls name.rke-wan01 / rke-wan02 (edge role, WAN IPs\n79.137.114.193/.194; DNS-owned via tofu tfvars).flowchart LR\n WAN[\"internet (WAN 80/443)\"]\n subgraph EDGE[\"edge nodes rke-wan01/02 (hostNetwork)\"]\n NFT[\"nftables edge_lb: 80->8000, 443->8443\"]\n TRAEF[\"Traefik DS (rke2-traefik 40.1.009) \u2014 entryPoints 8000/8443\"]\n end\n GW[\"Gateway wetopi-edge (kube-system) \u2014 https TLS Terminate; certs: bootstrap + argo-tls + docs-tls + vault-tls\"]\n subgraph ROUTES[\"HTTPRoutes\"]\n RD[\"docs-site route (ns docs-site)\"]\n RA[\"argo route + basic-auth (ns kube-system)\"]\n RV[\"vault route, deny-first (ns vault)\"]\n end\n BACK[\"backends: docs-site / argocd-server / vault-ui Services\"]\n WAN --> NFT --> TRAEF --> GW --> RD & RA & RV --> BACK\nBootstrap-time, not GitOps-managed: the HelmChartConfig and the\nGatewayClass/Gateway objects are applied from wetopi-ops01 with kubectl\nafter the edge nodes join (an RKE2 addon HelmChartConfig cannot be installed\nby the Cilium-style chart flow \u2014 the chart already exists; RKE2's packaged\naddon reconciler drives upgrades). Verify gates after apply: helm job\nComplete, DS 2/2 on edge nodes only, :8000/:8443 LISTEN in the host\nnetns, 80/443 answered via the redirect. Gateway certificateRefs deltas\n(certs for new services) are reviewed declarative changes with a mandatory\n\"Certificate Ready \u2192 THEN apply Gateway\" ordering (see each Certificate\nheader).
Depends on: edge nodes + the edge_lb nftables role (Ansible); the\nRKE2 bundled chart (version rides the RKE2 pin); cert-manager for every LE\ncertificate on the https listener (SVC-CERT-MANAGER);\nDNS (tofu-owned A records) per hostname.
Depended on by: every public service of the v1 window \u2014\ndocs.wetopi.net (SVC-DOCS-SITE), argo.wetopi.net\n(SVC-ARGOCD), vault.wetopi.net (SVC-VAULT), and\nthe F3 acceptance name edge-test.wetopi.net. Any future external-client\nservice must land the L4 layer first (ADR-0006).
\nkubectl -n kube-system get gateway wetopi-edge \u2014\nAccepted=True, Programmed=True, https listener ResolvedRefs=True;\nss -tlnp on each edge node shows *:8000 + *:8443; per-route\nHTTPRoute status Accepted=True with the parent resolved.--for=condition=Ready; (2) add the certificateRef entry to the Gateway\n(reviewed change on kubernetes/edge/gateway-wetopi-edge.yaml); (3)\nHTTPRoute + optional Basic-Auth Middleware (one htpasswd Secret per\nservice \u2014 never shared); (4) DNS A records (tofu apply).certificateRef entry (git revert of the\nGateway delta) \u2014 the listener falls back to the remaining certs; the\nSecret stays (cert-manager keeps renewing it).kubernetes/edge/README.md (F2 shape + gates),\nper-service READMEs under kubernetes/<service>/.DEPLOYED. F2 applied 2026-08-30 (task t_64e49873, all gates green):\nGateway Accepted+Programmed, both listeners healthy, edge_lb redirect live,\nthree HTTPRoutes attached (docs, argo, vault) with the four\ncertificateRefs in place. v1 window is bounded by ADR-0006 \u2014 the L4 layer\nremains mandatory before any external-client service.
wetopi-harness): the edge\ncurrently holds no Vault-sourced material; per-service htpasswd Secrets\n(argo-basic-auth, docs-basic-auth) are operator-managed cluster\nSecrets with the Vault as the long-term home (ADR-0009).\n\nMigrado desde
\nwetopi-platform/docs@main(services/vault.md) el 2026-09-06. Fuente can\u00f3nica: Git. Mirror de lectura.
Vault \u2014 Infisical self-hosted (ns vault)
\nInfisical is the platform's secrets manager for humans and for agent\nprofiles (machine identities via Universal Auth). ADR-0009 (accepted\n2026-08-31) fixes the shape: self-hosted in cluster wetopi3, namespace\nvault, with persistent PostgreSQL and Redis datastores in the same\ncluster, the Infisical \u2192 Kubernetes Operator for Secret sync, Universal Auth\nrestricted to the private network, and SOPS+age demoted to bootstrap/recovery\nonly. The operator's decision explicitly replaced the dedicated-VM variant\nwith the in-cluster deployment.
All objects Git-owned under kubernetes/vault/ (images digest-pinned):
vault \u2014 infisical/infisical:v0.164.1@sha256:d3ff9f82\u2026\n(multi-arch index digest), non-root uid 1001, replicas 1, Recreate\nstrategy, readiness/liveness on :8080/api/status. A ConfigMap\nvault-entrypoint wrapper builds DB_CONNECTION_URI at container start\nfrom Secret env (URL-encoded, never a YAML literal).postgres:17.6-alpine3.22 (digest-pinned, non-root uid 70)\nvault-postgres-data 5Gi; redis:7.4-alpine (digest-pinned,\nnoeviction + appendonly yes, uid 999) + PVC vault-redis-data 1Gi.\nBoth on StorageClass ceph-rbd \u2192 depends on Ceph CSI (below).vault (FULL API \u2014 in-cluster only, NO edge route) and\nvault-ui (edge UI backend).vault (ns vault) \u2192 Gateway wetopi-edge https\nlistener; Certificate vault-tls (LE PROD DNS-01, kube-system).vault-infra-credentials\n(ENCRYPTION_KEY, AUTH_SECRET) and vault-postgres-credentials\n(POSTGRES_USER, POSTGRES_PASSWORD); shapes: secret.infra.example.yaml,\nsecret.postgres.example.yaml.infisical/kubernetes-operator:v0.11.8\n(digest-pinned) in namespace infisical-operator-system\n(kubernetes/infisical-operator/); anchoring its Argo Application is a\nfuture reviewed change. CRDs are v1beta1 (Secrets Operator v1beta1).flowchart TD\n AGENTS[\"agents / profiles (machine identities)\"]\n HUMANS[\"operators (browser, login+TOTP)\"]\n subgraph CL[\"cluster wetopi3\"]\n subgraph NSV[\"namespace vault\"]\n APP[\"Deployment vault \u2014 Infisical v0.164.1\"]\n PG[\"PostgreSQL 17.6 \u2014 PVC 5Gi\"]\n RD[\"Redis 7.4 noeviction \u2014 PVC 1Gi\"]\n SVAULT[\"Service vault (FULL API, in-cluster only)\"]\n SUI[\"Service vault-ui (UI edge backend)\"]\n end\n OP[\"Infisical Operator v0.11.8 (ns infisical-operator-system)\"]\n GW[\"Gateway wetopi-edge (kube-system) + cert vault-tls\"]\n SC[\"StorageClass ceph-rbd (Ceph CSI)\"]\n end\n CEPH[\"Ceph cluster 192.168.96.0/20 (VLAN 2)\"]\n AGENTS -->|\"http://vault.vault.svc.cluster.local (NEVER the edge)\"| SVAULT\n HUMANS -->|\"https://vault.wetopi.net\"| GW --> SUI\n SVAULT --> APP\n SUI --> APP\n APP --> PG\n APP --> RD\n PG --> SC\n RD --> SC\n SC --> CEPH\n OP -->|\"pulls from internal endpoint\"| SVAULT\nGitOps: the Argo CD Application vault (anchored in the root path at\nkubernetes/platform/vault.yaml) syncs kubernetes/vault/ with an\ninclude-only filter ({namespace.yaml,postgres.yaml,redis.yaml,infisical.yaml,certificate-vault.yaml,httproute-vault.yaml}) and prune + selfHeal \u2014\nthe two operator Secrets are excluded on purpose and must exist before\nthe first sync (runbook: kubernetes/vault/README.md \u00a7provisioning \u2014\nAPPLY card t_89a254ce). TLS: Certificate vault-tls must be Ready\nBEFORE the Gateway certificateRefs edit is applied (fail-safe order in\ncertificate-vault.yaml header). The Infisical Operator directory has its\nown lifecycle (not yet anchored to the root app).
Depends on: Ceph CSI / StorageClass ceph-rbd for both PVCs\n(SVC-CEPH-CSI) \u2014 if CSI is not live, the datastores cannot\nbind; edge Gateway wetopi-edge + the nftables 443\u21928443 redirect\n(SVC-TRAEFIK-EDGE); cert-manager for vault-tls\n(SVC-CERT-MANAGER); DNS A records vault.wetopi.net \u2192\nedge IPs (tofu-owned, tfvars); operator Secrets vault-infra-credentials
vault-postgres-credentials (APPLY S0\u2013S2, card t_89a254ce); RKE2\netcd encryption at rest is the at-rest story (etcd keys on ops01).Depended on by: every machine identity (agent profiles) \u2014 the ai-hub\nvault.* contract (vault.get/put/update/delete/list/exists) resolves\nagainst http://vault.vault.svc.cluster.local:80; long-term the ceph-csi\ncsi-rbd-secret and other bootstrap secrets migrate from ops01 files into\nthe Vault (ADR-0009).
https://vault.wetopi.net \u2014 native Infisical login\nhttp://vault.vault.svc.cluster.local:80 only \u2014\nthe public route carries DENY rules (D1\u2013D6: universal-auth login,\nidentity, /api/v1|v2|v3|v4 secret surfaces all point at the\nnonexistent vault-api-deny Service \u2014 fail-closed 5xx by Gateway API\nspec). The proof contract (public 5xx / internal 200 drill) is in\nkubernetes/vault/README.md \u00a75 and runs on the ops card after apply.kubectl -n vault get pods (vault/postgres/redis Ready),\nGET http://vault.vault.svc.cluster.local/api/status from inside the\ncluster; edge probe: vault-tls Ready + https listener ResolvedRefs.pg_dump + age-encrypted off-host\nbackup, restore REHEARSED before the service is considered production\n(recovery contract compensates the accepted correlated-failure trade-off\nof in-cluster datastores).LIVE \u2014 vault.wetopi.net serving HTTPS 200 with LE cert. End-to-end\nverify after the operator's tofu DNS apply (vault.wetopi.net \u2192\n79.137.114.193/.194): Web UI 200 via both edge IPs, certificate\nvault-tls (Let's Encrypt, valid 2026-08-31\u21922026-11-29), pods 3/3\nRunning (infisical + postgres + redis on ceph-rbd PVCs), /api/status\nOk with Redis configured. Two lead-direct hotfixes landed in Git during\nthe apply: commit 3a87ee94 (entrypoint ConfigMap defaultMode: 0755 \u2014\nrunAsUser 1001 exec permission, crashloop fix) and 7e0b3a0b (infisical\nmemory limit 1Gi\u21922Gi after an OOMKill at 19:59Z, V8 heap 2.3GB).\nThe deny rules D1\u2013D3 (machine-API paths \u2192 nonexistent Service\nvault-api-deny) are fail-closed BY SPEC \u2014 verified: /api/v1/secrets\nreturns 500 from the public edge; the Argo Degraded on the app is the\ndocumented false positive of this mechanism (ResolvedRefs=False is the\ndeny working, not a fault). PENDING: operator admin bootstrap + TOTP QR\n(human step).
wetopi-harness): this is\nthe service that hosts the rest of the platform's secrets \u2014 machine\nidentities and secret paths live under project wetopi-harness (e.g. the\ndrill identity drill-probe-<date> in the README proof). Secret values\nare never in this repo or the docs; reference paths only (DocOps \u00a78).\u00bfQu\u00e9 es? Wiki de conocimiento vivo (book.wetopi.net): el almac\u00e9n \"Living Knowledge\" del Knowledge Plane. Sede del Workbench de agentes (book 90), la evidencia de investigaciones y los how-to del d\u00eda a d\u00eda. NO es el SSOT de decisiones \u2014 eso es Git.
\n\u00bfQu\u00e9 hace? Books por dominio (00-90), chapters, p\u00e1ginas con tags #draft/#ready/#superseded, b\u00fasqueda global, historial de revisiones, API REST program\u00e1tica para los agentes.
\n\u00bfC\u00f3mo est\u00e1 desplegado?
\nbookstack: Deployment bookstack (lscr.io/linuxserver/bookstack@sha256:90a3df31\u2026, v26.05.4) + StatefulSet bookstack-mariadb-0 (mariadb 11.8 @sha256:2439dcd7\u2026, ceph-rbd 10Gi)bookstack-data (ceph-rbd 10Gi), strategy Recreate (RWO single-writer)allowlist-middleware-bookstack (.87/.89) \u2192 Gateway wetopi-edgebook-tls (kube-system, LE PROD DNS-01, v\u00e1lido hasta 2026-12-05)bookstack \u2192 platform-bookstack@main (Synced+Healthy), ancla en platform-gitops (include-glob 7 manifests)\u00bfC\u00f3mo se opera?
\nservices/bookstack/agents/<agent>/{BOOKSTACK_TOKEN_ID,BOOKSTACK_TOKEN_SECRET} (E2E 200 verificado)php artisan bookstack:create-admin --initial para resetservices/bookstack/bookstack (9 keys)\u00bfC\u00f3mo se rompe y se arregla?
\nbookstack-secrets (Infisical) \u2014 el Secret no sobrevive al ns borrado\u00bfQu\u00e9 es? El plano de memoria de la flota (memory.wetopi.net): Hindsight 0.9.2 (ns wetopi-memory) con banks personales user-u_{carlos,joan,josep} (provider nativo) + corporate (plugin wetopi-memory, corporate-only por decisi\u00f3n A1).
\n\u00bfQu\u00e9 hace? Retain/recall/reflect de hechos sem\u00e1nticos con entity graph; punteros a evidencia (regla: la memoria apunta, no almacena). Clasificaci\u00f3n STATE/EVENT/DECISION/PROCEDURE/LEARNING.
\n\u00bfC\u00f3mo est\u00e1 desplegado?
\n\u00bfC\u00f3mo se opera?
\n\u00bfC\u00f3mo se rompe y se arregla?
\n\u00bfQu\u00e9 es? Registry OCI interno (Zot v2.1.2) previsto para builds de CI self-hosted y pulls LAN.
\n\u00bfQu\u00e9 hace? (dise\u00f1o) Serve self-hosted CI builds (push) + cluster pulls (LAN-fast), path imperativo para despliegues fuera de ArgoCD.
\n\u00bfC\u00f3mo est\u00e1 desplegado? \u26a0 NO desplegado \u2014 verificado live 2026-09-06: ns platform existe pero NO hay workloads. Los manifests viven en platform-registry (rama feat/zot-initial, sin anclar en Argo). La CI de platform-docs-runtime (R4) apunta a zot.platform.svc:5000 que NO existe a\u00fan \u2014 repin del digest docs-runtime pendiente.
\u00bfC\u00f3mo se opera? (cuando exista) htpasswd (runner/admin, Secret zot-auth operator-side), LAN Service :5000, dedupe+GC 24h.
\n\u00bfC\u00f3mo se rompe y se arregla? local-path NO existe en este cluster (solo ceph-rbd \u2014 lecci\u00f3n R2); NodePort prohibido en edge-WAN (invariante RC-5).
\n" }, { "id": 73, "title": "SVC-LITELLM \u2014 infer.wetopi.net", "slug": "svc-litellm-inferwetopinet", "book_id": 3, "chapter_id": 21, "description": null, "created_at": "2026-09-06T13:39:32.000000Z", "updated_at": "2026-09-06T21:45:53.000000Z", "markdown": "---\nid: SVC-LITELLM\ntitle: LiteLLM \u2014 proxy LLM de la flota (infer.wetopi.net)\ntype: service\nstatus: active\nowner: wetopi-lead\nsummary: OpenAI-compatible proxy (infer.wetopi.net) con Postgres full-mode, virtual keys por agente.\nlast_verified: 2026-09-06\n---\n\n# SVC-LITELLM: LiteLLM Proxy\n\n**\u00bfQu\u00e9 es?** Proxy OpenAI-compatible (infer.wetopi.net) para toda la flota Hermes, con virtual key por agente (carlos/joan/josep) y spend attribution por key_alias.\n\n**\u00bfQu\u00e9 hace?** Routing a upstreams LLM, budgets, usage tracking (SpendLogs \u2192 BBDD litellm-postgres \u2192 dashboards LLM-spend en Grafana).\n\n**\u00bfC\u00f3mo est\u00e1 desplegado?**\n- ns litellm: Deployment litellm v1.99.0 (2Gi limits \u2014 sizing OOM lecci\u00f3n L12) + postgres:16\n- Edge: infer.wetopi.net, allowlist .87/.89, LE cert\n- Argo: 2 apps (litellm + litellm-postgres) \u2192 platform-litellm@main\n\n**\u00bfC\u00f3mo se opera?**\n- Keys de agentes en vault root `LITELLM-\u00bfQu\u00e9 es? Proxy OpenAI-compatible (infer.wetopi.net) para toda la flota Hermes, con virtual key por agente (carlos/joan/josep) y spend attribution por key_alias.
\n\u00bfQu\u00e9 hace? Routing a upstreams LLM, budgets, usage tracking (SpendLogs \u2192 BBDD litellm-postgres \u2192 dashboards LLM-spend en Grafana).
\n\u00bfC\u00f3mo est\u00e1 desplegado?
\n\u00bfC\u00f3mo se opera?
\nLITELLM-<agent>_hermes_wetopi_keykubectl -n litellm rollout restart deployment/litellm\u00bfC\u00f3mo se rompe y se arregla?
\nEl book 90 es el L1 del circuito de conocimiento: borradores, evidencia\nde investigaciones y docs intermedios de los agentes. No es el SSOT de nada\n\u2014 cuando algo se consolida, se promociona a Git (PR) y esta copia muere.
\ncarlos, joan, josep) \u2014 nunca escribir en el de otro.WIP-<fecha>-<tema>, DRAFT-ADR-<nnnn>-<slug>, EVIDENCE-<slug>.#draft \u2192 #ready \u2192 #superseded (tras promocionar).#draft.#ready y abre PR al destino formal (docs / ai-hub).PROMOVED \u2192 <link al doc en Git> y tag #superseded.services/bookstack/agents/<agent>/wetopi-docops (flota completa)docs/objectives/active/OBJ-KDOCPANE-001.md (Git)\u00bfQu\u00e9 es? El canal ACTIVO de docs.wetopi.net (desde 2026-09-06, flip G1+G2): un pod que\nrebuild-ea MkDocs Material desde wetopi-platform/docs@main al recibir el webhook de GitHub.\nCommit a main = publicado en <10s, sin rehacer imagen.
\u00bfQu\u00e9 hace?
\n\u00bfC\u00f3mo est\u00e1 desplegado?
\ndocs-site: Deployment docs-runtime (imagen can\u00f3nica GHCR @sha256:b00efab8\u2026)\ndocs-runtime \u2192 platform-docs-runtime@main (Synced+Healthy)\u00bfC\u00f3mo se opera?
\n\u00bfC\u00f3mo se rompe y se arregla?
\n\u00bfQu\u00e9 es? 3 instancias de Hermes Agent (carlos/joan/josep) corriendo en agents01\n(92.222.245.89, host bigardo), en /opt/hermes-<n>/.
\u00bfQu\u00e9 hace? Operaci\u00f3n de plataforma, skills compartidas, memoria (nativo personal +\nplugin corporate), cron jobs, delegaci\u00f3n a subagentes.
\n\u00bfC\u00f3mo est\u00e1 desplegado?
\nLITELLM-<agent>_hermes_wetopi_key)\u00bfC\u00f3mo se opera?
\nssh -i /opt/data/.ssh/bigardo root@92.222.245.89 (config lan-config del harness)/opt/hermes-<n>/data/skills/ + verificaci\u00f3n sha256\u00bfC\u00f3mo se rompe y se arregla?
\n\u00bfQu\u00e9 es? Sistema dual de memoria (decisi\u00f3n A1 2026-09-06):
\n\u00bfQu\u00e9 hace? Retain/recall/reflect con fact extraction, entity graph, y clasificaci\u00f3n\nSTATE/EVENT/DECISION/PROCEDURE/LEARNING. Principio: la memoria apunta, no almacena\n(hecho esencial + timestamp + puntero a URL, nunca el documento).
\n\u00bfC\u00f3mo est\u00e1 desplegado?
\n\u00bfC\u00f3mo se opera?
\n\u00bfC\u00f3mo se rompe y se arregla?
\nipBlock 0.0.0.0/0 bloqueaba SYN de pods remotos al\nnamespace nuevo (drop silencioso pre-veth, invisible al monitor). Fix: SIN netpols\n(patr\u00f3n vault/litellm: edge allowlist + ClusterIP + app auth). Commit 8a0c796.php artisan bookstack:create-admin --initial (user 1 = operator / carlos@wetopi.net)book/bookstack \u2192 allowlist-bookstack \u00b7 docs \u2192 edge-ip-allowlist (legacy) \u00b7 argo \u2192 edge-ip-allowlist\ninfer \u2192 allowlist-litellm \u00b7 alertmanager/cerebro/grafana/otlp/prometheus \u2192 allowlist-observability\nvault \u2192 edge-ip-allowlist \u00b7 memory \u2192 allowlist-memory (2 rules: /health+/v1/ API, resto CP)
\nargo, book, cerebro, docs, edge-test, grafana, infer, memory, otlp, prometheus, vault, alertmanager
\nargocd, bookstack, cert-manager, docs-site, docs-runtime, gitops, hindsight, litellm, litellm-postgres, observability, vault
\nns platform EXISTE pero VAC\u00cdO (sin deploy) \u2014 platform-registry en feat/zot-initial sin anclar
\n" }, { "id": 80, "title": "ARCH-PROXMOX \u2014 cluster wetopi7", "slug": "arch-proxmox-cluster-wetopi7", "book_id": 2, "chapter_id": 12, "description": null, "created_at": "2026-09-06T21:47:30.000000Z", "updated_at": "2026-09-06T21:47:30.000000Z", "markdown": "---\nid: ARCH-PROXMOX\ntitle: Proxmox cluster wetopi7\ntype: architecture\nstatus: active\nowner: wetopi-lead\nsummary: Cluster Proxmox de 8 nodos + Ceph sobre OVH vRack, VLAN 4 = node LAN.\nlast_verified: 2026-09-06\n---\n\n# Proxmox cluster wetopi7\n\n**\u00bfQu\u00e9 es?** Cluster Proxmox (8 nodos) + Ceph sobre OVH vRack. VLAN 4 / vmbr4 = node LAN 10.72.0.0/23.\n\n**Nodos:** wetopi-ops01 (bastion 92.222.245.91 WAN / 10.72.0.10), rke-cp01/02/03 (.21/.22/.23),\nrke-platform01/02/03 (.41/.42/.43), rke-wan01/02 (.51/.52 \u2014 hostNetwork edge), wetopi-natgw01 (.2, WAN .83).\n\nFuente can\u00f3nica: `docs/architecture/infrastructure/proxmox-cluster.md` (Git).", "html": "\u00bfQu\u00e9 es? Cluster Proxmox (8 nodos) + Ceph sobre OVH vRack. VLAN 4 / vmbr4 = node LAN 10.72.0.0/23.
\nNodos: wetopi-ops01 (bastion 92.222.245.91 WAN / 10.72.0.10), rke-cp01/02/03 (.21/.22/.23),\nrke-platform01/02/03 (.41/.42/.43), rke-wan01/02 (.51/.52 \u2014 hostNetwork edge), wetopi-natgw01 (.2, WAN .83).
\nFuente can\u00f3nica: docs/architecture/infrastructure/proxmox-cluster.md (Git).
Fuente can\u00f3nica: docs/architecture/infrastructure/ceph.md + ADR-0007 (Git).
Fuente: docs/architecture/infrastructure/ovh-and-network.md + ADR-0014 (Git).
Fuente: docs/architecture/platform/rke2-cluster.md + ADR-0001/0004 (Git).
ipBlock 0.0.0.0/0 en ns nuevo bloquea SYN de pods\nremotos (drop silencioso pre-veth) \u2014 patr\u00f3n casa: SIN netpols para apps edgeFuente: ADR-0005 + docs/architecture/platform/cilium-network.md (Git).
wetopi-edge (kube-system): listener web:8000 HTTP + websecure:8443 HTTPS\n(nftables edge redirige 80\u21928000, 443\u21928443). allowedRoutes from: All en httpsFuente: platform-edge repo + ADR-0006/0014 (Git).
\n" }, { "id": 86, "title": "ARCH-GITOPS \u2014 app-of-apps Argo CD", "slug": "arch-gitops-app-of-apps-argo-cd", "book_id": 3, "chapter_id": 19, "description": null, "created_at": "2026-09-06T21:47:32.000000Z", "updated_at": "2026-09-06T21:47:32.000000Z", "markdown": "---\nid: ARCH-GITOPS\ntitle: GitOps (Argo CD)\ntype: architecture\nstatus: active\nowner: wetopi-lead\nsummary: App-of-apps (index platform-gitops), 11 apps, automated prune+selfHeal, include-glob contracts.\nlast_verified: 2026-09-06\n---\n\n# GitOps (Argo CD)\n\n- v3.4.8 non-HA (sin Postgres \u2014 Redis only); WebUI argo.wetopi.net\n- App-of-apps: app `gitops` \u2192 platform-gitops@main (apps/*.yaml) con recurse:true\n- 11 apps: argocd, bookstack, cert-manager, docs-site, docs-runtime, gitops, hindsight,\n litellm, litellm-postgres, observability, vault \u2014 todas Synced+Healthy\n- include-glob CONTRACT (Ley 4): a\u00f1adir/quitar ficheros del include EN EL MISMO PR\n- repo Secrets: `repo-gitops \u2192 platform-gitops@main (apps/*.yaml) con recurse:truerepo-<name>-https (auto-discovery por label, con url/username/password/project)Fuente: platform-gitops README (reglas R1-R5) + memo gitops-reorg v3.1 (Git).
\n" }, { "id": 87, "title": "ARCH-CERTM \u2014 LE DNS-01", "slug": "arch-certm-le-dns-01", "book_id": 3, "chapter_id": 20, "description": null, "created_at": "2026-09-06T21:47:32.000000Z", "updated_at": "2026-09-06T21:47:32.000000Z", "markdown": "---\nid: ARCH-CERTM\ntitle: Cert-Manager\ntype: architecture\nstatus: active\nowner: wetopi-lead\nsummary: cert-manager v1.21.1 con ClusterIssuer letsencrypt-prod (DNS-01 Cloudflare).\nlast_verified: 2026-09-06\n---\n\n# Cert-Manager\n\n- v1.21.1 (chart quay.io/jetstack/charts, pinned)\n- ClusterIssuer `letsencrypt-prod`: DNS-01 via Cloudflare (token del vault services/cloudflare)\n- Certs por host en kube-system (same-namespace del Gateway, sin ReferenceGrant)\n- 12 certs activos, todos Ready (verificaci\u00f3n: `kubectl -n kube-system get certificates`)\n- **Regla de flujo**: Certificate Ready ANTES de certRef en el Gateway (fail-safe co-tenant)\n\nFuente: `docs/services/cert-manager.md` (Git).", "html": "letsencrypt-prod: DNS-01 via Cloudflare (token del vault services/cloudflare)kubectl -n kube-system get certificates)Fuente: docs/services/cert-manager.md (Git).
| Ficha | \nHost | \nEstado | \n
|---|---|---|
| SVC-DOCS-RUNTIME | \ndocs.wetopi.net | \nACTIVO (flip G1+G2 hoy) | \n
| SVC-BOOKSTACK | \nbook.wetopi.net | \nACTIVO | \n
| SVC-HINDSIGHT | \nmemory.wetopi.net | \nACTIVO | \n
| SVC-LITELLM | \ninfer.wetopi.net | \nACTIVO | \n
| SVC-VAULT | \nvault.wetopi.net | \nACTIVO | \n
| SVC-ARGOCD | \nargo.wetopi.net | \nACTIVO | \n
| SVC-DOCS-SITE | \ndocs.wetopi.net | \nLEGACY (retirada G3 ~13-09) | \n
| SVC-REGISTRY | \nzot.platform.svc | \nPLANNED (ns vac\u00edo \u2014 sin desplegar) | \n
| SVC-CEPH-CSI | \nstorage driver | \nACTIVO (infra) | \n
| SVC-CERT-MANAGER | \ncert-manager ns | \nACTIVO (infra) | \n
| SVC-TRAEFIK-EDGE | \nedge WAN | \nACTIVO (infra) | \n
| SVC-OBSERVABILITY | \ngrafana/prometheus/osd/alertmanager/cerebro | \nACTIVO (OTel plano inerte, reactivable) | \n
| SVC-HERMES-FLEET | \nagents01 (docker) | \nACTIVO (book 60) | \n
Todos los hosts: allowlist .87/.89 (excepto /hooks/rebuild \u2014 HMAC).
\n" }, { "id": 89, "title": "REF-SKILLS-FLOTA \u2014 skills y distribuci\u00f3n", "slug": "ref-skills-flota-skills-y-distribucion", "book_id": 7, "chapter_id": 25, "description": null, "created_at": "2026-09-06T21:47:43.000000Z", "updated_at": "2026-09-06T21:47:43.000000Z", "markdown": "---\nid: REF-SKILLS-FLOTA\ntitle: Skills de la flota (estado y distribuci\u00f3n)\ntype: reference\nstatus: active\nowner: wetopi-lead\nsummary: Skills activas por agente y mecanismo de replicaci\u00f3n con sha256.\nlast_verified: 2026-09-06\n---\n\n# Skills de la flota\n\n**Activas por decisi\u00f3n de Carlos (09-02):** wetopi-* (4), hermes-agent, github-* (5),\nserving-llms-vllm, huggingface-hub, evaluating-llms-harness, dev/debug (7), agentes de\ndelegaci\u00f3n (3), llm-wiki, design-md, humanizer, architecture-diagram, pretext, obsidian.\n\n**DocOps (nuevo 09-06):** `wetopi-docops` replicada a los 3 agentes (sha256 MATCH verificado).\n\u26a0 PENDIENTE CONSOLIDACI\u00d3N: la skill can\u00f3nica `docops` v1.1.0 (con plantillas) solapa con\n`wetopi-docops` v0.1.0 \u2014 plan: extender docops a v1.2.0 con el Knowledge Plane y retirar\nwetopi-docops (un solo contrato).\n\n**Replicaci\u00f3n:** local `/opt/data/skills/...` \u2192 `ssh -i /opt/data/.ssh/bigardo root@92.222.245.89`\n\u2192 `/opt/hermes-Activas por decisi\u00f3n de Carlos (09-02): wetopi-* (4), hermes-agent, github-* (5),\nserving-llms-vllm, huggingface-hub, evaluating-llms-harness, dev/debug (7), agentes de\ndelegaci\u00f3n (3), llm-wiki, design-md, humanizer, architecture-diagram, pretext, obsidian.
\nDocOps (nuevo 09-06): wetopi-docops replicada a los 3 agentes (sha256 MATCH verificado).\n\u26a0 PENDIENTE CONSOLIDACI\u00d3N: la skill can\u00f3nica docops v1.1.0 (con plantillas) solapa con\nwetopi-docops v0.1.0 \u2014 plan: extender docops a v1.2.0 con el Knowledge Plane y retirar\nwetopi-docops (un solo contrato).
Replicaci\u00f3n: local /opt/data/skills/... \u2192 ssh -i /opt/data/.ssh/bigardo root@92.222.245.89\n\u2192 /opt/hermes-<n>/data/skills/... \u2192 verificar sha256 en ambos.
Nota: el skill loader es per-sesi\u00f3n \u2014 skills nuevas visibles al reiniciar la sesi\u00f3n.
\n" }, { "id": 90, "title": "\u00cdNDICE del Workbench", "slug": "indice-del-workbench", "book_id": 10, "chapter_id": 30, "description": null, "created_at": "2026-09-06T21:47:43.000000Z", "updated_at": "2026-09-06T21:47:43.000000Z", "markdown": "---\nid: REF-WORKBENCH-INDEX\ntitle: \u00cdndice del Workbench (book 90)\ntype: reference\nstatus: active\nowner: wetopi-lead\nsummary: Qu\u00e9 es el Workbench, estructura por agente, naming y ciclo #draft/#ready/#superseded.\nlast_verified: 2026-09-06\n---\n\n# Agents Workbench \u2014 \u00edndice\n\nEl book 90 es el **L1 del circuito** (borradores y evidencia). NO es el SSOT de nada:\nlo que se consolida se promociona a Git (PR) y la copia aqu\u00ed muere (`PROMOVED \u2192 `).\n\n- `carlos/` \u2014 EVIDENCE-cluster-live-20260906 (inventario live del cluster)\n- `joan/` \u2014 (vac\u00edo, escribir aqu\u00ed)\n- `josep/` \u2014 (vac\u00edo, escribir aqu\u00ed)\n\nNaming: `WIP-El book 90 es el L1 del circuito (borradores y evidencia). NO es el SSOT de nada:\nlo que se consolida se promociona a Git (PR) y la copia aqu\u00ed muere (PROMOVED \u2192 <link>).
carlos/ \u2014 EVIDENCE-cluster-live-20260906 (inventario live del cluster)joan/ \u2014 (vac\u00edo, escribir aqu\u00ed)josep/ \u2014 (vac\u00edo, escribir aqu\u00ed)Naming: WIP-<fecha>-<tema>, DRAFT-ADR-<nnnn>-<slug>, EVIDENCE-<slug> + fecha.\nTags: #draft \u2192 #ready \u2192 #superseded.
No escribir en el chapter de otro agente. Tokens: Infisical services/bookstack/agents/.
\n" }, { "id": 91, "title": "\u00cdNDICE de mirrors (l\u00e9eme primero)", "slug": "indice-de-mirrors-leeme-primero", "book_id": 8, "chapter_id": null, "description": null, "created_at": "2026-09-06T21:47:43.000000Z", "updated_at": "2026-09-06T21:47:43.000000Z", "markdown": "---\nid: REF-MIRROR-INDEX\ntitle: \u00cdndice de mirrors (book 70)\ntype: reference\nstatus: active\nowner: wetopi-lead\nsummary: Este book es SOLO mirror de lectura \u2014 Git es el SSOT de ADRs y Architecture Records.\nlast_verified: 2026-09-06\n---\n\n# Mirrors (solo lectura)\n\n**Git es el SSOT.** Los ADRs viven en `wetopi-platform/docs/docs/decisions/`\n(ADR-0001..0009, 0010..0013 rescatados 2026-09-06, ADR-0014) y los design docs en\n`docs/design/`. Esta copia es para navegaci\u00f3n humana en BookStack \u2014 si diverge, manda Git.\n\nRegla de promoci\u00f3n: cualquier cambio se hace en Git; este mirror se regenera.", "html": "Git es el SSOT. Los ADRs viven en wetopi-platform/docs/docs/decisions/\n(ADR-0001..0009, 0010..0013 rescatados 2026-09-06, ADR-0014) y los design docs en\ndocs/design/. Esta copia es para navegaci\u00f3n humana en BookStack \u2014 si diverge, manda Git.
Regla de promoci\u00f3n: cualquier cambio se hace en Git; este mirror se regenera.
\n" } ]