Skip to content

Observability suite (ns observability)

What is it?

The platform's full observability stack: one Git repo (platform-observability), one Argo anchor, one namespace (observability), and three pipelines —

  • Metrics — node/kube/service exporters scraped by Prometheus (15d in-server retention); alert rules feed Alertmanager.
  • Logs — an OTel Collector DaemonSet on every node tails /var/log/pods (filelog + k8s_attributes enrichment) and ships OTLP/gRPC to Data Prepper, which indexes into OpenSearch (14-day ISM retention on logs-*).
  • Traces — apps export OTLP to the collector (gRPC :4317 / HTTP :4318); a traces pipeline forwards them to Tempo (14d retention), browsable in Grafana.

Presentation: Grafana (metrics + logs + traces in one UI) and dashboards.wetopi.net (OpenSearch Dashboards — the log search entry point; renamed from cerebro.wetopi.net on 2026-09-11).

Go-live: 2026-09-05 (F7.2 apply gate + F7.3 public-contract probes, all green — see Estado honesto).

Public hostnames, TLS and auth posture

Five hostnames, all served by the WAN edge (Gateway kube-system/wetopi-edge, RKE2 Traefik, https listener) and all gated by ONE shared source-IP allowlist middleware before the app's own auth — no HTTP Basic Auth at the edge (same posture as litellm / vault / docs routes, ADR-0014):

Hostname Backend App-level auth behind the IP gate
grafana.wetopi.net grafana:3000 Grafana admin login (Infisical services/observability/grafana)
prometheus.wetopi.net prometheus:9090 none — the IP gate IS the auth (as argo/vault)
alertmanager.wetopi.net alertmanager:9093 none — the IP gate IS the auth
dashboards.wetopi.net opensearch-dashboards:5601 OpenSearch Dashboards login with the DFM credential (Infisical services/observability/opensearch-dashboards)
otlp.wetopi.net otel-collector:4318 none — OTLP/HTTP ingestion, token-free by design (the IP gate IS the auth; fleet exporters authenticate nothing at the edge)
  • Allowlist: 51.254.106.87/32 (ops01 / operator VPN egress) + 92.222.245.89/32 (agents01 egress). Any other source gets 403 at the edge and never reaches a backend.
  • TLS: five Let's Encrypt PROD certificates (DNS-01) — kube-system/grafana-tls, prometheus-tls, alertmanager-tls, dashboards-tls, otlp-tls — declared in the infrastructure repo (kubernetes/cert-manager/certificate-{grafana,prometheus,alertmanager,dashboards}.yaml merged 2026-09-04, infra PR #44; certificate-otlp.yaml landed with the F-GSP 2026-09-05 Gateway certRef delta) and wired into the Gateway's https listener at the F1.5 edge apply (operator-gated).
  • DNS: 10 A-records (5 hosts × .193/.194, Cloudflare, TTL 300) in infrastructure (tofu; merged 2026-09-04, infra PR #43; apply gate F1.6; otlp pair live-verified 2026-09-09; dashboards pair + cerebro removal applied 2026-09-11 with the rename cutover, infra PR #52).

Components and pinned versions

All objects Git-owned in platform-observability (kubernetes/observability/<component>/), ns observability. Desired state is the source of truth — this page only explains it. Every image is tag-pinned (never a floating tag):

Component Image (pinned) Shape PVC (ceph-rbd, RWO)
Prometheus prom/prometheus:v3.9.1 Deployment, Recreate; 1Gi/2Gi prometheus 20Gi (retention 15d in-server)
Alertmanager quay.io/prometheus/alertmanager:v0.34.0 Deployment, 256Mi/512Mi data-alertmanager-0 2Gi
node-exporter quay.io/prometheus/node-exporter:v1.12.1 DaemonSet, 64Mi/128Mi —
kube-state-metrics registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.9.2 Deployment, 128Mi/256Mi —
Tempo grafana/tempo:3.0.3 StatefulSet, 1Gi/2Gi tempo-data 10Gi (14d retention)
Grafana grafana/grafana:13.0.8 Deployment, 512Mi/1Gi grafana-data 2Gi
OTel Collector otel/opentelemetry-collector-contrib:0.160.0 DaemonSet (1/node), 256Mi/512Mi —
Data Prepper opensearchproject/data-prepper:2.16.0 Deployment, 512Mi/1Gi —
OpenSearch opensearchproject/opensearch:3.8.0 StatefulSet (single node, raw manifests), 2Gi/3Gi (heap 1Gi) opensearch-data 50Gi
OpenSearch Dashboards opensearchproject/opensearch-dashboards:3.8.0 Deployment, 512Mi/1Gi —

StorageClass: ceph-rbd only (there is no local-path on RKE2); every PVC pins it explicitly. No NodePort anywhere — all Services are ClusterIP; public exposure goes through the edge only.

Notable v1 postures (documented in-manifest, not duplicated here):

  • OpenSearch 3.8.0 runs the image's demo-installer security posture: self-signed demo transport+HTTP TLS generated at pod boot, admin seeded from OPENSEARCH_INITIAL_ADMIN_PASSWORD (Secret opensearch), and a bootstrap-security.sh (in-pod, apply-time) that creates the app users (OSD dfm, data-prepper) with minimal roles. :9200 is HTTPS + auth; probes are tcpSocket accordingly. ISM policy logs-retention-14d + composable template on logs-* applied out-of-band (scripts/ism-apply.sh).
  • Data Prepper 2.16.0 does not interpolate env vars in pipeline YAML — its initContainer render generates the runtime pipeline at startup from the two data-prepper-creds keys (single Secret consumer).
  • Grafana installs the standalone plugin grafana-opensearch-datasource@2.34.3 (pinned) so the OpenSearch datasource can load.
  • NetworkPolicies (ingress-only v1): default-deny ingress + explicit allows (traefik → UIs, intra-namespace full, remote node-UIs); scrape egress stays open in v1.

Edge wiring (repo layout)

kubernetes/observability/edge/ carries the five HTTPRoutes (grafana / prometheus / alertmanager / dashboards / otlp, ns observability) and the single shared Middleware allowlist-middleware-observability (ipAllowList, the two ranges above). Law 4 (PR #36 lesson): the middleware ships in the SAME directory/PR as its routes — a desynced ExtensionRef would fail-closed with 500 while Argo shows Healthy; the otlp route (HTTPRoute otlp → otel-collector:4318) followed exactly that discipline, landing in the same edge directory + PR discipline as its Gateway certRef delta (F-GSP 2026-09-05). TLS terminates at the Gateway; the routes are TLS-free.

Pipelines

flowchart TD
  subgraph EDGE["WAN edge (Gateway wetopi-edge, LE TLS)"]
    R["5 HTTPRoutes + Middleware ipAllowList (51.254.106.87/32, 92.222.245.89/32)"]
  end
  subgraph NS["namespace observability"]
    subgraph M["metrics"]
      EX["node-exporter DS + kube-state-metrics + apps /metrics"]
      PR["Prometheus v3.9.1 (PVC 20Gi, ret 15d)"]
      AM["Alertmanager v0.34.0 (UI-only v1)"]
    end
    subgraph L["logs"]
      OC["OTel Collector contrib 0.160.0 (DaemonSet) — filelog + k8s_attributes"]
      DP["Data Prepper 2.16.0 (render-at-startup)"]
      OS["OpenSearch 3.8.0 (PVC 50Gi, ISM 14d)"]
      OSD["OSD 3.8.0 — dashboards.wetopi.net"]
    end
    subgraph T["traces"]
      TMP["Tempo 3.0.3 (PVC 10Gi, ret 14d)"]
    end
    GR["Grafana 13.0.8 — grafana.wetopi.net (datasources: prometheus, alertmanager, tempo, opensearch)"]
  end
  OP["allowlisted clients (.87 / .89)"]
  APPS["apps exporting OTLP (litellm = first real traces producer)"]
  FLEET['Hermes fleet + external OTLP producers']
  OP -->|HTTPS| R -->|403 for all other sources| EDGE
  FLEET -->|OTLP/HTTPS otlp.wetopi.net:4318| R
  R --> GR & PR & AM & OSD
  EX --> PR -->|rules| AM
  OC -->|OTLP/gRPC :2189| DP -->|sink| OS
  APPS -->|OTLP 4317/4318| OC -->|otlp/traces| TMP
  OS -->|datasource| GR
  TMP --> GR
  PR --> GR

Dashboards and datasources

  • Datasources (provisioned at Grafana start): prometheus (default, http://prometheus:9090), alertmanager (http://alertmanager:9093), tempo (http://tempo:3200), opensearch (https://opensearch.observability.svc:9200, type grafana-opensearch-datasource, Basic-Auth with the DFM credential).
  • Dashboards ("Wetopi Obs" folder, file provider): k8s-overview, prometheus-self, litellm, argocd, opensearch-health (F6.3) + edge-traefik, alertmanager-fireboard (F6.4).
  • Out-of-band rendering (load-bearing): the grafana-datasources ConfigMap in Git is a TEMPLATE OF RECORD carrying the Argo skip marker +argocd:skip-file-rendering; scripts/apply-observability.sh renders it out-of-band at apply time (envsubst of ${OSD_DFM_USERNAME} / ${OSD_DFM_PASSWORD} from Infisical) because Grafana's provisioner does not expand ${...} — a synced placeholder CM would give a 401 datasource. Being outside the Argo tracked set, selfHeal can neither create the placeholder nor revert the rendered CM.

Alerting — UI-only v1

Operator decision for v1: no external alert channels. Alertmanager's default receiver is the no-op null receiver (no email, webhook or pager); Alertmanager only stores and displays alerts in its UI. The v1 rule set (prometheus-rules-configmap.yaml) covers node health (NodeNotReady, NodeFilesystemAlmostFull, NodeClockSkew), pod health (PodCrashLooping, PodNotReady, ContainerOOMKilled), storage (PVCAlmostFull) and self-monitoring (PrometheusTargetDown), each with a severity label (warning|critical). The edge group adds TraefikDown — absent(up{job="traefik"}) for 5m, severity critical — covering total loss of the edge's own scrape targets (partial target loss stays covered by PrometheusTargetDown). Adding a real receiver later = appending to receivers: and pointing a route sub-tree at it — no image or PVC change. (Backlog v1.1: Telegram/email channels.)

Log-pipeline scope (live-verified 2026-09-09): the collector's filelog receiver EXCLUDES /var/log/pods/kube-system_*/*/*.log — Traefik pod logs (the edge access-log source) do NOT reach OpenSearch today. Enabling Traefik access logs would require an ingestion delta (un-exclude + possible volume sizing), flagged for the traffic-observability study (F1).

Secrets (Infisical paths only — never values)

Project harness, env prod. Four out-of-band k8s Secrets, created at apply time by scripts/apply-observability.sh from Infisical (the single source of truth for the contract is its secret_contract()):

k8s Secret (ns observability) Infisical path Keys Consumed by
opensearch services/observability/opensearch OPENSEARCH_ADMIN_PASSWORD, OPENSEARCH_ADMIN_HASH OpenSearch STS (admin seed), bootstrap-security, ism-apply
osd-dfm services/observability/opensearch-dashboards OSD_ADMIN_USER, OSD_ADMIN_PASSWORD OSD Deployment; rendered into the grafana-datasources CM at apply
grafana-admin services/observability/grafana GF_ADMIN_USER, GF_ADMIN_PASSWORD Grafana Deployment (admin login)
data-prepper-creds services/observability/data-prepper DATA_PREPPER_OPENSEARCH_PASSWORD, OPENSEARCH_ENDPOINTS (non-secret) Data Prepper initContainer render

Only secret.example.yaml (placeholders) lives in Git — root kubernetes/observability/secret.example.yaml (consolidated shape) plus the per-component examples. These are EXCLUDED from the Argo include glob and from any dir-wide apply (litellm-gate RC-2: an example Secret inside an Argo-synced tree clobbers the real one while Argo stays green — the 2026-09-04 incident that fixed the exclude glob to a single brace glob).

How is it deployed?

GitOps with two layers:

  1. Argo anchor — platform-gitops/apps/observability.yaml: Application observability (project platform) syncs platform-observability:main path kubernetes/observability, directory mode recurse: true with exclude: '{*example*,*.json,README.md}' (single brace glob — Argo exclude takes ONE glob), automated: prune: true, selfHeal: true, CreateNamespace=false (the namespace is created by the apply script / namespace.yaml first), PruneLast=true (the anchor was hardened in F7.4). The anchor argocd/application.yaml inside platform-observability is a legacy placeholder — the live anchor is the platform-gitops one.
  2. Apply script — scripts/apply-observability.sh (OPERATOR-GATED; the worker never runs it). It mirrors the proven apply-litellm.sh pattern and runs on the harness host with no local kubectl — every cluster op goes through ssh rke-cp01 sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml. Phases: preflight → Infisical fetch → create the 4 Secrets → file-by-file apply in a fixed order (never dir-wide; never touches *example* or *.json) → OpenSearch security bootstrap + ISM policy (in-pod, out-of-band) → WAIT gates (all pods Ready; certs verified) → in-cluster probes + summary. --allow-existing makes re-runs idempotent (re-fetch from Infisical and overwrite the Secrets). scripts/ism-apply.sh (separately re-runnable) loads the 14-day ISM policy + logs-* template with the OpenSearch 3.8.0 schema fixes live-verified at F7.2.

Dependencies

Depends on: Ceph CSI / StorageClass ceph-rbd for all five PVCs (SVC-CEPH-CSI); the edge Gateway wetopi-edge + nftables 443→8443 redirect (SVC-TRAEFIK-EDGE); cert-manager for the five obs certificates (SVC-CERT-MANAGER); DNS A records in infrastructure (tofu-owned, Cloudflare); Infisical (SVC-VAULT) for the four apply-time Secrets; the scrape targets across the cluster (kubernetes-SD) and OTLP producers for logs/traces (litellm is the first real traces producer — SVC-LITELLM).

Depended on by: every platform operation that needs metrics, log search or traces — incident response, OOM sizing decisions (litellm/vault lessons), E2E acceptance gates (F8.2 logs / F8.3 traces / F8.4 metrics).

Operation

kubectl is run on the control plane (no local kubectl); every command is via ssh rke-cp01 sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml.

  • Health: kubectl -n observability get pods (all 1/1, restarts low, no OOMKilled in lastState); kubectl -n argocd get application observability (Synced/Healthy); the four public probes (DNS → 2 IPs, TLS LE CN, 403 from a non-allowlisted source, 200/302 from .89/.87).
  • Logs: kubectl -n observability logs -l app.kubernetes.io/name=<component>; for node-level filelog ingestion check the collector DaemonSet pods (group-0 read on /var/log/pods is what makes pod logs readable).
  • Adding a dashboard: drop the JSON into the right ConfigMap under kubernetes/observability/grafana/ (see the repo README "Adding a dashboard" section), PR it; the wetopi-obs file provider picks it up from its mounted path on the next Grafana start (or the provider rescan). CM and mount changes are coupled (Law 4 — ship both in the same PR).
  • Adding a datasource: same pattern via grafana-datasources-configmap.yaml — but remember that file is the out-of-band template of record: a datasource needing credentials must keep the ${VAR} placeholder, gain a matching render step in apply-observability.sh, and a new Infisical key; plain http datasources are safe to edit directly (still: re-run the apply render so the live CM matches).
  • Rotating credentials: rotate the value in Infisical at its path, then re-run scripts/apply-observability.sh --allow-existing (re-fetches and overwrites the Secrets), then roll the consuming workloads (kubectl -n observability rollout restart deployment/<name> / statefulset/opensearch only when the OpenSearch admin password changes). The OpenSearch admin password is seeded at first boot only (OPENSEARCH_INITIAL_ADMIN_PASSWORD) — changing it later requires a security plugin user change, not a PVC wipe.
  • OpenSearch re-bootstrap (new PVC / DR): opensearch/bootstrap-security.sh (users + roles) and scripts/ism-apply.sh (retention) are the two out-of-band steps that re-run after a fresh OpenSearch data volume; both carry the live-verified OpenSearch 3.8.0 schema fixes (policy body wrapped as {"policy": {...}}, no enabled field in ism_template).

Runbook — apply (F7.2, operator-gated)

The full go-live apply is scripts/apply-observability.sh — see its header for the phase list and the Wetopi laws baked in (explicit file list, never dir-wide; Secrets out-of-band; name must match secretKeyRef; no NodePort; set -e). Operator steps:

  1. Seed Infisical (project harness, env prod) at the four vault paths above with the listed keys.
  2. From the repo root: VAULT_API_CLIENTID=*** VAULT_API_KEY=*** \ scripts/apply-observability.sh (idempotent re-runs: add --allow-existing).
  3. Wait for the script's WAIT gates (all pods Ready, 15m budget; OpenSearch bootstrap 10m budget), then verify the F7.3 public-contract probes.

Preconditions already in place at go-live: the four LE certificates Ready, the Gateway delta applied (F1.5), DNS A-records live (F1.6), the Argo anchor merged (F2.3/F7.4).

Rollback plan (plan v1 §7)

  • Edge: git-revert the Gateway certRef delta in platform-edge — the https listener falls back to the remaining certs; the four *-tls Secrets stay and cert-manager keeps renewing them. (Invariant R1: a certRef whose Secret is missing takes down the ENTIRE https listener, so edge changes are always fail-safe ordered and individually reversible.)
  • Whole stack: the manifests live under the Argo app with prune: true — revert the anchor merge or argocd app delete observability, then kubectl delete ns observability manually (the namespace is not Argo-created). PVCs carry the Delete policy, so metrics/logs/traces data is lost on rollback — accepted for v1, everything is re-derivable (scrapes and logs re-ingest; retention is by config, not by recreating PVCs — R6: never delete PVCs from a PR).
  • DNS: record removal is a tofu PR (the records carry prevent_destroy deliberately).
  • Blast radius: nothing outside ns observability except the shared https listener (mitigated by R1).

Estado honesto (live)

GO-LIVE 2026-09-05 (UTC). The suite is deployed, serving, and end-to-end operational.

  • Apply gate (F7.2): apply-observability.sh executed; all pods Running, Argo observability Synced/Healthy.
  • Public-contract probes (F7.3): 21/21 green — DNS 4×3 resolvers (.193/.194), TLS LE certs ×4 (correct CN per host), positive 200 from 92.222.245.89 (agents01 egress) via both edge IPs, negative 403 from 92.222.245.91 (ops01, outside the allowlist) on all four hostnames, in-cluster probes on all four backends, pods at 0 restarts.
  • Incident (resolved pre-gate): OSD (then at cerebro.wetopi.net) returned 503 at the OpenSearch Dashboards system-index step — root cause: the OSD config index fell under the demo-installer's system-index guard. Fix: OSD config index pointed to observability-osd-config (platform-observability PR #38 + hotfix commits; opensearchDashboards.index schema-verified from the 3.8.0 image) plus the dfm role PUT to its final live-verified shape (commit bea78ae).
  • Known limitations (honest):
  • Alerting is display-only — no external channels in v1 (operator decision; backlogged for v1.1).
  • Allowlist entry 51.254.106.87/32 is possibly stale (flagged 2026-09-09): ops01's real egress is 92.222.245.91, so the .87 range may be dead weight; an operator decision is pending — the middleware is UNTOUCHED until then (no edge mutation from a doc task).
  • otlp.wetopi.net route documented as of the 2026-09-09 live verification (HTTPRoute, cert, DNS, TraefikDown rule, filelog exclusion all re-read from the cluster); the fleet-side ingestion was verified the same day (3/3 hermes_gateway_up series in Prometheus).
  • OpenSearch security posture is the demo installer (self-signed transport/HTTP certs generated at boot, single node) — acceptable v1, not a production-grade TLS story.
  • Logs e2e evidence (a pod line visible in OpenSearch Dashboards) is the F8.2 lead gate — the pipeline is green (collector → prepper → OpenSearch) but the public-facing screenshot evidence lands with F8.2.
  • Hubble flow-metrics and egress NP hardening are out of scope v1 (plan §8).