ADR + Architectural Pack — Wetopi Network & Workload Security Architecture¶

Document type: ADR + Architectural Pack
Project: Wetopi Managed WordPress Platform — RKE2 Migration
Status: Proposed / Ready for decomposition into implementation tasks
Audience: Platform Engineering, SRE, Security, AI agents operating the Wetopi platform
Scope: North-south security, east-west isolation, egress governance, IDS/runtime visibility, WAF, TLS, threat feeds and security telemetry
Primary workload: Wetopi wepod — managed WordPress workload composed of nginx + PHP-FPM + MariaDB, currently packaged together and exposed through Wetopi load-balancing infrastructure
Target platform: RKE2 / Kubernetes, GitOps-managed
Decision owner: Wetopi Platform
Last updated: 2026-09-09

1. Executive decision¶

Wetopi will move from a security model based mainly on host firewalls, Docker/Swarm networking, NAT gateways and Suricata on workers toward a layered Kubernetes-native security architecture.

The target architecture will separate five responsibilities:

Cheap perimeter filtering with centrally compiled firewall/GeoIP lists
Known malicious IPs, Wetopi hard allow/deny lists, attack-time blocklists and global GeoIP policies will be compiled centrally and rejected as early and cheaply as possible using nftables + large nft sets on Internet-facing nodes.
Kubernetes workload isolation and network policy
Cilium will provide workload-aware L3/L4/L7 policy enforcement, default-deny segmentation, controlled DNS access and egress policy.
Controlled Internet egress
Wepods will no longer inherit unrestricted Internet access through the worker node default gateway. Outbound traffic will be forced through explicit Wetopi egress gateways / NAT-GW pools.
Detection and attribution
Hubble, Tetragon and Suricata will be used for complementary purposes:
Hubble: network-flow attribution and policy visibility.
Tetragon: workload/process/runtime attribution.
Suricata: IDS/protocol/signature analysis at network chokepoints.
Selective application-layer protection
Coraza + OWASP CRS will be available as an optional WAF controlled per site, with off, detect and block modes.

TLS will continue to terminate at the Wetopi L7 ingress layer. Traffic between ingress and workloads may remain HTTP at application level while being protected at transport level by Cilium/WireGuard between nodes.

The architecture is intentionally designed so that security policy is declared through high-level Wetopi profiles, rather than duplicated as raw rules in every customer workload manifest.

2. Context¶

Wetopi is a managed WordPress hosting platform.

The current backend consists of hundreds of customer workloads ("wepods"), historically implemented as Docker containers combining:

s6
nginx
PHP-FPM
MariaDB

The workloads expose their HTTP service internally. Wetopi load balancers publish those services to the Internet.

Historically:

inbound Internet traffic reaches Wetopi L4/L7 infrastructure;
nginx reverse proxies perform SSL/TLS offloading;
traffic from L7 proxy to customer container travels over the internal LAN as HTTP;
customer containers use the default gateway of their Docker/Swarm worker to reach the Internet;
known malicious address lists, hard allow/deny lists and attack blocklists are distributed to Internet-facing hosts with iptables + ipset;
GeoIP country/continent lists are currently generated from downloaded IP/range databases and used for geographic access control;
the lists are refreshed using mechanisms including consul-template;
Suricata is used on Swarm workers and NAT gateways to detect suspicious traffic.

The migration to RKE2/Kubernetes is driven in significant part by security and operational isolation requirements.

The new platform must preserve useful protections from the legacy design while improving:

tenant isolation;
workload attribution;
egress control;
incident containment;
observability;
automation;
policy consistency;
GitOps management.

3. Problem statement¶

The new platform must answer the following questions reliably:

How are very large external malicious-IP lists distributed and enforced on every Internet-facing node?
How do we prevent one compromised WordPress workload from reaching other tenants or arbitrary internal services?
How do we prevent compromised or abusive customer workloads from damaging the reputation of Wetopi public egress IPs?
How do we identify which exact site/pod/process generated suspicious traffic?
Where should Suricata run in Kubernetes?
Should TLS continue to terminate at the L7 ingress, or should every workload receive HTTPS internally?
How can a WAF be enabled only for sites that need it, without forcing all customers through a potentially noisy rule set?
How do we expose a small, stable security API to the Wetopi control plane instead of embedding implementation details in every wepod manifest?

4. Goals¶

4.1 Primary goals¶

The system MUST:

reject known hostile traffic before it consumes expensive Kubernetes resources;
implement Kubernetes-native default-deny network isolation;
prevent direct east-west communication between unrelated customer workloads;
route customer Internet egress through controlled NAT gateways;
block direct outbound SMTP by default;
provide site-level attribution for outbound network activity;
make WAF enforcement configurable per wepod/site;
provide runtime and network telemetry suitable for incident analysis;
preserve Wetopi's current L7 TLS-termination model unless a concrete requirement justifies changing it;
support GitOps deployment and reconciliation;
keep operational complexity proportional to the value of each layer;
permit staged rollout and fast rollback.

4.2 Secondary goals¶

The system SHOULD:

support security profiles rather than hand-written per-site policies;
permit quarantine of a compromised site without stopping the entire platform;
permit dynamic threat-feed updates without restarting workloads;
expose Prometheus metrics for every security control;
emit structured logs into OpenSearch;
allow future automated security scoring/remediation;
minimize per-packet cost for known blocklist traffic;
avoid introducing service mesh unless separately justified.

5. Non-goals¶

This ADR does NOT attempt to:

redesign the internal composition of the wepod;
split nginx, PHP-FPM and MariaDB into separate pods/containers;
define a complete SIEM/SOC product;
replace customer WordPress security plugins;
guarantee detection of every compromised workload;
create a fully autonomous AI security remediation system in phase 1;
build a custom IDS;
implement zero-trust mTLS between every application process;
replace existing Wetopi L4 DSR/IPVS/VRRP load balancing unless separately decided.

6. Architectural principles¶

6.1 Security by chokepoint¶

Controls should be applied at the cheapest point that has enough information to make the decision.

Examples:

known malicious source IP → edge firewall;
pod-to-pod authorization → Cilium policy;
site-specific HTTP exploit rule → WAF;
malware/C2 network signature → Suricata;
suspicious process execution → Tetragon.

Do not ask a higher and more expensive layer to perform work that a lower layer can handle reliably.

6.2 Default deny, explicit allow¶

Customer workloads are considered untrusted workloads.

The target state is:

Ingress: default deny
Egress:  default deny
East-west: default deny

Traffic is allowed only through approved Wetopi profiles.

6.3 Platform policy, not customer policy¶

The wepod manifest MUST NOT become a copy of the entire network security configuration.

A workload declares intent:

wetopi.io/network-profile: wordpress-standard
wetopi.io/egress-pool: customer-a
wetopi.io/waf-mode: detect

The platform translates that intent into:

CiliumNetworkPolicy / CiliumClusterwideNetworkPolicy;
EgressGatewayPolicy;
Traefik middleware;
WAF configuration;
security telemetry labels.

6.4 Stable identity everywhere¶

Every security event should be attributable to at least:

site_id
customer_id
namespace
pod
node
egress_pool
public_egress_ip

Where possible also:

container
process
binary
destination_ip
destination_port
dns_name
HTTP host
security_rule_id

7. High-level architecture¶

                                  INTERNET
                                      │
                                      ▼
                        ┌────────────────────────┐
                        │ Wetopi WAN / L4 Edge   │
                        │ IPVS / VRRP / DSR      │
                        │                        │
                        │ nftables + nft sets    │
                        │ optional Suricata IDS  │
                        └───────────┬────────────┘
                                    │
                                    ▼
                        ┌────────────────────────┐
                        │ Traefik L7 Ingress     │
                        │ TLS termination        │
                        │ HTTP routing           │
                        │ rate limits            │
                        │ optional Coraza WAF    │
                        └───────────┬────────────┘
                                    │
                              HTTP application
                                    │
                         Cilium policy enforcement
                                    │
                                    ▼
                       ┌──────────────────────────┐
                       │ Customer wepod           │
                       │ nginx + PHP + MariaDB    │
                       │                          │
                       │ identity = labels        │
                       └────────────┬─────────────┘
                                    │
                             controlled egress
                                    │
                         Cilium Egress Gateway
                                    │
                                    ▼
                       ┌──────────────────────────┐
                       │ Wetopi NAT-GW            │
                       │                          │
                       │ nftables                 │
                       │ Suricata IDS             │
                       │ fixed public egress IP   │
                       └────────────┬─────────────┘
                                    │
                                    ▼
                                 INTERNET

Observability plane:

Cilium/Hubble ──────┐
Tetragon ───────────┤
Suricata EVE ───────┤
Traefik logs ───────┼──> collectors/pipeline ──> OpenSearch
Coraza logs ────────┤
nftables metrics ───┘

Prometheus <── exporters / native metrics
Grafana    <── Prometheus + OpenSearch

8. ADR-001 — Use Cilium as the workload network security enforcement layer¶

Status¶

Proposed — recommended for adoption

Decision¶

Use Cilium for:

Kubernetes network policy enforcement;
default-deny workload segmentation;
L3/L4 rules;
selected L7-aware controls where useful;
DNS/FQDN-aware egress rules;
Hubble network observability;
egress gateway functionality;
optional transparent node-to-node WireGuard encryption.

Rationale¶

The requirement is larger than basic Kubernetes NetworkPolicy.

Wetopi needs:

workload identity based on Kubernetes labels;
rich flow visibility;
controlled egress;
scalable policy;
correlation of flows with pods/namespaces;
eventual integration with automated security responses.

Cilium places these controls in one coherent data plane.

Consequence¶

Cilium becomes a critical platform component and must be treated as infrastructure, not as an optional add-on.

It requires:

explicit version pinning;
GitOps-managed configuration;
upgrade strategy;
observability;
staged deployment;
rollback plan.

9. ADR-002 — Central Firewall List Compiler + Node-local Edge Firewall Controller¶

Status¶

Proposed — recommended for implementation

Decision¶

Wetopi will implement the dynamic edge-firewall distribution plane as two separate components with a strict responsibility boundary:

wetopi-firewall-list-controller
Central lightweight service, deployed as a Kubernetes Deployment / ReplicaSet with multiple replicas. It is the only component that downloads, parses, curates, deduplicates, merges and compiles external/internal firewall lists.
wetopi-edge-fw-controller
Very small privileged node-local component, deployed as a DaemonSet only on selected WAN/NAT-GW nodes. It does not curate feeds or build policy. It only receives a revision notification, downloads the already-built artifact, validates it and atomically reconciles the local firewall state.

The node-local component MUST NOT independently download FireHOL, abuse feeds, GeoIP databases or any other upstream source.

All edge nodes must converge on an immutable revision generated centrally.

10. Firewall List / GeoIP Distribution Plane¶

10.1 Desired architecture¶

                   External sources
        FireHOL / abuse feeds / GeoIP DBs / etc.
                          │
                          ▼
              ┌─────────────────────────┐
              │ wetopi-firewall-list-   │
              │ controller              │
              │                         │
              │ Deployment / ReplicaSet │
              │                         │
              │ fetch                   │
              │ parse                   │
              │ validate                │
              │ normalize               │
              │ deduplicate             │
              │ aggregate CIDRs         │
              │ curate                  │
              │ compile                 │
              └────────────┬────────────┘
                           │
             Wetopi runtime firewall state
                           │
        ┌──────────────────┼────────────────────┐
        │                  │                    │
        ▼                  ▼                    ▼
 hard whitelist      hard blacklist       attack blocks
 temporary blocks    custom firewall       GeoIP policy
                     rules / ACLs
        │                  │                    │
        └──────────────────┼────────────────────┘
                           ▼
                  immutable revision
                     `fw-rev-18472`
                           │
              ┌────────────┴────────────┐
              │ artifact repository     │
              │ + revision API          │
              └────────────┬────────────┘
                           │
               revision-changed notification
                           │
       ┌───────────────────┼────────────────────┐
       ▼                   ▼                    ▼
 WAN-01                WAN-02               NATGW-01
 edge-fw-controller    edge-fw-controller   edge-fw-controller
 DaemonSet             DaemonSet            DaemonSet
       │                   │                    │
       └────── download + verify + atomic apply ┘
                           │
                           ▼
                    nftables / nft sets

The central service produces artifacts.

The node-local controllers consume artifacts.

There MUST be no per-node feed compilation.

10.2 Source of truth¶

Three distinct concepts MUST not be mixed.

A. Desired runtime firewall state¶

The Wetopi backend/control-plane is the source of truth for dynamic firewall intent:

hard whitelist
hard blacklist
temporary blacklist
attack blacklist
custom firewall rules
enabled external feed sources
GeoIP policy assignments
firewall profile targeting

This state may live in the Wetopi backend database/API.

It is not GitOps state because some of it changes quickly during incidents.

Examples:

block 48,000 source IPs now
whitelist customer office CIDR
expire temporary block in 2 hours
disable Brazil for site 19382
globally block country X temporarily

These actions MUST NOT require:

Git commit
Argo CD sync
node restart
pod restart

B. Compiler configuration¶

Git/GitOps owns stable configuration of the system:

which external feed providers exist
provider URLs/endpoints
parser type
refresh interval
validation thresholds
curation policy
aggregation policy
list precedence
artifact format
edge profiles
base nftables policy
software deployment/configuration

Example:

sources:
  - id: firehol-level1
    type: cidr-list
    enabled: true
    refresh: 30m

  - id: provider-abuse
    type: cidr-list
    enabled: true
    refresh: 15m

  - id: geoip-country-db
    type: geoip-ranges
    enabled: true
    refresh: 24h

C. Materialized runtime revision¶

The compiler creates an immutable, versioned artifact:

fw-rev-18472

That artifact is the exact desired firewall state for a given edge profile at that point in time.

Example:

profile=edge-ingress
revision=18472
sha256=...
generated_at=...

Every applicable node should ultimately report:

desired_revision = 18472
active_revision  = 18472

The local nftables state is never a source of truth.

10.3 Central component — `wetopi-firewall-list-controller`¶

Deployment model¶

Deploy as a normal Kubernetes Deployment with multiple replicas.

Initial recommendation:

replicas: 2 or 3

The service must tolerate loss/restart of one replica.

Compilation of a single logical revision MUST be serialized to avoid multiple replicas publishing competing revisions.

Acceptable implementation:

Kubernetes Lease / leader election

Only the current leader performs scheduled compilation/publication.

Do not make compiler state depend only on local pod filesystem.

10.4 Central compiler responsibilities¶

The compiler MUST support:

Source acquisition¶

HTTP/HTTPS list
compressed files
plain CIDR/IP files
GeoIP/range databases
Wetopi internal API/database lists
manual emergency lists

Parsing¶

Support:

single IPv4
single IPv6
IPv4 CIDR
IPv6 CIDR
start-end ranges
country/continent metadata
comments/metadata

Validation¶

Examples:

reject malformed addresses
reject impossible ranges
detect suspiciously empty feed
detect massive unexpected size changes
validate download status
validate freshness

A broken upstream source MUST NOT result in an empty production firewall set.

Normalization¶

Convert addresses to canonical form.

Deduplication¶

Duplicates across providers must not result in duplicate runtime entries.

The compiler SHOULD retain source provenance even if the deployed set contains the address only once.

CIDR/range aggregation¶

Where semantically safe, contiguous ranges may be compacted.

Aggregation MUST NOT accidentally include addresses that were not in the source policy.

Allowlist subtraction¶

Hard allowlists are applied before publication according to explicit precedence rules.

Compilation¶

Generate efficient artifacts appropriate for:

nftables/nft sets
Traefik GeoIP policy consumption
inspection/debugging

Publication¶

Publish an immutable revision and update the desired revision pointer only after all validation succeeds.

Notification¶

Notify connected edge controllers that a new desired revision exists.

10.5 Notification model¶

The central service SHOULD actively notify wetopi-edge-fw-controller instances when a new revision is published.

The notification itself MUST NOT contain the firewall state and MUST NOT be the source of truth.

It is only a wake-up signal:

revision 18473 available

The edge controller then performs:

GET desired revision
GET manifest
GET artifact
validate
apply
report active revision

Recommended lightweight implementation:

long-lived SSE/stream or long-poll connection
edge-fw-controller → central service

This avoids the central controller needing routable addresses for every DaemonSet pod.

A periodic fallback poll MUST also exist.

Example:

notification path: immediate
fallback reconciliation: every 30–60 seconds

Therefore loss of a notification never creates permanent drift.

Do NOT introduce Kafka/NATS/another message bus solely for this mechanism.

10.6 Artifact repository¶

The compiler MUST publish immutable artifacts to durable shared storage.

The exact implementation can be:

S3-compatible object storage
HTTP artifact repository
other durable shared object storage

Required logical structure:

/firewall/
  edge-ingress/
    current.json
    revisions/
      18471/
      18472/
      18473/

  edge-egress/
    current.json
    revisions/
      ...

current.json points to the desired revision.

Example:

{
  "profile": "edge-ingress",
  "revision": 18473,
  "sha256": "sha256:...",
  "generated_at": "2026-09-09T12:52:10Z"
}

A published revision MUST be immutable.

Rollback is performed by changing the desired revision pointer to a previous known-good artifact, not by editing the old artifact.

10.7 Firewall profiles¶

Different node roles may receive different bundles.

Initial profiles:

edge-ingress
edge-egress

Possible future profiles:

edge-ingress-ddos
edge-management
edge-special

Node selection example:

wetopi.io/security-edge: "true"
wetopi.io/firewall-profile: "edge-ingress"

The edge controller reads its assigned profile and reconciles only that profile.

10.8 Node-local component — `wetopi-edge-fw-controller`¶

Deployment¶

Kubernetes DaemonSet.

Runs only on explicitly selected nodes.

Because it changes host firewall state it will require carefully constrained privileged/host-network capabilities.

The security review MUST minimize capabilities rather than granting unrestricted privilege without justification.

Responsibilities¶

The node-local controller MUST remain intentionally small.

It performs:

determine its assigned firewall profile;
connect to/watch the central controller for revision notifications;
periodically reconcile as a fallback;
obtain desired revision metadata;
download the immutable artifact;
verify checksum/signature;
validate local applicability;
atomically load the new nftables rules/sets;
verify the resulting state;
report active revision and health;
retain enough previous local state for emergency recovery.

It MUST NOT:

download FireHOL
download abuse feeds
download GeoIP databases
deduplicate addresses
decide list precedence
aggregate CIDRs
interpret customer policy
generate country mappings
invent firewall rules

All of those belong to the central compiler/control-plane.

10.9 Firewall data classes¶

The compiler should preserve separate semantic sets even where they all ultimately cause DROP.

Initial classes:

hard_whitelist_v4
hard_whitelist_v6

hard_blacklist_v4
hard_blacklist_v6

attack_blacklist_v4
attack_blacklist_v6

temporary_blacklist_v4
temporary_blacklist_v6

threatfeed_v4
threatfeed_v6

Do not flatten all inputs into one opaque blacklist.

Separate sets provide:

provenance;
debugging;
independent counters;
different expiration semantics;
controlled precedence;
easier rollback;
incident visibility.

10.10 High-volume attack blocking¶

Wetopi explicitly requires the ability to add thousands or tens of thousands of attacking IPs quickly.

Example:

incident detected
    ↓
Wetopi backend adds 47,821 attack sources
    ↓
compiler generates revision 18474
    ↓
publishes artifact
    ↓
notifies edge controllers
    ↓
nodes download
    ↓
atomic nft set update

This path must not require GitOps.

For large lists, use nftables sets rather than one firewall rule per address.

10.11 Custom allow/deny rules¶

Wetopi also requires hard custom firewall policy beyond threat feeds.

Examples:

hard whitelist an office CIDR
hard blacklist an IP/CIDR
allow TCP/8443 only from management networks
block a protocol/port on egress gateways
allow a monitoring network
temporary emergency rule

The preferred model is structured policy in the Wetopi backend.

Example:

id: fw-rule-1844
target: edge-ingress
action: accept
source:
  cidrs:
    - 192.0.2.0/24
protocol: tcp
destinationPorts:
  - 8443
priority: hard-whitelist

The central compiler translates this representation into nftables.

Avoid allowing arbitrary free-form nftables fragments as the normal API.

10.12 Sets versus individual rules¶

For high-cardinality IP matching:

DO:
nft set + one/few lookup rules

DO NOT:
generate 50,000 individual nftables rules

Per-destination/VIP policy can use concatenated keys where useful:

destination_ip . source_ip

Individual firewall rules are reserved for policy that genuinely includes protocol/port/direction/interface/destination/connection-state semantics.

10.13 Precedence¶

Precedence MUST be explicit and testable.

Initial conceptual precedence:

1. hard whitelist
2. explicit high-priority custom allow rules
3. explicit hard blacklist
4. attack blacklist
5. temporary blacklist
6. external threat feeds
7. normal/base firewall policy

The compiler owns precedence.

The node-local controller merely applies the compiled result.

10.14 GeoIP list generation¶

GeoIP is promoted from a future idea to a first-class responsibility of the central compiler.

Wetopi currently builds country lists by downloading databases containing IP/range-to-geography mappings, parsing them and generating IP/range lists per country.

The new service MUST absorb this functionality.

Pipeline:

GeoIP source databases
        ↓
download
        ↓
validate
        ↓
parse IP/range → country/continent
        ↓
normalize
        ↓
deduplicate
        ↓
CIDR/range compact where safe
        ↓
publish country/continent artifacts

Required logical products:

geo_country_ES_v4
geo_country_ES_v6
geo_country_FR_v4
geo_country_FR_v6
geo_country_US_v4
geo_country_US_v6
...
geo_continent_EU_v4
geo_continent_EU_v6
geo_continent_AS_v4
geo_continent_AS_v6
...

Country identity should use stable ISO-style two-letter codes.

10.15 GeoIP consumers¶

The same compiled GeoIP source should serve two different enforcement scopes.

Global/system-wide country blocking¶

For:

block country X everywhere

prefer edge nftables sets because this is the earliest and cheapest enforcement point.

Per-site/customer GeoIP policy¶

For:

site 12345:
  block CN
  block RU

or:

site 12345:
  allow ES/PT/FR only

the decision requires site/HTTP-host identity.

This should happen at the L7 ingress / Traefik layer.

Target:

client IP
    ↓
Traefik
    ↓
Host/SNI → site_id
    ↓
site GeoIP policy
    ↓
allow / deny

The central compiler therefore SHOULD publish a Traefik-consumable GeoIP artifact in addition to nftables sets.

Do not duplicate independent GeoIP download/parsing logic in Traefik.

The architectural requirement is:

one GeoIP compilation pipeline, multiple consumers.

Possible consumption mechanisms to evaluate:

generated CIDR artifacts consumed by middleware
compact radix/prefix database loaded by a small Traefik middleware/plugin
dedicated lightweight GeoIP middleware/service

The exact mechanism is deferred.

10.16 GeoIP policy model¶

Wetopi control-plane intent should support at least:

geoip:
  mode: deny
  countries:
    - CN
    - RU

and:

geoip:
  mode: allow
  countries:
    - ES
    - PT
    - FR

Scopes:

global
site
possibly customer/account in future

Recommended precedence:

global hard deny
      ↓
site allow/deny policy
      ↓
normal access

10.17 GeoIP update semantics¶

The compiler MUST:

record dataset version/date;
generate a new immutable revision when mappings change;
expose country-set cardinality;
keep the previous known-good revision;
detect malformed/empty country data;
avoid publishing partial country datasets.

Required metrics:

wetopi_geoip_dataset_age_seconds
wetopi_geoip_country_prefixes
wetopi_geoip_compile_failure_total

10.18 Atomicity and failure behaviour¶

A node MUST never transition through:

old rules
→ empty rules
→ new rules

Required:

old revision active
        ↓
download new revision
        ↓
validate
        ↓
prepare new nft state
        ↓
atomic switch/replacement
        ↓
verify
        ↓
mark active revision

On failure:

KEEP previous known-good active revision
REPORT failure
RETRY

Never flush protection because a feed, GeoIP source, compiler instance, notification or artifact download failed.

10.19 Drift detection¶

Each controller reports:

node
profile
desired_revision
active_revision
last_apply_timestamp
status

Example:

WAN01   edge-ingress  desired=18473 active=18473 OK
WAN02   edge-ingress  desired=18473 active=18473 OK
WAN03   edge-ingress  desired=18473 active=18472 DRIFT
NAT01   edge-egress   desired=918   active=918   OK

Persistent mismatch must alert.

10.20 Metrics¶

Central compiler:

wetopi_fw_compile_success_total
wetopi_fw_compile_failure_total
wetopi_fw_active_revision
wetopi_fw_compile_duration_seconds

wetopi_fw_source_last_success_timestamp
wetopi_fw_source_age_seconds
wetopi_fw_source_entries
wetopi_fw_source_errors_total

wetopi_fw_entries{class="hard_blacklist"}
wetopi_fw_entries{class="attack_blacklist"}
wetopi_fw_entries{class="temporary_blacklist"}
wetopi_fw_entries{class="threatfeed"}

wetopi_geoip_dataset_age_seconds
wetopi_geoip_compile_failure_total
wetopi_geoip_country_prefixes

Node controller:

wetopi_edgefw_desired_revision
wetopi_edgefw_active_revision
wetopi_edgefw_apply_success_total
wetopi_edgefw_apply_failure_total
wetopi_edgefw_last_success_timestamp
wetopi_edgefw_ruleset_entries
wetopi_edgefw_blocked_packets_total
wetopi_edgefw_blocked_bytes_total

Do not use arbitrary source IPs/site IDs as Prometheus labels.

10.21 Initial implementation boundary¶

The first version does NOT need:

AI
automatic attack classification
complex distributed messaging
full SIEM integration
per-site autonomous remediation
arbitrary rule DSL

It DOES need:

centralized feeds
curation
deduplication
GeoIP generation
hard allow/deny inputs
attack blocklist
immutable revisions
notification
fallback reconciliation
atomic nftables apply
revision reporting

Keep this component small and boring.

11. ADR-003 — Wepods use platform-defined security profiles¶

Decision¶

Each wepod receives a small set of high-level labels/fields.

Example:

metadata:
  labels:
    wetopi.io/workload: wordpress
    wetopi.io/site-id: "12345"
    wetopi.io/customer-id: "567"
    wetopi.io/network-profile: wordpress-standard
    wetopi.io/egress-pool: natgw-customer-a
    wetopi.io/waf-mode: "off"

The platform owns these labels.

Customers MUST NOT be able to arbitrarily assign privileged security profiles.

12. Ingress policy¶

12.1 Baseline¶

Every customer workload should be default-deny for ingress.

Conceptual policy:

DENY ingress
ALLOW Traefik → wepod HTTP port
ALLOW required Wetopi management traffic
ALLOW explicitly documented platform dependencies

Do not permit:

Internet → pod directly
customer pod → unrelated customer pod
arbitrary namespace → wepod

12.2 Internal management traffic¶

Any operational access required by:

Wetopi SRE agents;
backup jobs;
health checks;
migration tooling;
monitoring;

must be represented as explicit identities/policies.

Avoid broad rules such as:

allow entire 10.0.0.0/8

unless unavoidable.

Prefer:

namespaceSelector
serviceAccount
labels
explicit ports

13. East-west isolation¶

Decision¶

Customer workloads are isolated from each other by default.

Required property:

wepod A cannot initiate arbitrary traffic to wepod B

This remains true even if both workloads are scheduled on the same worker.

This protects against:

compromised WordPress lateral scanning;
credential harvesting;
exploitation of another tenant;
service discovery attacks;
accidental cross-tenant access.

14. ADR-004 — Internet egress is controlled through Wetopi Egress Gateways¶

Decision¶

The legacy pattern:

wepod
  ↓
worker default gateway
  ↓
Internet

will be removed.

Target:

wepod
  ↓
Cilium egress policy
  ↓
designated Wetopi NAT-GW / egress node
  ↓
SNAT to known Wetopi public IP
  ↓
Internet

Benefits¶

stable public source IP assignment;
attribution by egress pool;
centralized IDS visibility;
reputation containment;
customer grouping;
easier emergency blocking;
consistent routing independent of pod placement.

15. Egress profiles¶

Initial profiles:

`wordpress-standard`¶

Intended for normal managed WordPress sites.

Allowed:

DNS → approved cluster DNS
TCP/80 → Internet
TCP/443 → Internet

Denied:

direct SMTP
arbitrary UDP
arbitrary TCP
east-west traffic
private internal networks unless explicitly permitted

`wordpress-restricted`¶

For suspicious, high-risk or tightly controlled sites.

Possible rules:

DNS → approved DNS
HTTPS → approved FQDNs only
HTTP → optionally disabled
everything else → deny

Potential permitted FQDNs:

api.wordpress.org
downloads.wordpress.org
*.wordpress.org
approved payment APIs
approved plugin/vendor APIs

This profile SHOULD NOT be the default until real-world plugin compatibility is understood.

`wordpress-mail-relay`¶

Same as standard, but permits traffic to:

Wetopi SMTP relay only

It MUST NOT grant generic SMTP access to the Internet.

`quarantine`¶

For compromised/suspicious workloads.

Example:

Ingress:
  optionally maintenance/control-plane only

Egress:
  Wetopi forensic/update endpoints only
  no arbitrary Internet
  no SMTP

East-west:
  deny

Quarantine must be activatable through one high-level site state.

`custom`¶

Exceptional profile.

Requires:

explicit justification;
documented owner;
expiry/review date where possible;
GitOps-visible configuration.

Custom profiles must not become the normal path.

16. SMTP policy¶

Decision¶

Direct outbound SMTP from customer WordPress workloads is denied by default.

At minimum block:

TCP/25
TCP/465
TCP/587

except towards explicitly approved infrastructure.

Recommended path:

WordPress
   │
   ▼
Wetopi SMTP relay
   │
authentication
tenant attribution
rate limiting
quotas
abuse detection
   │
   ▼
external SMTP provider / Internet

This is one of the highest-value controls for protecting Wetopi IP reputation.

Future relay requirements:

per-site credentials or identity;
rate limit;
recipient limits;
anomaly detection;
logging;
revocation;
optional external reputation provider integration.

17. DNS policy¶

Wepods MUST use approved cluster DNS infrastructure.

Do not allow pods to bypass policy through arbitrary external resolvers.

Conceptual policy:

ALLOW UDP/TCP 53 → kube-dns/CoreDNS only
DENY UDP/TCP 53 → arbitrary Internet

If DNS over HTTPS becomes a meaningful bypass concern, handle it separately; do not attempt to solve it prematurely in phase 1.

18. Private/internal destination protection¶

Customer workloads should not have generic access to:

RFC1918 infrastructure
Kubernetes API
Ceph
RKE2 control plane
hypervisors
management networks
monitoring systems
Infisical
Argo CD
internal databases
other Wetopi control-plane services

Explicit denies or absence of allows must guarantee this property.

High-risk destinations should additionally be guarded at infrastructure boundaries, not only through pod policy.

19. ADR-005 — Hubble is the primary network-flow observability layer¶

Decision¶

Deploy Hubble with enough retention/export to answer:

Which site contacted this IP?
Which destination caused this policy drop?
Which pods are generating most outbound connections?
Which pod attempted SMTP?
Which workloads are scanning internal addresses?

Hubble data should contain Wetopi labels wherever possible.

Required useful dimensions:

site_id
customer_id
namespace
pod
source_ip
destination_ip
destination_port
protocol
verdict
policy
node

Do not store every possible flow forever.

Retention and export should be tuned according to operational value and cost.

20. ADR-006 — Tetragon provides runtime attribution¶

Decision¶

Use Tetragon selectively to correlate suspicious network/runtime behaviour with actual processes.

Example desired event:

site_id=12345
pod=wepod-abc
process=php-fpm
pid=...
event=connect
destination=198.51.100.42:4444

Other high-value detections:

php-fpm spawning shell
nginx spawning shell
unexpected curl/wget execution
unexpected package managers
reverse-shell patterns
execution from writable upload directories
suspicious privilege changes

Rollout strategy¶

Do NOT begin with hundreds of aggressive runtime rules.

Start with a small set of high-signal detections.

Tetragon should initially be treated as:

visibility + detection

not broad autonomous enforcement.

Enforcement can be introduced later for rules with proven low false-positive rates.

21. ADR-007 — Suricata moves to network chokepoints¶

Decision¶

Do not replicate the old pattern of installing Suricata on every Kubernetes worker unless future evidence demonstrates a concrete blind spot.

Primary locations:

WAN ingress nodes
NAT-GW / egress nodes

This provides high-value visibility while reducing duplication and complexity.

22. Suricata responsibilities¶

Suricata is responsible for network/protocol-level detection such as:

known C2;
malware signatures;
exploit signatures;
suspicious scanning;
protocol anomalies;
DNS anomalies;
TLS metadata;
abnormal outbound traffic;
known botnet destinations.

Suricata is NOT the source of truth for workload identity.

Identity comes from correlation with:

Cilium/Hubble
Kubernetes metadata
NAT gateway routing information
Tetragon where required

23. Suricata output¶

Use EVE JSON.

Events of interest include:

alert
flow
dns
http
tls
fileinfo
anomaly

Do not ingest every available event type without considering volume.

Send selected structured events to OpenSearch.

Important correlation fields:

timestamp
src_ip
src_port
dest_ip
dest_port
proto
flow_id
alert.signature_id
alert.signature
alert.category
tls.sni
dns.rrname
http.hostname

24. Correlation model¶

Target incident reconstruction:

Suricata:
  source=10.42.17.83
  dest=198.51.100.66:443
  SID=known-c2

        ↓ correlate source IP/time

Hubble:
  10.42.17.83
  namespace=wp-12345
  pod=wepod-12345-xyz
  site_id=12345

        ↓ optional runtime correlation

Tetragon:
  process=php-fpm
  binary=/usr/sbin/php-fpm
  connect() → 198.51.100.66:443

Final security event:

site_id: 12345
severity: critical
type: outbound-known-c2
process: php-fpm
destination: 198.51.100.66
network_signature: ...
recommended_action: quarantine

This correlation is more valuable than placing Suricata everywhere without workload identity.

25. ADR-008 — Continue TLS termination at Traefik¶

Decision¶

Continue:

Internet ─HTTPS─> Traefik ─HTTP─> wepod nginx

at the application layer.

Reasons:

centralized certificate lifecycle;
simpler customer workload;
no duplicated certificate management;
simpler WordPress backend;
consistent ingress policy;
easier WAF integration.

Do not introduce internal per-wepod TLS merely for conceptual purity.

26. Internal traffic encryption¶

If protection of inter-node plaintext traffic is required, prefer transparent Cilium WireGuard encryption.

Target:

Traefik
   │
   │ HTTP packet
   ▼
Cilium/WireGuard encrypted node transport
   │
   ▼
wepod

Application components still see HTTP.

Physical/network transit between nodes is encrypted.

This avoids introducing a service mesh solely to obtain transport encryption.

27. ADR-009 — WAF is per-site and has three modes¶

Decision¶

Deploy Coraza + OWASP Core Rule Set behind/in Traefik as an optional middleware.

Every site supports:

off
detect
block

`off`¶

No WAF evaluation.

`detect`¶

Evaluate requests and log what would have been blocked.

Do not block the request.

`block`¶

Enforce configured CRS policy.

28. WAF activation model¶

Example intent:

wetopi.io/waf-mode: detect

or control-plane configuration:

security:
  waf:
    mode: detect
    profile: wordpress-default

The platform reconciles this into the corresponding Traefik middleware/routing configuration.

The customer workload itself does not need to know Coraza exists.

29. WAF rollout philosophy¶

Do NOT globally force CRS blocking across all production WordPress sites on day one.

WordPress ecosystems contain:

page builders;
arbitrary plugin endpoints;
WooCommerce callbacks;
REST APIs;
XML-RPC;
admin AJAX;
webhook receivers;
unusual multipart/form-data requests.

Global blocking without observation is likely to create false positives.

Recommended progression:

OFF
  ↓
DETECT
  ↓ observe
site-specific exclusions if required
  ↓
BLOCK

Later Wetopi may decide that detect or low-paranoia block becomes default for selected plans.

That is a later product/security decision.

30. WAF configuration hierarchy¶

Recommended hierarchy:

global CRS baseline
      │
      ├── WordPress base exclusions
      │
      ├── WooCommerce exclusions
      │
      ├── known plugin compatibility packs
      │
      └── per-site overrides

Avoid hundreds of unstructured site-specific rule files.

Overrides need:

site_id
rule_id
reason
created_at
owner
optional expiry/review date

31. Rate limiting¶

Rate limiting should remain conceptually distinct from WAF.

Useful controls at Traefik:

per-IP request rate
burst
login endpoint limits
xmlrpc.php limits
wp-login.php limits
admin-ajax anomaly controls

Do not use WAF signatures for simple volumetric behaviours that a cheaper rate limiter can handle.

32. Future Wetopi Security Controller¶

Status¶

Future phase, not required for initial migration

Build a small platform service that consumes security signals and calculates a site-level state.

Inputs:

Hubble
Tetragon
Suricata
Traefik
Coraza
nftables counters
possibly WordPress application signals

Example state:

site_id: 12345
risk_score: 90
state: critical
signals:
  - suricata-known-c2
  - php-fpm-shell-exec
  - outbound-connection-spike

Potential state machine:

NORMAL
  ↓
OBSERVE
  ↓
WAF_DETECT
  ↓
WAF_BLOCK
  ↓
EGRESS_RESTRICTED
  ↓
QUARANTINE

Initial versions MUST remain human-controlled for disruptive actions.

Recommended first automation:

detect → recommend action

before:

detect → automatically quarantine

33. Security profile API¶

The long-term control-plane interface should look approximately like:

apiVersion: wetopi.io/v1
kind: SiteSecurityProfile

metadata:
  name: site-12345

spec:
  siteId: "12345"

  network:
    profile: wordpress-standard

  egress:
    pool: natgw-customer-a
    policy: wordpress-standard

  waf:
    mode: detect
    profile: wordpress-default

  runtime:
    profile: wordpress-observe

  incidentMode:
    quarantine: false

This CRD is illustrative.

Do not implement a CRD merely because Kubernetes supports CRDs. First determine whether the Wetopi control plane should own the data and render standard resources, or whether a CRD/operator genuinely reduces complexity.

34. Ownership boundaries¶

Wetopi control plane¶

Owns:

site security intent
profile assignment
WAF mode
egress pool assignment
quarantine request
customer/site metadata

GitOps/platform repository¶

Owns:

security profile definitions
Cilium baseline policies
gateway policies
Traefik middleware templates
Coraza base configuration
Suricata configuration
threat feed sources
Tetragon tracing policies

Security controllers/operators¶

Own:

translation/reconciliation
runtime state
health
telemetry
temporary dynamic responses

35. Namespace and label requirements¶

Every wepod namespace/workload must expose consistent metadata.

Minimum labels:

wetopi.io/site-id
wetopi.io/customer-id
wetopi.io/workload
wetopi.io/network-profile
wetopi.io/egress-pool
wetopi.io/waf-mode

Recommended:

wetopi.io/environment
wetopi.io/plan
wetopi.io/region
wetopi.io/platform-generation

These labels become security-critical.

Mutation must be restricted to trusted platform identities.

36. NAT-GW design¶

Wetopi already uses distinct NAT gateway ranges/public egress IPs.

Preserve this concept in Kubernetes.

Each egress pool should expose:

pool_id
gateway nodes
public source IP(s)
customer/site selector
health
capacity
Suricata sensor identity

Example:

natgw-a
  public_ip = 203.0.113.10

natgw-b
  public_ip = 203.0.113.11

natgw-infra
  public_ip = 203.0.113.20

Avoid mixing trusted platform infrastructure and untrusted customer WordPress traffic behind the same public IP where practical.

37. Egress reputation isolation¶

Recommended categories:

customer-general
customer-high-risk
platform-trusted
mail-relay
internal-agents

Do not necessarily dedicate an IP per customer.

The goal is to limit blast radius.

A customer workload compromise should not poison the reputation of:

Wetopi APIs
platform automation
email infrastructure
critical outbound integrations

38. Network path requirements¶

Inbound¶

Internet
→ L4 edge
→ threat-feed nftables
→ Traefik
→ optional WAF
→ Cilium ingress policy
→ wepod

Outbound¶

wepod
→ Cilium egress policy
→ egress gateway selection
→ NAT-GW
→ Suricata
→ nftables policy
→ Internet

East-west¶

wepod
→ Cilium policy
→ DENY unless explicitly authorized

39. Observability architecture¶

Metrics — Prometheus¶

Collect:

Cilium
Hubble
Traefik
Coraza if available
Suricata exporters
firewall-list-controller
edge-fw-controller
NAT-GW
Tetragon health

Logs/events — OpenSearch¶

Recommended streams/index families:

security-suricata-*
security-hubble-*
security-tetragon-*
security-waf-*
security-edgefw-*
access-traefik-*

Do not blindly duplicate all raw Kubernetes logs into security indices.

40. Required dashboards¶

Initial dashboards:

`Wetopi Security Overview`¶

blocked traffic at edge;
Suricata alert count by severity;
outbound suspicious traffic by site;
top sites by denied egress;
direct SMTP attempts;
WAF detect/block events;
quarantined sites.

`Site Security`¶

Filter by site_id.

Show:

ingress request rate;
WAF events;
Hubble egress destinations;
denied network flows;
Suricata alerts;
Tetragon runtime detections;
assigned egress IP;
active security profile.

`Edge Firewall`¶

feed age;
entries per source;
active revision per node;
blocked packets;
update failures;
node drift.

41. Alerting¶

Initial high-value alerts:

threat feed older than threshold
edge node failed to update ruleset
edge nodes have different revisions
Cilium agent unhealthy
Hubble unavailable
NAT-GW unavailable
Suricata sensor down
high-severity Suricata C2 alert
unexpected SMTP attempt
large outbound scan from one site
Tetragon detects PHP → shell
WAF block surge

Do not create dozens of low-signal alerts initially.

42. Failure-mode design¶

Threat feed unavailable¶

continue with last-known-good
alert

Edge-fw-agent failure¶

existing nftables rules remain active
alert

Suricata failure¶

Traffic SHOULD continue unless Wetopi later explicitly chooses inline IPS behaviour.

Initial deployment is IDS-oriented.

WAF failure¶

Decision required during implementation:

Preferred initial behaviour:

fail open for WAF component

while preserving:

Traefik routing
Cilium policy
edge firewall

Rationale: WAF outage must not cause platform-wide outage.

This must be tested.

Hubble failure¶

Networking continues.

Observability degrades.

Tetragon failure¶

Networking continues.

Runtime visibility degrades.

NAT-GW failure¶

Requires high availability or fast re-selection.

This is traffic-path critical and deserves explicit HA design.

43. Why Suricata is not enough¶

Suricata sees network traffic but lacks Kubernetes-native tenant/process identity.

Example:

10.42.4.17 → malicious.example

is insufficient on its own.

Wetopi needs:

site 9283
wepod xyz
php-fpm
→ malicious.example

Therefore:

Suricata + Hubble + Tetragon

are complementary.

44. Why NetworkPolicy is not enough¶

Network policy answers:

is this connection allowed?

It does not necessarily answer:

is this HTTPS connection malicious?
did PHP spawn a reverse shell?
is the destination a known C2?
is this WordPress receiving SQL injection attempts?

Hence multiple layers remain necessary.

45. Why the giant threat list should not live primarily inside workload policy¶

Large fast-changing malicious-IP lists are best rejected at edge nodes because:

the decision only depends on IP;
rejection should occur before L7/Kubernetes work;
updating a shared nft set is operationally cheap;
it avoids exploding workload policy definitions;
it preserves a proven Wetopi control pattern.

Cilium lists can still be used for semantically meaningful Kubernetes policy groups.

46. GitOps repository proposal¶

Possible layout:

platform-security/
├── README.md
├── adr/
│   ├── ADR-001-cilium-security-plane.md
│   ├── ADR-002-edge-threatfeeds.md
│   ├── ADR-003-egress-gateway.md
│   ├── ADR-004-suricata-placement.md
│   └── ADR-005-waf.md
│
├── cilium/
│   ├── values/
│   ├── policies/
│   │   ├── baseline/
│   │   ├── wordpress-standard/
│   │   ├── wordpress-restricted/
│   │   └── quarantine/
│   └── egress-gateway/
│
├── threatfeeds/
│   ├── controller/
│   ├── agent/
│   ├── sources.yaml
│   └── policies/
│
├── suricata/
│   ├── ingress/
│   ├── egress/
│   └── rules/
│
├── tetragon/
│   └── tracing-policies/
│
├── waf/
│   ├── coraza/
│   ├── crs/
│   ├── wordpress/
│   └── exclusions/
│
├── observability/
│   ├── prometheus/
│   ├── dashboards/
│   ├── alerts/
│   └── opensearch/
│
└── docs/
    ├── runbooks/
    ├── profiles/
    └── architecture/

If Wetopi already has a preferred platform repo structure, integrate these directories instead of creating another repository without reason.

47. Deployment phases¶

The deployment MUST be incremental.

Phase 0 — Inventory and baseline¶

Current platform constraint: no wepods exist yet¶

At the time of this ADR update, there is not yet any wepod running on the new RKE2 platform.

The implementation agent MUST NOT wait for the final production wepod before validating the network/security path.

For E2E testing, deploy a deliberately disposable WordPress Docker test workload that approximates the relevant network behaviour:

temporary WordPress Docker hello-world
WordPress + web server + PHP
database only as required for the test
HTTP service exposed internally
Wetopi site/security labels attached

The goal is not to define the future production wepod architecture.

The goal is to obtain a real WordPress endpoint to validate:

Traefik ingress
TLS termination
Cilium ingress policy
DNS
HTTP/HTTPS egress
Cilium Egress Gateway / NAT-GW
Hubble attribution
GeoIP policy when implemented
WAF later
Suricata/Tetragon correlation later

This workload must be clearly disposable and must not become the accidental production wepod design.

Before changing enforcement, document:

current WAN nodes
current NAT-GW nodes
current public IP pools
current threat-feed sources
current ipset rule semantics
current Suricata placement
current Suricata custom rules
current WordPress outbound traffic patterns
current SMTP behaviour
current ingress routes
existing CNI status/configuration
existing Traefik deployment

Deliverable:

security-current-state.md

Acceptance:

no unknown critical legacy rule is discarded;
current egress IP mapping is documented;
current blocklist semantics are understood.

Phase 1 — Cilium foundation¶

Deploy/configure:

Cilium
Hubble
baseline observability

Initially avoid disruptive default-deny policies.

Validate:

node connectivity;
pod connectivity;
service routing;
Traefik ingress;
DNS;
Hubble attribution.

Deliverable:

cilium-platform-baseline

Phase 2 — NetworkPolicy in observe/test namespaces¶

Create:

wordpress-standard

for non-production/test workloads first.

Apply:

default deny ingress
allow Traefik
explicit DNS
controlled normal web egress

Validate common WordPress actions:

plugin updates
theme updates
wp-cron
WooCommerce callbacks
REST
external APIs
webhooks
image fetching
license checks

Do not attempt exhaustive plugin compatibility testing across the ecosystem.

Focus on core platform paths.

Phase 3 — Egress gateway¶

Deploy/configure:

Cilium Egress Gateway
Wetopi NAT-GW nodes
egress pools

Verify:

pod migration does not change public egress IP
site→pool mapping is deterministic
NAT-GW failure behaviour is acceptable
Hubble attribution remains available

Phase 4 — SMTP containment¶

Deploy blocking policy.

Initial action:

detect/log direct SMTP attempts

Then:

deny direct SMTP

Ensure Wetopi-approved relay path exists before applying to sites that rely on mail.

This phase requires explicit customer-compatibility consideration.

Phase 5 — Threatfeed controller + edge-fw-controller¶

Build and deploy:

wetopi-firewall-list-controller
wetopi-edge-fw-controller

Start on:

test edge node

Then:

one production edge node

Then:

all WAN/NAT-GW nodes

Compare drop counters with legacy system before retiring old ipset mechanism.

Phase 6 — Suricata chokepoints¶

Deploy or migrate Suricata to:

WAN
NAT-GW

Enable EVE export.

Integrate into OpenSearch.

Only after observability is confirmed should Suricata on legacy worker nodes be removed.

Phase 7 — Tetragon¶

Deploy runtime observability.

Start with high-confidence tracing policies:

php-fpm → shell
nginx → shell
unexpected connect patterns
execution from WordPress uploads

Initially:

observe only

Phase 8 — WAF¶

Deploy:

Coraza
OWASP CRS
Traefik integration

Start with one internal/test WordPress in:

detect

Build minimal WordPress compatibility exclusions.

Then offer:

off/detect/block

per site.

Phase 9 — Security correlation¶

Normalize identifiers.

Build enough pipeline logic to answer:

Suricata IP → Kubernetes pod → site_id

Then optionally:

pod → Tetragon process

A full custom security controller is not required yet.

Phase 10 — Quarantine profile¶

Implement one operationally safe action:

set site → quarantine

which atomically/reliably applies the restricted network posture.

Document recovery procedure.

48. Initial implementation backlog¶

The following is intentionally task-oriented so an implementation agent can decompose it further.

EPIC SEC-01 — Current-state inventory¶

SEC-01.01¶

Inventory existing threat-feed sources and refresh intervals.

SEC-01.02¶

Export current iptables/ipset rules from all WAN and NAT-GW roles.

SEC-01.03¶

Document current allowlist precedence.

SEC-01.04¶

Inventory NAT-GW public IPs and customer assignment rules.

SEC-01.05¶

Inventory Suricata configuration/custom rules.

SEC-01.06¶

Document current outbound SMTP usage.

SEC-01.07¶

Confirm current RKE2 CNI configuration and compatibility requirements.

EPIC SEC-02 — Cilium/Hubble foundation¶

SEC-02.01¶

Define pinned Cilium version.

SEC-02.02¶

Create GitOps Helm values.

SEC-02.03¶

Enable Hubble.

SEC-02.04¶

Define Hubble retention/export strategy.

SEC-02.05¶

Create health checks and Prometheus scraping.

SEC-02.06¶

Create a basic Cilium/Hubble Grafana dashboard.

SEC-02.07¶

Document rollback procedure.

Acceptance:

all nodes healthy
Traefik works
DNS works
wepods communicate only as expected before enforcement
Hubble resolves pod/namespace identity

EPIC SEC-03 — Baseline workload policy¶

SEC-03.01¶

Define required Wetopi security labels.

SEC-03.02¶

Implement label assignment in wepod deployment pipeline.

SEC-03.03¶

Create default-deny ingress policy.

SEC-03.04¶

Allow Traefik → wepod.

SEC-03.05¶

Allow approved monitoring/management paths.

SEC-03.06¶

Deny cross-tenant traffic.

SEC-03.07¶

Create test workload validating denied lateral movement.

EPIC SEC-04 — Egress policy¶

SEC-04.01¶

Create wordpress-standard.

SEC-04.02¶

Restrict DNS to CoreDNS.

SEC-04.03¶

Allow HTTP/HTTPS.

SEC-04.04¶

Explicitly deny/prove blocked access to sensitive internal networks.

SEC-04.05¶

Create wordpress-restricted.

SEC-04.06¶

Create quarantine.

SEC-04.07¶

Document exception/custom policy workflow.

EPIC SEC-05 — Controlled NAT egress¶

SEC-05.01¶

Define NAT-GW Kubernetes node role.

SEC-05.02¶

Define egress pools.

SEC-05.03¶

Deploy Cilium Egress Gateway configuration.

SEC-05.04¶

Map wepod label → egress pool.

SEC-05.05¶

Verify public source IP stability.

SEC-05.06¶

Test pod rescheduling.

SEC-05.07¶

Test NAT-GW failure.

SEC-05.08¶

Add egress pool observability.

EPIC SEC-06 — SMTP protection¶

SEC-06.01¶

Measure existing outbound SMTP traffic.

SEC-06.02¶

Identify legitimate customer requirements.

SEC-06.03¶

Define Wetopi mail relay architecture or approved external relay policy.

SEC-06.04¶

Add SMTP-attempt observability.

SEC-06.05¶

Block TCP/25 direct.

SEC-06.06¶

Block/limit TCP/465 and TCP/587 according to approved policy.

SEC-06.07¶

Create alert on blocked SMTP attempts.

EPIC SEC-07 — Central firewall-list compiler¶

SEC-07.01¶

Define source/provider configuration schema.

SEC-07.02¶

Implement HTTP/file source acquisition.

SEC-07.03¶

Implement IPv4/IPv6/CIDR/range parsing.

SEC-07.04¶

Implement validation and broken/empty-feed safeguards.

SEC-07.05¶

Implement normalization and deduplication.

SEC-07.06¶

Implement safe CIDR/range aggregation.

SEC-07.07¶

Implement source provenance tracking.

SEC-07.08¶

Implement hard-whitelist subtraction and precedence.

SEC-07.09¶

Implement hard blacklist, attack blacklist and temporary blacklist inputs.

SEC-07.10¶

Implement structured custom firewall-rule input.

SEC-07.11¶

Implement immutable revision generation.

SEC-07.12¶

Implement durable artifact publication.

SEC-07.13¶

Implement desired/current revision metadata.

SEC-07.14¶

Implement leader election / single-writer compilation.

SEC-07.15¶

Implement revision notification stream/watch.

SEC-07.16¶

Implement metrics and health.

SEC-07.17¶

Deploy as a multi-replica Deployment via Argo CD.

EPIC SEC-08 — GeoIP + edge firewall controller¶

SEC-08.01¶

Choose and document approved GeoIP source datasets.

SEC-08.02¶

Implement GeoIP IP/range parser.

SEC-08.03¶

Generate IPv4/IPv6 country sets.

SEC-08.04¶

Generate continent sets.

SEC-08.05¶

Define country/continent identifiers and dataset metadata.

SEC-08.06¶

Publish nftables-compatible GeoIP artifacts for global policy.

SEC-08.07¶

Publish Traefik-consumable GeoIP artifacts for per-site policy.

SEC-08.08¶

Define global/site GeoIP policy schema and precedence.

SEC-08.09¶

Define edge node labels/firewall profiles.

SEC-08.10¶

Implement wetopi-edge-fw-controller DaemonSet.

SEC-08.11¶

Implement central revision watch/notification.

SEC-08.12¶

Implement fallback periodic reconciliation.

SEC-08.13¶

Implement artifact checksum/signature verification.

SEC-08.14¶

Implement atomic nftables/nft-set replacement.

SEC-08.15¶

Implement last-known-good failure behaviour.

SEC-08.16¶

Implement desired/active revision reporting.

SEC-08.17¶

Implement metrics and drift alerting.

SEC-08.18¶

Deploy to one test edge node.

SEC-08.19¶

Compare against the legacy iptables/ipset behaviour.

SEC-08.20¶

Progressively deploy to WAN and NAT-GW nodes.

EPIC SEC-09 — Suricata¶

SEC-09.01¶

Define exact packet capture point on WAN nodes.

SEC-09.02¶

Define exact packet capture point on NAT-GW nodes.

SEC-09.03¶

Port existing high-value rules.

SEC-09.04¶

Configure EVE JSON.

SEC-09.05¶

Create OpenSearch ingestion.

SEC-09.06¶

Create Suricata alert dashboard.

SEC-09.07¶

Correlate source pod IP to Kubernetes metadata.

SEC-09.08¶

Remove redundant worker sensors only after equivalent visibility is demonstrated.

EPIC SEC-10 — Tetragon¶

SEC-10.01¶

Deploy Tetragon.

SEC-10.02¶

Create PHP-FPM shell-spawn detection.

SEC-10.03¶

Create nginx shell-spawn detection.

SEC-10.04¶

Observe execution from writable WordPress directories.

SEC-10.05¶

Create OpenSearch ingestion.

SEC-10.06¶

Correlate Tetragon events with site_id.

EPIC SEC-11 — WAF¶

SEC-11.01¶

Choose exact Coraza/Traefik integration method/version.

SEC-11.02¶

Deploy Coraza.

SEC-11.03¶

Deploy OWASP CRS.

SEC-11.04¶

Create WordPress baseline configuration.

SEC-11.05¶

Implement off.

SEC-11.06¶

Implement detect.

SEC-11.07¶

Implement block.

SEC-11.08¶

Map wetopi.io/waf-mode to routing middleware.

SEC-11.09¶

Build WAF event ingestion.

SEC-11.10¶

Document exclusion workflow.

SEC-11.11¶

Test WooCommerce/webhooks/admin endpoints.

EPIC SEC-12 — TLS/internal encryption¶

SEC-12.01¶

Keep TLS termination at Traefik.

SEC-12.02¶

Document trusted internal HTTP path.

SEC-12.03¶

Evaluate/enable Cilium WireGuard if required.

SEC-12.04¶

Measure performance overhead.

SEC-12.05¶

Document key rotation/operational implications.

EPIC SEC-13 — Security observability¶

SEC-13.01¶

Define canonical security event fields.

SEC-13.02¶

Ensure every source includes/can derive site_id.

SEC-13.03¶

Create OpenSearch index templates.

SEC-13.04¶

Define retention tiers.

SEC-13.05¶

Create Security Overview dashboard.

SEC-13.06¶

Create Site Security dashboard.

SEC-13.07¶

Create Edge Firewall dashboard.

SEC-13.08¶

Create high-signal alerts only.

EPIC SEC-14 — Quarantine¶

SEC-14.01¶

Define quarantine policy.

SEC-14.02¶

Define permitted forensic/management traffic.

SEC-14.03¶

Implement control-plane switch.

SEC-14.04¶

Implement audit event.

SEC-14.05¶

Create runbook.

SEC-14.06¶

Test recovery from quarantine.

49. Recommended first sprint¶

Do not begin every epic simultaneously.

Recommended first implementation slice:

1. Inventory current security behaviour
2. Confirm/pin Cilium architecture
3. Enable Hubble
4. Standardize site/security labels
5. Deploy one disposable WordPress Docker hello-world workload because no real wepod exists yet
6. Route it through Traefik
7. Apply ingress default-deny to that test workload
8. Apply controlled DNS + HTTP/HTTPS egress
9. Build first Cilium Egress Gateway path
10. Verify site → NAT-GW → public-IP attribution

This establishes the foundation on which the remaining security controls depend.

The next slice:

9. Build nftables threatfeed replacement
10. Move/validate Suricata on NAT-GW/WAN
11. Block direct SMTP
12. Deploy Tetragon high-signal detection
13. Add Coraza in detect mode
14. Add quarantine profile

50. Acceptance criteria for the architecture¶

The initial security architecture is considered operational when all the following are true.

Network isolation¶

[ ] A random customer wepod cannot contact another customer wepod.
[ ] A random customer wepod cannot contact the Kubernetes API.
[ ] A random customer wepod cannot contact Ceph/management networks.
[ ] Only approved ingress identities can reach the wepod HTTP endpoint.

Egress¶

[ ] Every customer wepod exits through an explicitly selected egress pool.
[ ] Rescheduling a pod does not accidentally change its intended egress pool.
[ ] Direct SMTP is blocked according to policy.
[ ] DNS is restricted to approved resolvers.
[ ] site_id → egress public IP is observable.

Threat feeds¶

[ ] Every WAN/NAT-GW node reports its active feed revision.
[ ] Failed feed downloads preserve the last known-good configuration.
[ ] A test blocked IP is dropped.
[ ] An allowlisted test IP is not accidentally blocked.
[ ] No node silently runs a stale/empty ruleset.

IDS/runtime¶

[ ] A Suricata test alert can be traced back to a site.
[ ] A high-signal Tetragon test event can be traced back to a site.
[ ] Sensor failure produces an alert.

WAF¶

[ ] off adds no WAF enforcement.
[ ] detect logs but does not block.
[ ] block rejects a known CRS test case.
[ ] Site-specific exclusions can be applied without editing global rules.

TLS¶

[ ] Public traffic remains HTTPS.
[ ] Certificate lifecycle stays centralized at Traefik.
[ ] If WireGuard is enabled, inter-node workload traffic is verified as encrypted.

51. Security event schema¶

Create a canonical event envelope so future controllers/agents do not need source-specific logic everywhere.

Example:

{
  "@timestamp": "2026-09-09T11:00:00Z",
  "source_type": "suricata",
  "event_type": "known-c2",
  "severity": "critical",

  "site_id": "12345",
  "customer_id": "567",

  "kubernetes": {
    "namespace": "wp-12345",
    "pod": "wepod-12345-7cf88",
    "node": "worker-12"
  },

  "network": {
    "src_ip": "10.42.17.83",
    "src_port": 45120,
    "dst_ip": "198.51.100.66",
    "dst_port": 443,
    "transport": "tcp"
  },

  "egress": {
    "pool": "natgw-a",
    "public_ip": "203.0.113.10"
  },

  "detection": {
    "rule_id": "SURICATA-XXXX",
    "message": "Known C2"
  }
}

Do not force every source to emit this natively.

A normalization pipeline may enrich events.

52. Auditability¶

Every security-state change should create an audit event.

Examples:

site WAF detect → block
site network standard → restricted
site → quarantine
manual IP block added
allowlist changed
threatfeed source changed
egress pool changed

Audit event fields:

timestamp
actor
source
site_id if applicable
old_state
new_state
reason
change/reference ID

Actors can be:

human
control-plane
GitOps
automation
AI agent

53. Agent operating rules¶

An implementation agent working from this document MUST:

Prefer GitOps-managed resources.
Avoid manual production-only configuration.
Keep changes small enough to roll back.
Preserve existing service availability during security migration.
Use test workloads before broad enforcement.
Report discovered incompatibilities rather than weakening global policy silently.
Never introduce broad allow all rules as permanent fixes.
Document every exception.
Avoid adding a new platform component when an existing selected component already solves the problem.
Treat Cilium/NAT-GW changes as traffic-path critical.
Prefer high-signal security telemetry over maximal telemetry.
Do not build the future Security Controller before the basic data plane is stable.

54. Decisions explicitly deferred¶

The following require implementation-time validation or future ADRs:

D-01 — Exact Cilium version/features¶

Pin after compatibility review with the exact RKE2 version.

D-02 — WAF integration mechanism¶

Confirm the supported Coraza integration mode for the selected Traefik release.

D-03 — Suricata packet acquisition mechanism¶

Decide according to physical/virtual NAT-GW topology:

AF_PACKET
NFQUEUE
mirror interface
other supported capture path

Initial preference is passive IDS rather than inline IPS.

D-04 — Security controller implementation¶

Do not build yet.

D-05 — CRD vs control-plane generated resources¶

Decide only when the control-plane integration is implemented.

D-06 — WireGuard enablement¶

Recommended if inter-node plaintext is considered a meaningful threat in the final physical design, but benchmark first.

D-07 — SMTP relay product¶

Needs a separate design if Wetopi does not already have a suitable outbound mail path.

55. Consequences¶

Positive¶

much stronger tenant isolation;
stable and auditable egress;
greatly reduced chance of outbound spam abuse;
better public-IP reputation protection;
better attribution of incidents;
less need for Suricata on every worker;
WAF can be introduced without forcing all WordPress sites through it;
legacy threat-feed investment is preserved;
security controls become GitOps-friendly;
future AI/SRE automation receives structured evidence.

Negative¶

Cilium becomes critical infrastructure;
egress gateways become part of the availability path;
policy mistakes can break WordPress integrations;
WAF configuration needs ongoing WordPress-specific tuning;
security telemetry volume can become expensive;
multi-layer event correlation requires a normalized identity model.

56. Rejected alternatives¶

Put every hostile IP into Kubernetes NetworkPolicy¶

Rejected as the primary mechanism.

Reason:

wrong abstraction for giant rapidly changing perimeter blocklists;
unnecessary workload-policy churn;
rejection occurs later in the path.

Keep unrestricted worker-node egress¶

Rejected.

Reason:

weak attribution;
poor containment;
public-IP reputation blast radius;
pod placement affects traffic path.

Install Suricata on every worker¶

Not the preferred target architecture.

Reason:

duplicates sensors;
operational cost;
Kubernetes-aware telemetry can provide attribution more effectively;
NAT-GW provides a natural outbound chokepoint.

May be reconsidered if real blind spots are demonstrated.

Force HTTPS between Traefik and every wepod¶

Rejected for now.

Reason:

certificate lifecycle complexity;
limited security benefit if node transport is encrypted;
application-layer HTTP is acceptable inside an appropriately protected cluster.

Force WAF blocking globally¶

Rejected.

Reason:

WordPress plugin/ecosystem false positives;
unsafe migration;
need for detect mode and per-site policy.

Service mesh for mTLS¶

Rejected for this requirement alone.

Reason:

excessive operational complexity compared with transparent node encryption.

57. Definition of done for the migration security layer¶

The security migration is not complete merely because components are installed.

It is complete when Wetopi can answer and enforce all of the following:

Which site is this packet from?
Which public IP will this site use?
Can this site talk to another customer?
Can this site talk to the control plane?
Can this site send SMTP directly?
Can we restrict this site without moving it?
Can we quarantine this site?
Can we enable WAF only for this site?
Can we see what the WAF would block before enforcing it?
Can we attribute an IDS event to a site?
Can we attribute selected suspicious behaviour to PHP/nginx?
Are every WAN and NAT-GW firewall ruleset synchronized?
Can a broken threat feed update leave the platform unprotected?
Can a failed WAF take down all customer traffic?

Required answers:

site attribution              → yes
stable egress identity        → yes
cross-tenant arbitrary access → no
control-plane arbitrary access→ no
direct SMTP                   → no
site-level restriction        → yes
quarantine                    → yes
per-site WAF                  → yes
WAF detect mode               → yes
IDS→site correlation          → yes
runtime attribution           → yes for selected detections
firewall sync visibility      → yes
empty-feed failure mode       → no
WAF global SPOF               → no

58. Recommended architectural outcome¶

The target Wetopi platform should converge on this separation:

Firewall List Controller
    = centralized feed/ACL/GeoIP curation and immutable firewall revisions

Edge FW Controller
    = node-local reconciliation of centrally compiled revisions

nftables
    = known-bad sources, hard allow/deny, attack blocks and global GeoIP rejection

Cilium
    = who may talk to whom

Egress Gateway
    = through which public identity traffic may leave

Hubble
    = who talked to whom

Tetragon
    = which process did it

Suricata
    = whether network behaviour looks malicious

Traefik
    = application ingress / TLS termination / routing

Coraza
    = whether the HTTP request should be blocked

OpenSearch
    = investigation and security-event history

Wetopi control plane
    = desired security state per site

This separation is deliberate.

No single component is expected to become the firewall, IDS, WAF, runtime sensor and policy engine simultaneously.

59. Immediate instruction to the implementation agent¶

Use this document as the parent architecture.

First:

inspect the current RKE2/Cilium/Traefik state;
compare it against this desired state;
produce a short gap analysis;
turn Phase 0 through Phase 3 into concrete implementation tasks;
identify any blockers that would invalidate the Cilium Egress Gateway design;
avoid deploying WAF/Tetragon/Security Controller before the basic network and egress foundation is stable;
keep Ceph and unrelated platform services untouched;
make all new platform configuration GitOps-managed unless technically impossible;
preserve existing Wetopi ingress availability throughout the migration.

The first milestone is:

One disposable WordPress Docker hello-world workload, standing in for the future wepod, with explicit Wetopi identity, default-deny isolation, allowed Traefik ingress, controlled DNS/HTTP(S) egress, and deterministic NAT-GW/public-IP egress, fully observable through Hubble.

There are currently no wepods running on the new platform, so the implementation agent is explicitly authorized to deploy this temporary WordPress test workload to validate the complete E2E path.

Do not wait for the production wepod design and do not broaden the first milestone.

Once that works reliably, proceed to threat feeds, Suricata, SMTP enforcement, Tetragon and WAF.

60. Final architecture summary¶

                               INTERNET
                                  │
                                  ▼
                   ┌────────────────────────┐
                   │  WAN / IPVS / VRRP     │
                   │  nftables threat sets  │
                   │  Suricata              │
                   └────────────┬───────────┘
                                │
                                ▼
                   ┌────────────────────────┐
                   │ Traefik                │
                   │ TLS termination        │
                   │ rate limiting          │
                   │ Coraza optional        │
                   └────────────┬───────────┘
                                │
                     Cilium ingress policy
                                │
                                ▼
                   ┌────────────────────────┐
                   │       WEPOD            │
                   │ nginx/php/mariadb      │
                   │ site/customer labels   │
                   └────────────┬───────────┘
                                │
                     Cilium egress policy
                                │
                                ▼
                     Cilium Egress Gateway
                                │
                                ▼
                   ┌────────────────────────┐
                   │ NAT-GW                 │
                   │ fixed public egress IP │
                   │ nftables               │
                   │ Suricata               │
                   └────────────┬───────────┘
                                │
                                ▼
                             INTERNET


Hubble    → network attribution
Tetragon  → process/runtime attribution
Suricata  → IDS/network detection
Coraza    → WAF
OpenSearch→ investigation/history
Prometheus→ health/metrics
GitOps    → desired platform state

End of ADR / Architectural Pack.

ADR + Architectural Pack — Wetopi Network & Workload Security Architecture¶

1. Executive decision¶

2. Context¶

3. Problem statement¶

4. Goals¶

4.1 Primary goals¶

4.2 Secondary goals¶

5. Non-goals¶

6. Architectural principles¶

6.1 Security by chokepoint¶

6.2 Default deny, explicit allow¶

6.3 Platform policy, not customer policy¶

6.4 Stable identity everywhere¶

7. High-level architecture¶

8. ADR-001 — Use Cilium as the workload network security enforcement layer¶

Status¶

Decision¶

Rationale¶

Consequence¶

9. ADR-002 — Central Firewall List Compiler + Node-local Edge Firewall Controller¶

Status¶

Decision¶

10. Firewall List / GeoIP Distribution Plane¶

10.1 Desired architecture¶

10.2 Source of truth¶

A. Desired runtime firewall state¶

B. Compiler configuration¶

C. Materialized runtime revision¶

10.3 Central component — wetopi-firewall-list-controller¶

Deployment model¶

10.4 Central compiler responsibilities¶

Source acquisition¶

Parsing¶

Validation¶

Normalization¶

Deduplication¶

CIDR/range aggregation¶

Allowlist subtraction¶

Compilation¶

Publication¶

Notification¶

10.5 Notification model¶

10.6 Artifact repository¶

10.7 Firewall profiles¶

10.8 Node-local component — wetopi-edge-fw-controller¶

Deployment¶

Responsibilities¶

10.9 Firewall data classes¶

10.10 High-volume attack blocking¶

10.11 Custom allow/deny rules¶

10.12 Sets versus individual rules¶

10.13 Precedence¶

10.14 GeoIP list generation¶

10.15 GeoIP consumers¶

Global/system-wide country blocking¶

Per-site/customer GeoIP policy¶

10.16 GeoIP policy model¶

10.17 GeoIP update semantics¶

10.18 Atomicity and failure behaviour¶

10.19 Drift detection¶

10.20 Metrics¶

10.21 Initial implementation boundary¶

11. ADR-003 — Wepods use platform-defined security profiles¶

Decision¶

12. Ingress policy¶

12.1 Baseline¶

12.2 Internal management traffic¶

13. East-west isolation¶

Decision¶

14. ADR-004 — Internet egress is controlled through Wetopi Egress Gateways¶

Decision¶

Benefits¶

15. Egress profiles¶

wordpress-standard¶

wordpress-restricted¶

wordpress-mail-relay¶

quarantine¶

custom¶

16. SMTP policy¶

Decision¶

10.3 Central component — `wetopi-firewall-list-controller`¶

10.8 Node-local component — `wetopi-edge-fw-controller`¶

`wordpress-standard`¶

`wordpress-restricted`¶

`wordpress-mail-relay`¶

`quarantine`¶

`custom`¶

`off`¶

`detect`¶

`block`¶

`Wetopi Security Overview`¶

`Site Security`¶

`Edge Firewall`¶