Platform Architecture
Ozone runs as two independent tiers joined by exactly one narrow, outbound channel: the multi-tenant Ozone Orchestrator (the SaaS control plane at ozone.techforcz.com) in the cloud and a fleet of single-tenant Ozone Data Collectors inside customer networks. This page gives a high-level model of how those tiers are wired, what crosses the boundary between them, and where telemetry lives along the way.
The service groups and data flows described here reflect the current platform build.
1. High-level topology
The platform is two separate stacks that never share a database. The collector runs a light, local copy of each capability so a customer branch stays fully observable offline, and it does its own telemetry ingestion and its own notification delivery on-prem. The SaaS runs the heavy, fleet-wide copy across many tenants.
The one rule that explains the whole diagram: there is a single rope across the canyon, saas-sync (collector) to the api-gateway ingest path (SaaS). Everything else is internal to one side. Notifications and tickets are delivered by the collector directly from on-prem, so they keep working even when the WAN is down. No SaaS service ever opens a connection into a customer network, the collector initiates every byte that crosses the boundary. The security model behind that rope is detailed in Security & Compliance.
2. Why two tiers
| Driver | What the split buys you |
|---|---|
| Data minimisation | Device credentials (SNMP communities, SSH keys, iLO/iDRAC passwords) and the raw telemetry firehose never leave the premises. The collector forwards a curated subset; lower licence tiers forward only critical (P1/P2) problems, not the raw stream. |
| Resilience | The collector owns local databases and a local message bus, so it keeps polling, correlating, and alerting with the WAN link down. The bus replays the backlog on reconnect: zero loss, no inbound ports. |
| Scale by fleet | The SaaS scales by managing many independent collectors through a registry, heartbeats, and per-tenant isolation, not by holding thousands of direct device sessions itself. |
The collector is autonomous. Losing the SaaS degrades only registration state, central config push, fleet visibility, and upgrades, never local collection or alerting. That single fact closes a large class of customer-reported "outages."
3. The Ozone Orchestrator (SaaS control plane)
The Ozone Orchestrator (ozone.techforcz.com) is a multi-tenant stack of FastAPI microservices fronted by an API gateway, backed by a managed PostgreSQL database as the relational system of record and Keycloak for identity. All cloud telemetry stores are scoped by tenant. The whole stack sits on a cloud VM behind an outbound tunnel with no inbound ports; collectors dial out to it over HTTPS.
3.1 Service responsibilities (control plane)
The orchestrator runs a set of focused microservices. The table below describes each at a functional level.
| Service | Responsibility |
|---|---|
api-gateway | Single front door: routing, user-JWT validation (except public paths), per-user rate limiting, the public collector ingest receiver with per-tenant ingest rate limiting and fan-out to tenant-scoped subjects, the billing router, and SPA serving |
auth-service | Authentication, JWT issuance, Keycloak integration, ingest-token resolution, edge JWKS |
tenant-service | Tenant lifecycle and isolation metadata |
collector-registry | Collector registration, one-time token to long-lived JWT issuance, heartbeats, and the authoritative fleet-wide collector state and lifecycle |
correlation-engine | Per-tenant causal graphs to correlated problems |
ai-engine | Anomaly detection, capacity forecasting, graph-based RCA |
alerting-engine | Dedup, suppression, escalation, routing |
licensing-service | Licence tiers and signing; caps on collectors, devices, and retention |
billing-service | Usage metering and checkout |
inventory-service | Cloud-side device inventory |
notification-service | Internal dispatcher for alert and escalation events. Customer-facing operational integrations (Slack, Teams, PagerDuty, ServiceNow, Jira, webhook) are configured and delivered on-prem by the collector, see Integrations |
audit-service | Immutable audit trail |
ai-triage | LLM triage brain: multi-stage agent, read-only tools, budget-capped model gateway, retrieval-augmented context |
frontend | React customer portal |
Same name, two tiers.
correlation-engineandalerting-engineexist on both tiers on purpose: a light local build on the collector, a heavy fleet-wide build in the cloud. They are different codebases doing the same job at different scales. Network topology, SLO/error-budget tracking, and capacity forecasting are delivered at the collector tier rather than as separate cloud services.
4. The Ozone Data Collector
A self-contained virtual appliance (Ubuntu, Docker Compose, shipped as an OVA). The appliance is single-tenant by design and runs its application services locally. It has no PostgreSQL and no Keycloak: a deliberate design thesis of minimal external dependencies and a single outbound egress path. All local state is kept in embedded databases.
4.1 Service responsibilities (edge)
| Service | Responsibility |
|---|---|
universal-collector | Multi-protocol onboarding and polling across many device categories, including scheduled SNMP polling (there is no separate snmp-poller service) |
auto-discovery | Subnet scanning and device fingerprinting |
ebpf-collector | Kernel-level host metrics |
synthetic-prober | Active ICMP/TCP/HTTP/DNS synthetic checks |
inventory-engine (+passive) | Device inventory of record; HA active/standby pair |
ha-sync | Heartbeat and failover via a cache-backed leader key |
correlation-engine | Local event correlation into problems |
problem-engine | Groups related alerts into problem cards; hosts the auto-remediation rule engine |
alerting-engine | Local alert rules, firing, ack/silence |
slo-engine | Local SLO / error-budget tracking |
capacity-forecast | Local forecasting (relays to the SaaS forecaster on Enterprise) |
security-posture | NVD CVE sync, risk scoring, firmware EOL |
business-impact | Maps technical faults to business services |
unified-search | Cross-source local search (powers the command palette) |
netshot-sync | Bridge to Netshot for config backup and compliance; mirrors config-change events onto the local bus |
webui | Local appliance UI (built React served through a same-origin reverse proxy) |
saas-sync | The one bridge to the cloud: registration, heartbeat, event forwarding |
Local infrastructure containers provide the message bus, cache, time-series store, and log/event store.
netshot-sync authenticates to Netshot over an outbound-only connection and polls on a fixed interval. It powers the WebUI "Backup & Compliance" tab and mirrors Netshot config-change events onto the local bus, so the correlation engine can fold "config changed 12 minutes before the alert storm" into a problem statement.
Installation, host footprint, firewall, and proxy configuration for this appliance live in Data Collector Deployment. The full collection catalog and the onboarding wizard live in Collection Methods & Protocols.
4.2 Local intelligence at the edge
The edge is deliberately lightweight, but it does real work with no cloud round-trip:
- capacity-forecast runs a local regression over time-series history to project usage. On Enterprise the box relays to the SaaS forecaster through the saas-sync AI bridge and falls back to local regression when offline.
- problem-engine RCA builds a graph of the topology and picks the most upstream affected node, returning an evidence chain and blast radius.
- correlation-engine does cache-backed fingerprint deduplication and a schema-aware trap classifier that maps SNMP traps to a name and severity. Correlation is Professional-tier gated; Free keeps only a local buffer.
4.3 Edge governance, remediation, and hardening
The collector includes an operator-facing control layer:
- Audit trail. An append-only local audit log records actor, action, target, and details, readable from the WebUI, which can export it to CSV and PDF.
- RBAC. The frontend derives the operator role from the JWT and gates WebUI actions accordingly.
- Automated remediation. The problem-engine ships a declarative rule engine that fans remediation events to connectors. Rules cover critical/multi-device escalation, interface flap, CPU/memory pressure, and disk full. The active-remediation guardrail ships disabled, so remediation is advisory only until an operator opts in.
- Web hardening. The WebUI sets standard security response headers (HSTS, nosniff, frame options, referrer policy, a restrictive permissions policy). All egress is outbound on 443, gzip with optional payload encryption. Edge TLS terminates at the cloud load balancer with a managed certificate.
- Break-glass login. A locally signed admin token lets an operator into the WebUI during a full SaaS outage.
- Offline licensing. The collector verifies a signed licence (tier, device limit, validity window) offline. A valid signed licence wins over SaaS register/heartbeat, which wins over Free; an expired or tampered licence falls back to Free.
5. End-to-end data pipeline lifecycle
This is the path a single metric or event takes from a device to a ticket. The cloud stream bus provides persistent store-and-forward, filling the role a stream tier would in a generic design.
The seven stages, and where each one is documented in depth:
- Acquisition.
universal-collectorpolls or listens passively. Nothing is injected inline. See Data Collector. - Normalise to local bus. Each payload becomes a uniform JSON envelope on the local bus.
- Local processing. Correlation, problem grouping, and threshold alerting fire on-prem with no cloud round-trip.
- Local persistence. Metrics go to the local time-series store, logs and flows to the local log store, inventory/alerts/problems to embedded databases.
- Store-and-forward. The local bus buffers cloud-bound events;
saas-syncflushes them over strictly outbound TLS. - Cloud ingestion.
api-gatewayresolves the token, rate-limits per tenant, normalises the batch, and republishes to tenant-scoped subjects. See SaaS Workflows. - Fleet correlation and action.
correlation-engineandai-engineproduce problems and RCA;ai-triagereasons over each incident;alerting-enginededups and escalates fleet-wide problems into the console. Operational notifications and tickets are delivered on-prem by the collector's Integrations, so paging and ticketing keep working during a WAN outage.
The AI/ML layer that sits on top of this pipeline, the statistical detection "eyes" and the LLM-triage "brain" (agent, retrieval-augmented context, gated remediation), is documented in AI Intelligence.
6. Storage & retention
Each tier runs its own data stores. The storage roles below reflect the current platform build.
| Store | Tier | Holds | Retention |
|---|---|---|---|
| Embedded databases | Collector | Local inventory, alerts, problems, correlation state, per-collector buffers, plus HA passive copies | local lifetime + nightly backup |
| Time-series store | Both | Time-series metrics | 30 days at the edge, 90 days in the cloud |
| Log/event store | Both | Logs, flows, events, alert/audit history | roughly 30 to 90 days by data type |
| Message bus | Both | Internal bus + store-and-forward buffer | until acked / flushed |
| Cache | Both | Cache, rate-limit buckets, HA leader election | ephemeral |
| Managed PostgreSQL | SaaS only | Relational system of record across the platform's logical databases, including the AI/RAG store | persistent |
| Keycloak | SaaS only | Identity | persistent |
At a high level, the collector uses embedded databases (single files, zero external dependencies) so the appliance stays simple and self-healing, while the SaaS uses a managed relational database because relational integrity across many tenants only matters in the cloud. Collector databases live on a host path that survives container teardown and rebuilds, with nightly backup and restore.
On the collector, the time-series store and bus monitoring endpoints are not published on the host. They are reached only through the WebUI reverse proxy.
6.1 Cloud retention is enforced in config, not by a cleanup job
Time-series retention is a startup setting; log-store retention is a per-table time-to-live that the engine enforces automatically on merge. Neither needs a cron job. Tenant isolation is also physical: cloud data is partitioned per tenant so a query for one tenant never touches another tenant's data on disk. The encryption, IAM, and token-rotation layers that sit on top are covered in Security & Compliance.
6.2 Per-tenant ingest rate limits
Rate limits are enforced per tenant at the api-gateway, matched to the licence tier:
| Tier | Ingest rate limit | Retention |
|---|---|---|
| Trial | lowest | 7 days |
| Professional | medium | 30 days |
| Enterprise | highest | 90 days |
Trial allows a single collector and up to 50 devices with no AI. Professional allows 5 collectors and up to 250 devices with AI anomaly detection and capacity forecasting. Enterprise allows unlimited collectors and devices with full AI (including LLM incident triage), cross-collector HA dedup, payload encryption, and consolidated batch forwarding. On the collector side the tier names are Free/Professional/Enterprise, where Free is Trial-equivalent (local buffer only). The canonical per-tier limits and feature matrix live in Licensing & Tiers.
7. The boundary: one outbound bridge
saas-sync is the only service that crosses the WAN, and it only ever dials out over HTTPS/443. It owns three jobs against the SaaS:
| Channel | Cadence | Credential |
|---|---|---|
| Registration | once | a one-time registration token returns a long-lived collector JWT plus collector and tenant identifiers |
| Heartbeat | periodic | the collector JWT as a bearer token |
| Ingest | continuous (store-and-forward) | an unguessable ingest token in the URL path |
The collector egress path is HTTPS-only, honours an outbound proxy where one is configured, gzips the ingest body, and on Enterprise adds application-layer payload encryption on top of TLS. Collector identity is persisted locally so a reboot does not force re-onboarding, and a collector that misses several heartbeats is shown as offline in the fleet view.
7.1 Three token types
The platform uses three distinct token types, and it helps to keep them straight:
- User JWT. Issued by auth-service, verified by both the api-gateway (before proxying) and the target service.
- Collector JWT. Long-lived, minted at registration with a collector claim.
- Edge token. Offline-verifiable via a published keyset, so the on-prem WebUI can verify it during a SaaS outage. This is what makes collector break-glass and offline auth possible.
The registration token itself is one-time, hashed at rest, expires after a fixed window, and is consumed on first successful register.
The exact registration handshake, token claims, and rotation are in Security & Compliance. When the cloud cannot see a collector, the diagnostic order is fixed: (1) is saas-sync registered? (2) are heartbeats landing? (3) is the ingest token valid (the gateway returns 404 on a bad token)? The full flowchart is in Collector Troubleshooting.
8. Deployment topology
The Ozone Data Collector ships as an OVA appliance and cold-boots clean with no manual steps, so fixes are baked into images rather than applied to a live container. One Compose stack runs per appliance.
The Ozone Orchestrator runs on a single cloud VM behind an outbound tunnel, with a managed PostgreSQL database. Kubernetes charts exist for a clustered topology, but the adopted production topology is single-node Compose. Production runs on real public DNS, managed TLS, and Keycloak-backed auth with tenant isolation.
ozone.techforcz.com -> Ozone Orchestrator: customer portal, collector control & ingest
On-prem appliance -> Ozone Data Collector (in-network; dials out to ozone.techforcz.com)
9. Network & port reference
Only the ports a customer needs are listed here: the device-facing listeners on the collector's LAN, and the single outbound flow to the cloud.
| Port | Direction | Purpose |
|---|---|---|
| 161 / 162 | Device to collector | SNMP poll / trap |
| 22 / 443 / 623 | Device to collector | SSH / Redfish / IPMI |
| 514 / 2055 / 9995 / 4739 / 6343 | Device to collector | Syslog / NetFlow v5 / NetFlow v9 / IPFIX / sFlow listeners |
| 4317 / 4318 / 8125 | Device to collector | OTLP gRPC / OTLP HTTP / StatsD listeners |
| 443 outbound | Collector to cloud | saas-sync to the SaaS ingress, the only cross-WAN flow |
Where to go next
| You want to… | Read |
|---|---|
| Understand the AI/ML layer: anomaly detection, the LLM triage agent, retrieval-augmented context, and auto-remediation | AI Intelligence |
| Understand multi-tenancy, encryption, IAM/RBAC, token rotation, SOC2/GDPR | Security & Compliance |
| Install the collector, size the host, open firewalls/proxies | Data Collector Deployment |
| See the full collection catalog and onboarding wizard | Collection Methods & Protocols |
| Onboard a tenant, configure alerts, build dashboards, set ingestion policy | SaaS Workflows |
| Trace ingestion failures and edge/cloud faults | Troubleshooting |