Data Collector Troubleshooting
Edge-side faults: the collector keeps polling and buffering locally even when the cloud link is down, so most "data missing" reports here are transport problems, not data loss. This page covers network drops, local buffer growth, TLS handshake failures, NATS bus loss, SNMP poll failures, SQLite faults, and the service-down signatures, each as symptom, cause, fix, escalate. The cross-tier isolation flow is in Troubleshooting Overview.
1. Network drops & store-and-forward
Symptom. Collector shows OFFLINE in the fleet view; cloud dashboards stop updating for this site.
Cause. The outbound 443 path to ozone.techforcz.com is broken: circuit down, firewall change, DNS, or proxy. Local collection is unaffected.
What actually happens (so you can reassure the customer). saas-sync forwarding is best-effort. With the WAN down, events stay in NATS JetStream and SQLite; the collector keeps polling, correlating, and firing local alerts. On reconnect, the heartbeat loop re-establishes and the buffered backlog replays, so there is zero loss and no inbound ports. A collector is judged "connected" only if a heartbeat landed within 3 intervals (about 90s).
Fix.
# Confirm it is transport, not collection: local data should be fresh
docker compose logs --tail 20 universal-collector # devices still polling?
# Walk the egress path
nslookup ozone.techforcz.com # DNS
nc -vz ozone.techforcz.com 443 # egress 443
systemctl status techforcz-heartbeat.timer # heartbeat firing?
docker compose logs --tail 50 saas-sync # last forward attempt + error
Restore DNS, firewall, or proxy, then watch saas-sync logs for a successful heartbeat. No data replay step is needed; it is automatic.
Escalate to SaaS-side if egress, DNS, and NTP are clean but the orchestrator still rejects the collector (see SaaS Troubleshooting).
2. Local buffer overflow
Symptom. /data disk usage climbing; a long WAN outage; eventually disk-pressure warnings.
Cause. Store-and-forward is doing its job: JetStream is buffering cloud-bound events while the link is down. The risk is only the disk filling before the link returns.
Fix.
# Where is the space going?
du -sh /data/* # JetStream (nats), VictoriaMetrics, SQLite
docker system df # image/layer bloat
docker exec aiops-nats nats stream report # pending/unacked messages in TELEMETRY (if nats CLI present)
- If JetStream backlog is the consumer: restore the WAN link; the backlog drains on reconnect.
- If VictoriaMetrics retention is the consumer: tune retention before deleting anything (VictoriaMetrics
--retentionPeriod). The on-prem metrics default is 30d. - Treat greater than 90% disk as SEV2.
Never
docker compose down(and above all neverdocker compose down -v) to "reclaim space" on a customer system. That drops the buffering services, and-vwould try to remove volumes. The SQLite stores live on a host bind mount (AIOPS_DATA_DIR, default/opt/aiops/data), not a Docker named volume, so-vwill not delete them, but it is still a forbidden habit. Restart individual containers only.
Escalate to L3 for JetStream store inspection if NATS itself crash-loops (possible JetStream store corruption under /data/nats).
3. TLS handshake failures
Symptom. saas-sync logs show TLS/SSL errors, such as certificate verify failed, handshake timeout, or SSLError, on every forward.
Causes & fixes, in order of likelihood.
| Cause | Signature | Fix |
|---|---|---|
| Clock skew | certificate is not yet valid / expired; skew over 5 min | fix NTP: timedatectl set-ntp true; TLS validates against the clock |
| Proxy TLS interception | verify failed, issuer is the corporate proxy CA | trust the proxy CA on the appliance, or exempt ozone.techforcz.com from interception |
| Missing or untrusted server CA | unable to get local issuer certificate | ensure the system CA bundle is current; production must serve a real CA cert, not tls internal |
| Wrong endpoint or TLS downgrade | handshake reset, protocol mismatch | confirm SAAS_URL=https://ozone.techforcz.com; floor is TLS 1.2 |
# Inspect what the appliance actually negotiates with the Orchestrator
openssl s_client -connect ozone.techforcz.com:443 -servername ozone.techforcz.com </dev/null 2>/dev/null \
| openssl x509 -noout -issuer -subject -dates
timedatectl | grep -E 'synchronized|NTP'
The collector-to-SaaS channel is token/JWT over TLS, not mTLS, so there is no client certificate to rotate here. Do not chase a missing client cert. See Security & Compliance, Collector authentication.
4. Core service-down signatures
Symptom. A FastAPI service is "Up" but its HTTP port returns nothing; logs repeat Waiting for application startup plus nats: no servers available / socket.gaierror.
Cause. NATS is down or the container is not attached to the collector's internal Docker network. The app blocks in its startup hook retrying NATS, so it never serves. Fix NATS and networking first, never debug the app.
cd /opt/techforcz/collector/deploy/docker-compose
docker compose up -d nats
docker inspect <svc> --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' # expect an address on the collector network
# If empty, reattach the container to the collector's compose network and restart it:
docker compose restart <svc>
Other common edge faults include "port already allocated" when a duplicate stack is running, and devices "disappearing" after a container is recreated without the /data bind mount. Confirm a single running stack and an intact /data mount first; if these persist, contact TechForcz support.
5. NATS bus unavailability (silent telemetry drop)
Symptom. Everything looks "Up", but nothing new lands in VictoriaMetrics and no alerts fire. The pipeline is quietly dropping telemetry.
Cause. NATS JetStream is the spine of the collector. When the shared NatsPool has no live connections, publishers get no transport and telemetry is dropped silently rather than erroring loudly. This differs from §4 (where a service refuses to start): here services stay up but the bus underneath them is empty.
Fix. Restart in dependency order so publishers reconnect to a healthy bus:
cd /opt/techforcz/collector/deploy/docker-compose
docker compose restart nats
# wait for NATS to report healthy, then bring the publishers back
docker compose ps nats # wait until it shows healthy
docker compose restart universal-collector
docker compose restart correlation-engine
Confirm streams are present and receiving (docker exec aiops-nats nats stream report if the CLI is available). The correlation-engine has no HTTP health endpoint, so verify it by container state (docker ps), not by curling a port.
Escalate to L3 if NATS crash-loops or JetStream reports store corruption under /data/nats.
6. SNMP poll failure
Symptom. A device onboarded for SNMP shows no metrics; the universal-collector logs report authentication or reachability errors.
Causes & fixes.
| Signature | Cause | Fix |
|---|---|---|
wrongDigest on SNMPv3 | auth/priv password or algorithm mismatch | re-check the SNMPv3 credentials; the collector supports auth SHA / MD5 / SHA-224 / SHA-256 / SHA-384 / SHA-512 and priv AES-128 / AES-192 / AES-256 / DES / 3DES. Security level is derived from which passwords are set (noAuthNoPriv / authNoPriv / authPriv) |
| Timeout / unreachable | device down, ACL, or wrong UDP path | confirm the device answers SNMP from the appliance; check firewall and community/context |
| SNMP import or decode errors | library version conflict inside the collector image | reinstall the shipped collector image rather than upgrading dependencies in place; contact TechForcz support if it persists |
SNMP polling runs inside the universal-collector (there is no separate snmp-poller service), so read its logs and confirm it is Up with docker ps. All active pollers ship disabled with empty target tables, so a fresh collector polls nothing until devices are onboarded.
For inbound SNMP traps on 162/udp, the receiver needs root or a lowered net.ipv4.ip_unprivileged_port_start sysctl, because 162 is a privileged port. If traps never arrive, check that first.
7. SQLite faults (lock, empty alerts.db, disk full)
The collector uses SQLite only (no PostgreSQL on the edge), roughly 15 databases on the AIOPS_DATA_DIR host bind mount (default /opt/aiops/data): collector.db, inventory.db, alerts.db, problems.db, correlation.db, and others. Because it is a host bind mount, the data survives docker compose down, volume prune, system prune, and image rebuilds.
Database is locked.
# WAL mode should already be on; confirm and let writers drain
docker exec aiops-universal-collector sqlite3 /data/collector.db 'PRAGMA journal_mode;' # expect wal
SQLite with WAL allows concurrent readers with a single writer. A persistent lock usually means two processes are writing the same file (for example a stray second container). Resolve the duplicate rather than deleting the lock file.
alerts.db is empty (no alerts even though devices are up). On the Free tier the correlation and alerting engines are intentionally dormant (local buffer only). An empty alerts.db on a Free-tier collector is expected, not a fault. Correlation is Professional-tier gated. If the collector should be licensed higher, check the tier resolution in saas-sync before treating this as a bug.
Permissions / disk full.
du -sh /opt/aiops/data/* # which db is growing
ls -l /opt/aiops/data # files should be owned by uid/gid 1000:1000
sudo chown -R 1000:1000 /opt/aiops/data # containers run as uid 1000; wrong owner blocks writes
If a database is corrupt, restore from the nightly backup (scripts/aiops-restore.sh, SQLite online .backup, 14-day retention, cron at 02:30). Never delete a .db file to "reset" it; that is the data.
Escalate to L3 for SQLite recovery if a file is corrupt and no clean backup exists.
Quick reference
# State capture (attach to every case)
python3 /opt/techforcz/bin/diagnostics.py
docker compose ps # every container up and healthy?
# Component logs
docker compose logs --tail 100 saas-sync # egress bridge
docker compose logs --tail 50 universal-collector # collection plane
docker compose logs --tail 50 correlation-engine # correlation
If the fault is past the boundary (404 on ingest, 429 throttling, webhook or RBAC errors), cross to SaaS Platform Troubleshooting.