SaaS Platform Troubleshooting
Cloud-side faults sit past the ingestion boundary: a collector is connected and its local data is fresh, but the Orchestrator is shedding, rejecting, or failing to act on traffic. This page covers ingestion throttling and rate limits, unknown-token rejections, webhook delivery failures, and RBAC/403 issues. The cross-tier isolation flow is in Troubleshooting Overview.
The Orchestrator is a hosted, multi-tenant service. The checks below are things you can see or do as a tenant administrator. For anything that needs platform-side logs, capture the symptom and the affected route and contact TechForcz support.
1. Ingestion throttling (429)
Symptom. A tenant's data is partially landing, ingestion returns 429, and a portion of events is dropped.
Cause. The per-tenant ingestion quota is exceeded. This is a sizing or policy condition, not an outage; the limiter is protecting fleet-wide ingest. Per-tenant ceilings scale with tier (roughly Trial about 1,000, Professional about 50,000, and Enterprise about 500,000 events per minute).
Fix.
- Raise the tenant's quota in the ingestion policy if the volume is legitimate, or move the tenant to a higher tier. See SaaS Workflows.
- Or lower what the collector forwards: drop its forward tier so only critical problems escalate instead of the full stream.
- Confirm it is not a misbehaving collector re-sending the same batch (check the
saas-synclogs on the appliance, which you do control).
2. API rate limits (gateway 429)
Symptom. Portal or API calls fail with 429 Too Many Requests and a Retry-After: 60 header.
Cause. The platform enforces a per-user, sliding-window API rate limit (default about 120 requests per minute). This is independent of the per-tenant ingestion quota in section 1.
Fix.
- Back off the client; the window resets each minute, honor
Retry-After. - For legitimate automation that needs higher throughput, move the integration to the collector ingest path (
/ingest/{token}), which is not on the per-user API limiter, or ask support to raise the limit for your account.
The two 429s are different controls. Section 1 is the per-tenant telemetry rate limit; section 2 is the per-user API limit. The Retry-After header and the endpoint you called tell you which one you hit.
3. Unknown ingest token (404)
Symptom. A collector's ingest posts return 404 Unknown ingest token; that tenant's telemetry never appears, though heartbeats are fine.
Cause. The ingest token does not resolve to a tenant. It was rotated, disabled, or never bound.
Fix.
# A bad token returns 404; a good post returns 202
curl -sS -o /dev/null -w '%{http_code}\n' https://ozone.techforcz.com/ingest/<token>
- Re-issue or confirm the tenant's ingest token in the integration config, and update the collector's
saas-syncconfig. - Heartbeat and ingest are separate credentials, so a working heartbeat does not imply a valid ingest token. See Security & Compliance.
4. Integration delivery failures
Integration delivery is a collector function, not an Orchestrator portal one. Connectors (Slack, Microsoft Teams, Webex, PagerDuty, ServiceNow, Jira, and generic webhook) are configured and delivered on the on-prem Ozone Data Collector, in its WebUI Integrations page. If tickets or pages are not arriving, triage them on the collector, not in the SaaS portal.
Symptom. Tickets or pages are not arriving in Slack, PagerDuty, ServiceNow, or a generic webhook, though the problem fired.
Cause. A downstream endpoint error: a bad URL or routing key, an expired credential, or the destination returning 4xx/5xx.
Fix by channel.
| Channel | Common cause | Fix |
|---|---|---|
| Slack / Teams / Webex | revoked or incorrect webhook URL | re-issue the incoming webhook, update the connector config |
| PagerDuty | wrong or expired routing key | correct the routing key, verify Events API reachability |
| ServiceNow / Jira | auth or instance URL wrong, field validation | re-check the instance URL and credentials |
| Generic webhook | endpoint 4xx/5xx | check the receiver, confirm the payload contract |
Each connector has a Test action in the collector's Integrations page that sends a sample event, use it to confirm a fix without waiting for a real problem. For delivery status and deeper diagnosis, see Collector Troubleshooting.
5. RBAC / 403 issues
Symptom. A user is authenticated but receives 403 on portal actions or API routes, or a user sees the wrong tenant's scope (treat cross-tenant visibility as a P1 security issue).
Cause. A role or scope mismatch. The platform uses five RBAC roles: SuperAdmin, TenantAdmin, Engineer, ReadOnly, and Auditor. Each action is checked against the caller's role.
Fix.
- A TenantAdmin can correct the user's role assignment (one of the five roles above) in the portal.
- If a user can see another tenant's data, that is a security defect, not a configuration nit. Open a P1 with TechForcz support and include the failing route. Never paste full tokens into a ticket.
6. Other Orchestrator faults
| Signature | First move |
|---|---|
| Collector OFFLINE but customer says fine | heartbeat path broken, not collection, check the appliance timer, then the edge network chain (Collector Troubleshooting) |
| Registration token rejected | expired or already-consumed (one-time, 72h expiry) or tenant mismatch, issue a new token |
| Cert renewal failures fleet-wide | a platform-side issue, escalate to support as a high-severity incident |
ozone.techforcz.com up but API errors | capture the failing route and any error id shown in the portal, then contact support |
| AI triage not producing an RCA | triage is advisory and its write tools ship disabled, so a triage outage never blocks alerting or ticketing, confirm your plan includes AI triage and contact support if it stays silent |
For faults you cannot resolve from these playbooks, capture the failing route and the time, and escalate to TechForcz support. Architecture context is in Platform Architecture.