Server troubleshooting

Classify the failure before changing state: availability, authentication, authorization, metadata database, blob storage, event dispatch, package registry, policy, or execution.

Triage sequence

1. Check health and readiness plus database and blob dependencies.
2. Capture the request trace ID and structured error code.
3. Confirm OIDC issuer, audience, clock, organization, role, and project.
4. For file failures, compare metadata commit state with blob operation state.
5. For stale clients, inspect event cursor age and reconciliation.
6. For remote runs, separate queue, staging, worker, Runtime, and artifact-finalization states.
7. Check quota or entitlement decisions before retrying a rejected operation.

Preserve metadata, blobs, and request evidence during diagnosis. Use documented reconciliation or retry paths, and verify idempotency before replaying a mutation. An incident report should include Server version, deployment mode, time window, trace IDs, resource IDs safe to share, error codes, dependency health, and the smallest reproduction with secrets removed.

Availability

Check liveness, then readiness. Readiness failures should identify database, blob, migration, or required-provider state. Compare the first failure time with deploys, secret rotations, database failover, certificate changes, and provider incidents. Use request and trace IDs to follow one operation across the gateway and domain services.

Authentication and authorization

A burst of 401 responses points to issuer/audience configuration, JWKS availability, clock skew, expired sessions, or key revocation. 403 means the identity was accepted; inspect active organization, resource membership, role, key scope, and policy decision. For SSO routing, verify domain ownership and connection state.

Files and storage

Locate the operation phase: policy preflight, upload target issuance, byte transfer, content verification, metadata commit, or event dispatch. An expired signed URL can be reissued. A hash mismatch requires retransmission. An ambiguous commit requires reading current metadata before retrying. Compare live and retained bytes for quota cases.

Missing content with present metadata is a high-priority consistency fault. Preserve the metadata row, content hash, blob key, request IDs, and storage-provider response. Leave the affected path untouched until reconciliation identifies the correct content.

Stale clients and events

Inspect SSE connection duration, heartbeat, proxy timeout, cursor, dispatch backlog, and resume result. Force an authoritative scope refresh after a cursor-expired response. Duplicate events are expected around reconnect. Every client should still converge on the authoritative state.

Packages and remote runs

Package resolution failures need the full coordinate, constraint/lock, registry scope, and expected hash. Remote runs have distinct queue, lease, staging, execution, and finalization phases. Queue delay points to eligibility/capacity. Staging points to project/package/blob access. Execution points to Runtime or workload. Finalization points to artifact or terminal-state commit.

Incident record

Record exact UTC window, Server version and deployment mode, affected organizations/projects in a private channel, request/trace IDs, stable error codes, dependency health, recent changes, and sanitized reproduction. State what recovered the service and what evidence remains before cleanup.