Health and Monitoring

Is IdentityScribe healthy? This page shows you how to check, what the numbers mean, and what to do when they look wrong.

For the full list of /observe/* endpoints and their response shapes, see the Endpoints Reference.

Examples use port 8080 (the default). The bundled monitoring stack uses port 9001 via socket separation. These endpoints are also listed in the Endpoints Reference with full response schemas. Observe JSON endpoints require an operator session or accepted credential by default; examples below use a bearer token. Kubernetes-style /livez, /readyz, and /startedz probes remain unauthenticated.

The smartest health check. Returns threshold-based assessments, per-service status, and actionable recommendations.

set -o pipefail
curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/doctor | jq

Focused on saturation metrics only. Shows per-entry-type breakdown of queue and task pressure, plus query_permit_queue (threads waiting for permits). Use this when you want a quick saturation check without the full doctor report.

set -o pipefail
curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/pressure | jq

For Kubernetes liveness, readiness, and startup checks, see the probe configuration in the deployment guide.

Use /readyz as the Kubernetes readiness probe. Use /observe/convergence for deployment scripts, smoke tests, and dashboards that need strict completion instead of traffic readiness.

Pressure gauges are the first thing to check when something feels off. Each one measures how close a resource is to saturation. For the full metric definitions with labels, units, and scrape details, see the Telemetry Reference.

Look for sustained elevation, not spikes. A brief spike during a bulk import is normal. Sustained high pressure with rising latency or rejections means you need to act.

After a deployment or restart:

set -o pipefail

# 1. Services started?
# Keep the query URL quoted so Bash and zsh pass it literally.
curl -fsS 'http://localhost:8080/startedz?verbose'

# 2. Uptime and status
curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/status | jq

# 3. Sync and index convergence complete?
curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/convergence | jq

# 4. Any recent restarts?
curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/doctor | jq '.checks[] | select(.name == "services.restarts_5m")'

If data looks stale, check ingest status:

set -o pipefail
curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/stats/ingest | jq

Alert: scribe_query_permit_pressure > 0.8

1. Check /observe/doctor for query.permit_pressure
2. Find slow queries via traces or stage durations
3. Consider raising database.channel-pool-size
4. Check /observe/hints?severity=warning for missing indexes

Alert: scribe_ingest_task_pressure > 1.2

1. Check /observe/stats/ingest for per-entry-type breakdown
2. Verify source LDAP connectivity
3. Open Observe → Services and look for a saturated database pool
4. Check /observe/stats/ingest for one entry type dominating queue depth
5. Consider raising ingest workers only after the database pool has headroom

Alert: jvm_memory_pressure > 0.8

1. Check /observe/doctor context section
2. Review -Xmx heap size
3. Profile for leaks if persistent
4. Restart as last resort

Alert: scribe_service_restarts_5m > 1

1. Check /observe/doctor for services.restarts_5m
2. Review logs for crash causes
3. Check resource limits (CPU, memory)
4. Check external dependencies (DB, LDAP)