Skip to content

Health and Monitoring

  • Alive?

    Check whether the process is running.

  • Ready?

    Check whether it can serve traffic.

  • Caught up?

    Check sync and search preparation progress.

Health checks answer different questions about a running instance.

You can separate a running process from a ready service and identify what still needs attention.

Next: Troubleshooting

Is IdentityScribe healthy? This page shows you how to check, what the numbers mean, and what to do when they look wrong.

For the full list of /observe/* endpoints and their response shapes, see the Endpoints Reference.

Examples use port 8080 (the default). The bundled monitoring stack uses port 9001 via socket separation. These endpoints are also listed in the Endpoints Reference with full response schemas. Observe JSON endpoints require an operator session or accepted credential by default; examples below use a bearer token. Kubernetes-style /livez, /readyz, and /startedz probes remain unauthenticated.

The smartest health check. Returns threshold-based assessments, per-service status, and actionable recommendations.

Terminal window
set -o pipefail
curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/doctor | jq
StatusMeaning
healthyAll checks pass
degradedWarning thresholds exceeded — investigate soon
criticalCritical thresholds exceeded — act now

Focused on saturation metrics only. Shows per-entry-type breakdown of queue and task pressure, plus query_permit_queue (threads waiting for permits). Use this when you want a quick saturation check without the full doctor report.

Terminal window
set -o pipefail
curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/pressure | jq

For Kubernetes liveness, readiness, and startup checks, see the probe configuration in the deployment guide.

EndpointPurpose
/livezIs the process alive?
/readyzCan it safely receive traffic? Fails while searches are stalled.
/startedzHave services initialized?
/healthzCombined health check, including a search stall
/observe/convergenceIs long-running initial ingest and index work fully complete?

Use /readyz as the Kubernetes readiness probe. Use /observe/convergence for deployment scripts, smoke tests, and dashboards that need strict completion instead of traffic readiness.

All permits in useSearches waitor turned awayStill readysearches finishSearches stalledafter 60 sReadiness/readyz → 503new searchreleasedno releasefailsstall ends
All permits in useSearches waitor turned awayStill readysearches finishSearches stalledafter 60 sReadiness/readyz → 503new searchreleasedno releasefailsstall ends
High load stays ready while searches finish; only a sustained, complete search stall takes an instance out of traffic.

/readyz, /healthz and /observe/health/ready return 503 while searches are stalled: every query permit is in use, a search has waited for a permit or was turned away since the last release, and no permit has been released for monitoring.readiness.query-stall-after (default 60s). A turned-away search still counts after it gave up. Taking a stalled instance out of rotation lets the other instances serve searches. High load without a stall shows up in scribe_query_permit_pressure instead. When a permit is released, the probes return 200 again without a restart. /livez is not affected. The JSON from /observe/health/ready reports the stall in its query-stall check.

To keep readiness independent of searches, set monitoring.readiness.fail-on-query-stall = false (or SCRIBE_MONITORING_READINESS_FAIL_ON_QUERY_STALL=false). The query.permit_stall doctor check and the scribe_query_permit_stall_seconds gauge still report the stall. Raise monitoring.readiness.query-stall-after (SCRIBE_MONITORING_READINESS_QUERY_STALL_AFTER) above your longest legitimate search if long exports hold every permit for a minute or more.

Pressure gauges are the first thing to check when something feels off. Most measure how close a resource is to its limit; ingest task pressure measures whether processing keeps up.

FRACTION OF A LIMIT · 0 TO 1KEEP-UP RATIO · CAN EXCEED 1Permits in useof all permitsBuffered changesof buffer sizeHeap in useof max heapProcessing demandvs capacityQuery permit pressurewarn 0.8 · crit 0.95Ingest queue pressurewarn 0.8 · crit 0.95Memory pressurewarn 0.8 · crit 0.95Ingest task pressure~1 · warn 1.2 · crit 2fractionfill levelfractiondemand ÷ capacity
FRACTION OF A LIMIT · 0 TO 1KEEP-UP RATIO · CAN EXCEED 1Permits in useof all permitsBuffered changesof buffer sizeHeap in useof max heapProcessing demandvs capacityQuery permit pressurewarn 0.8 · crit 0.95Ingest queue pressurewarn 0.8 · crit 0.95Memory pressurewarn 0.8 · crit 0.95Ingest task pressure~1 · warn 1.2 · crit 2fractionfill levelfractiondemand ÷ capacity
Ingest task pressure is a keep-up ratio, not a percentage of a full buffer: it sits near 1 and rises above 1 when processing falls behind.
For the full metric definitions with labels, units, and scrape details, see the Telemetry Reference.

MetricRangeWarningCriticalWhat it means
scribe_query_permit_pressure0..10.80.95Query capacity used
scribe_query_permit_queue0..N510Threads waiting for query permits
scribe_query_permit_stall_secondsseconds—> 0Seconds all permits have stayed in use without a release while searches are stalled; 0 otherwise
scribe_ingest_queue_pressure0..10.80.95Ingest buffer fill level
scribe_ingest_task_pressure~11.22.0Processing keep-up ratio (>1 = falling behind)
jvm_memory_pressure0..10.80.95Heap utilization

Look for sustained elevation, not spikes. A brief spike during a bulk import is normal. Sustained high pressure with rising latency or rejections means you need to act.

MetricThreshold
scribe_db_connections_pending> 0 means pool saturation
scribe_ingest_lag_seconds> 60s warning, > 300s critical
scribe_query_rejected_5m> 0 means queries are being dropped
scribe_ldap_upstream_state1 degraded, ≥ 2 critical: the directory is not answering sign-ins

After a deployment or restart:

Terminal window
set -o pipefail
# 1. Services started?
# Keep the query URL quoted so Bash and zsh pass it literally.
curl -fsS 'http://localhost:8080/startedz?verbose'
# 2. Uptime and status
curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/status | jq
# 3. Sync and index convergence complete?
curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/convergence | jq
# 4. Any recent restarts?
curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/doctor | jq '.checks[] | select(.name == "services.restarts_5m")'

If data looks stale, check ingest status:

Terminal window
set -o pipefail
curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/stats/ingest | jq
SymptomLikely causeAction
High lag, low task pressureSource LDAP slowCheck source LDAP performance
High lag, high task pressureProcessing bottleneckIncrease workers or check DB
High queue pressureDB writes slowOpen Observe → Services for pool saturation, then check /observe/stats/ingest for stuck entry types
ingest_completed = falseInitial sync still runningCheck /observe/convergence and /observe/stats/ingest

Alert: scribe_query_permit_pressure > 0.8

  1. Check /observe/doctor for query.permit_pressure
  2. Find slow queries via traces or stage durations
  3. Consider raising database.channel-pool-size
  4. Check /observe/hints?severity=warning for missing indexes

Alert: scribe_query_permit_stall_seconds > 0

  1. Check /observe/doctor for query.permit_stall
  2. Collect a thread dump with SIGQUIT (kill -3 <pid>)
  3. If the log shows query_permit_stall_cleared, the gauge is back to 0, and /readyz returns 200, the stall has ended and no restart is needed; otherwise restart the instance
  4. Send the thread dump to support either way

Alert: scribe_ingest_task_pressure > 1.2

  1. Check /observe/stats/ingest for per-entry-type breakdown
  2. Verify source LDAP connectivity
  3. Open Observe → Services and look for a saturated database pool
  4. Check /observe/stats/ingest for one entry type dominating queue depth
  5. Consider raising ingest workers only after the database pool has headroom

Alert: jvm_memory_pressure > 0.8

  1. Check /observe/doctor context section
  2. Review -Xmx heap size
  3. Profile for leaks if persistent
  4. Restart as last resort

Alert: max_over_time(scribe_ldap_upstream_state[2m]) >= 2

The auth.ldap.upstream doctor check reads the same gauge:

ValueDoctorMeaning
0healthyThe directory answers sign-in checks
1degradedAt least three sign-in checks and 10 % of them failed in the last minute
2criticalChecking recovery: the next sign-in that needs the directory first checks whether it answers again
3criticalSign-ins that need the directory fail immediately with LDAP 52, HTTP 503, or “Sign-in temporarily unavailable”

The gauge and the check exist only when LDAP authentication is configured. Scribe answers sign-ins immediately once auth.ldap.breaker.threshold checks in a row have failed and Scribe’s own check of the directory has failed too, instead of letting every client wait for auth.ldap.request-timeout. A client cannot cause this with sign-ins that fail because of what it sent, such as oversized credentials, but it can make the check report degraded. Local accounts, browser sessions, cached sign-ins, and searches keep working, and readiness stays green on purpose: the directory serves every instance, so taking instances out of rotation would not help.

  1. Check the directory servers and the network path from Scribe to them
  2. Look for ldap_upstream_unavailable warnings in the logs; ldap_upstream_recovered follows when the directory answers again and reports in auth.ldap.upstream.failed_fast how many sign-ins failed fast meanwhile. Those sign-ins are not logged one by one, so the log stays small during an outage; scribe_channel_requests_total{result="unavailable"} still counts each one as it happens
  3. Recovery needs no restart: after the directory answers again, sign-ins may keep failing for up to auth.ldap.breaker.open-duration until the next check
  4. If one client with a single account causes most sign-ins, lower auth.ldap.breaker.threshold so an outage is detected sooner

See the breaker settings for the defaults and how to turn it off.

Alert: scribe_service_restarts_5m > 1

  1. Check /observe/doctor for services.restarts_5m
  2. Review logs for crash causes
  3. Check resource limits (CPU, memory)
  4. Check external dependencies (DB, LDAP)