Health and Monitoring
- Alive?
Check whether the process is running.
- Ready?
Check whether it can serve traffic.
- Caught up?
Check sync and search preparation progress.
You can separate a running process from a ready service and identify what still needs attention.
Next: Troubleshooting
Is IdentityScribe healthy? This page shows you how to check, what the numbers mean, and what to do when they look wrong.
For the full list of /observe/* endpoints and their response shapes, see the Endpoints Reference.
Built-in tools
Section titled “Built-in tools”Health endpoints
Section titled “Health endpoints”Examples use port 8080 (the default). The bundled monitoring stack uses port 9001 via socket separation. These endpoints are also listed in the Endpoints Reference with full response schemas. Observe JSON endpoints require an operator session or accepted credential by default; examples below use a bearer token. Kubernetes-style
/livez,/readyz, and/startedzprobes remain unauthenticated.
Doctor (/observe/doctor)
Section titled “Doctor (/observe/doctor)”The smartest health check. Returns threshold-based assessments, per-service status, and actionable recommendations.
set -o pipefailcurl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/doctor | jq| Status | Meaning |
|---|---|
healthy | All checks pass |
degraded | Warning thresholds exceeded — investigate soon |
critical | Critical thresholds exceeded — act now |
Pressure (/observe/pressure)
Section titled “Pressure (/observe/pressure)”Focused on saturation metrics only. Shows per-entry-type breakdown of queue and task pressure, plus query_permit_queue (threads waiting for permits). Use this when you want a quick saturation check without the full doctor report.
set -o pipefailcurl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/pressure | jqKubernetes probes
Section titled “Kubernetes probes”For Kubernetes liveness, readiness, and startup checks, see the probe configuration in the deployment guide.
| Endpoint | Purpose |
|---|---|
/livez | Is the process alive? |
/readyz | Can it safely receive traffic? Fails while searches are stalled. |
/startedz | Have services initialized? |
/healthz | Combined health check, including a search stall |
/observe/convergence | Is long-running initial ingest and index work fully complete? |
Use /readyz as the Kubernetes readiness probe. Use /observe/convergence for deployment scripts, smoke tests, and dashboards that need strict completion instead of traffic readiness.
/readyz, /healthz and /observe/health/ready return 503 while searches are stalled: every query permit is in use, a search has waited for a permit or was turned away since the last release, and no permit has been released for monitoring.readiness.query-stall-after (default 60s). A turned-away search still counts after it gave up. Taking a stalled instance out of rotation lets the other instances serve searches. High load without a stall shows up in scribe_query_permit_pressure instead. When a permit is released, the probes return 200 again without a restart. /livez is not affected. The JSON from /observe/health/ready reports the stall in its query-stall check.
To keep readiness independent of searches, set monitoring.readiness.fail-on-query-stall = false (or SCRIBE_MONITORING_READINESS_FAIL_ON_QUERY_STALL=false). The query.permit_stall doctor check and the scribe_query_permit_stall_seconds gauge still report the stall. Raise monitoring.readiness.query-stall-after (SCRIBE_MONITORING_READINESS_QUERY_STALL_AFTER) above your longest legitimate search if long exports hold every permit for a minute or more.
Pressure gauges
Section titled “Pressure gauges”Pressure gauges are the first thing to check when something feels off. Most measure how close a resource is to its limit; ingest task pressure measures whether processing keeps up.
| Metric | Range | Warning | Critical | What it means |
|---|---|---|---|---|
scribe_query_permit_pressure | 0..1 | 0.8 | 0.95 | Query capacity used |
scribe_query_permit_queue | 0..N | 5 | 10 | Threads waiting for query permits |
scribe_query_permit_stall_seconds | seconds | — | > 0 | Seconds all permits have stayed in use without a release while searches are stalled; 0 otherwise |
scribe_ingest_queue_pressure | 0..1 | 0.8 | 0.95 | Ingest buffer fill level |
scribe_ingest_task_pressure | ~1 | 1.2 | 2.0 | Processing keep-up ratio (>1 = falling behind) |
jvm_memory_pressure | 0..1 | 0.8 | 0.95 | Heap utilization |
Look for sustained elevation, not spikes. A brief spike during a bulk import is normal. Sustained high pressure with rising latency or rejections means you need to act.
Connection health
Section titled “Connection health”| Metric | Threshold |
|---|---|
scribe_db_connections_pending | > 0 means pool saturation |
scribe_ingest_lag_seconds | > 60s warning, > 300s critical |
scribe_query_rejected_5m | > 0 means queries are being dropped |
scribe_ldap_upstream_state | 1 degraded, ≥ 2 critical: the directory is not answering sign-ins |
Deployment verification
Section titled “Deployment verification”After a deployment or restart:
set -o pipefail
# 1. Services started?# Keep the query URL quoted so Bash and zsh pass it literally.curl -fsS 'http://localhost:8080/startedz?verbose'
# 2. Uptime and statuscurl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/status | jq
# 3. Sync and index convergence complete?curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/convergence | jq
# 4. Any recent restarts?curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/doctor | jq '.checks[] | select(.name == "services.restarts_5m")'Data freshness
Section titled “Data freshness”If data looks stale, check ingest status:
set -o pipefailcurl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/stats/ingest | jq| Symptom | Likely cause | Action |
|---|---|---|
| High lag, low task pressure | Source LDAP slow | Check source LDAP performance |
| High lag, high task pressure | Processing bottleneck | Increase workers or check DB |
| High queue pressure | DB writes slow | Open Observe → Services for pool saturation, then check /observe/stats/ingest for stuck entry types |
ingest_completed = false | Initial sync still running | Check /observe/convergence and /observe/stats/ingest |
Alert runbooks
Section titled “Alert runbooks”High permit pressure
Section titled “High permit pressure”Alert: scribe_query_permit_pressure > 0.8
- Check
/observe/doctorforquery.permit_pressure - Find slow queries via traces or stage durations
- Consider raising
database.channel-pool-size - Check
/observe/hints?severity=warningfor missing indexes
Searches stalled
Section titled “Searches stalled”Alert: scribe_query_permit_stall_seconds > 0
- Check
/observe/doctorforquery.permit_stall - Collect a thread dump with
SIGQUIT(kill -3 <pid>) - If the log shows
query_permit_stall_cleared, the gauge is back to 0, and/readyzreturns200, the stall has ended and no restart is needed; otherwise restart the instance - Send the thread dump to support either way
Ingest falling behind
Section titled “Ingest falling behind”Alert: scribe_ingest_task_pressure > 1.2
- Check
/observe/stats/ingestfor per-entry-type breakdown - Verify source LDAP connectivity
- Open Observe → Services and look for a saturated database pool
- Check
/observe/stats/ingestfor one entry type dominating queue depth - Consider raising ingest workers only after the database pool has headroom
Memory pressure
Section titled “Memory pressure”Alert: jvm_memory_pressure > 0.8
- Check
/observe/doctorcontext section - Review
-Xmxheap size - Profile for leaks if persistent
- Restart as last resort
Directory not answering sign-ins
Section titled “Directory not answering sign-ins”Alert: max_over_time(scribe_ldap_upstream_state[2m]) >= 2
The auth.ldap.upstream doctor check reads the same gauge:
| Value | Doctor | Meaning |
|---|---|---|
0 | healthy | The directory answers sign-in checks |
1 | degraded | At least three sign-in checks and 10 % of them failed in the last minute |
2 | critical | Checking recovery: the next sign-in that needs the directory first checks whether it answers again |
3 | critical | Sign-ins that need the directory fail immediately with LDAP 52, HTTP 503, or “Sign-in temporarily unavailable” |
The gauge and the check exist only when LDAP authentication is configured. Scribe answers sign-ins immediately once auth.ldap.breaker.threshold checks in a row have failed and Scribe’s own check of the directory has failed too, instead of letting every client wait for auth.ldap.request-timeout. A client cannot cause this with sign-ins that fail because of what it sent, such as oversized credentials, but it can make the check report degraded. Local accounts, browser sessions, cached sign-ins, and searches keep working, and readiness stays green on purpose: the directory serves every instance, so taking instances out of rotation would not help.
- Check the directory servers and the network path from Scribe to them
- Look for
ldap_upstream_unavailablewarnings in the logs;ldap_upstream_recoveredfollows when the directory answers again and reports inauth.ldap.upstream.failed_fasthow many sign-ins failed fast meanwhile. Those sign-ins are not logged one by one, so the log stays small during an outage;scribe_channel_requests_total{result="unavailable"}still counts each one as it happens - Recovery needs no restart: after the directory answers again, sign-ins may keep failing for up to
auth.ldap.breaker.open-durationuntil the next check - If one client with a single account causes most sign-ins, lower
auth.ldap.breaker.thresholdso an outage is detected sooner
See the breaker settings for the defaults and how to turn it off.
Service flapping
Section titled “Service flapping”Alert: scribe_service_restarts_5m > 1
- Check
/observe/doctorforservices.restarts_5m - Review logs for crash causes
- Check resource limits (CPU, memory)
- Check external dependencies (DB, LDAP)