Signals

Your system is running. What numbers should you watch, and when should you worry?

This page covers the signals that tell you whether IdentityScribe is healthy. For the full metrics inventory, see the generated Telemetry Reference. For pressure gauge playbooks, see Health and Monitoring.

Scribe exposes the Four Golden Signals for both query and ingest sides.

Five signals tell you whether the query side is healthy: latency, traffic, server errors, client errors, and saturation. Each is available as a pre-computed gauge — no PromQL needed.

Per-channel breakdowns use the channel label (e.g., scribe_signals_channel_latency_p95{channel="ldap"}).

Four signals cover the sync side: task latency, replication lag, change detection rate, and task failure rate. Per-entry-type breakdowns use the entry_type label.

LDAP searches that can't be satisfied locally get delegated upstream. Track delegation rate with scribe_signals_ldap_delegation_rate_percent and per-reason counts with scribe_ldap_search_delegated_total. See LDAP channel docs for delegation reasons.

The ldap.delegation.scope span event records scope decisions for an individual search. It now also fires when a broad subtree search that names no entry type is answered locally (untyped-search = local) — those events carry a local-resolution cause and a local target, so a dashboard that filters on ldap.delegation.scope will include local-serve decisions alongside delegations. Filter on the cause/target to separate them.

For the full metric inventory with labels, units, and scrape details, see the Telemetry Reference.

Observe JSON endpoints require an operator session or accepted credential by default; the example below uses a bearer token.

curl -s -H "Authorization: Bearer <token>" http://localhost:8080/observe/signals | jq

Returns a JSON summary of all golden signals. The built-in Operator UI at /ui/observe visualizes these in real time.

Before it runs your search, the query planner does extra database work — sizing and readiness checks that decide how to execute the query — and each connection it borrows may need per-request session setup. That preparation cost used to be invisible. Two signal families now expose it.

Connection setup per pool. scribe.database.connection_hints.roundtrips.total (labeled by pool and op) counts the per-request session setup a connection pool still performs; scribe.database.connection_hints.mitigated.total (labeled by pool) counts borrows served from a pool's baked session profile with no setup cost. A pool showing roundtrips but little or no mitigated is a candidate for a baked session profile — wire one and the per-request cost drops to zero.

Per-query estimation. scribe.plan.estimation.duration.seconds (histogram) records how long row-estimation ran for a query; derive p50/p95 with histogram_quantile. scribe.plan.probe.invocations.total (labeled by kind) counts estimation invocations by kind. Estimation cost is surfaced today for cursor-paginated searches; other query routes read "not captured".

Where to see it. The Operator UI surfaces both in Observe → Inbox, in a recommendation's evidence-health panel under "Estimation & GUC cost", and per query in the diagnostic report's technical appendix. Values read "evidence pending" until the first sample arrives.

With trace export on (monitoring.traces.enabled = true), the background estimation and plan-mode checks also appear on the trace timeline with their timing and frequency.

Exact exported metric names, labels, and units are in the generated Telemetry Reference.

Not every error deserves the same response. A user typo shouldn't page you at 3am. A database timeout should.

Server errors are problems inside Scribe — internal failures, timeouts, resource exhaustion. These affect health status because they mean the service itself has a problem.

Client errors are problems with the request — bad input, missing auth, resources that don't exist. A spike might mean a misconfigured caller or a scanning attack, but Scribe itself is fine.

NOISY is intentionally separate from DEGRADED. A flood of 404s from a broken client shouldn't mask a real database problem, and shouldn't trigger your pager.

The defaults work for most deployments. Adjust them if your traffic pattern is unusual:

monitoring.signals {
    min-requests = 200           # Ignore thresholds below this volume
    server-error-rate.degraded = 0.5
    server-error-rate.critical = 2.0
    client-error-rate.noisy = 10.0
}

Different channels can have different tolerances. LDAP clients often need tighter latency than REST:

channels.ldap.signals = ${monitoring.signals} {
    latency-p99.degraded = 0.3
}

See monitoring.signals for all thresholds.

By default, client errors pass through log rules and may get filtered out. Server errors and auth failures always log — you want to see those regardless.

To log all client errors (useful when debugging a broken caller):

monitoring.log.filter-client-errors = false

PromQL examples live in the reference: PromQL Recipes.