Signals

Your system is running. What numbers should you watch, and when should you worry?

This page covers the signals that tell you whether IdentityScribe is healthy. For the full metrics inventory, see the generated Telemetry Reference. For pressure gauge playbooks, see Health and Monitoring.

Scribe exposes the Four Golden Signals for both query and ingest sides.

Five signals tell you whether the query side is healthy: latency, traffic, server errors, client errors, and saturation. Each is available as a pre-computed gauge — no PromQL needed.

Per-channel breakdowns use the channel label (e.g., scribe_signals_channel_latency_p95{channel="ldap"}).

Four signals cover the sync side: task latency, replication lag, change detection rate, and task failure rate. Per-entry-type breakdowns use the entry_type label.

LDAP searches that can’t be satisfied locally get delegated upstream. Track delegation rate with scribe_signals_ldap_delegation_rate_percent and per-reason counts with scribe_ldap_search_delegated_total. See LDAP channel docs for delegation reasons.

The ldap.delegation.scope span event records scope decisions for an individual search. It now also fires when a broad subtree search that names no entry type is answered locally (untyped-search = local) — those events carry a local-resolution cause and a local target, so a dashboard that filters on ldap.delegation.scope will include local-serve decisions alongside delegations. Filter on the cause/target to separate them.

For the full metric inventory with labels, units, and scrape details, see the Telemetry Reference.

Observe JSON endpoints require an operator session or accepted credential by default; the example below uses a bearer token.

set -o pipefail
curl -fsS -H "Authorization: Bearer <token>" http://localhost:8080/observe/signals | jq

Returns a JSON summary of all golden signals.

Before a search runs, IdentityScribe may size the work, check that supporting data is ready, and prepare a borrowed database connection. Observe makes that overhead visible without requiring a custom dashboard.

Connection setup. The evidence-health view shows which connection pools still perform setup for individual requests and which reuse an established session profile. Persistent setup work can point to a deployment-level optimization.

Per-search preparation. A query’s diagnostic report shows the preparation time and whether enough evidence has been collected. Values read “evidence pending” until the first sample arrives, and “not captured” when that request does not collect preparation detail.

Where to see it. Open Observe → Inbox, then inspect a recommendation’s evidence-health panel or the query’s diagnostic report. With trace export on (monitoring.traces.enabled = true), the same preparation work appears on the request timeline.

The supporting measurements are exported for built-in operator features, but their names and internal label sets are not a supported public telemetry contract. Supported public exports remain listed in the generated Telemetry Reference.

Not every error deserves the same response. A user typo shouldn’t page you at 3am. A database timeout should.

Server errors are problems inside Scribe — internal failures, timeouts, resource exhaustion. These affect health status because they mean the service itself has a problem.

Client errors are problems with the request — bad input, missing auth, resources that don’t exist. A spike might mean a misconfigured caller or a scanning attack, but Scribe itself is fine.

NOISY is intentionally separate from DEGRADED. A flood of 404s from a broken client shouldn’t mask a real database problem, and shouldn’t trigger your pager.

The defaults work for most deployments. Adjust them if your traffic pattern is unusual:

monitoring.signals {
    min-requests = 200           # Ignore thresholds below this volume
    server-error-rate.degraded = 0.5
    server-error-rate.critical = 2.0
    client-error-rate.noisy = 10.0
}

Different channels can have different tolerances. LDAP clients often need tighter latency than REST:

channels.ldap.signals = ${monitoring.signals} {
    latency-p99.degraded = 0.3
}

See monitoring.signals for all thresholds.

By default, client errors pass through log rules and may get filtered out. Server errors and auth failures always log — you want to see those regardless.

To log all client errors (useful when debugging a broken caller):

monitoring.log.filter-client-errors = false

PromQL examples live in the reference: PromQL Recipes.