Metrics
- Requests
Response times, errors and traffic.
- Synchronization
Freshness, work and failures.
- Resources
Capacity, waiting work and service health.
Look up metric names, units and labels below.
Next: Signals guide
This page documents all metrics emitted by IdentityScribe. Metric names use OTel dot notation (e.g., scribe.channel.requests.total) which is auto-converted to Prometheus underscore notation for scraping.
Categories
Section titled “Categories”- Application info metrics
- Channel connection metrics
- Channel metrics
- Hints metrics
- Index hints metrics
- Ingest metrics
- MCP tool metrics
- Maintenance metrics
- Query pipeline metrics
- Reconciliation metrics
- Runtime state metrics
- Segment leak detection metrics
- Service metrics
- Store metrics
- JVM / Process
Application info metrics
Section titled “Application info metrics”scribe.info
Section titled “scribe.info”Static application information gauge (always 1). Labels contain name, vendor, and version. Use for deployment inventory dashboards.
| Property | Value |
|---|---|
| Prometheus | scribe_info |
| Type | Gauge |
| Unit | unitless |
scribe.concurrency
Section titled “scribe.concurrency”Configured query concurrency limit. Maximum concurrent queries allowed. Compare with inflight metrics for utilization.
| Property | Value |
|---|---|
| Prometheus | scribe_concurrency |
| Type | Gauge |
| Unit | unitless |
scribe.license.expiration
Section titled “scribe.license.expiration”License expiration timestamp (Unix epoch seconds). Monitor for upcoming license renewals. Value of 0 indicates no license or perpetual.
| Property | Value |
|---|---|
| Prometheus | scribe_license_expiration |
| Type | Gauge |
| Unit | seconds |
scribe.db.connections.active
Section titled “scribe.db.connections.active”Aggregate active DB connections across all pools. Used for scheduling thresholds and capacity planning.
| Property | Value |
|---|---|
| Prometheus | scribe_db_connections_active |
| Type | Gauge |
| Unit | connections |
scribe.db.connections.pending
Section titled “scribe.db.connections.pending”Aggregate pending DB connections (threads waiting) across all pools. Non-zero values indicate connection pool exhaustion.
| Property | Value |
|---|---|
| Prometheus | scribe_db_connections_pending |
| Type | Gauge |
| Unit | connections |
scribe.ldap.connections.active
Section titled “scribe.ldap.connections.active”Aggregate active LDAP connections across all hosts. Used for scheduling thresholds and upstream load tracking.
| Property | Value |
|---|---|
| Prometheus | scribe_ldap_connections_active |
| Type | Gauge |
| Unit | connections |
scribe.ldap.upstream.state
Section titled “scribe.ldap.upstream.state”Whether the directory answers sign-ins that need it. 0 = answering; 1 = degraded, at least three and 10 % of those sign-in checks failed in the last minute; 2 = checking recovery, the next such sign-in is let through as a trial; 3 = unavailable, such sign-ins fail immediately (LDAP 52, HTTP 503). Cached sign-ins, local accounts, sessions and searches are not affected. Drives the auth.ldap.upstream doctor check; reported only when LDAP authentication is configured.
| Property | Value |
|---|---|
| Prometheus | scribe_ldap_upstream_state |
| Type | Gauge |
| Unit | unitless |
scribe.signals.latency.p95
Section titled “scribe.signals.latency.p95”Channel request latency p95 (aggregate, 5-minute window). 95th percentile response time across all channels.
| Property | Value |
|---|---|
| Prometheus | scribe_signals_latency_p95 |
| Type | Gauge |
| Unit | seconds |
scribe.signals.requests.per.second
Section titled “scribe.signals.requests.per.second”Channel requests per second (aggregate, 5-minute window). Request rate across all channels.
| Property | Value |
|---|---|
| Prometheus | scribe_signals_requests_per_second |
| Type | Gauge |
| Unit | requests/second |
scribe.signals.traffic.ratio
Section titled “scribe.signals.traffic.ratio”Traffic ratio vs baseline (aggregate). Current request rate divided by adaptive baseline. Values >2 indicate traffic spike.
| Property | Value |
|---|---|
| Prometheus | scribe_signals_traffic_ratio |
| Type | Gauge |
| Unit | ratio |
scribe.signals.error.rate.percent
Section titled “scribe.signals.error.rate.percent”Channel error rate percentage (aggregate, 5-minute window). Percentage of requests that resulted in errors. (0-100)
| Property | Value |
|---|---|
| Prometheus | scribe_signals_error_rate_percent |
| Type | Gauge |
| Unit | percent |
scribe.signals.client.error.rate.percent
Section titled “scribe.signals.client.error.rate.percent”Client error rate percentage (aggregate, 5-minute window). Percentage of requests that resulted in client errors (caller’s fault). (0-100)
| Property | Value |
|---|---|
| Prometheus | scribe_signals_client_error_rate_percent |
| Type | Gauge |
| Unit | percent |
scribe.signals.server.error.rate.percent
Section titled “scribe.signals.server.error.rate.percent”Server error rate percentage (aggregate, 5-minute window). Percentage of requests that resulted in server errors (system fault). (0-100)
| Property | Value |
|---|---|
| Prometheus | scribe_signals_server_error_rate_percent |
| Type | Gauge |
| Unit | percent |
scribe.signals.ingest.task.duration.p95
Section titled “scribe.signals.ingest.task.duration.p95”Ingest task duration p95 (aggregate, 5-minute window). 95th percentile ingest task processing time.
| Property | Value |
|---|---|
| Prometheus | scribe_signals_ingest_task_duration_p95 |
| Type | Gauge |
| Unit | seconds |
scribe.signals.ingest.lag.max.seconds
Section titled “scribe.signals.ingest.lag.max.seconds”Maximum ingest replication lag across all entry types. Time since last sync from upstream. High values indicate sync backlog.
| Property | Value |
|---|---|
| Prometheus | scribe_signals_ingest_lag_max_seconds |
| Type | Gauge |
| Unit | seconds |
scribe.signals.ingest.changes.per.second
Section titled “scribe.signals.ingest.changes.per.second”Ingest changes per second (aggregate, 5-minute window). Rate of change detection from upstream.
| Property | Value |
|---|---|
| Prometheus | scribe_signals_ingest_changes_per_second |
| Type | Gauge |
| Unit | changes/second |
scribe.signals.ingest.events.per.second
Section titled “scribe.signals.ingest.events.per.second”Ingest events per second (aggregate, 5-minute window). Rate of events written to store.
| Property | Value |
|---|---|
| Prometheus | scribe_signals_ingest_events_per_second |
| Type | Gauge |
| Unit | events/second |
scribe.signals.ingest.failed.rate.percent
Section titled “scribe.signals.ingest.failed.rate.percent”Ingest task failure rate percentage (aggregate, 5-minute window). Percentage of ingest tasks that failed (per task, not per change). (0-100)
| Property | Value |
|---|---|
| Prometheus | scribe_signals_ingest_failed_rate_percent |
| Type | Gauge |
| Unit | percent |
scribe.signals.ldap.delegation.rate.percent
Section titled “scribe.signals.ldap.delegation.rate.percent”LDAP search delegation rate percentage (5-minute window). Percentage of LDAP search requests forwarded to the upstream LDAP backend. (0-100)
| Property | Value |
|---|---|
| Prometheus | scribe_signals_ldap_delegation_rate_percent |
| Type | Gauge |
| Unit | percent |
scribe.signals.channel.latency.p95
Section titled “scribe.signals.channel.latency.p95”Channel request latency p95 by channel.
| Property | Value |
|---|---|
| Prometheus | scribe_signals_channel_latency_p95 |
| Type | Gauge |
| Unit | seconds |
| Dimensions | channel |
scribe.signals.channel.requests.per.second
Section titled “scribe.signals.channel.requests.per.second”Channel requests per second by channel.
| Property | Value |
|---|---|
| Prometheus | scribe_signals_channel_requests_per_second |
| Type | Gauge |
| Unit | requests/second |
| Dimensions | channel |
scribe.signals.channel.error.rate.percent
Section titled “scribe.signals.channel.error.rate.percent”Channel error rate percentage by channel. (0-100)
| Property | Value |
|---|---|
| Prometheus | scribe_signals_channel_error_rate_percent |
| Type | Gauge |
| Unit | percent |
| Dimensions | channel |
scribe.signals.channel.client.error.rate.percent
Section titled “scribe.signals.channel.client.error.rate.percent”Channel client error rate percentage by channel. (0-100)
| Property | Value |
|---|---|
| Prometheus | scribe_signals_channel_client_error_rate_percent |
| Type | Gauge |
| Unit | percent |
| Dimensions | channel |
scribe.signals.channel.server.error.rate.percent
Section titled “scribe.signals.channel.server.error.rate.percent”Channel server error rate percentage by channel. (0-100)
| Property | Value |
|---|---|
| Prometheus | scribe_signals_channel_server_error_rate_percent |
| Type | Gauge |
| Unit | percent |
| Dimensions | channel |
scribe.signals.ingest.entry.task.duration.p95
Section titled “scribe.signals.ingest.entry.task.duration.p95”Ingest task duration p95 by entry type.
| Property | Value |
|---|---|
| Prometheus | scribe_signals_ingest_entry_task_duration_p95 |
| Type | Gauge |
| Unit | seconds |
| Dimensions | entry_type |
scribe.signals.ingest.entry.changes.per.second
Section titled “scribe.signals.ingest.entry.changes.per.second”Ingest changes per second by entry type.
| Property | Value |
|---|---|
| Prometheus | scribe_signals_ingest_entry_changes_per_second |
| Type | Gauge |
| Unit | changes/second |
| Dimensions | entry_type |
scribe.signals.ingest.entry.events.per.second
Section titled “scribe.signals.ingest.entry.events.per.second”Ingest events per second by entry type.
| Property | Value |
|---|---|
| Prometheus | scribe_signals_ingest_entry_events_per_second |
| Type | Gauge |
| Unit | events/second |
| Dimensions | entry_type |
scribe.signals.ingest.entry.failed.rate.percent
Section titled “scribe.signals.ingest.entry.failed.rate.percent”Ingest task failure rate percentage by entry type. (0-100)
| Property | Value |
|---|---|
| Prometheus | scribe_signals_ingest_entry_failed_rate_percent |
| Type | Gauge |
| Unit | percent |
| Dimensions | entry_type |
scribe.signals.ingest.entry.lag.seconds
Section titled “scribe.signals.ingest.entry.lag.seconds”Ingest replication lag by entry type.
| Property | Value |
|---|---|
| Prometheus | scribe_signals_ingest_entry_lag_seconds |
| Type | Gauge |
| Unit | seconds |
| Dimensions | entry_type |
scribe.memory.live_set.bytes
Section titled “scribe.memory.live_set.bytes”Post-GC live-set floor in bytes observed by the memory-leak sensor.
| Property | Value |
|---|---|
| Prometheus | scribe_memory_live_set_bytes |
| Type | Gauge |
scribe.memory.growth.per_hour
Section titled “scribe.memory.growth.per_hour”Rate in bytes per hour at which the post-GC live-set floor is growing.
| Property | Value |
|---|---|
| Prometheus | scribe_memory_growth_per_hour |
| Type | Gauge |
scribe.memory.leak.suspected
Section titled “scribe.memory.leak.suspected”1 when the memory-leak sensor suspects unbounded growth, otherwise 0.
| Property | Value |
|---|---|
| Prometheus | scribe_memory_leak_suspected |
| Type | Gauge |
Channel connection metrics
Section titled “Channel connection metrics”scribe.channel.connections.total
Section titled “scribe.channel.connections.total”Total connections established to protocol channels. Counts new connections (not requests).
| Property | Value |
|---|---|
| Prometheus | scribe_channel_connections_total |
| Type | Counter |
| Unit | connections |
| Dimensions | channel, scribe.socket |
scribe.channel.connections.active
Section titled “scribe.channel.connections.active”Current number of active connections. Open client connections across all channels.
| Property | Value |
|---|---|
| Prometheus | scribe_channel_connections_active |
| Type | Gauge |
| Unit | connections |
| Dimensions | channel, scribe.socket |
scribe.channel.forwarded.total
Section titled “scribe.channel.forwarded.total”Total requests forwarded to upstream directory. Requests that could not be served locally.
| Property | Value |
|---|---|
| Prometheus | scribe_channel_forwarded_total |
| Type | Counter |
| Unit | requests |
| Dimensions | channel, op |
scribe.ldap.search.delegated.total
Section titled “scribe.ldap.search.delegated.total”LDAP search requests delegated to upstream by reason. Tracks delegation breakdown for observability.
| Property | Value |
|---|---|
| Prometheus | scribe_ldap_search_delegated_total |
| Type | Counter |
| Unit | requests |
| Dimensions | ldap.delegation.reason |
Channel metrics
Section titled “Channel metrics”scribe.channel.requests.total
Section titled “scribe.channel.requests.total”Total requests received across all protocol channels. Incremented at the end of each request regardless of outcome. Filter by the result dimension to separate successful from failed requests.
| Property | Value |
|---|---|
| Prometheus | scribe_channel_requests_total |
| Type | Counter |
| Unit | requests |
| Dimensions | channel, op, result |
scribe.channel.request.duration.seconds
Section titled “scribe.channel.request.duration.seconds”Duration of request processing in seconds. Measures wall-clock time from request receipt to response completion. Use percentiles (p50, p95, p99) for latency SLOs.
| Property | Value |
|---|---|
| Prometheus | scribe_channel_request_duration_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | channel, op, result |
scribe.access.rules.evaluation.seconds
Section titled “scribe.access.rules.evaluation.seconds”Duration of access rule evaluation in seconds. Measures time spent evaluating access rules for authorization decisions.
| Property | Value |
|---|---|
| Prometheus | scribe_access_rules_evaluation_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | decision |
scribe.channel.inflight
Section titled “scribe.channel.inflight”Current number of in-flight requests. Useful for detecting request queuing and concurrency limits. A sustained high value may indicate resource exhaustion.
| Property | Value |
|---|---|
| Prometheus | scribe_channel_inflight |
| Type | UpDownCounter |
| Unit | requests |
| Dimensions | channel |
Hints metrics
Section titled “Hints metrics”scribe.hints.queue.size
Section titled “scribe.hints.queue.size”Current number of hints in the persistence queue. Hints are query patterns used for index recommendations. A growing queue indicates high query diversity.
| Property | Value |
|---|---|
| Prometheus | scribe_hints_queue_size |
| Type | Gauge |
| Unit | unitless |
scribe.hints.queue.dropped.total
Section titled “scribe.hints.queue.dropped.total”Total hints dropped due to queue overflow. Indicates the hint queue is saturated. Consider increasing queue capacity if dropping frequently.
| Property | Value |
|---|---|
| Prometheus | scribe_hints_queue_dropped_total |
| Type | Counter |
| Unit | unitless |
scribe.hints.persisted.total
Section titled “scribe.hints.persisted.total”Total hints successfully persisted. Counts query patterns saved for analysis.
| Property | Value |
|---|---|
| Prometheus | scribe_hints_persisted_total |
| Type | Counter |
| Unit | unitless |
scribe.hints.excluded.total
Section titled “scribe.hints.excluded.total”Total hints excluded by persistence rules. Counts hints that matched an exclude rule and were not persisted to database. These hints are still emitted via metrics and logs.
| Property | Value |
|---|---|
| Prometheus | scribe_hints_excluded_total |
| Type | Counter |
| Unit | unitless |
| Dimensions | hint_type |
Index hints metrics
Section titled “Index hints metrics”scribe.hints.index.usage.total
Section titled “scribe.hints.index.usage.total”Total queries that used an index effectively. Filter by category and attribute dimensions.
| Property | Value |
|---|---|
| Prometheus | scribe_hints_index_usage_total |
| Type | Counter |
| Unit | unitless |
| Dimensions | category, attribute |
scribe.hints.index.recommendation.total
Section titled “scribe.hints.index.recommendation.total”Total index recommendations generated. Missing indexes detected from query patterns.
| Property | Value |
|---|---|
| Prometheus | scribe_hints_index_recommendation_total |
| Type | Counter |
| Unit | unitless |
| Dimensions | category, attribute |
scribe.hints.index.seqscan.total
Section titled “scribe.hints.index.seqscan.total”Total queries that triggered sequential scans. Queries without suitable index support. Consider adding indexes for frequent patterns.
| Property | Value |
|---|---|
| Prometheus | scribe_hints_index_seqscan_total |
| Type | Counter |
| Unit | unitless |
| Dimensions | category, attribute |
scribe.hints.seqscan.suppressed.total
Section titled “scribe.hints.seqscan.suppressed.total”Total sequential scan hints suppressed because no missing coverage or reviewable performance impact was found.
| Property | Value |
|---|---|
| Prometheus | scribe_hints_seqscan_suppressed_total |
| Type | Counter |
| Unit | unitless |
Ingest metrics
Section titled “Ingest metrics”scribe.ingest.lag.seconds
Section titled “scribe.ingest.lag.seconds”Current replication lag in seconds per entry type. Time since the last change from the upstream directory was processed. High lag indicates slow synchronization or upstream issues.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_lag_seconds |
| Type | Gauge |
| Unit | seconds |
| Dimensions | entry_type |
scribe.ingest.queue.pressure
Section titled “scribe.ingest.queue.pressure”Current ingest queue utilization ratio (0.0 to 1.0). Ratio of pending changes to queue capacity. Values near 1.0 indicate change processing cannot keep up.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_queue_pressure |
| Type | Gauge |
| Unit | unitless |
| Dimensions | entry_type |
scribe.ingest.task.pressure
Section titled “scribe.ingest.task.pressure”Current ingest task pressure ratio. Ratio of accepted to finished ingest tasks over a sliding window. Values near 1.0 indicate steady state; values above 1.0 indicate backlog growth because arrivals are outpacing completion.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_task_pressure |
| Type | Gauge |
| Unit | unitless |
| Dimensions | entry_type |
scribe.ingest.changes.total
Section titled “scribe.ingest.changes.total”Total directory changes detected from upstream. Counts add, modify, delete, and move events. Filter by entry_type and event_type dimensions.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_changes_total |
| Type | Counter |
| Unit | events |
| Dimensions | entry_type, event_type |
scribe.ingest.events.written.total
Section titled “scribe.ingest.events.written.total”Total change events successfully written to the store. Difference between detected changes and written events indicates filtered or deduplicated entries.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_events_written_total |
| Type | Counter |
| Unit | events |
| Dimensions | entry_type, event_type |
scribe.ingest.tasks.active
Section titled “scribe.ingest.tasks.active”Current number of active ingest tasks. Tasks process batches of directory changes. A sustained high value indicates heavy sync load.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_tasks_active |
| Type | Gauge |
| Unit | tasks |
| Dimensions | entry_type |
scribe.ingest.tasks.throttled
Section titled “scribe.ingest.tasks.throttled”Current number of throttled ingest tasks. Tasks paused due to rate limiting or backpressure. Non-zero values indicate the system is self-regulating.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_tasks_throttled |
| Type | Gauge |
| Unit | tasks |
| Dimensions | entry_type |
scribe.ingest.tasks.total
Section titled “scribe.ingest.tasks.total”Total ingest tasks started. Compare with failed total to calculate success rate.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_tasks_total |
| Type | Counter |
| Unit | tasks |
| Dimensions | entry_type |
scribe.ingest.tasks.failed.total
Section titled “scribe.ingest.tasks.failed.total”Total ingest tasks that failed. Includes retryable and non-retryable failures. Check logs for failure details.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_tasks_failed_total |
| Type | Counter |
| Unit | tasks |
| Dimensions | entry_type, reason |
scribe.ingest.task.duration.seconds
Section titled “scribe.ingest.task.duration.seconds”Duration of ingest task execution. Time to process a batch of changes including DB writes. High durations may indicate database performance issues.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_task_duration_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | entry_type |
scribe.ingest.throttle.wait.seconds
Section titled “scribe.ingest.throttle.wait.seconds”Time spent waiting due to throttling. Cumulative delay from rate limiting. Monitors the cost of backpressure.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_throttle_wait_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | entry_type |
scribe.ingest.unchanged.total
Section titled “scribe.ingest.unchanged.total”Total entries skipped because they were unchanged. Directory reported a change but entry content was identical. High values may indicate noisy upstream notifications.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_unchanged_total |
| Type | Counter |
| Unit | entries |
| Dimensions | entry_type |
scribe.ingest.skipped.total
Section titled “scribe.ingest.skipped.total”Total entries skipped due to filter rules. Entries excluded by transcribe configuration filters.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_skipped_total |
| Type | Counter |
| Unit | entries |
| Dimensions | entry_type |
scribe.ingest.incomplete_notification.total
Section titled “scribe.ingest.incomplete_notification.total”Total incomplete change notifications detected during ingest. A persistent-search MODIFY notification missing objectClass (a structural LDAP invariant every live entry satisfies) is incomplete. reason=repaired counts notifications re-fetched into a complete entry; reason=skipped counts those left for reconciliation to avoid corrupting stored data. Any non-zero value warrants inspecting the source directory for dropped attributes.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_incomplete_notification_total |
| Type | Counter |
| Unit | entries |
| Dimensions | entry_type, reason |
scribe.ingest.queue.blocked.total
Section titled “scribe.ingest.queue.blocked.total”Total times ingest was blocked waiting for queue capacity. Indicates queue saturation events. Frequent blocking suggests increasing queue size.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_queue_blocked_total |
| Type | Counter |
| Unit | events |
| Dimensions | entry_type |
scribe.ingest.queue.wait.seconds
Section titled “scribe.ingest.queue.wait.seconds”Time spent waiting for queue capacity. Measures blocking delay when queues are full.
| Property | Value |
|---|---|
| Prometheus | scribe_ingest_queue_wait_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | entry_type |
MCP tool metrics
Section titled “MCP tool metrics”scribe.mcp.tool.requests.total
Section titled “scribe.mcp.tool.requests.total”MCP tool invocations, grouped by endpoint, tool, and result. Endpoints are mcp and mcp-observe. Tools are describe, help, search, ref, lookup, changes, hotChanges, and observe. Results are ok or a bounded failure classification.
| Property | Value |
|---|---|
| Prometheus | scribe_mcp_tool_requests_total |
| Type | Counter |
| Unit | requests |
| Dimensions | endpoint, tool, result |
scribe.mcp.tool.duration.seconds
Section titled “scribe.mcp.tool.duration.seconds”MCP tool execution duration, grouped by endpoint, tool, and result. Endpoints are mcp and mcp-observe. Tools are describe, help, search, ref, lookup, changes, hotChanges, and observe. Results are ok or a bounded failure classification.
| Property | Value |
|---|---|
| Prometheus | scribe_mcp_tool_duration_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | endpoint, tool, result |
Maintenance metrics
Section titled “Maintenance metrics”scribe.maintenance.duration.seconds
Section titled “scribe.maintenance.duration.seconds”Duration of maintenance operations. Periodic cleanup tasks like vacuuming and statistics refresh.
| Property | Value |
|---|---|
| Prometheus | scribe_maintenance_duration_seconds |
| Type | Histogram |
| Unit | seconds |
scribe.maintenance.failed.total
Section titled “scribe.maintenance.failed.total”Total maintenance operations that failed. Check logs for failure reasons.
| Property | Value |
|---|---|
| Prometheus | scribe_maintenance_failed_total |
| Type | Counter |
| Unit | unitless |
Query pipeline metrics
Section titled “Query pipeline metrics”scribe.query.stage.duration.seconds
Section titled “scribe.query.stage.duration.seconds”Duration of individual query pipeline stages. Helps identify which stage is the bottleneck (normalize, plan, compile, execute). Filter by phase dimension for per-stage analysis.
| Property | Value |
|---|---|
| Prometheus | scribe_query_stage_duration_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | channel, phase |
scribe.query.planner.share
Section titled “scribe.query.planner.share”Fraction of one request’s completed pipeline time spent in planning.
| Property | Value |
|---|---|
| Prometheus | scribe_query_planner_share |
| Type | Histogram |
| Unit | unitless |
| Dimensions | channel |
scribe.query.shapes.total
Section titled “scribe.query.shapes.total”Total queries by shape classification. Tracks query patterns: base lookups, list queries, paginated queries. Useful for understanding query workload characteristics.
| Property | Value |
|---|---|
| Prometheus | scribe_query_shapes_total |
| Type | Counter |
| Unit | requests |
| Dimensions | channel, shape |
scribe.query.permit.pressure
Section titled “scribe.query.permit.pressure”Current query permit utilization ratio (0.0 to 1.0). Indicates how close the system is to query concurrency limits. Values approaching 1.0 suggest imminent query rejection.
| Property | Value |
|---|---|
| Prometheus | scribe_query_permit_pressure |
| Type | Gauge |
| Unit | unitless |
scribe.query.permit.queue
Section titled “scribe.query.permit.queue”Number of threads waiting to acquire query permits. Indicates queuing pressure. High values suggest queries are waiting longer than expected for permits. Monitor alongside permit_pressure to understand concurrency bottlenecks.
| Property | Value |
|---|---|
| Prometheus | scribe_query_permit_queue |
| Type | Gauge |
| Unit | threads |
scribe.query.permit.stall.seconds
Section titled “scribe.query.permit.stall.seconds”Seconds all query permits have stayed in use without a release while stalled. A stall means every permit is in use, a search waited or was turned away since the last release, and no permit was released for monitoring.readiness.query-stall-after. 0 otherwise, including under full load while permits are still released: searches are wedged only while stalled and the instance usually needs a restart. Drives the query.permit_stall doctor check and, unless disabled, fails /readyz and /healthz.
| Property | Value |
|---|---|
| Prometheus | scribe_query_permit_stall_seconds |
| Type | Gauge |
| Unit | seconds |
scribe.query.permit.wait.duration.seconds
Section titled “scribe.query.permit.wait.duration.seconds”Time spent waiting for query execution admission.
| Property | Value |
|---|---|
| Prometheus | scribe_query_permit_wait_duration_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | channel |
scribe.query.connection.held.duration.seconds
Section titled “scribe.query.connection.held.duration.seconds”Query database connection residence time.
| Property | Value |
|---|---|
| Prometheus | scribe_query_connection_held_duration_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | channel |
scribe.query.first_row.duration.seconds
Section titled “scribe.query.first_row.duration.seconds”Time from query execution start to the first delivered result.
| Property | Value |
|---|---|
| Prometheus | scribe_query_first_row_duration_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | channel |
scribe.query.rejected.total
Section titled “scribe.query.rejected.total”Total queries rejected due to resource limits. Indicates backpressure is being applied. Check result dimension for rejection reason (resource_exhausted, deadline_exceeded, etc.).
| Property | Value |
|---|---|
| Prometheus | scribe_query_rejected_total |
| Type | Counter |
| Unit | requests |
| Dimensions | channel, result |
scribe.db.temp_files.rate_5m
Section titled “scribe.db.temp_files.rate_5m”5-minute rate of PostgreSQL temporary-file creations.
| Property | Value |
|---|---|
| Prometheus | scribe_db_temp_files_rate_5m |
scribe.db.temp_bytes.rate_5m
Section titled “scribe.db.temp_bytes.rate_5m”5-minute rate of PostgreSQL temporary bytes written.
| Property | Value |
|---|---|
| Prometheus | scribe_db_temp_bytes_rate_5m |
scribe.db.deadlocks.rate_5m
Section titled “scribe.db.deadlocks.rate_5m”5-minute rate of PostgreSQL deadlocks.
| Property | Value |
|---|---|
| Prometheus | scribe_db_deadlocks_rate_5m |
scribe.db.cache.hit_ratio
Section titled “scribe.db.cache.hit_ratio”PostgreSQL block-cache hit ratio.
| Property | Value |
|---|---|
| Prometheus | scribe_db_cache_hit_ratio |
scribe.db.spills_per_commit_5m
Section titled “scribe.db.spills_per_commit_5m”Derived 5-minute temp-file creations per committed transaction.
| Property | Value |
|---|---|
| Prometheus | scribe_db_spills_per_commit_5m |
Reconciliation metrics
Section titled “Reconciliation metrics”scribe.reconciliation.entries.verified.total
Section titled “scribe.reconciliation.entries.verified.total”Total entries verified during reconciliation. Full-sync operations compare cached entries with upstream.
| Property | Value |
|---|---|
| Prometheus | scribe_reconciliation_entries_verified_total |
| Type | Counter |
| Unit | entries |
| Dimensions | entry_type |
scribe.reconciliation.entries.deleted.total
Section titled “scribe.reconciliation.entries.deleted.total”Total stale entries deleted during reconciliation. Entries that exist locally but not upstream are removed.
| Property | Value |
|---|---|
| Prometheus | scribe_reconciliation_entries_deleted_total |
| Type | Counter |
| Unit | entries |
| Dimensions | entry_type |
scribe.reconciliation.drain.timeouts.total
Section titled “scribe.reconciliation.drain.timeouts.total”Total reconciliation delete batches that exceeded the completion timeout. Use this counter to alert on repeated reconciliation delete batches that are not completing within the bounded timeout.
| Property | Value |
|---|---|
| Prometheus | scribe_reconciliation_drain_timeouts_total |
| Type | Counter |
| Unit | batches |
| Dimensions | entry_type |
scribe.reconciliation.duration.seconds
Section titled “scribe.reconciliation.duration.seconds”Duration of reconciliation operations. Full-sync duration depends on entry count and upstream latency.
| Property | Value |
|---|---|
| Prometheus | scribe_reconciliation_duration_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | entry_type |
Runtime state metrics
Section titled “Runtime state metrics”scribe.runtime_state.flush.bytes
Section titled “scribe.runtime_state.flush.bytes”Bytes written during a runtime_state namespace flush (namespace label).
| Property | Value |
|---|---|
| Prometheus | scribe_runtime_state_flush_bytes |
scribe.plan.work.duration.seconds
Section titled “scribe.plan.work.duration.seconds”Planning work duration grouped by a closed component vocabulary.
| Property | Value |
|---|---|
| Prometheus | scribe_plan_work_duration_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | scribe.plan.component |
scribe.plan.work.invocations.total
Section titled “scribe.plan.work.invocations.total”Planning work invocations grouped by closed component and outcome vocabularies.
| Property | Value |
|---|---|
| Prometheus | scribe_plan_work_invocations_total |
| Type | Counter |
| Unit | invocations |
| Dimensions | scribe.plan.component, outcome |
Segment leak detection metrics
Section titled “Segment leak detection metrics”scribe.telemetry.segment.leaks.total
Section titled “scribe.telemetry.segment.leaks.total”Total segments that exceeded the leak detection threshold. Indicates potential resource leaks in span tracking. Investigate if this counter increases.
| Property | Value |
|---|---|
| Prometheus | scribe_telemetry_segment_leaks_total |
| Type | Counter |
| Unit | unitless |
scribe.telemetry.segment.leaks.ended_after.total
Section titled “scribe.telemetry.segment.leaks.ended_after.total”Total leaked segments that eventually ended (post-leak closure). Segments closed after being flagged as leaks. Useful for distinguishing true leaks from slow operations.
| Property | Value |
|---|---|
| Prometheus | scribe_telemetry_segment_leaks_ended_after_total |
| Type | Counter |
| Unit | unitless |
Service metrics
Section titled “Service metrics”scribe.service.restarts.total
Section titled “scribe.service.restarts.total”Total restarts of internal services. Counts automatic recovery attempts. Frequent restarts indicate instability.
| Property | Value |
|---|---|
| Prometheus | scribe_service_restarts_total |
| Type | Counter |
| Unit | unitless |
| Dimensions | service |
scribe.service.up
Section titled “scribe.service.up”Current service health status (1=up, 0=down). Use for availability dashboards and alerts. A value of 0 indicates the service is not running.
| Property | Value |
|---|---|
| Prometheus | scribe_service_up |
| Type | Gauge |
| Unit | unitless |
| Dimensions | service |
scribe.service.transitions.total
Section titled “scribe.service.transitions.total”Total service state transitions. Counts state changes like starting, running, stopping. Filter by from and to dimensions.
| Property | Value |
|---|---|
| Prometheus | scribe_service_transitions_total |
| Type | Counter |
| Unit | unitless |
| Dimensions | service, from, to |
scribe.service.transition.duration.seconds
Section titled “scribe.service.transition.duration.seconds”Duration of service state transitions. Time spent transitioning between states. Long transitions may indicate initialization issues.
| Property | Value |
|---|---|
| Prometheus | scribe_service_transition_duration_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | service, from, to |
Store metrics
Section titled “Store metrics”scribe.store.commit.duration.seconds
Section titled “scribe.store.commit.duration.seconds”Duration of store commit operations. Time to persist changes to the database. Spikes may indicate transaction contention.
| Property | Value |
|---|---|
| Prometheus | scribe_store_commit_duration_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | entry_type |
scribe.store.commit.wait.duration.seconds
Section titled “scribe.store.commit.wait.duration.seconds”Time spent waiting for commit lock. Serialization delay when multiple writers contend. High values suggest commit throughput bottleneck.
| Property | Value |
|---|---|
| Prometheus | scribe_store_commit_wait_duration_seconds |
| Type | Histogram |
| Unit | seconds |
| Dimensions | entry_type |
JVM / Process
Section titled “JVM / Process”These are standard OTel runtime metrics. See OTel Semantic Conventions for details.
jvm.memory.used
Section titled “jvm.memory.used”| Property | Value |
|---|---|
| Prometheus | jvm_memory_used |
jvm.memory.committed
Section titled “jvm.memory.committed”| Property | Value |
|---|---|
| Prometheus | jvm_memory_committed |
jvm.memory.limit
Section titled “jvm.memory.limit”| Property | Value |
|---|---|
| Prometheus | jvm_memory_limit |
jvm.memory.pressure
Section titled “jvm.memory.pressure”| Property | Value |
|---|---|
| Prometheus | jvm_memory_pressure |
jvm.thread.count
Section titled “jvm.thread.count”| Property | Value |
|---|---|
| Prometheus | jvm_thread_count |
jvm.cpu.count
Section titled “jvm.cpu.count”| Property | Value |
|---|---|
| Prometheus | jvm_cpu_count |
process.uptime
Section titled “process.uptime”| Property | Value |
|---|---|
| Prometheus | process_uptime |