Skip to content

Change Detection

  • Live changes

    Notifications or polling pick up directory changes.

  • Reconciliation

    A separate check finds changes that were missed.

Live detection and reconciliation work together to keep entries current.

You can understand what affects freshness and which settings are relevant to your directory.

Next: Monitoring freshness

Scribe picks up live changes with persistent search where the directory supports it and with polling where it does not. Reconciliation runs separately and checks for anything live detection missed.

LIVE · ONE OFVERIFICATIONLDAPdirectoryPersistent searchif supportedPollingif not supportedStartup or scheduleReconciliationfull scan · when idleTranscriptionPostgreSQLpushesis polledchangeschangesstartsis scanneddifferencesappends
LIVE · ONE OFVERIFICATIONLDAPdirectoryPersistent searchif supportedPollingif not supportedStartup or scheduleReconciliationfull scan · when idleTranscriptionPostgreSQLpushesis polledchangeschangesstartsis scanneddifferencesappends
Persistent search or polling supplies live changes; reconciliation separately checks what they may have missed.

This page covers each mechanism, when it runs, and the knobs that affect sync lag.

For the high-level data flow, see Architecture. For tuning worker counts and queue sizes, see Transcribes configuration.

When the LDAP directory supports the persistent search control, Scribe registers for real-time change notifications. The directory pushes changes as they happen.

This is the fastest path. Lag is typically sub-second — bounded by network round-trip and PostgreSQL write latency. Scribe holds the connection open and reconnects automatically if it drops.

Not all directories support persistent search, and some only support it for certain subtrees. When persistent search isn’t available for a transcribe, Scribe falls back to polling.

Scribe periodically queries the directory for entries modified since the last check. The idle period between polls defaults to 5 seconds (ldap.idle-period). The effective interval is the search duration plus that idle period.

Lag equals the poll interval plus processing time. For a directory with moderate change rates (hundreds of changes per interval), processing adds negligible overhead. For high-rate sources (thousands of changes per poll), consider lowering the interval or increasing workers.

Polling uses modifyTimestamp or an equivalent attribute to detect changes. It sees only the state at each poll: a value that was changed and changed back between two polls is never recorded, and reconciliation cannot recover it either.

Reconciliation is the safety net. It walks the entire directory tree for a transcribe and compares each entry against the local copy, so anything that persistent search or polling missed gets picked up here.

Typical triggers for missed changes:

  • Network partition during a persistent search connection
  • Directory failover to a replica with slightly different state
  • Entries modified while Scribe was shut down
  • Clock skew causing modifyTimestamp queries to miss a window

Reconciliation is pressure-aware. It waits for the system to be idle — low ingest queue pressure, no active tasks consuming significant resources — before starting a full scan. This prevents reconciliation from competing with real-time change processing during peak load.

After startup, Scribe runs an initial reconciliation to catch anything that changed while it was down. Scheduled runs happen only when you set ldap.reconciliation.interval or ldap.reconciliation.cron (there is no default interval).

Every entry has an attribute hash (ETag) computed from its current attribute values.

Directory entryscannedCompare ETagwith stored copyNo new eventcheck recordedTranscriptionrecords changeETagequaldifferent
Directory entryscannedCompare ETagwith stored copyNo new eventcheck recordedTranscriptionrecords changeETagequaldifferent
An unchanged ETag avoids a new change event during reconciliation; only entries that differ are transcribed.

When the ETags match, Scribe writes no change event and only records that the entry was verified. A full scan of a directory where nothing changed writes no change events.

The gap between a change happening in the directory and that change being queryable through Scribe depends on several factors:

FactorImpactTuning
Persistent search supportSub-second lag when available; falls back to poll interval otherwiseDirectory-dependent — enable the persistent search control on the directory if possible
Poll intervalDirectly adds to lag for polled transcribesldap.idle-period
Worker countMore workers process changes faster under loadtranscribes.<name>.transcription.workers
PostgreSQL write throughputBottleneck under high change ratesConnection pool size, disk I/O, routine maintenance
Network latency to LDAPAdds to each poll and persistent search reconnectNetwork topology

Monitor lag with scribe_ingest_lag_seconds per entry type. Sustained lag above 60 seconds warrants investigation — see Monitoring for the diagnostic workflow.