Version History

v2.6.0 (2025-07-21)

Features

LDAP Entry Reconciliation: Automatic reconciliation of entries in the database with the LDAP server.
- The reconciliation is triggered by a maintenance task that runs periodically if either the interval or cron is set. See the reference.conf for the configuration options and defaults.
- One reconciliation run is triggered after the initial sync is complete and the continuous search is started, regardless of the schedule, to ensure that all deletions during down time are caught.
- Enable scheduled reconciliation if:
  - Your LDAP server has unreliable persistent search delete notifications
  - You need guaranteed consistency checks for compliance
  - You experience frequent network partitions
- New Prometheus metrics for reconciliation:
  - scribe_reconciliation_entries_verified_total (counter): Number of entries verified as present in LDAP during reconciliation.
  - scribe_reconciliation_entries_deleted_total (counter): Number of entries deleted (synthetic deletes emitted) during reconciliation.
  - scribe_reconciliation_duration_seconds (summary/timer): Total time taken for a full reconciliation run.
  - scribe_reconciliation_last_run_timestamp_seconds (gauge): Unix timestamp of the last completed reconciliation run.
Maintenance Task Scheduling: Added support for scheduling regular maintenance tasks
- Please check the reference.conf for the configuration options and defaults.
- The following services support maintenance tasks:
  - Database: for vacuuming and re-indexing the database instead of running these during startup
    - New Prometheus metrics for database maintenance:
      - database_maintenance_duration_seconds (timer): Duration of each maintenance run.
      - database_maintenance_failed_total (counter): Number of failed maintenance runs.
      - database_maintenance_last_run_timestamp_seconds (gauge): Unix timestamp of the last attempted maintenance run.
  - Scribe: for automatice reconciliation of entries in the database with the LDAP server.
    - New Prometheus metrics for scribe maintenance: are descibed above
Intelligent Data Compression: Automatic compression of LDAP entry data to reduce storage footprint and network transfer sizes.
- System automatically selects optimal compression algorithm based on data characteristics
- Completely transparent to applications - no configuration changes required
- Up to 70% reduction in storage usage for typical LDAP entries
- New Prometheus metrics track compression effectiveness and performance
LDAP Identity Resolution Reliability: Enhanced the way the system matches and updates LDAP entries, ensuring that even in rare cases—such as after directory restores or renames—entries are always correctly identified and updated.
- Added extra safeguards to prevent rare synchronization issues, further protecting data integrity during both initial and ongoing syncs.
- These improvements make the synchronization process more robust and reliable, especially in complex or high-throughput environments.
Monitoring: Metrics are now refreshed and cached at a regular interval to prevent slow responses.
- Added a new monitoring.prometheus.scrapeInterval configuration property to control the refresh interval. The default is 15 seconds.
```
monitoring.prometheus.scrapeInterval = 15s
```

Fixes

License Verification: Improved license verification to prevent the service from crashing in case of failed license verification due to connectivity issues to third party services like LDAP or databases. The service will now retry up to 5 times.
Attribute Change Descriptions: Fixed an issue where change logs and event descriptions could use lowercased attribute names instead of the configured casing. All change tracking now consistently uses the attribute casing as defined in your configuration, ensuring clarity and compatibility with downstream systems.
Operational Attribute Handling: Resolved a bug that could falsely report the removal of certain attributes (such as createTimestamp, modifyTimestamp, or entryUUID) during entry reconciliation. The system now correctly ignores unobserved operational attributes when comparing entries, preventing spurious removal events and ensuring only relevant changes are tracked.

Breaking Changes

Metrics Optimization: This release includes significant changes to metrics that may impact existing dashboards and monitoring configurations.
- Performance Impact
  - ~50% fewer metrics: Reduced from ~1,600 to ~800 total metrics
  - Memory optimization: Rolling windows with 5-minute expiry to reduce memory usage and provide more accurate metrics
  - Reduced cardinality: Service transitions, LDAP searches optimized
  - Better precision: Summary percentiles vs histogram approximations
  - Lower storage: Fewer time series to store and query, reducing storage requirements
- Monitoring Recommendations
  - Update dashboards to use new metric formats before upgrading
  - Test queries in staging environment first to ensure the new metrics are working as expected
  - Review alerts that depend on histogram buckets to ensure they are still working as expected
  - Monitor memory usage after upgrade (should decrease)
Detailed Changes
- Service Transition Metrics
  
  Before:
```
service_transition_seconds{service="X",from="new",to="starting",...}
service_transition_seconds{service="X",from="starting",to="running",...}
service_transition_seconds{service="X",from="running",to="failed",...}
```
  After:
```
service_transition_seconds{service="X",type="startup",...}
service_transition_seconds{service="X",type="restart",...}
service_transition_seconds{service="X",type="failure",...}
```
  Changes:
  - Removed: from and to tags (high cardinality)
  - Added: type tag with meaningful categories (startup, restart, failure, shutdown)
  - Filtered: Startup noise transitions (new → starting)
  - Percentiles: Reduced from 5 to 2 percentiles (0.5, 0.95)
  Migration:
  - Update dashboards to use type instead of from/to tags
  - Categories: startup, restart, failure, shutdown
  Terminal window
```
# Old query
service_transition_seconds{from="starting",to="running"}

# New query
service_transition_seconds{type="startup"}
```
- LDAP Search Metrics
  
  Before: Histogram with 50+ buckets
```
channel_ldap_search_time_seconds_bucket{...,le="0.001"} 0
channel_ldap_search_time_seconds_bucket{...,le="0.002"} 1
[... 50+ buckets ...]
```
  After: Summary with optimized percentiles
```
channel_ldap_search_time_seconds{...,quantile="0.5"} 0.043
channel_ldap_search_time_seconds{...,quantile="0.95"} 0.051
channel_ldap_search_time_seconds{...,quantile="0.99"} 0.052
```
  Changes:
  - Format: Histogram → Summary
  - Percentiles: Reduced to 0.5, 0.95, 0.99 (high cardinality optimization)
  - Buckets: Removed all histogram buckets (50+ → 0)
  Migration:
  - Replace histogram_quantile() with direct quantile label access
  - Update SLI/SLO calculations to use summary percentiles
  Terminal window
```
# Old query
histogram_quantile(0.95, rate(channel_ldap_search_time_seconds_bucket[5m]))

# New query
channel_ldap_search_time_seconds{quantile="0.95"}
```
- Processing Time Metrics
  
  Before: Histogram with SLO buckets
```
scribe_processing_time_seconds_bucket{entryType="user",phase="diffing",le="0.001"} 5
scribe_processing_time_seconds_bucket{entryType="user",phase="diffing",le="0.002"} 25
[... many buckets ...]
```
  After: Summary with full percentiles
```
scribe_processing_time_seconds{entryType="user",phase="diffing",quantile="0.5"} 0.004
scribe_processing_time_seconds{entryType="user",phase="diffing",quantile="0.75"} 0.007
scribe_processing_time_seconds{entryType="user",phase="diffing",quantile="0.9"} 0.015
scribe_processing_time_seconds{entryType="user",phase="diffing",quantile="0.95"} 0.050
scribe_processing_time_seconds{entryType="user",phase="diffing",quantile="0.99"} 0.487
```
  Changes:
  - Format: Histogram → Summary
  - Percentiles: Added 0.5, 0.75, 0.9, 0.95, 0.99 (standard set)
  - Buckets: Removed all histogram buckets
  Migration:
  - Update queries to use quantile labels instead of histogram_quantile()
  - SLIs can now use direct percentile values
  Terminal window
```
# Old query
histogram_quantile(0.90, rate(scribe_processing_time_seconds_bucket[5m]))

# New query
scribe_processing_time_seconds{quantile="0.9"}
```
- New Entry Codec Metrics
  
  Added new compression and entropy metrics:
```
scribe_entry_encode_bytes{codec="lz4|zstd|none",kind="raw|compressed",quantile="0.5"} 1440
scribe_entry_decode_bytes{codec="lz4|zstd|none",kind="raw|compressed",quantile="0.5"} 1440
scribe_entry_encode_compression{codec="lz4|zstd",le="5.0"} 4
scribe_entry_encode_entropy{le="25.0"} 137598
```
  Features:
  - Encode/Decode sizes: Track compression effectiveness by codec
  - Compression percentage: Business-critical buckets at 85% and 95% thresholds
  - Entropy analysis: Optimized buckets for compression decision making
  - Percentiles: Standard set 0.5, 0.75, 0.9, 0.95, 0.99

v2.5.0 (2025-06-26)

Features

LDAP Socket Tuning: Added support for tuning the LDAP socket settings using environment variables and configuration properties, with optimized defaults for production environments.
Monitoring: Use dynamic balanced thread pool for metrics scraping to improve performance.

Fixes

Query Syntax Correction: Removed invalid USE INDEX hint from LDAP presence filters.
Prometheus Tag Standardization: All scribe_events_count metrics now use a consistent set of tag keys (event, op, target) to comply with Prometheus requirements. For event types where op or target are not applicable, the value "none" is used. This applies to event=add, event=move, and event=delete. This resolves meter registration errors and ensures reliable metric collection and querying.

v2.4.0 (2025-06-25)

Features

Faster and More Reliable Lookups: New database indexes have been added to speed up directory entry searches, especially for large datasets.
Binary LDAP Entry Serialization: Storing LDAP entries in a binary format, for faster processing during sync and retrieval
Optimized SQL Query Performance: SQL queries are now dynamically constructed for highly selective searches, ensuring PostgreSQL always uses the most efficient query plan.
Initial Sync Performance: Improved initial synchronization speed by removing unnecessary locking.
Sync State Query Optimization: Enhanced synchronization speed by optimizing queries that determine the state of entries

Fixes

Service Info: Improved scribe service info to include the starting state as a healthy state, preventing health check failures during slow startups.
Health Check: Shows the correct service name in the health check response.
Monitoring: Suppressed noisy error logs for client disconnects (e.g., “Broken pipe”, “Connection reset”) on metrics and health endpoints. These are now logged at debug level instead of error, reducing log noise from normal client disconnects.
Scribe Service: Improved error handling and logging in case of disconnects for scribe service.
Scribe Service: After a restart, the scribe service will now continue with the incremental sync instead of the initial sync.

Breaking

Attribute Sets Removed: Attribute sets are no longer supported and should be removed from the configuration

When removing attributeSets from your configuration, make sure to add all attributes that were previously only listed in attributeSets to the main attributes configuration. This ensures that all attributes are still observed and processed as before.

# Before (deprecated)
attributeSets = [
 "uid, mail, displayName"
]
attributes = "cn sn"

# After (supported)
attributes = """
 cn, sn
 uid, mail, displayName
"""

Removed Prometheus Metrics: This version removes the following Prometheus metrics
scribe.entries.refresh.cache.count
scribe.entries.refresh.cache.time

v2.3.0 (2025-05-15)

Fixes

Improved initial data synchronization speed and prevented Out-Of-Memory errors by implementing a high-watermark memory management strategy.
Eliminated LDAP message size limits to prevent failures with large LDAP server responses.
Optimized Database connection pool handling to prevent failed acquisitions.
Enhanced memory and CPU usage metrics for more accurate monitoring.
Improved performance of health and healthy check responses.
Streamlined logging levels and messages for better clarity and reduced noise.

Breaking

Removed built-in Prometheus metrics for PostgreSQL to mitigate excessive database load. For production environments, using the dedicated Prometheus Exporter is now recommended.

v2.2.0 (2025-03-21)

Features

implement size categorization for entries_data with optimized indexing and monitoring view
materialized views for improved performance and type handling

Fixes

enhance service orchestration and database upgrade coordination
correct LDAP channel listen
enhance memory and CPU usage metrics
improve health and healthy check response

v2.1.2 (2025-02-18)

Fixes

validate elements in attribute set to ensure they are strings in Settings class

v2.1.1 (2025-02-17)

Fixes

ensure postgres ssl config works
loading of referenced files

v2.1.0 (2024-07-05)

Fixes

relative file/path handling for file/path references in the configuration
prevent ssl.enabled from being propagated to the database configuration

v2.0.0 (2024-03-30)

Breaking

new licenses must be generated for the new version of the software
the data column on the entries table has been removed and replaced with a entries_data table that is partitioned by the uoid column
existing data will be migrated to the new table
use timestamps without time zone
new events will use createdAt and updatedAt properties in ISO8601 UTC instead of createTimestamp and modifyTimestamp properties
entries tables data column has been removed in favor of the entries_data table
entries tables meta column has been removed in favor of the uuid, dn, etag, createdAt, and updatedAt columns
changed ldap.idlePeriodForCheckpoint -> ldap.idlePeriod with the default of 5 seconds

What’s new

the uuid and dn columns have been added to the entries
use product based detection to resolve the attribute names of entryUUID, createTimestamp, modifyTimestamp, and eTag attributes
objectClass and structuralObjectClass are always observed
new entries_data tables which holds the data of the entries as attribute-value pairs
use more bytes for the hash for the etag calculation to reduce the chance of collisions
entries.dn lookups are now case-insensitive
use new commits table instead of transactions ids to prevent collisions when the transactions ids roll-over
the modifyTimestamp and revision attributes are no longer used for continuous searches
this prevents change notifications from being sent for every change to an entry
allow to define sets of attributes to pre-compute their data which can be used to speed up search responses
allow to define attributes which should be indexed for optimized partial matching (substring and approximate) searches
new channels to access the data via LDAP
only search operations where all attributes (filter, sort, and requested attributes) are observed will be executed
supported search controls are Simple Paged Results, Server Side Sorting, and Virtual List View
additionally to the well known scopes (base, one, and sub) the subordinate_subtree (3) scope is supported — it indicates that any subordinate entries (to any depth) below the entry specified by the base DN should be considered, but the base entry itself should not be considered, as described in draft-sermersheim-ldap-subordinate-scope
additionally to the well known filters (and, or, not, equality, substring, greaterOrEqual, lessOrEqual, and present) the two following filters are supported:
- approximate: used to determine whether an entry contains at least one value for a specified attribute that is approximately equal to a given value — for example, a filter of (givenName~=John) will match entries with givenName values of either John or Jon.
- extensible dn match: used to determine whether an entry contains at least one value for a specified attribute that matches a given value, where the match is based on the DN of the entry — for example, a filter of (ou:dn:=Engineering) will match entries with an ou attribute value of Engineering that are subordinate to the base DN.
the etag attribute is based on the values of the observed attributes — if any of the observed attributes change, the etag will change
all other operations will be forwarded to the source LDAP server
detailed metrics are available via the prometheus endpoint
to only use channels (eg load balancing) start the application with one of the following options:
- the --readonly (-r) flag
- IDENTITY_SCRIBE_READONLY=1, IDENTITY_SCRIBE_READONLY=true, or IDENTITY_SCRIBE_READONLY=yes environment variable
- IDENTITY_SCRIBE_TRANSCRIBE_ENABLED=0, IDENTITY_SCRIBE_TRANSCRIBE_ENABLED=false, or IDENTITY_SCRIBE_TRANSCRIBE_ENABLED=no environment variable

Fixes

do not emit events with empty operations
auto-generated table partitions are now created with the correct range (off-by-one error)
renaming an entry now correctly updates the naming attributes values
all enviroment variables can now be set either via system environment variables with prefix IDENTITY_SCRIBE_<...> or using command line properties -D<...> — for example IDENTITY_SCRIBE_READONLY=1 identity-scribe or identity-scribe -DREADONLY=1
issued licenses are now valid when used with clustered LDAP servers