Logging and Traces
- Logs
Find failures, warnings and slow operations.
- Traces
Inspect the time spent inside one operation.
You can choose the evidence needed to investigate a problem and keep its correlation identifiers.
Next: Error handling
You saw something in the signals. Now you need to find out what happened.
IdentityScribe uses two investigation tools: wide event logs that capture one structured log line per operation, and distributed traces via OpenTelemetry. Both work together — wide logs include trace IDs, and traces link back to the operation that produced them.
Wide logs are your first stop. They show failures, warnings, and slow operations without any extra infrastructure. Traces go deeper — they break each operation into stages so you can see exactly where time is spent. Traces require a backend (Tempo, Jaeger) but give you the full picture when logs aren’t enough.
Process output capture
Section titled “Process output capture”IdentityScribe writes operational output through two paths:
| Output path | Controlled by | What it contains |
|---|---|---|
| Application logger stream | log.file, --logfile, SCRIBE_LOG_FILE | Wide event logs and logger categories |
| Process stdout and stderr | Container runtime, service manager, shell, or log agent | Startup banner, command-line diagnostics, startup warnings, and failure summaries |
log.file, --logfile, and SCRIBE_LOG_FILE route only the application logger stream to System.out (the default),
System.err, or a file. Collect both stdout and stderr at the deployment boundary.
| Environment | Live access | Retention and support evidence |
|---|---|---|
| Kubernetes | kubectl logs deployment/identity-scribe --all-containers -f | Cluster logging, node log rotation, or a kubectl logs export for the incident window |
| Linux system service | journalctl -u identity-scribe -f | Persistent journal storage or a journalctl export for the incident window |
| File-based service policy | Host service manager, wrapper, or log agent that captures both stdout and stderr | A rotated file owned by the host logging policy |
| Manual shell | `./identity-scribe 2>&1 | tee -a identity-scribe.log` |
For a file without console output, redirect both streams:
./identity-scribe > identity-scribe.log 2>&1For a file and live console output, use tee:
./identity-scribe 2>&1 | tee -a identity-scribe.logWide event logging
Section titled “Wide event logging”IdentityScribe accumulates context throughout each request, then emits a single structured log line when the operation finishes — but only when it’s worth looking at.
An operation always logs when it:
- Failed on the server side or with a security-relevant failure, such as a rejected sign-in or denied access → ERROR level
- Had warnings or exceptions → WARN level
- Was explicitly marked noteworthy
Everything else, including client errors such as invalid input by default, goes through the log rules. The default rules suppress fast successful operations and fast client errors, so slow operations and the remaining failures still log. That’s by design — it keeps logs focused on things that need your attention.
Logger categories
Section titled “Logger categories”Five loggers, all under com.kenoxa.scribe.*:
| Logger | Env Override | Purpose |
|---|---|---|
SuperVisor | SCRIBE_LOG_SUPERVISOR | Startup, shutdown, orchestration |
Ingest | SCRIBE_LOG_INGEST | Transcription pipeline, event store |
Monitoring | SCRIBE_LOG_MONITORING | Wide-log output |
License | SCRIBE_LOG_LICENSE | License verification |
Config | SCRIBE_LOG_CONFIG | Configuration parsing |
Each category logs at info, or at log.level when that is more verbose. So --debug, --trace or a debug
log.level raise them, while --quiet or a warn level keeps their operation logs. To quiet one, set it explicitly:
SCRIBE_LOG_MONITORING=off # Disable wide logs entirelySCRIBE_LOG_INGEST=debug # Verbose transcription debuggingSCRIBE_LOG_LICENSE=error # Quiet license checksLibrary log levels
Section titled “Library log levels”Libraries log at warn by default. Raise one to diagnose a problem in that area:
| Library | Setting | Env Override |
|---|---|---|
| Kenoxa internal | log.com.kenoxa | SCRIBE_LOG_KENOXA |
| Connection pool (HikariCP) | log.com.zaxxer.hikari | SCRIBE_LOG_HIKARI |
| Database migrations (Flyway) | log.org.flywaydb | SCRIBE_LOG_FLYWAY |
| JWT and JWK handling (jose4j) | log.org.jose4j | SCRIBE_LOG_JOSE4J |
| Web server (Helidon) | log.io.helidon | SCRIBE_LOG_HELIDON |
SCRIBE_LOG_HIKARI=debug # Connection pool activitySee log for all logging settings.
Log format
Section titled “Log format”Pretty format (default in dev mode with a TTY):
LDAP.Search ok dur=1.5s warnings=1 events=5 trace=abc123 span=def456 db: system=postgresql duration.seconds=0.42 row_count=42 scribe: entry_type=user search.kind=paged scribe.query: scope="ou=users,o=org" where="(&(objectClass=*))" sort="cn" events: + 5.1ms 12.3ms Query.Plan warning code=unsupported_control message="VLV not available" + 18.0ms 401.9ms DB.Fetch row_count=42JSON format (default in production):
{“trace_id”:"abc123","span_id":"def456","duration_seconds":1.5,"result":"ok","scribe.operation":"LDAP.Search",...}Failure output includes full details and stacktrace:
scribe REST.Modify internal dur=311ms trace=trace123 span=span456 failure: kind=INTERNAL code=SERVER_ERROR message="Database connection failed" cause_type=java.sql.SQLException cause_message="Connection refused" stacktrace: java.sql.SQLException: Connection refused at org.postgresql.Driver.connect(Driver.java:285)Color detection uses TERM, FORCE_COLOR, NO_COLOR, and TTY presence. Disable colors with NO_COLOR=1.
Configuration
Section titled “Configuration”monitoring.log { enabled = true # default format = "pretty" # pretty | json | auto sample-rate = 100 # 0-100%, for ops that don't match any rule
# Redaction strategies (glob patterns for keys) redaction { "*dn*" = hash # Marked short hash: [hash:...] "*email*" = truncate # Marked prefix/suffix: [trunc:...] "auth.subject_id" = hash # Correlate a subject without the raw identifier "auth.username" = hash # Correlate a username without the raw name "client.address" = hash # Correlate a client without the raw address "*.raw" = replace # Replace with [REDACTED] } # Always replaced: passwords, credentials, tokens, secrets, API keys, # authorization headers, and cookies. # Authentication method, issuer, key id, and failure reason are shown by default.}Rule-based filtering
Section titled “Rule-based filtering”Rules control which operations get logged. They evaluate in order — first match wins. Operations without a matching rule fall through to random sampling.
Client errors go through the rules while monitoring.log.filter-client-errors = true (the default); set it to false to log every client error. Sign-ins that fail fast while the directory is down are counted, not logged one by one: they log only when a rule includes them.
Rule syntax:
monitoring.log.rules = [ { id = "suppress-fast-ok", action = exclude, where = "scribe.result=ok duration.seconds<=50ms" } { id = "log-slow", action = include, where = "duration.seconds>=5s" } { action = exclude, where = "scribe.operation~=Hints.* duration.seconds<=100ms" } { action = exclude, where = "scribe.operation~=Metrics.* duration.seconds<=500ms" }]| Field | Required | Description |
|---|---|---|
action | Yes | include or exclude |
where | No | LDAP-style filter on attributes; use scribe.operation for operation matching |
id | No | Identifier for debugging (defaults to Rule #N) |
Duration values accept plain seconds (0.05), HOCON style (50ms, 5s), or ISO 8601 (PT5S).
Common patterns:
{ action = include, where = "scribe.operation~=LDAP.*" } # All LDAP ops
{ action = exclude, where = "(&(scribe.result=ok)(duration.seconds<=50ms))" } # Fast successes
{ action = include, where = "(db.duration.seconds>=1s)" } # Slow DB queries
{ action = exclude } # Catch-allSee monitoring.log.rules for the complete reference. For all logging configuration options (redaction, leak detection, shutdown noise), see monitoring.log.
Segment tracking
Section titled “Segment tracking”Child segments can appear as events in wide logs, giving you an operation breakdown without opening a trace viewer.
| Mode | What’s captured | Overhead |
|---|---|---|
auto | Full in dev, minimal in prod (default) | Varies |
off | Nothing | None |
minimal | Name, offset, duration | Low |
full | Name, offset, duration, attributes | Moderate |
monitoring.log.childs { mode = "auto" rules = [ { action = include, where = "(duration.seconds>=1ms)" } { action = exclude } ]}Events appear in start-time order (not end-time), so the log reads chronologically. Segment attributes follow the same redaction rules as operation-level attributes.
Time LDAP writes
Section titled “Time LDAP writes”Add, Modify and Delete requests that IdentityScribe forwards to the directory log as LDAP.Add, LDAP.Modify and LDAP.Delete; a rename (Modify DN) logs as LDAP.Modify. The default rules drop successful LDAP operations up to 500 ms, so to log every write, put these rules before the defaults:
monitoring.log.rules = [ { id = "time-ldap-writes", action = include, where = "scribe.operation=LDAP.Modify" } { action = include, where = "scribe.operation=LDAP.Add" } { action = include, where = "scribe.operation=LDAP.Delete" }] ${monitoring.log.rules}Each write logs its result and ldap.result_code, and an LDAP.Delegation event: the time from handing the request to the directory until the directory answered. The rest of the operation’s duration is IdentityScribe’s own work, such as access rules and proxy authorization. When the client connection has no directory connection yet, the event includes connecting and signing in to the directory; with monitoring.log.childs.mode = "full" (SCRIBE_LOG_CHILDS_MODE=full) it then shows ldap.delegation.new_connection=true. A write whose LDAP.Delegation takes nearly all of its duration spent its time in the directory or on the network to it.
The same writes are counted in scribe_channel_request_duration_seconds with op="add", op="modify" or op="delete".
Trace layout
Section titled “Trace layout”When tracing is enabled, each query produces a span hierarchy:
LDAP.Search (or REST.Search, etc.) ← Channel entry span└── Query.Normalize ← Normalization└── Query.Plan ← Planning└── Query.Compile ← builds the database query└── Query.Execute ← DB execution + streamingSpan attributes
Section titled “Span attributes”| Attribute | Description | Example |
|---|---|---|
scribe.result | Outcome | ok, cancelled, deadline_exceeded |
scribe.search.kind | Pagination mode | simple, paged, vlv |
scribe.query.signature | Query signature hash | a1b2c3d4 |
scribe.entry_type | Entry type(s) in scope | inetOrgPerson |
Phase names
Section titled “Phase names”The phase label in scribe.query.stage.duration.seconds maps directly to span names:
| Phase | Span Name | What it does |
|---|---|---|
normalize | Query.Normalize | Attribute mapping, filter canonicalization |
plan | Query.Plan | Index selection, predicate classification |
compile | Query.Compile | Logical plan → SQL |
execute | Query.Execute | database execution, result streaming |
Latency debugging with traces
Section titled “Latency debugging with traces”- Find the slow stage — look at
scribe_query_stage_duration_secondsby phase - Get a sample trace — filter by trace ID or high duration
- Drill into Execute — if it’s slow, check for lock contention, missing indexes (
/observe/hints), or pool saturation (scribe_db_connections_pending > 0) - Check Plan — if it’s slow, the query may be too complex
Enabling tracing
Section titled “Enabling tracing”monitoring.traces { enabled = true endpoint = "http://otel-collector:4317" protocol = "grpc" # or "http/protobuf" for port 4318}Or via environment variables:
export SCRIBE_TRACES_ENABLED=trueexport SCRIBE_TRACES_ENDPOINT=http://otel-collector:4317Wide event fields
Section titled “Wide event fields”| Field | Type | Description |
|---|---|---|
trace_id | string | OpenTelemetry trace ID (when tracing enabled) |
span_id | string | OpenTelemetry span ID |
duration_seconds | double | Operation duration |
result | string | ok or failure kind |
failure | object | Details when failed: kind, code, message, stacktrace |
events | array | Warnings and segment timing |
scribe.operation | string | Operation name (e.g., LDAP.Search, REST.GET) |
Support collection
Section titled “Support collection”When opening a support case, collect the output for the incident window around the first error. Include the command used to collect the file and the local timezone of the timestamps.
Kubernetes:
kubectl logs deployment/identity-scribe --all-containers --timestamps --since=2h > identity-scribe.logIf the pod restarted during the incident, collect the previous container output as well:
kubectl logs pod/<pod-name> -c identity-scribe --previous --timestamps --since=2h >> identity-scribe.logLinux system service:
journalctl -u identity-scribe --since "<start-time>" --until "<end-time>" -o short-iso > identity-scribe.logFile-based service policy:
cp /var/log/identity-scribe/identity-scribe.log identity-scribe.logManual shell:
Use the file produced by the redirection or tee command above.