Skip to content

Logging and Traces

  • Logs

    Find failures, warnings and slow operations.

  • Traces

    Inspect the time spent inside one operation.

Start with logs; add traces when you need a detailed timing breakdown.

You can choose the evidence needed to investigate a problem and keep its correlation identifiers.

Next: Error handling

You saw something in the signals. Now you need to find out what happened.

IdentityScribe uses two investigation tools: wide event logs that capture one structured log line per operation, and distributed traces via OpenTelemetry. Both work together — wide logs include trace IDs, and traces link back to the operation that produced them.

Wide logs are your first stop. They show failures, warnings, and slow operations without any extra infrastructure. Traces go deeper — they break each operation into stages so you can see exactly where time is spent. Traces require a backend (Tempo, Jaeger) but give you the full picture when logs aren’t enough.

IdentityScribe writes operational output through two paths:

Output pathControlled byWhat it contains
Application logger streamlog.file, --logfile, SCRIBE_LOG_FILEWide event logs and logger categories
Process stdout and stderrContainer runtime, service manager, shell, or log agentStartup banner, command-line diagnostics, startup warnings, and failure summaries
SCRIBE PROCESSApplication loggerwide event logsProcess diagnosticsstartup · CLILog fileonly if log.filestdout + stderrLog collectore.g. journaldwriteslog.filealways writescaptures
SCRIBE PROCESSApplication loggerwide event logsProcess diagnosticsstartup · CLILog fileonly if log.filestdout + stderrLog collectore.g. journaldwriteslog.filealways writescaptures
Capturing both stdout and stderr preserves process diagnostics beyond the application logger.

log.file, --logfile, and SCRIBE_LOG_FILE route only the application logger stream to System.out (the default), System.err, or a file. Collect both stdout and stderr at the deployment boundary.

EnvironmentLive accessRetention and support evidence
Kuberneteskubectl logs deployment/identity-scribe --all-containers -fCluster logging, node log rotation, or a kubectl logs export for the incident window
Linux system servicejournalctl -u identity-scribe -fPersistent journal storage or a journalctl export for the incident window
File-based service policyHost service manager, wrapper, or log agent that captures both stdout and stderrA rotated file owned by the host logging policy
Manual shell`./identity-scribe 2>&1tee -a identity-scribe.log`

For a file without console output, redirect both streams:

./identity-scribe > identity-scribe.log 2>&1

For a file and live console output, use tee:

Terminal window
./identity-scribe 2>&1 | tee -a identity-scribe.log

IdentityScribe accumulates context throughout each request, then emits a single structured log line when the operation finishes — but only when it’s worth looking at.

An operation always logs when it:

  • Failed on the server side or with a security-relevant failure, such as a rejected sign-in or denied access → ERROR level
  • Had warnings or exceptions → WARN level
  • Was explicitly marked noteworthy

Everything else, including client errors such as invalid input by default, goes through the log rules. The default rules suppress fast successful operations and fast client errors, so slow operations and the remaining failures still log. That’s by design — it keeps logs focused on things that need your attention.

Five loggers, all under com.kenoxa.scribe.*:

LoggerEnv OverridePurpose
SuperVisorSCRIBE_LOG_SUPERVISORStartup, shutdown, orchestration
IngestSCRIBE_LOG_INGESTTranscription pipeline, event store
MonitoringSCRIBE_LOG_MONITORINGWide-log output
LicenseSCRIBE_LOG_LICENSELicense verification
ConfigSCRIBE_LOG_CONFIGConfiguration parsing

Each category logs at info, or at log.level when that is more verbose. So --debug, --trace or a debug log.level raise them, while --quiet or a warn level keeps their operation logs. To quiet one, set it explicitly:

Terminal window
SCRIBE_LOG_MONITORING=off # Disable wide logs entirely
SCRIBE_LOG_INGEST=debug # Verbose transcription debugging
SCRIBE_LOG_LICENSE=error # Quiet license checks

Libraries log at warn by default. Raise one to diagnose a problem in that area:

LibrarySettingEnv Override
Kenoxa internallog.com.kenoxaSCRIBE_LOG_KENOXA
Connection pool (HikariCP)log.com.zaxxer.hikariSCRIBE_LOG_HIKARI
Database migrations (Flyway)log.org.flywaydbSCRIBE_LOG_FLYWAY
JWT and JWK handling (jose4j)log.org.jose4jSCRIBE_LOG_JOSE4J
Web server (Helidon)log.io.helidonSCRIBE_LOG_HELIDON
Terminal window
SCRIBE_LOG_HIKARI=debug # Connection pool activity

See log for all logging settings.

Pretty format (default in dev mode with a TTY):

LDAP.Search ok dur=1.5s warnings=1 events=5
trace=abc123 span=def456
db: system=postgresql duration.seconds=0.42 row_count=42
scribe: entry_type=user search.kind=paged
scribe.query: scope="ou=users,o=org" where="(&(objectClass=*))" sort="cn"
events:
+ 5.1ms 12.3ms Query.Plan
warning code=unsupported_control message="VLV not available"
+ 18.0ms 401.9ms DB.Fetch row_count=42

JSON format (default in production):

{“trace_id”:"abc123","span_id":"def456","duration_seconds":1.5,"result":"ok","scribe.operation":"LDAP.Search",...}

Failure output includes full details and stacktrace:

scribe REST.Modify internal dur=311ms trace=trace123 span=span456
failure:
kind=INTERNAL code=SERVER_ERROR
message="Database connection failed"
cause_type=java.sql.SQLException
cause_message="Connection refused"
stacktrace:
java.sql.SQLException: Connection refused
at org.postgresql.Driver.connect(Driver.java:285)

Color detection uses TERM, FORCE_COLOR, NO_COLOR, and TTY presence. Disable colors with NO_COLOR=1.

monitoring.log {
enabled = true # default
format = "pretty" # pretty | json | auto
sample-rate = 100 # 0-100%, for ops that don't match any rule
# Redaction strategies (glob patterns for keys)
redaction {
"*dn*" = hash # Marked short hash: [hash:...]
"*email*" = truncate # Marked prefix/suffix: [trunc:...]
"auth.subject_id" = hash # Correlate a subject without the raw identifier
"auth.username" = hash # Correlate a username without the raw name
"client.address" = hash # Correlate a client without the raw address
"*.raw" = replace # Replace with [REDACTED]
}
# Always replaced: passwords, credentials, tokens, secrets, API keys,
# authorization headers, and cookies.
# Authentication method, issuer, key id, and failure reason are shown by default.
}

Rules control which operations get logged. They evaluate in order — first match wins. Operations without a matching rule fall through to random sampling.

Operation endsRequired?failure · warningLog rulesfirst match winsSample ratedefault 100 %LoggedSuppressedcheckyesnoincludeexcludeno matchsampled
Operation endsRequired?failure · warningLog rulesfirst match winsSample ratedefault 100 %LoggedSuppressedcheckyesnoincludeexcludeno matchsampled
Required failures and warnings are logged before the ordered log rules and sampling are considered; client errors go through the rules by default.

Client errors go through the rules while monitoring.log.filter-client-errors = true (the default); set it to false to log every client error. Sign-ins that fail fast while the directory is down are counted, not logged one by one: they log only when a rule includes them.

Rule syntax:

monitoring.log.rules = [
{ id = "suppress-fast-ok", action = exclude, where = "scribe.result=ok duration.seconds<=50ms" }
{ id = "log-slow", action = include, where = "duration.seconds>=5s" }
{ action = exclude, where = "scribe.operation~=Hints.* duration.seconds<=100ms" }
{ action = exclude, where = "scribe.operation~=Metrics.* duration.seconds<=500ms" }
]
FieldRequiredDescription
actionYesinclude or exclude
whereNoLDAP-style filter on attributes; use scribe.operation for operation matching
idNoIdentifier for debugging (defaults to Rule #N)

Duration values accept plain seconds (0.05), HOCON style (50ms, 5s), or ISO 8601 (PT5S).

Common patterns:

{ action = include, where = "scribe.operation~=LDAP.*" } # All LDAP ops
{ action = exclude, where = "(&(scribe.result=ok)(duration.seconds<=50ms))" } # Fast successes
{ action = include, where = "(db.duration.seconds>=1s)" } # Slow DB queries
{ action = exclude } # Catch-all

See monitoring.log.rules for the complete reference. For all logging configuration options (redaction, leak detection, shutdown noise), see monitoring.log.

Child segments can appear as events in wide logs, giving you an operation breakdown without opening a trace viewer.

ModeWhat’s capturedOverhead
autoFull in dev, minimal in prod (default)Varies
offNothingNone
minimalName, offset, durationLow
fullName, offset, duration, attributesModerate
monitoring.log.childs {
mode = "auto"
rules = [
{ action = include, where = "(duration.seconds>=1ms)" }
{ action = exclude }
]
}

Events appear in start-time order (not end-time), so the log reads chronologically. Segment attributes follow the same redaction rules as operation-level attributes.

Add, Modify and Delete requests that IdentityScribe forwards to the directory log as LDAP.Add, LDAP.Modify and LDAP.Delete; a rename (Modify DN) logs as LDAP.Modify. The default rules drop successful LDAP operations up to 500 ms, so to log every write, put these rules before the defaults:

monitoring.log.rules = [
{ id = "time-ldap-writes", action = include, where = "scribe.operation=LDAP.Modify" }
{ action = include, where = "scribe.operation=LDAP.Add" }
{ action = include, where = "scribe.operation=LDAP.Delete" }
] ${monitoring.log.rules}

Each write logs its result and ldap.result_code, and an LDAP.Delegation event: the time from handing the request to the directory until the directory answered. The rest of the operation’s duration is IdentityScribe’s own work, such as access rules and proxy authorization. When the client connection has no directory connection yet, the event includes connecting and signing in to the directory; with monitoring.log.childs.mode = "full" (SCRIBE_LOG_CHILDS_MODE=full) it then shows ldap.delegation.new_connection=true. A write whose LDAP.Delegation takes nearly all of its duration spent its time in the directory or on the network to it.

The same writes are counted in scribe_channel_request_duration_seconds with op="add", op="modify" or op="delete".

When tracing is enabled, each query produces a span hierarchy:

LDAP.Search (or REST.Search, etc.) ← Channel entry span
└── Query.Normalize ← Normalization
└── Query.Plan ← Planning
└── Query.Compile ← builds the database query
└── Query.Execute ← DB execution + streaming
AttributeDescriptionExample
scribe.resultOutcomeok, cancelled, deadline_exceeded
scribe.search.kindPagination modesimple, paged, vlv
scribe.query.signatureQuery signature hasha1b2c3d4
scribe.entry_typeEntry type(s) in scopeinetOrgPerson

The phase label in scribe.query.stage.duration.seconds maps directly to span names:

PhaseSpan NameWhat it does
normalizeQuery.NormalizeAttribute mapping, filter canonicalization
planQuery.PlanIndex selection, predicate classification
compileQuery.CompileLogical plan → SQL
executeQuery.Executedatabase execution, result streaming
  1. Find the slow stage — look at scribe_query_stage_duration_seconds by phase
  2. Get a sample trace — filter by trace ID or high duration
  3. Drill into Execute — if it’s slow, check for lock contention, missing indexes (/observe/hints), or pool saturation (scribe_db_connections_pending > 0)
  4. Check Plan — if it’s slow, the query may be too complex
monitoring.traces {
enabled = true
endpoint = "http://otel-collector:4317"
protocol = "grpc" # or "http/protobuf" for port 4318
}

Or via environment variables:

Terminal window
export SCRIBE_TRACES_ENABLED=true
export SCRIBE_TRACES_ENDPOINT=http://otel-collector:4317
FieldTypeDescription
trace_idstringOpenTelemetry trace ID (when tracing enabled)
span_idstringOpenTelemetry span ID
duration_secondsdoubleOperation duration
resultstringok or failure kind
failureobjectDetails when failed: kind, code, message, stacktrace
eventsarrayWarnings and segment timing
scribe.operationstringOperation name (e.g., LDAP.Search, REST.GET)

When opening a support case, collect the output for the incident window around the first error. Include the command used to collect the file and the local timezone of the timestamps.

Kubernetes:

Terminal window
kubectl logs deployment/identity-scribe --all-containers --timestamps --since=2h > identity-scribe.log

If the pod restarted during the incident, collect the previous container output as well:

Terminal window
kubectl logs pod/<pod-name> -c identity-scribe --previous --timestamps --since=2h >> identity-scribe.log

Linux system service:

Terminal window
journalctl -u identity-scribe --since "<start-time>" --until "<end-time>" -o short-iso > identity-scribe.log

File-based service policy:

Terminal window
cp /var/log/identity-scribe/identity-scribe.log identity-scribe.log

Manual shell:

Use the file produced by the redirection or tee command above.