Skip to content

Connection Issues

Clients can't connect, or connections drop intermittently. Work through the checks below to isolate the problem.

Connection path

Where failures happen

Client
Connection refused Port, firewall
Network / Firewall
503 Service Unavailable Starting up, not ready
IdentityScribe
Intermittent timeouts Pool exhaustion, backend down
LDAP Backend
Database errors Auth, connectivity, SSL
PostgreSQL

Observe JSON endpoints require an operator session or accepted credential by default; the examples below use a bearer token. For trusted local evaluation only, you may opt out with monitoring.observe.auth.enabled=false.

Terminal window
# Traffic readiness
curl http://localhost:8080/readyz
# Channel status
curl -s -H "Authorization: Bearer <token>" http://localhost:8080/observe/channels | jq
# Database connectivity
curl -s -H "Authorization: Bearer <token>" http://localhost:8080/observe/doctor | jq '.checks[] | select(.name | startswith("db"))'
Terminal window
curl -s -H "Authorization: Bearer <token>" http://localhost:8080/observe/channels | jq '.ldap'

Expected: enabled: true, running: true

If disabled: Confirm channels.ldap.listen contains at least one port — the LDAP channel auto-enables when one or more listen entries are configured.

Terminal window
curl -s -H "Authorization: Bearer <token>" http://localhost:8080/observe/channels | jq '.ldap.bindings[].actualPort'

The actual port may differ from the configured port when using ephemeral ports.

Verify traffic is allowed to the LDAP port:

Terminal window
nc -zv identity-scribe-host 10389
Terminal window
curl http://localhost:8080/startedz

If not started: Check logs for startup errors.

Terminal window
curl -s -H "Authorization: Bearer <token>" http://localhost:8080/observe/doctor | jq '.checks[] | select(.name | startswith("db"))'
Terminal window
curl http://localhost:8080/readyz

If readiness fails, the pod should not receive traffic yet. To check whether long-running initial ingest and index work is fully complete, inspect convergence:

Terminal window
curl -s -H "Authorization: Bearer <token>" http://localhost:8080/observe/convergence | jq

Symptom: /api returns 404, or the UI shows no entries

Section titled “Symptom: /api returns 404, or the UI shows no entries”

This is usually a disabled, hidden, or wrong-socket endpoint — not a connectivity fault.

  • /api returning 404 while /api/entries/... answers is expected in production: interactive documentation is hidden by default.
  • The Operator UI showing no entries usually means the REST channel is off, the UI and REST are on different sockets, the session is unauthenticated, or initial sync has not converged.

The full decision path — with the exact settings and curl checks — is in API Access & Networking — Troubleshooting.

Terminal window
curl -s http://localhost:8080/metrics \
| grep -E 'scribe_(db_connections_(pending|active)|db_pool_pressure|query_permit_(pressure|queue))'

If scribe_db_connections_pending > 0 frequently or scribe_db_pool_pressure stays above 0.8, increase the relevant database pool size.

If scribe_query_permit_pressure approaches 1.0 or scribe_query_permit_queue grows, queries are waiting on concurrency permits before they reach the database.

Check ingest lag:

Terminal window
curl -s -H "Authorization: Bearer <token>" http://localhost:8080/observe/stats/ingest | jq '.entry_types[].lag_seconds'

High lag points to source LDAP problems.

If an LDAPS connection fails during TLS negotiation, the log may include a unified ldap.ssl recommendation with the protocols and cipher suites that completed the handshake probe:

ldap.ssl {
ca = "/etc/scribe/tls/ldap-ca.pem"
protocols = [TLSv1.2, TLSv1.3]
cipher-suites = [
TLS_AES_256_GCM_SHA384
TLS_RSA_WITH_AES_256_GCM_SHA384
]
}

If the diagnostic says certificate verification was reached, the protocol and cipher are compatible; confirm ldap.ssl.ca points to the CA that issued the LDAP server certificate. If no compatible pair is found, check LDAPS versus StartTLS, the port, and whether the LDAP server requires ldap.ssl.cert and ldap.ssl.key.

Increase timeouts if the network is slow:

ldap.so-timeout = 30000
database.query-http-acquisition-timeout = 10s
CheckCommand
PostgreSQL runningpg_isready -h localhost -p 5432
Credentials correctCheck SCRIBE_DATABASE_USER/PASSWORD
SSL requiredSet ?sslmode=require in URL
Network reachablenc -zv db-host 5432
Pool not exhaustedCheck scribe_db_connections_pending and scribe_db_pool_pressure