Slow Queries
Use this guide when queries take longer than expected.
Quick diagnosis
Section titled “Quick diagnosis”# Check pressure metricscurl -s http://localhost:8080/observe/pressure | jq
# Check health statuscurl -s http://localhost:8080/observe/doctor | jq '.checks[] | select(.status != "healthy")'Symptom: All Queries Are Slow
Section titled “Symptom: All Queries Are Slow”Check 1: Query Permit Pressure
Section titled “Check 1: Query Permit Pressure”curl -s http://localhost:8080/observe/pressure | jq '.query_permit'| Value | Meaning | Action |
|---|---|---|
| < 0.8 | Normal | Not the bottleneck |
| 0.8-0.95 | High | Queries queuing — check database |
| > 0.95 | Saturated | Increase database.channelPoolSize |
Check 2: Database Connection Pool
Section titled “Check 2: Database Connection Pool”curl -s http://localhost:8080/metrics | grep hikari.*pendingIf hikaricp_connections_pending > 0:
- Queries are waiting for database connections
- Increase pool size or reduce query concurrency
Check 3: Memory Pressure
Section titled “Check 3: Memory Pressure”curl -s http://localhost:8080/observe/pressure | jq '.jvm_memory'If > 0.85: Increase heap size (-Xmx)
Symptom: Specific Queries Are Slow
Section titled “Symptom: Specific Queries Are Slow”Check 1: Slow Query Analysis (pg_stat_statements)
Section titled “Check 1: Slow Query Analysis (pg_stat_statements)”If you have pg_stat_statements enabled in PostgreSQL, the built-in monitoring shows slow queries:
curl -s http://localhost:8080/observe/stats/queries | jqExample response:
{ "queries": [ { "queryId": "1234567890", "calls": 15420, "meanMs": 45.2, "totalMs": 697344.0, "rows": 30840, "queryPreview": "SELECT * FROM entries WHERE dn = $1..." } ], "timestamp": "2026-01-24T12:00:00Z"}If you see "error": "pg_stat_statements extension not available", see Enabling pg_stat_statements below.
Check 2: Missing Indexes
Section titled “Check 2: Missing Indexes”curl -s http://localhost:8080/observe/hints | jq '.hints[] | select(.severity == "warning")'Add suggested indexes to improve performance.
Check 3: Query Complexity
Section titled “Check 3: Query Complexity”Enable tracing to identify slow stages:
SCRIBE_TRACES_ENABLED=true ./identity-scribeCheck spans for bottlenecks:
Query.Execute— Database execution timeQuery.Plan— Planning overhead (complex filters)Query.Map— Result mapping (too many attributes)
Common fixes
Section titled “Common fixes”| Problem | Solution |
|---|---|
| High permit pressure | Increase database.channelPoolSize |
| Pending DB connections | Increase database.maxPoolSize |
| Memory pressure | Increase -Xmx heap size |
| Missing indexes | Add indexes per /observe/hints |
| Complex filters | Add type constraint, simplify filter |
| Large result sets | Use pagination (paged results, cursors) |
Enabling pg_stat_statements
Section titled “Enabling pg_stat_statements”The pg_stat_statements extension provides detailed query execution statistics. IdentityScribe’s monitoring UI and /observe/stats/queries endpoint use this extension when available.
PostgreSQL Configuration
Section titled “PostgreSQL Configuration”Add to postgresql.conf:
shared_preload_libraries = 'pg_stat_statements'pg_stat_statements.track = allThen restart PostgreSQL and create the extension:
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;Managed PostgreSQL services
Section titled “Managed PostgreSQL services”| Provider | How to Enable |
|---|---|
| AWS RDS | Add pg_stat_statements to parameter group’s shared_preload_libraries, reboot instance |
| Azure Database | Enable via Server Parameters → shared_preload_libraries |
| Google Cloud SQL | Enable via Database Flags → cloudsql.enable_pg_stat_statements |
| DigitalOcean | Enabled by default on managed databases |
Verification
Section titled “Verification”After enabling, verify the extension is active:
curl -s http://localhost:8080/observe/stats/queries | jqIf successful, you’ll see query statistics. If you see "error": "pg_stat_statements extension not available", the extension is not installed or not loaded.
View refresh after extension install
Section titled “View refresh after extension install”If you install pg_stat_statements after the initial IdentityScribe deployment, the v_slow_queries database view may need to be refreshed. The view is created during migration and checks for the extension at that time.
To refresh the view, either:
- Re-run the migration by restarting IdentityScribe (Flyway will detect and apply pending migrations)
- Manually recreate the view by executing the view DDL from the migration file
If the endpoint returns an empty array despite having slow queries, this view refresh is likely needed.
Without pg_stat_statements
Section titled “Without pg_stat_statements”If you cannot enable pg_stat_statements, you can still diagnose slow queries using:
- Tracing: Enable
SCRIBE_TRACES_ENABLED=trueto see per-query execution times - Hints: Check
/observe/hintsfor missing index recommendations - PostgreSQL logs: Set
log_min_duration_statement = 100to log queries over 100ms
Related
Section titled “Related”- Monitoring — Operational workflows
- Observability — Metrics reference