perf(server): enable WAL for the SQLite store; relax sync only for SSH session issuance

[n1hility]

Summary

On-disk SQLite stores ran with sqlx defaults (rollback journal, synchronous=FULL), so every autocommit write paid several fsyncs and blocked readers. openshell forward service does two such writes per forwarded TCP connection (session token minted and revoked), which made connection setup through a forward linear in the number of concurrent connections and pushed bursts into the per-sandbox connection cap. This switches on-disk stores to WAL. Commits stay at synchronous=FULL (one WAL fsync each) so acknowledged writes such as SSH session revocations survive a power loss; only SSH session issuance, whose loss just invalidates a token, runs at synchronous=NORMAL.

Related Issue

Fixes #3494

Changes

• SqliteStore::connect: for on-disk URLs, switch the file to journal_mode=WAL once on a single connection before the pool opens (entering WAL needs exclusive access; done up front so pool connections only ever re-apply the pragma to a file already in WAL mode and a failure surfaces as one clear connect error), then build the main pool with journal_mode=WAL and synchronous=FULL, plus a single-connection synchronous=NORMAL pool for relaxed writes. Both pools share one WAL, so the next FULL commit also makes earlier relaxed commits durable. In-memory databases are unchanged and use one pool. A failed switch reports the file path and the exclusive-access requirement.
• Store::create_relaxed: MustCreate insert that is allowed to be lost in a crash. On file-backed SQLite it runs on the NORMAL pool; on Postgres it is an ordinary durable insert. handle_create_ssh_session uses it; revocation and every other write keep the durable put_if path, so relaxed durability is opt-in.
• Tests (persistence/tests.rs): fresh on-disk store reports wal with synchronous=2 on the main pool and synchronous=1 on the relaxed pool, and its -wal/-shm sidecars are 0600; an existing rollback-journal database is switched on connect; create_relaxed rejects duplicates and its insert can be revoked through the durable pool; concurrent readers proceed under a burst of relaxed-insert-then-durable-update writes on a file-backed store (so both pools contend for the writer lock); the stale comment about non-WAL production is reworded.
• Docs: architecture/gateway.md (durability settings and the relaxed issuance path, backup with sqlite3 .backup/VACUUM INTO, local filesystem requirement), docs/reference/gateway-config.mdx, deploy/rpm/CONFIGURATION.md, Helm values.yaml/README note that the SQLite volume must be local block storage.

Out of scope, noted for follow-up: an explicit store close on gateway shutdown for a final checkpoint; minting one session token per forward process instead of per connection; making the per-sandbox forward connection cap configurable.

Testing

• cargo fmt -p openshell-server -- --check, cargo clippy -p openshell-server --all-targets --features test-support -- -D warnings, python3 scripts/update_license_headers.py --check all clean on the rebased branch (main @ the base of this PR).
• cargo test -p openshell-server --features test-support persistence: 85 passed, 0 failed, 3 pre-existing ignores; the three new tests were also repeated 40 times with --test-threads=8 without a failure.
• mise run pre-commit clean after the durability follow-up commit; SQLite persistence and SSH session tests (33) pass. mise run ci not run locally.
• Unit tests added/updated
• E2E tests: not applicable to this change (no e2e path exercises store journal mode); measured behaviour instead:

Before/after with the same source, gateway on a hosted runner with the Docker driver, a sandbox serving loopback HTTP, openshell forward service in front, N simultaneous connections each doing one request (reproducer and scripts: https://github.com/n1hility/OpenShell/tree/forward-sweep-repro, workflow forward-sweep.yml; details on #3494):

Re-measured on the current head (d5737da6: WAL, synchronous=FULL on the main pool, relaxed issuance) with the same reproducer, both sets on one ubuntu-latest runner (fdatasync p50 about 0.4 ms; run: https://github.com/rh-forge/openshell/actions/runs/35930454359, patched gateway built from a checkout carrying this PR's two commits on the same base as the baseline image):

simultaneous connections	before: wall / completed	after (head): wall / completed	after: mean per conn (ms)
1	0.010-0.047 s / 1	0.005-0.006 s / 1	5
6	0.097-0.138 s / 6	0.007-0.008 s / 6	6.5
16	0.34-0.54 s / 16	0.015-0.017 s / 16	11-12
32	0.34-0.69 s / 32	0.045-0.058 s / 30-32	18-25
64	0.38-0.48 s / 47-57	0.12 s / 64	58-61
10 sequential	0.405 s (40 ms mean)	0.053 s (5.3 ms mean)

On this disk the head is indistinguishable from the first commit's all-NORMAL numbers (0.056 s at 32, 0.089 s at 64 in the earlier run). The difference appears on slow-fsync storage: each closed connection's revocation now holds the SQLite writer lock for one WAL fsync, and a new connection's issuance queues behind it, so burst time grows linearly with concurrent connections at roughly one fsync each. A store-level harness with a 4.5 ms fsync measured about 9 ms per connection for the head versus 35-50 ms for main and under 1 ms for the all-NORMAL variant; on a virtio volume with 8-12 ms fsync that projects to about 0.5 s at 32 connections and 1.2 s at 64, against 3.3 s and 6 s before this change. That is the durability trade made in the follow-up commit; removing the per-connection write altogether (one session per forward process) is the follow-up noted on #3494.

On a 2 vCPU VM with SQLite on a network block volume the same sweep went from 1.52 s to 0.06 s at 16 connections and from 6.05 s to 0.12 s at 64. Remaining refusals above 20 connections are the fixed per-sandbox cap, independent of this change.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

[n1hility]

I have read the DCO document and I hereby sign the DCO.

[mrunalp]

/ok to test 5cced0e

[github-actions]

Label test:e2e applied for 5cced0e. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[johntmyers]

gator-agent

PR Review Status

The accepted SQLite concurrency fix is project-valid, its operator-facing durability and storage constraints are documented, and the current required checks including E2E are green. One security-durability blocker remains before pipeline handoff.

Action required: Preserve durable acknowledgement for revocations and other authorization-reducing writes, then add a recovery regression test.

Blocking findings:

• GATOR-5cced0e3-01: WAL with synchronous=NORMAL may lose an acknowledged security revocation after power failure.

Carried findings:

• None

Non-blocking suggestions:

• None

Gator metadata

• Validation: Fixes accepted issue #3494 with a focused SQLite persistence change
• Docs: Fern gateway configuration, architecture, Helm, and RPM documentation updated
• Checks: Current-head required checks are green
• E2E: test:e2e applied; OpenShell / E2E is green for the current head
• Head SHA: 5cced0e39e74c5fa317b4a07e8602526158e4216
• Base SHA: 24706c175b25574640cb5e0d508ec063d4320770
• Merge base SHA: 24706c175b25574640cb5e0d508ec063d4320770
• Patch ID: 62bb61f4dc0a5708cdce40a174489fc22c80083c
• Gator payload: 9
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[mrunalp]

/ok to test d5737da

[johntmyers]

gator-agent

PR Review Status

Thanks @mrunalp. I checked your durability update against the prior finding: authorization-reducing writes now use the synchronous=FULL pool, only fail-closed SSH session issuance uses the relaxed pool, and the settings tests pin that separation. No blocking findings remain, and the current-head Branch Checks and E2E workflows are running.

Blocking findings:

• No blocking findings remain

Carried findings:

• GATOR-5cced0e3-01: Resolved by the current head; the Gator review thread is now resolved

Gator metadata

• Validation: Fixes accepted issue #3494 with a focused SQLite persistence change
• Docs: Fern gateway configuration, architecture, Helm, and RPM documentation updated
• Checks: Current-head required checks are queued, running, or complete; Branch Checks remain pending
• E2E: test:e2e is applied and the current-head E2E workflow is running
• Head SHA: d5737da6edefc6ae53c0c430ea11e7f378a397aa
• Base SHA: 24706c175b25574640cb5e0d508ec063d4320770
• Merge base SHA: 24706c175b25574640cb5e0d508ec063d4320770
• Patch ID: a870b525e3ae5f1cada7dc7cfc0aca65af8391b3
• Gator payload: 9
• Review mode: follow_up
• Previous reviewed SHA: 5cced0e39e74c5fa317b4a07e8602526158e4216
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:watch-pipeline

[n1hility]

Re-ran the forward-service sweep against the current head (d5737da6) and replaced the numbers in the description: on a hosted runner (fdatasync p50 ~0.4 ms) 32 simultaneous connections complete in 0.058 s (32/32) and 64 in 0.12 s (64/64), versus 0.34-0.69 s and 0.38-0.48 s for the baseline image; sequential mean 5.3 ms versus 40 ms. On this disk that matches the earlier all-NORMAL numbers. The description also notes where the two designs diverge: on slow-fsync storage each closed connection's durable revocation adds one WAL fsync on the writer lock, so bursts scale linearly at roughly one fsync per connection (projected ~0.5 s at 32 and ~1.2 s at 64 on a virtio volume, against 3.3 s and 6 s before).