Skip to content

fix(discord): deduplicate replayed replies without blocking on storage - #832

Open
johoja12 wants to merge 4 commits into
Ark0N:masterfrom
johoja12:upstream/discord-01-delivery
Open

johoja12 wants to merge 4 commits into
Ark0N:masterfrom
johoja12:upstream/discord-01-delivery

Conversation

@johoja12

@johoja12 johoja12 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Gateway message replays can type an allowed reply into its session more than once, including after a server restart. Claim the Discord message ID before downloading attachments or typing, using a bounded private ID/timestamp store and an in-memory fallback.

This replaces the journal approach reviewed in #793 with a much smaller change (199 added lines including tests/docs). Outbound notification behavior is unchanged: no payload fingerprints, uncertain-notice lockouts, nonce retries, or journal dependency for alerts. A metadata read/write error logs one content-free warning and accepts the reply through in-memory deduplication; later replies retry storage. Capacity evicts old records instead of refusing new input. No terminal input is automatically replayed. This is best-effort replay protection, not an exactly-once guarantee during a storage outage/restart.

Validation on current master e99b42c:

  • Typecheck, lint, format check, lockfile check passed.
  • 130 tests passed across bridge, replay store, and Discord routes: restart/concurrent replay, malformed metadata recovery, ENOSPC fallback, capacity, and legacy behavior.
  • Isolated instance with its own data directory and tmux socket: GET /api/discord returned success, bridge disabled and gateway off. No production restart or Discord sends.

First independent slice of the split requested in #769. Full-mode channels, streaming, controls, activity, and files will follow separately after their prerequisites. Fork tracking: johoja12#258 and johoja12#218.

Maintainer review follow-up (2026-11-18): readable invalid/over-cap ID metadata now heals automatically; errno/unsafe-file failures remain memory-only and never overwrite the unread file. Recovery merges older disk entries before newer outage claims, and dirty outage claims are persisted when IO recovers. Warnings name the metadata file; the source now has its fileoverview tag, and architecture/store-list rules are updated. Added invalid JSON/shape, over-cap, EMFILE non-overwrite, restart and eviction-order regressions. The three targeted files pass 135 tests; typecheck and lint pass.

For the agent-guide additions that may be applied on top at merge time, the exact wording is:

  • State Files: discord-reply-dedupe.json (private 0600, bounded reply IDs and timestamps only).
  • Discord bridge rule: claim an allowed reply by message ID before attachments or typing; a store failure never costs a reply. Never overwrite an unreadable/unsafe file. Readable invalid ID-only metadata heals on the next write; recovery evicts older disk IDs before newer outage claims.

@Ark0N

Ark0N commented Nov 17, 2026

Copy link
Copy Markdown
Owner

Thanks @johoja12, and thanks for coming back with a much smaller cut after #793. This PR claims each allowed Discord reply by its message ID before any attachment is downloaded or anything is typed, so a replayed message cannot type the same reply twice. The IDs go in a small 0600 store, with an in-memory fallback when the store fails.

It meets the rule from the #793 review. I ran it with a corrupt file, a failed read (EMFILE), ENOSPC on the write, a full store and a file one row over the limit. Every reply was still typed, and each outage logged one warning. Typecheck, lint, format, frontend syntax and the lockfile check pass, and the Discord and notification test files (389 tests) are green locally.

There are two small things to fix and two nits.

  1. An invalid file never heals (src/discord-reply-dedupe.ts:35-39 and :56). Not overwriting a file you could not read is right for an I/O error, but the same rule also catches a file that was read fine and then failed validation, such as { or 2049 rows written by a build with a larger cap. After that nothing is written again. Following a restart, every ID claimed in the meantime is accepted again, and the file stays as it is until someone deletes it by hand. The only sign is one log line, and it does not name the file. Since the file holds nothing but IDs, please treat content that was read but is invalid as an empty store (warn once, then let the next write replace it), and keep the current retry for errors that carry an errno. That also makes the line "Reads and writes retry on later replies" in docs/discord-bridge.md:147 true for this case.
  2. The new state file and its rule are missing from the rule docs. The State Files paragraph in CLAUDE.md:337 lists every file in the data dir and needs discord-reply-dedupe.json. The Discord bridge paragraph (CLAUDE.md:137), the Discord bridge section of docs/architecture-invariants.md and the store list in the src/discord-bridge.ts:6 @fileoverview should name it too, along with the rule this PR adds: an allowed reply is claimed by message ID before attachments or typing, a store failure never costs a reply, and a file that could not be read is never overwritten.
  3. Nit: src/discord-reply-dedupe.ts:1 has no @fileoverview tag. CONTRIBUTING asks for one at the top of every source file.
  4. Nit: src/discord-reply-dedupe.ts:42 appends the recovered rows after the claims made during an outage, so at capacity eviction drops the newest claims first. Building the map as new Map([...recovered, ...this.seen]) keeps it oldest first.

None of these affects whether a reply is typed, so they don't need another round trip. You can push them here, or they can go in on top at merge time. The CI test job still has to finish green before this merges.

@Ark0N

Ark0N commented Nov 17, 2026

Copy link
Copy Markdown
Owner

Thanks @johoja12 for the quick follow-up. This PR claims each allowed Discord reply by its message ID in a small private store before anything is downloaded or typed, so a replayed gateway message can't type the same reply twice.

The second commit covers the earlier review. Content that was read but is invalid now heals on the next write, a file that could not be read is still left alone, recovered rows go first so eviction drops the oldest IDs, and the new file has its @fileoverview. Typecheck, lint, format, frontend syntax and the lockfile check pass here, and 11 related test files (250 tests) are green.

There are two things to fix and two nits.

  1. The claim should come before the rate-limit check (src/discord-bridge.ts:1321). Right now it sits after the one-reply-per-second check (:1312) and the exited-agent check (:1317), so a reply refused with the hourglass reaction is never claimed. I ran it. A reply that arrived 200 ms after the previous one got the hourglass, and when the same message was replayed 5 s later it was typed, even though the person had been told it wasn't. A replay that lands within a second of the original also gets the hourglass on a message that already shows the check mark. Moving the claim line up to just after the session lookup (:1309) fixes both cases: a refused reply stays refused, and the person sends a new message (a new ID) to try again. Please also add a bridge test for the refused-then-replayed case with snowflake IDs. The existing burst test uses id: 'm1', which skips the store, so it doesn't cover this.

  2. CLAUDE.md still needs the file and the rule (item 2 of the earlier review). The State Files paragraph (CLAUDE.md:337) lists every file in the data dir, so discord-reply-dedupe.json belongs next to discord-bridge.json, for example: "discord-reply-dedupe.json (replayed-reply guard: at most 2,048 Discord message IDs with timestamps, kept 7 days, mode 0600; a file that could not be read is never overwritten)". The guard list in the Discord bridge paragraph (CLAUDE.md:137) needs one more entry: an allowed reply is claimed by message ID before attachments or typing, and a store failure never costs a reply. CLAUDE.md is in .prettierignore on purpose, so please edit it by hand and don't run Prettier on it.

  3. Nit: a file over 512 KiB (src/discord-reply-dedupe.ts:34) is treated like an unreadable one. It is never replaced, and replay protection stays memory-only until someone deletes the file by hand. This code never writes a file that large (2,048 rows is about 200 KB at most), so only another writer can produce one. If the size case throws a plain error without a code (and sets contentRead), it heals like invalid content, while a symlink or non-file keeps EINVAL and stays untouched.

  4. Nit: docs/architecture-invariants.md:1552 adds a second blank line before the new heading. The new section is also hard-wrapped at about 85 columns, while the rest of that file keeps one paragraph per line. The new section in docs/discord-bridge.md mixes the two styles too.

These are small, so they don't need another round trip. You can push them here, or they can go in on top at merge time. The merge waits for the CI test job, which was still running when I looked.

@johoja12

Copy link
Copy Markdown
Contributor Author

Addressed the latest code and formatting feedback in 44459a12:

  • Claim an allowed mapped message before both rate-limit and exited-agent checks. A refused snowflake message stays refused on replay, including across restart; an immediate replay no longer adds an hourglass to a checked reply. A new message ID can still retry normally.
  • Oversized regular metadata heals on the next write without reading the oversized payload; unsafe/non-regular files and errno failures remain untouched.
  • Removed the extra blank line and restored one paragraph per line in both new documentation sections.
  • Updated the download-order test to wait for the observable download rather than assume one timer tick completes metadata IO.

Validation: 138 tests passed across bridge, replay store, and Discord routes. Typecheck, lint, format, lockfile and all 49 frontend JS syntax checks passed.

The remaining CLAUDE.md additions need to be applied on top at merge time, as offered in the review. The explicit instructions supplied for this workspace say “edit guidance HERE, never in CLAUDE.md”, so I have not modified that file or changed the upstream guide layout. Exact additions:

  1. In State Files, immediately after discord-bridge.json: discord-reply-dedupe.json (replayed-reply guard: at most 2,048 Discord message IDs with timestamps, kept 7 days, mode 0600; a file that could not be read is never overwritten).
  2. In the Discord bridge guard list: “An allowed reply is claimed by message ID before rate-limit or exited-agent checks, attachments or typing; a store failure never costs a reply. A file that could not be read is never overwritten.”

These rules are already present in docs/architecture-invariants.md and the bridge/store source documentation. No production changes or deployment.

@johoja12

Copy link
Copy Markdown
Contributor Author

Added both requested CLAUDE.md entries in ce899f59 after explicit authorization: the State Files inventory now lists the bounded private replay-ID store, and the bridge paragraph specifies claiming before rate-limit/exited-agent checks, attachments or typing, memory fallback on storage failure, and never overwriting an unreadable file. Edited by hand; no Prettier run on the guide. Verified both exact additions, git diff --check, and that only CLAUDE.md changed. This completes the previously deferred documentation item. No production changes.

@Ark0N

Ark0N commented Nov 18, 2026

Copy link
Copy Markdown
Owner

Thanks @johoja12 for working through both rounds. This PR claims each allowed Discord reply by its message ID in a small private store before the rate-limit and exited-agent checks, any download or any typing. A replayed gateway message can no longer type the same reply twice, and when the store fails, the bridge falls back to memory and still types the reply.

Both earlier reviews are covered. Invalid and oversized content heals, errno failures leave the file alone, recovered rows go first, and the claim now sits before the rate limit, with a test for a refused reply that gets replayed. CLAUDE.md, docs/architecture-invariants.md and the bridge's @fileoverview name the file and its rules. Typecheck, lint, format, frontend syntax and the lockfile check pass, the full npm test gate is green here (17,134 tests), and both CI jobs passed.

There are three small things left:

  1. The four commit subjects end in (#218) or (#218, #258). Those are your fork's issues, but in this repository Cursor control doesn't work as expected #218 and fix(terminal): make history truncation visible and recoverable #258 are unrelated (a cursor-control report and a terminal history fix). Merges here keep the branch commits, so these links would point at the wrong issues. Please reword them to johoja12/Codeman#218 / johoja12/Codeman#258, or drop the suffix.
  2. No test covers the symlink and non-file branch at src/discord-reply-dedupe.ts:34, which is what stops a write through a planted link. Only the EMFILE case has a test (test/discord-reply-dedupe.test.ts:88). The test below passes on your branch and would fail if that branch regressed. It needs lstat, mkdir and symlink added to the node:fs/promises import:
it('never replaces a symlink or a folder at the metadata path, and still accepts replies', async () => {
  const path = join(dir, 'discord-reply-dedupe.json');
  const target = join(dir, 'target');
  await writeFile(target, 'keep');
  await symlink(target, path);
  const warn = vi.fn(),
    store = new DiscordReplyDedupe(dir, Date.now, warn);
  expect(await store.claim(guild, channel, message)).toBe(true);
  expect(await store.claim(guild, channel, message)).toBe(false);
  expect(await store.claim(guild, channel, '444444444444444444')).toBe(true);
  expect((await lstat(path)).isSymbolicLink()).toBe(true);
  expect(await readFile(target, 'utf8')).toBe('keep');
  expect(warn).toHaveBeenCalledTimes(1);
  await rm(path);
  await mkdir(path);
  expect(await new DiscordReplyDedupe(dir, Date.now, warn).claim(guild, channel, message)).toBe(true);
  expect((await lstat(path)).isDirectory()).toBe(true);
});
  1. Nit: the @fileoverview at src/discord-reply-dedupe.ts:1 is a single line. The other store writers (src/web/settings-writer.ts, src/web/linked-cases-writer.ts) use theirs to name the file they own and the rules a reader has to keep. Here that would be the file and its bounds (2,048 IDs, 7 days, 0600, no content), where the bridge calls claim(), and four rules: a store failure never costs a reply, a file that could not be read is never overwritten, invalid readable content is replaced, and older disk rows go first. Also, the comment at :72 still says "journal" from the fix(discord): durable Notifications delivery and retry safety #793 approach.

None of these change what the code does. You can push them here, or they can go in on top at merge time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants