Skip to content

fix: handle replica lag in WebDAV server - #736

Open
jzunigax2 wants to merge 3 commits into
mainfrom
fix/webdav-replica-lag
Open

jzunigax2 wants to merge 3 commits into
mainfrom
fix/webdav-replica-lag

Conversation

@jzunigax2

@jzunigax2 jzunigax2 commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What

Makes WebDAV PUT robust to read-replica lag and replace failures.

  • Removes the trash-then-create fallback on replace errors.
  • Replaces only when the lookup says EXISTS; otherwise creates.
  • If replaceFile returns 404 or FILE_NOT_REPLACEABLE, creates the file instead, reusing the upload.
  • A 409 from createFile returns 503 + Retry-After: 5.
  • Trusts our own writes for 10s over a replica 404. This follows MOVEs and is cleared on DELETE.
  • findExistentFile matches the extension exactly.

Why

Hyper Backup and HBS3 users reported syncs stopping. Their logs show: PUT → 201, then a lookup of the same file 1–2s
later misses it (404), the PUT creates it and aborts. The lookups read from replicas thatlag behind the primary.

Also:

  • The old fallback could trash the user's file on any replace error.
  • Hyper Backup aborts the job on any 4xx but retries 5xx.
  • Hyper Backup uploads x.temp and then MOVEs it to x, so the protection has to follow moves.

How

  • PUT.handler.ts: create-on-gone after a failed replace. createFileOrRetryLater maps an API 409 to a 503 and
    clears the stale cache. Other errors pass th
  • errors.utils.ts: isFileNotReplaceable (matches errorCode) and isApiConflict (status 409).
  • drive-item.service.ts: rememberFileWrite, moveFileWrite, forgetFileWrites, used by MOVE/DELETE.
  • drive-file.service.ts: exact type match, and sizes parsed like API reads.
  • Tests for each path.

Depends on

@jzunigax2
jzunigax2 force-pushed the fix/webdav-replica-lag branch from b1e6f01 to 2253ccd Compare November 12, 2026 17:06
- Return replace errors instead of trashing the file and creating a new one
- Only replace when the file exists, otherwise create it
- Match the exact extension in findExistentFile
- Trust our own recent writes over the replica's not found
@jzunigax2
jzunigax2 force-pushed the fix/webdav-replica-lag branch from 2253ccd to 0e35958 Compare November 16, 2026 19:22
The lookup reads from a replica, so it can still see a file that the primary already trashed or deleted. When replaceFile answers 404 or FILE_NOT_REPLACEABLE, drop the cached uuid and create the file once, reusing the upload. Any other replace error is returned unchanged.
@jzunigax2
jzunigax2 marked this pull request as ready for review November 17, 2026 20:43
@sonarqubecloud

Copy link
Copy Markdown

if (originalDriveItem.itemType === 'file') {
DriveItemService.instance.moveFileWrite(resource.url, destinationResource.url);
} else {
DriveItemService.instance.forgetFileWrites(resource.url);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When originalDriveItem is a folder, forgetFileWrites(resource.url) clears recent-write protection for every file inside it, but moveFileWrite is never called to re-register them at their new destination paths (only the single-file branch above does that). A file PUT just before its containing folder is moved/renamed loses the trust window at the new path, so a GET right after the move can still hit the exact replica-lag 404 this PR is meant to prevent.

I dont know if I am missing something but here is an example: a backup client PUTs /Backups/2026-11-18_tmp/report.pdf (201, remembered for 10s). Within that window it issues MOVE /Backups/2026-11-18_tmp -> /Backups/2026-11-18 to atomically "publish" the finished backup (a common pattern: write into a staging folder, then rename it into place). The handler calls forgetFileWrites('/Backups/2026-11-18_tmp/'), dropping the remembered write for report.pdf, but never calls moveFileWrite for it at the new path. If a client now does GET /Backups/2026-11-18/report.pdf to verify the backup and that read hits a replica that hasn't caught up with the folder rename yet, the API returns 404 and there's no trust window to fall back on, since that whole pattern is gone, so the server returns a 404 for a file that was just written seconds ago

itemType: 'file',
name: existentFile.plainName ?? '',
uuid: existentFile.uuid,
size: existentFile.size,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could wrap it the same way as createFile/replaceFile above: size: DriveUtils.parseFileSize(existentFile.size), so a string size from checkDuplicatedFiles gets normalized too, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants