From 6d6e7a275e0857a92949d1ee582f0314ca1135c2 Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 15:46:39 -0700 Subject: [PATCH 1/6] docs(claude): rewrite in STE in the one mirror shape, daily at 17:42 --- CLAUDE.md | 105 ++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 70 insertions(+), 35 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 943fa27..44e7da2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,44 +1,79 @@ # github -A nightly mirror of every repository under `OWNERS` into one Proton Drive folder, one -git bundle each. `README.md` says what it mirrors, how to restore a bundle, how it works -and how to fork it; [katoptra/lib](https://github.com/katoptra/lib)'s README is the -manual for the proton engine and everything the mirrors share. This file is what a -change must not break. +This is a daily mirror of each repository of the owners in `OWNERS`. It puts one git bundle +for each repository into one Proton Drive folder. `README.md` tells: -Nothing in this repo starts a run: an external scheduler dispatches `sync.yml` nightly. -`Taskfile.yml` and its comments are the design of what is this mirror's own: `OWNERS` -and `LIST_FLOOR` in root vars, `list`, the engine's two hooks `stage` and `prune`, -`report-mirror` and `offline`. The pipeline is the engine's. +- Which repositories the mirror copies +- How to get a repository from a bundle +- How the mirror operates +- How to fork it. + +The README of [katoptra/lib](https://github.com/katoptra/lib) is the manual for the proton +engine and for the parts that more than one mirror uses. This file gives the rules that each +change must obey. + +No part of this repository starts a run. An external scheduler dispatches `sync.yml` daily, +at 17:42 UTC. + +## Constraints + +- Where a change goes: `Taskfile.yml` and its comments are the design of these parts of this + mirror: + - `OWNERS` and `LIST_FLOOR`, in the root vars + - `list` + - `stage` and `prune`, the two hooks of the engine + - `report-mirror` + - `offline`. +- All other changes go to the proton engine of lib, where each proton mirror gets them. The + pipeline is a part of the engine. +- The excludes are `report-mirror` on the toolbox include, and `stage` and `prune` on the + proton include. +- The root vars contain only the values of this mirror. No root var contains a copy of an + engine default + ([lib README, Rules a mirror keeps](https://github.com/katoptra/lib#rules-a-mirror-keeps)). +- For a change that adds storage, calculate the new total. Compare it with the baseline + (31 bundles and 325.7 MB, the run of 2026-10-08) and with the storage limit of the Proton + plan. +- Writing: obey ASD-STE100 and the rules in the + [Writing section of the org CONTRIBUTING](https://github.com/katoptra/.github/blob/main/CONTRIBUTING.md#writing). + Read that section before you write. ## Must knows -- **The skip rests on reproducible bundles.** `git bundle create --all` from two mirror - clones of an unchanged repository is byte-identical, which is why the engine's upload - skips it and why nothing but the session is kept in the bucket. `offline` proves it; - do not add anything to a bundle that varies between clones. -- **One token per owner, read-only, and `list` refuses a listing that names another - owner.** A fine-grained token has one resource owner. A token that lists the wrong - owner is a swapped credential, and a listing under `LIST_FLOOR` is a broken token or - API, never an emptied account; either fails the run before `prune` can trash on it. -- **The session is this mirror's own.** Two mirrors sharing one race its rotating refresh - token. Never run `sync` or `plan` from a laptop while an Actions run may be going. -- **The logs are public.** git's stderr goes to `.run/git.err`, the CLI's to - `.run/pd.err`, a failure names a repository by its position in the listing, and the - token reaches git as a header through its environment, never on a command line. -- **A renamed repository is trashed under its old name and uploaded under the new.** - Proton's trash keeps it; `empty-trash` is by hand and asks first. -- **Unverified until a real run shows it:** whether the CLI uploads the dotfile bundle - `katoptra/.github.bundle`, and the exact JSON shape of `filesystem list -j`, which - `destination` and `prune` unwrap as `{"value": ...}` the way dropbox's provider does. +- **A repository that did not change must give the same bundle.** `git bundle create --all` + makes the same bytes from two mirror clones of a repository that did not change. Thus, the + upload of the engine does not send that bundle again, and the bucket keeps only the + session. `offline` does a test of this. Do not add to a bundle an item that is different + from one clone to the next. +- **One read-only token for each owner, and `list` rejects a listing with a different + owner.** A fine-grained token has one resource owner. A listing with a repository of a + different owner is the result of interchanged tokens. A listing of less than `LIST_FLOOR` + repositories is the result of an incorrect token or an API problem, not an empty account. + In the two conditions, the run stops before `prune` can move a bundle to the trash. The + `vars` input of `sync.yml` can set `LIST_FLOOR` and `OWNERS` for `list` and `prune`, and + `LIST_FLOOR=0` there removes this guard. +- **This mirror has a session that no other mirror uses.** + [lib, The session](https://github.com/katoptra/lib#the-session) gives the rules for a + session. +- **Each person can read the run logs.** The stderr of git goes to `.run/git.err`, and the + stderr of the CLI goes to `.run/pd.err`. But the stderr of `list-folder`, which `prune` + uses, goes to the log. An error message gives the position of the repository in the + listing, not its name. The token goes to git in its environment, as a header, not on a + command line. +- **If a repository gets a new name, its bundle with the previous name goes to the trash.** + The mirror uploads a bundle with the new name. The trash in Proton Drive keeps the + previous bundle. Only a person runs `empty-trash`, and it shows a prompt first. +- **A run failure is the only alert.** The check in healthchecks.io has the cron + `42 17 * * *` UTC and a grace time of 3 h. ## Verifying a change -- `task check` renders every command of the pipeline inside the image and diffs it - against `render.txt`; `task render-update` accepts a change. The `check` workflow does - the same on every pull request, then `task run -- task offline`. -- `task run -- task offline`: a throwaway repository bundled twice inside the image; the - two must match. -- `task plan`: the real listing, clones and bundles, nothing uploaded. Needs the vault - and the session. -- The engine's own checks run in lib: `cd ../lib/examples/proton && task run -- task offline`. +- `task check` makes a render of each command of the pipeline in the image. Then it compares + the render with `render.txt`. `task render-update` accepts a change. +- The `check` workflow uses the check workflow of lib, with `offline: true`. Thus, on each + pull request, it runs `task check` and then `task run -- task offline`. +- `task run -- task offline` is the one check without credentials. It makes two bundles of a + test repository in the image, and the two bundles must be the same. +- `task plan` gets the listing from GitHub, and makes the clones and the bundles, but it + uploads no file. For `task plan`, the vault and the session are necessary. +- The checks of the engine are in lib: `cd ../lib/examples/proton && task run -- task offline`. From 37e9381841c0fefc1387644a059515d36faea6a2 Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 15:46:39 -0700 Subject: [PATCH 2/6] docs(readme): rewrite in STE; link lib for the session and the logs --- README.md | 264 ++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 166 insertions(+), 98 deletions(-) diff --git a/README.md b/README.md index 46a42c5..b3fb2eb 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@

github

-

A nightly mirror of GitHub repositories into Proton Drive, one bundle each.

+

A daily mirror of GitHub repositories into Proton Drive, one bundle for each repository.

sync @@ -17,20 +17,24 @@ mirror

-A nightly mirror of every repository under the GitHub owners it names into one Proton -Drive folder. Each repository becomes one git bundle, `/.bundle`: a mirror -clone's every ref, branches, tags and pull-request heads, in one file that `git clone` -restores. A repository that did not change uploads nothing, because its bundle's bytes -are the same and Proton's CLI skips a file whose content it already holds; one that -changed becomes a new revision, and Proton's version history is the history of the -mirror. Nothing is kept between runs but the Proton CLI session, encrypted in a bucket. +This mirror copies each repository of the GitHub owners in `OWNERS` into one Proton Drive +folder, daily. Each repository becomes one git bundle, `/.bundle`. A bundle is +one file with all the refs of a mirror clone (the branches, the tags and the pull-request +heads). `git clone` makes the repository again from that file. -The repository is public and holds no account: every credential and every account -identifier lives in one vault and reaches a run by name. +If a repository did not change, its bundle has the same bytes as the bundle of the last run. +The Proton CLI does not upload a file that is the same as the file in Proton Drive. Thus, the +run uploads no file for that repository. If a repository changed, its bundle becomes a new +revision. Thus, the version history in Proton Drive is the history of the mirror. + +Between runs, the mirror keeps only the session of the Proton CLI, encrypted, in a bucket. + +This is a public repository. It contains no credential and no account identifier. These are +in one vault, and a run gets each value by its name. ## How to use -A bundle is a repository. To get one back: +A bundle is a repository. Use these commands to get a repository from its bundle: ```sh git clone owner/name.bundle name # every branch, tag and pull-request head @@ -38,16 +42,19 @@ git -C name remote set-url origin https://github.com/owner/name.git git bundle list-heads owner/name.bundle # what it carries, without cloning ``` -Last night's copy is the current revision in Proton Drive; earlier nights are its -version history, as many as the plan keeps. A repository that left GitHub is in Proton's -trash under its old name, and a renamed one is trashed under the old name and uploaded -under the new. +The copy from the last run is the newest revision in Proton Drive. The copies from the +previous runs are its version history. The Proton plan has a limit on the number of +revisions. + +If the listing from GitHub does not contain a repository, the mirror moves its bundle to the +trash in Proton Drive. If a repository gets a new name, the mirror moves the bundle with the +previous name to the trash. Then it uploads a bundle with the new name. ## How it works -Once a night a GitHub Actions job runs this pipeline inside the toolbox image from -[katoptra/lib](https://github.com/katoptra/lib). Every solid box is a verb of lib's -proton engine; the dashed ones are this mirror's own. +A GitHub Actions job operates this pipeline daily, in the toolbox image of +[katoptra/lib](https://github.com/katoptra/lib). In the diagram, each dashed box is a verb of +this mirror. Each other box is a verb of the toolbox or of the proton engine of lib. ```mermaid flowchart LR @@ -58,83 +65,115 @@ flowchart LR class stage,list,prune,rm own ``` -What this mirror owns, in [`Taskfile.yml`](Taskfile.yml): - -- **Its identity**, in root vars: `OWNERS`, the GitHub users or organizations to mirror, - and `LIST_FLOOR`, the count under which a listing is refused, so a broken token can - never turn an empty listing into a trash list. -- **`stage`**, the engine's hook for filling the staging tree. It runs `list`, which asks - the API for every repository under each owner with that owner's token and refuses a - listing that names another owner's repositories, then clones each as a mirror, bundles - it with `git bundle create --all`, and deletes the clone. -- **`prune`**, the engine's hook for what upstream dropped: list each owner's folder in - Proton and trash the bundles whose repository is no longer listed. -- **Its rows of the run summary**, and `offline`, which bundles a throwaway repository - twice inside the image and compares: the skip rests on bundles being byte-for-byte - reproducible for an unchanged repository. - -The session, the destination check, the one-call upload and its confirmation are the -engine's and are documented once in -[lib's README](https://github.com/katoptra/lib#the-proton-engine). +[`Taskfile.yml`](Taskfile.yml) contains the parts of this mirror: + +- **The root vars** are the identity of the mirror. `OWNERS` gives the GitHub users or + organizations that the mirror copies. If a listing has less than `LIST_FLOOR` + repositories, `list` rejects it. Thus, the empty listing of an incorrect token cannot + cause `prune` to move bundles to the trash. But the `vars` input of `sync.yml` can set + `LIST_FLOOR` and `OWNERS` for `list` and `prune`, and `LIST_FLOOR=0` there removes this + guard. +- **`stage`** is the hook of the engine that fills the staging tree. First, it runs `list`. + `list` gets each repository of each owner from the API, with the token of that owner. It + rejects a listing that contains a repository of a different owner. Then `stage` does + these steps for each repository: + 1. It clones the repository as a mirror. + 2. It makes a bundle of the clone with `git bundle create --all`. + 3. It deletes the clone. +- **`prune`** is the hook of the engine for the repositories that upstream does not have. + It gets the listing of the folder of each owner in Proton Drive. Then it moves a bundle to + the trash if the listing from GitHub does not contain its repository. +- **`report-mirror`** adds the rows of this mirror to the run summary. +- **`offline`** makes two bundles of a test repository in the image, and compares them. A + repository that did not change must give the same bundle, byte for byte. If the bundle + changes, the upload sends it again at each run. + +These steps are verbs of the engine: + +- The session +- The destination check +- The upload in one CLI call +- The check of the upload. + +Only [lib's README](https://github.com/katoptra/lib#the-proton-engine) tells how they operate. ## Want your own? ### 1. Fork it -Fork [katoptra/github](https://github.com/katoptra/github). One line of `Taskfile.yml` is -yours to change, `OWNERS`; each owner needs a token line in `op.env` named -`MIRROR_GITHUB_TOKEN_`. Set `LIST_FLOOR` to about half the repositories you have. +Fork [katoptra/github](https://github.com/katoptra/github). Then set two root vars in +`Taskfile.yml`: + +- Set `OWNERS` to your GitHub users and organizations. +- Set `LIST_FLOOR` to approximately half the number of your repositories. + +For each owner, add a token line to `op.env`, with the name `MIRROR_GITHUB_TOKEN_`. ### 2. Storage -The bucket holds one object, the encrypted Proton CLI session, under `.state/`. Proton -holds the mirror. +The bucket contains one object: the encrypted session of the Proton CLI, in `.state/`. +Proton Drive contains the mirror. -| What | Why | +| Item | Function | |---|---| -| An R2 bucket, or any S3-compatible bucket | The session | -| An API token with Object Read & Write, scoped to that bucket | The `r2` values in step 3 | +| An R2 bucket, or a bucket of a different S3-compatible service | It contains the session | +| An API token with Object Read & Write, for that bucket only | It gives the `r2` values in step 3 | -What the engine keeps in a bucket and why the session gets no history: -[lib, Storage](https://github.com/katoptra/lib#storage). +[lib, Storage](https://github.com/katoptra/lib#storage) gives the keys that the engine keeps +in a bucket. It also gives the cause of a session without history. ### 3. Secrets -Ten values, in one vault item named `github`, one section per service: +Put ten values (for two owners) in one vault item with the name `github`. Use one section +for each service: -| Section | Field | What it is | Reaches the run as | +| Section | Field | Value | Name in the run | |---|---|---|---| -| `github` | `token_`, one per owner | A fine-grained token, step 4 | `MIRROR_GITHUB_TOKEN_` | +| `github` | `token_`, one for each owner | A fine-grained token, step 4 | `MIRROR_GITHUB_TOKEN_` | | `proton` | `destination` | The CLI path of the folder, `/my-files/GitHub` | `MIRROR_PROTON_DESTINATION` | -| `proton` | `destination_uid` | That folder's UID | `MIRROR_PROTON_DESTINATION_UID` | +| `proton` | `destination_uid` | The UID of that folder | `MIRROR_PROTON_DESTINATION_UID` | | `r2` | `access_key_id`, `secret_access_key` | The token from step 2 | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | | `r2` | `endpoint` | `https://.r2.cloudflarestorage.com` | `AWS_ENDPOINT_URL_S3` | -| `r2` | `bucket` | The bucket's name | `MIRROR_R2_BUCKET` | +| `r2` | `bucket` | The name of the bucket | `MIRROR_R2_BUCKET` | | `age` | `identity` | An `AGE-SECRET-KEY-...` line from `age-keygen` | `MIRROR_AGE_IDENTITY` | | `healthcheck` | `url` | Optional: a healthchecks.io ping URL | `HEALTHCHECK_URL` | -Put your vault's UUID into the references in [`op.env`](op.env), make a service account -that can read that vault, and store its token as the `OP_SERVICE_ACCOUNT_TOKEN` secret, -on the organization or on the repository. Finding a vault's UUID and why a UUID and not -a name: [lib, Secrets](https://github.com/katoptra/lib#secrets). +Put the UUID of your vault in the references in [`op.env`](op.env). Make a service account +that can read that vault. Put its token in the `OP_SERVICE_ACCOUNT_TOKEN` secret, on the +organization or on the repository. + +Each reference uses the UUID of the vault, not its name. +[lib, Secrets](https://github.com/katoptra/lib#secrets) tells how to find the UUID, and gives +the cause. ### 4. GitHub and Proton -**Tokens.** One fine-grained personal access token per owner, since a fine-grained token -has exactly one resource owner: for all repositories, with `Contents: read` and -`Metadata: read` and nothing else. The mirror can never write to GitHub. Store each as -the `token_` field of section `github`. +**Tokens.** Make one fine-grained personal access token for each owner. A fine-grained token +has only one resource owner. Give each token access to all repositories, with +`Contents: read`, `Metadata: read` and no other permission. Thus, the mirror cannot write to +GitHub. Put each token in the field `token_` of the section `github`. + +**The session.** Only a sign-in in a browser can make a session for the Proton CLI. Thus, you +make the session one time, on a laptop. Then each run gets the encrypted session from the +bucket. [lib, The session](https://github.com/katoptra/lib#the-session) gives: -**The session.** The Proton CLI can only be seeded by a browser sign-in, so the session -is made once on a laptop and carried to every run encrypted; the commands, the folder and -its UID, and `task session-seal -- .run/pd` are in -[lib, The session](https://github.com/katoptra/lib#the-session). Store the folder's CLI -path as `destination` and its `uid` from the listing as `destination_uid`. This mirror's -session is its own: two mirrors sharing one race its rotating refresh token. +- The commands +- How to make the folder and find its UID +- `task session-seal -- .run/pd`. -### 5. Prove it, run it, schedule it +Put the CLI path of the folder in the field `destination`. Put the `uid` of the folder from +the listing in the field `destination_uid`. This mirror has a session that no other mirror +uses. -On a laptop with go-task, the 1Password CLI and Docker or Apple `container`: +### 5. Do the checks, run it, schedule it + +Use a laptop with these tools: + +- go-task +- The 1Password CLI +- Docker or Apple `container`. + +Run these commands: ```sh task check # every pipeline command rendered inside the image, diffed against render.txt @@ -142,51 +181,80 @@ task run -- task offline # the bundle reproducibility check; no network task plan # the real thing, read-only: session, destination, list, clone and bundle; nothing uploaded ``` -Then Actions, sync, Run workflow. The first run uploads every bundle; a quiet night -afterwards uploads none. Nothing in this repository schedules a run: add a `schedule:` -trigger to `.github/workflows/sync.yml`, or dispatch it from outside as this mirror is. +Pause the healthcheck before the first run. Then, in Actions, open the sync workflow. Click +**Run workflow**. The first run uploads each bundle. After that, a run on a day with no change +uploads no bundle. + +No file in this repository starts a run on a schedule. To start runs on a schedule, add a +`schedule:` trigger to `.github/workflows/sync.yml`. Or dispatch the workflow from an external +scheduler. This mirror uses an external scheduler. ## Operating it -`task` alone prints the menu. Everything runs inside the toolbox. +`task` without a task name prints the menu. Each task operates in the toolbox. ```sh task sync # one run, the same thing Actions runs task plan # read-only: list, clone and bundle; prints what an upload would carry task session-seal -- .run/pd # encrypt a laptop Proton CLI session into the bucket task empty-trash # permanently delete Proton's trash; asks first; never scheduled +gh workflow run sync.yml # one run in Actions ``` -Never run `task sync` or `task plan` from a laptop while an Actions run may be in -progress: both hold the one Proton session, and the loser of a race needs a fresh login. - -Every run appends one table to its job page: when it started, the files and MB staged, -what Proton reported (uploaded, skipped as identical, failed), whether the session was -restored, repositories listed and bundled, bundles pruned. A quiet night reads "0 -uploaded, N skipped as identical", N being the bundles plus their owner folders. A failed -run is the only alert: healthchecks.io emails when a night passes without a ping. - -On a public repository the run logs are public. They carry counts and never a repository -name, a path in Proton, a token or an account identifier: git's stderr goes to a file, a -failure names a repository by its position in the listing, and the CLI's stderr goes to -`.run/pd.err`. - -- **The run fails at `session`.** The session is gone or its token was rotated out from - under it. Sign in again on a laptop and `task session-seal -- .run/pd`. -- **`destination` refuses the run.** The folder is not a direct child of its parent, or - its UID differs from the vault's. List the parent with `filesystem list -j` and fix the - field or the folder. -- **`list` refuses a listing.** Under `LIST_FLOOR`, or the token listed another owner's - repositories: a wrong or expired token. Nothing was trashed. -- **`confirm` fails.** Proton's summary did not account for every staged file. The next - night retries everything, which costs nothing for what already landed. +Do not run `task sync`, `task plan` or `task empty-trash` on a laptop while an Actions run can +be in progress. [lib, The session](https://github.com/katoptra/lib#the-session) gives the risk. + +Each run adds one table to its job page, with these rows: + +- The start of the run +- The files and the MB in the staging tree +- The result from Proton: "uploaded", "skipped as identical" and "failed" +- The session: "restored" or "not restored" +- The repositories in the listing, and the bundles that the run made +- The bundles that the run moved to the trash. + +After a run with no change, the Proton row shows "0 uploaded" and "N skipped as identical". +N is the number of bundles plus the number of owner folders. A run failure is the only +alert: if a day has no ping, healthchecks.io sends an e-mail. + +Each person can read the run logs: +[lib, Monitoring](https://github.com/katoptra/lib#monitoring) gives the data that a run writes +to them. + +If a run has a problem, find it in this list: + +- **The run stops at `session`.** The bucket has no session, or a different process changed + the token of the session. Make a new session on a laptop. Then run + `task session-seal -- .run/pd`. +- **`destination` stops the run.** The folder is not directly in its parent folder, or its + UID is different from the UID in the vault. Get the listing of the parent folder with + `filesystem list -j`. Then correct the field or the folder. +- **`list` rejects a listing.** The listing has less than `LIST_FLOOR` repositories, or it + contains a repository of a different owner. The cause is an incorrect token, or a token + after its expiration date. The run moved no bundle to the trash. +- **`confirm` stops the run.** The summary from Proton does not include each staged file. + The next run tries all the bundles again. The CLI does not send a bundle again if Proton + Drive has the same file. +- **The run did not start.** No part of this repository starts a run. Examine the scheduler + ([katoptra/dispatch](https://github.com/katoptra/dispatch#when-something-goes-wrong)). + Then run `gh workflow view sync.yml` to find if the workflow is disabled. Until you + correct the cause, start runs with `gh workflow run sync.yml`. ## Reference -Issues, pull-request discussion, release notes and attachments are not in git and -GitHub's migration export API is closed to ordinary accounts, so they are not here. A -wiki would be one more clone each; no repository under either owner has one today. +These items are not in git: + +- Issues +- The comments on pull requests +- Release notes +- Attached files. + +GitHub does not let usual accounts use its migration export API. Thus, this mirror does not +contain these items. + +The mirror does not copy wikis. Each wiki is one more clone for its repository. No +repository of the two owners has a wiki. -Pull requests are welcome. +You can send pull requests. MIT licensed. Built by [Josh Vaughen](https://ijosh.com). From 2c0058a6fd6f0fb943e4bc2f8d114b1a5952e19b Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 15:52:29 -0700 Subject: [PATCH 3/6] docs(readme): use gh workflow list --all; keep the closing line --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index b3fb2eb..78944f8 100644 --- a/README.md +++ b/README.md @@ -237,8 +237,9 @@ If a run has a problem, find it in this list: Drive has the same file. - **The run did not start.** No part of this repository starts a run. Examine the scheduler ([katoptra/dispatch](https://github.com/katoptra/dispatch#when-something-goes-wrong)). - Then run `gh workflow view sync.yml` to find if the workflow is disabled. Until you - correct the cause, start runs with `gh workflow run sync.yml`. + Then run `gh workflow list --all` to find if a person disabled the workflow. A disabled + workflow shows `disabled_manually`, not `active`. Until you correct the cause, start runs + with `gh workflow run sync.yml`. ## Reference @@ -255,6 +256,6 @@ contain these items. The mirror does not copy wikis. Each wiki is one more clone for its repository. No repository of the two owners has a wiki. -You can send pull requests. +Pull requests are welcome. MIT licensed. Built by [Josh Vaughen](https://ijosh.com). From 811e4946418afb445b02d61914d95a222543fae9 Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 16:37:04 -0700 Subject: [PATCH 4/6] docs(github): fix the fact-check and STE review findings --- CLAUDE.md | 51 +++++++++++++++++++++++----------------------- README.md | 61 +++++++++++++++++++++++++++++++------------------------ 2 files changed, 61 insertions(+), 51 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 44e7da2..1b242a1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,26 +15,26 @@ change must obey. No part of this repository starts a run. An external scheduler dispatches `sync.yml` daily, at 17:42 UTC. +`Taskfile.yml` and its comments are the design of the parts of this mirror. The pipeline is a +part of the engine. + ## Constraints -- Where a change goes: `Taskfile.yml` and its comments are the design of these parts of this - mirror: - - `OWNERS` and `LIST_FLOOR`, in the root vars - - `list` - - `stage` and `prune`, the two hooks of the engine - - `report-mirror` - - `offline`. -- All other changes go to the proton engine of lib, where each proton mirror gets them. The - pipeline is a part of the engine. -- The excludes are `report-mirror` on the toolbox include, and `stage` and `prune` on the - proton include. -- The root vars contain only the values of this mirror. No root var contains a copy of an - engine default +- **Where a change goes.** `Taskfile.yml` has these verbs: `list`, `stage`, `prune`, + `report-mirror` and `offline`. Make all other changes to verbs in lib's proton engine. Then + each proton mirror gets them. +- **Excludes.** The `excludes:` of the two includes contain these verbs: + - On the toolbox include: `report-mirror` + - On the proton include: `stage` and `prune`, the two hooks of the engine. +- **Root vars.** Root vars hold only the values of this mirror. Do not put an engine default + in a root var, because then the command line cannot set it ([lib README, Rules a mirror keeps](https://github.com/katoptra/lib#rules-a-mirror-keeps)). -- For a change that adds storage, calculate the new total. Compare it with the baseline - (31 bundles and 325.7 MB, the run of 2026-10-08) and with the storage limit of the Proton - plan. -- Writing: obey ASD-STE100 and the rules in the + `OWNERS` and `LIST_FLOOR` are the identity of the mirror. `API`, `CURL` and `ACCEPT` are + only for `list`. +- **Storage.** For each change that adds storage, calculate the new storage. Compare it with + the baseline (31 bundles and 325.7 MB, the run of 2026-10-08) and with the storage limit of + the Proton plan. +- **Writing.** Use ASD-STE100 and the rules in the [Writing section of the org CONTRIBUTING](https://github.com/katoptra/.github/blob/main/CONTRIBUTING.md#writing). Read that section before you write. @@ -43,9 +43,10 @@ at 17:42 UTC. - **A repository that did not change must give the same bundle.** `git bundle create --all` makes the same bytes from two mirror clones of a repository that did not change. Thus, the upload of the engine does not send that bundle again, and the bucket keeps only the - session. `offline` does a test of this. Do not add to a bundle an item that is different - from one clone to the next. -- **One read-only token for each owner, and `list` rejects a listing with a different + session. `offline` does a test to make sure that the two bundles are the same. Do not add + to a bundle an item that is different in each clone. The upload then sends the bundle + again at each run. +- **Each owner has one read-only token, and `list` rejects a listing with a different owner.** A fine-grained token has one resource owner. A listing with a repository of a different owner is the result of interchanged tokens. A listing of less than `LIST_FLOOR` repositories is the result of an incorrect token or an API problem, not an empty account. @@ -63,17 +64,17 @@ at 17:42 UTC. - **If a repository gets a new name, its bundle with the previous name goes to the trash.** The mirror uploads a bundle with the new name. The trash in Proton Drive keeps the previous bundle. Only a person runs `empty-trash`, and it shows a prompt first. -- **A run failure is the only alert.** The check in healthchecks.io has the cron - `42 17 * * *` UTC and a grace time of 3 h. +- **A run failure is the only alert.** The healthcheck in healthchecks.io has the cron + `42 17 * * *` UTC and a grace time of 3 hours. ## Verifying a change - `task check` makes a render of each command of the pipeline in the image. Then it compares the render with `render.txt`. `task render-update` accepts a change. - The `check` workflow uses the check workflow of lib, with `offline: true`. Thus, on each - pull request, it runs `task check` and then `task run -- task offline`. -- `task run -- task offline` is the one check without credentials. It makes two bundles of a - test repository in the image, and the two bundles must be the same. + pull request, it runs `task check` and then `task run -- task offline`, with no secrets. +- `task run -- task offline` makes two bundles of a test repository in the image, with no + credentials. The two bundles must be the same. - `task plan` gets the listing from GitHub, and makes the clones and the bundles, but it uploads no file. For `task plan`, the vault and the session are necessary. - The checks of the engine are in lib: `cd ../lib/examples/proton && task run -- task offline`. diff --git a/README.md b/README.md index 78944f8..7238319 100644 --- a/README.md +++ b/README.md @@ -27,10 +27,10 @@ The Proton CLI does not upload a file that is the same as the file in Proton Dri run uploads no file for that repository. If a repository changed, its bundle becomes a new revision. Thus, the version history in Proton Drive is the history of the mirror. -Between runs, the mirror keeps only the session of the Proton CLI, encrypted, in a bucket. +Between runs, the mirror keeps only the encrypted session of the Proton CLI in a bucket. This is a public repository. It contains no credential and no account identifier. These are -in one vault, and a run gets each value by its name. +in one vault. A run uses the name of each value to get it. ## How to use @@ -47,8 +47,8 @@ previous runs are its version history. The Proton plan has a limit on the number revisions. If the listing from GitHub does not contain a repository, the mirror moves its bundle to the -trash in Proton Drive. If a repository gets a new name, the mirror moves the bundle with the -previous name to the trash. Then it uploads a bundle with the new name. +trash in Proton Drive. If a repository gets a new name, the mirror uploads a bundle with the +new name. Then it moves the bundle with the previous name to the trash. ## How it works @@ -67,12 +67,16 @@ flowchart LR [`Taskfile.yml`](Taskfile.yml) contains the parts of this mirror: -- **The root vars** are the identity of the mirror. `OWNERS` gives the GitHub users or - organizations that the mirror copies. If a listing has less than `LIST_FLOOR` - repositories, `list` rejects it. Thus, the empty listing of an incorrect token cannot - cause `prune` to move bundles to the trash. But the `vars` input of `sync.yml` can set - `LIST_FLOOR` and `OWNERS` for `list` and `prune`, and `LIST_FLOOR=0` there removes this - guard. +- **Root vars.** Root vars hold only the values of this mirror. Do not put an engine default + in a root var, because then the command line cannot set it + ([lib README, Rules a mirror keeps](https://github.com/katoptra/lib#rules-a-mirror-keeps)). + Two root vars are the identity of the mirror: + - `OWNERS` gives the GitHub users or organizations that the mirror copies. + - `LIST_FLOOR` is the floor of a listing. `list` rejects a listing that has less than + `LIST_FLOOR` repositories. Thus, the empty listing of an incorrect token cannot cause + `prune` to move bundles to the trash. But the `vars` input of `sync.yml` can set + `LIST_FLOOR` and `OWNERS` for `list` and `prune`, and `LIST_FLOOR=0` there removes this + guard. - **`stage`** is the hook of the engine that fills the staging tree. First, it runs `list`. `list` gets each repository of each owner from the API, with the token of that owner. It rejects a listing that contains a repository of a different owner. Then `stage` does @@ -88,7 +92,7 @@ flowchart LR repository that did not change must give the same bundle, byte for byte. If the bundle changes, the upload sends it again at each run. -These steps are verbs of the engine: +The engine has these verbs: - The session - The destination check @@ -120,7 +124,8 @@ Proton Drive contains the mirror. | An API token with Object Read & Write, for that bucket only | It gives the `r2` values in step 3 | [lib, Storage](https://github.com/katoptra/lib#storage) gives the keys that the engine keeps -in a bucket. It also gives the cause of a session without history. +in a bucket. [lib, The session](https://github.com/katoptra/lib#the-session) gives the cause +of a session without history. ### 3. Secrets @@ -144,7 +149,7 @@ organization or on the repository. Each reference uses the UUID of the vault, not its name. [lib, Secrets](https://github.com/katoptra/lib#secrets) tells how to find the UUID, and gives -the cause. +the cause of this rule. ### 4. GitHub and Proton @@ -153,9 +158,9 @@ has only one resource owner. Give each token access to all repositories, with `Contents: read`, `Metadata: read` and no other permission. Thus, the mirror cannot write to GitHub. Put each token in the field `token_` of the section `github`. -**The session.** Only a sign-in in a browser can make a session for the Proton CLI. Thus, you -make the session one time, on a laptop. Then each run gets the encrypted session from the -bucket. [lib, The session](https://github.com/katoptra/lib#the-session) gives: +**The session.** Only a sign-in in a browser can make a session for the Proton CLI. Make the +session one time, on a laptop. Then each run gets the encrypted session from the bucket. +[lib, The session](https://github.com/katoptra/lib#the-session) gives: - The commands - How to make the folder and find its UID @@ -182,12 +187,12 @@ task plan # the real thing, read-only: session, destination, l ``` Pause the healthcheck before the first run. Then, in Actions, open the sync workflow. Click -**Run workflow**. The first run uploads each bundle. After that, a run on a day with no change -uploads no bundle. +**Run workflow**. The first run uploads each bundle. After that, a run uploads no bundle if no +repository changed. -No file in this repository starts a run on a schedule. To start runs on a schedule, add a -`schedule:` trigger to `.github/workflows/sync.yml`. Or dispatch the workflow from an external -scheduler. This mirror uses an external scheduler. +No part of this repository starts a run. To schedule runs, add a `schedule:` trigger to +`.github/workflows/sync.yml`. Or dispatch the workflow from an external scheduler. This mirror +uses an external scheduler. ## Operating it @@ -202,11 +207,14 @@ gh workflow run sync.yml # one run in Actions ``` Do not run `task sync`, `task plan` or `task empty-trash` on a laptop while an Actions run can -be in progress. [lib, The session](https://github.com/katoptra/lib#the-session) gives the risk. +be in progress. The two runs use one session, and then a new login can be necessary +([lib, The session](https://github.com/katoptra/lib#the-session)). Each run adds one table to its job page, with these rows: - The start of the run +- The image +- The next run - The files and the MB in the staging tree - The result from Proton: "uploaded", "skipped as identical" and "failed" - The session: "restored" or "not restored" @@ -214,8 +222,9 @@ Each run adds one table to its job page, with these rows: - The bundles that the run moved to the trash. After a run with no change, the Proton row shows "0 uploaded" and "N skipped as identical". -N is the number of bundles plus the number of owner folders. A run failure is the only -alert: if a day has no ping, healthchecks.io sends an e-mail. +N is the number of bundles plus the number of owner folders. + +A run failure is the only alert: if a day has no ping, healthchecks.io sends an e-mail. Each person can read the run logs: [lib, Monitoring](https://github.com/katoptra/lib#monitoring) gives the data that a run writes @@ -230,8 +239,8 @@ If a run has a problem, find it in this list: UID is different from the UID in the vault. Get the listing of the parent folder with `filesystem list -j`. Then correct the field or the folder. - **`list` rejects a listing.** The listing has less than `LIST_FLOOR` repositories, or it - contains a repository of a different owner. The cause is an incorrect token, or a token - after its expiration date. The run moved no bundle to the trash. + contains a repository of a different owner. The cause is an incorrect token or an expired + token. The run moved no bundle to the trash. - **`confirm` stops the run.** The summary from Proton does not include each staged file. The next run tries all the bundles again. The CLI does not send a bundle again if Proton Drive has the same file. From 28b44595546ef413342b65bd40b13189842656fe Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 16:39:39 -0700 Subject: [PATCH 5/6] docs(github): the final change line; LIST_FLOOR is not identity --- CLAUDE.md | 8 ++++---- README.md | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 1b242a1..f23b223 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -21,16 +21,16 @@ part of the engine. ## Constraints - **Where a change goes.** `Taskfile.yml` has these verbs: `list`, `stage`, `prune`, - `report-mirror` and `offline`. Make all other changes to verbs in lib's proton engine. Then - each proton mirror gets them. + `report-mirror` and `offline`. Make all other changes to verbs in lib: in the toolbox or in + the proton engine. Then each mirror that includes that file gets the change. - **Excludes.** The `excludes:` of the two includes contain these verbs: - On the toolbox include: `report-mirror` - On the proton include: `stage` and `prune`, the two hooks of the engine. - **Root vars.** Root vars hold only the values of this mirror. Do not put an engine default in a root var, because then the command line cannot set it ([lib README, Rules a mirror keeps](https://github.com/katoptra/lib#rules-a-mirror-keeps)). - `OWNERS` and `LIST_FLOOR` are the identity of the mirror. `API`, `CURL` and `ACCEPT` are - only for `list`. + `OWNERS` and `LIST_FLOOR` are root vars of this mirror. `API`, `CURL` and `ACCEPT` are + also root vars, only for `list`. - **Storage.** For each change that adds storage, calculate the new storage. Compare it with the baseline (31 bundles and 325.7 MB, the run of 2026-10-08) and with the storage limit of the Proton plan. diff --git a/README.md b/README.md index 7238319..24151aa 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,7 @@ flowchart LR - **Root vars.** Root vars hold only the values of this mirror. Do not put an engine default in a root var, because then the command line cannot set it ([lib README, Rules a mirror keeps](https://github.com/katoptra/lib#rules-a-mirror-keeps)). - Two root vars are the identity of the mirror: + `API`, `CURL` and `ACCEPT` are only for `list`. This mirror has two more root vars: - `OWNERS` gives the GitHub users or organizations that the mirror copies. - `LIST_FLOOR` is the floor of a listing. `list` rejects a listing that has less than `LIST_FLOOR` repositories. Thus, the empty listing of an incorrect token cannot cause From a92e3f6c9d903181ee19889845e4c835b9521112 Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 17:26:30 -0700 Subject: [PATCH 6/6] docs(readme): say minimum number, not floor, for LIST_FLOOR --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 24151aa..766d790 100644 --- a/README.md +++ b/README.md @@ -72,11 +72,11 @@ flowchart LR ([lib README, Rules a mirror keeps](https://github.com/katoptra/lib#rules-a-mirror-keeps)). `API`, `CURL` and `ACCEPT` are only for `list`. This mirror has two more root vars: - `OWNERS` gives the GitHub users or organizations that the mirror copies. - - `LIST_FLOOR` is the floor of a listing. `list` rejects a listing that has less than - `LIST_FLOOR` repositories. Thus, the empty listing of an incorrect token cannot cause - `prune` to move bundles to the trash. But the `vars` input of `sync.yml` can set - `LIST_FLOOR` and `OWNERS` for `list` and `prune`, and `LIST_FLOOR=0` there removes this - guard. + - `LIST_FLOOR` is the minimum number of repositories in a listing. `list` rejects a + listing that has less than `LIST_FLOOR` repositories. Thus, the empty listing of an + incorrect token cannot cause `prune` to move bundles to the trash. But the `vars` input + of `sync.yml` can set `LIST_FLOOR` and `OWNERS` for `list` and `prune`, and + `LIST_FLOOR=0` there removes this guard. - **`stage`** is the hook of the engine that fills the staging tree. First, it runs `list`. `list` gets each repository of each owner from the API, with the token of that owner. It rejects a listing that contains a repository of a different owner. Then `stage` does