diff --git a/CLAUDE.md b/CLAUDE.md index 943fa27..f23b223 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,44 +1,80 @@ # github -A nightly mirror of every repository under `OWNERS` into one Proton Drive folder, one -git bundle each. `README.md` says what it mirrors, how to restore a bundle, how it works -and how to fork it; [katoptra/lib](https://github.com/katoptra/lib)'s README is the -manual for the proton engine and everything the mirrors share. This file is what a -change must not break. +This is a daily mirror of each repository of the owners in `OWNERS`. It puts one git bundle +for each repository into one Proton Drive folder. `README.md` tells: -Nothing in this repo starts a run: an external scheduler dispatches `sync.yml` nightly. -`Taskfile.yml` and its comments are the design of what is this mirror's own: `OWNERS` -and `LIST_FLOOR` in root vars, `list`, the engine's two hooks `stage` and `prune`, -`report-mirror` and `offline`. The pipeline is the engine's. +- Which repositories the mirror copies +- How to get a repository from a bundle +- How the mirror operates +- How to fork it. + +The README of [katoptra/lib](https://github.com/katoptra/lib) is the manual for the proton +engine and for the parts that more than one mirror uses. This file gives the rules that each +change must obey. + +No part of this repository starts a run. An external scheduler dispatches `sync.yml` daily, +at 17:42 UTC. + +`Taskfile.yml` and its comments are the design of the parts of this mirror. The pipeline is a +part of the engine. + +## Constraints + +- **Where a change goes.** `Taskfile.yml` has these verbs: `list`, `stage`, `prune`, + `report-mirror` and `offline`. Make all other changes to verbs in lib: in the toolbox or in + the proton engine. Then each mirror that includes that file gets the change. +- **Excludes.** The `excludes:` of the two includes contain these verbs: + - On the toolbox include: `report-mirror` + - On the proton include: `stage` and `prune`, the two hooks of the engine. +- **Root vars.** Root vars hold only the values of this mirror. Do not put an engine default + in a root var, because then the command line cannot set it + ([lib README, Rules a mirror keeps](https://github.com/katoptra/lib#rules-a-mirror-keeps)). + `OWNERS` and `LIST_FLOOR` are root vars of this mirror. `API`, `CURL` and `ACCEPT` are + also root vars, only for `list`. +- **Storage.** For each change that adds storage, calculate the new storage. Compare it with + the baseline (31 bundles and 325.7 MB, the run of 2026-10-08) and with the storage limit of + the Proton plan. +- **Writing.** Use ASD-STE100 and the rules in the + [Writing section of the org CONTRIBUTING](https://github.com/katoptra/.github/blob/main/CONTRIBUTING.md#writing). + Read that section before you write. ## Must knows -- **The skip rests on reproducible bundles.** `git bundle create --all` from two mirror - clones of an unchanged repository is byte-identical, which is why the engine's upload - skips it and why nothing but the session is kept in the bucket. `offline` proves it; - do not add anything to a bundle that varies between clones. -- **One token per owner, read-only, and `list` refuses a listing that names another - owner.** A fine-grained token has one resource owner. A token that lists the wrong - owner is a swapped credential, and a listing under `LIST_FLOOR` is a broken token or - API, never an emptied account; either fails the run before `prune` can trash on it. -- **The session is this mirror's own.** Two mirrors sharing one race its rotating refresh - token. Never run `sync` or `plan` from a laptop while an Actions run may be going. -- **The logs are public.** git's stderr goes to `.run/git.err`, the CLI's to - `.run/pd.err`, a failure names a repository by its position in the listing, and the - token reaches git as a header through its environment, never on a command line. -- **A renamed repository is trashed under its old name and uploaded under the new.** - Proton's trash keeps it; `empty-trash` is by hand and asks first. -- **Unverified until a real run shows it:** whether the CLI uploads the dotfile bundle - `katoptra/.github.bundle`, and the exact JSON shape of `filesystem list -j`, which - `destination` and `prune` unwrap as `{"value": ...}` the way dropbox's provider does. +- **A repository that did not change must give the same bundle.** `git bundle create --all` + makes the same bytes from two mirror clones of a repository that did not change. Thus, the + upload of the engine does not send that bundle again, and the bucket keeps only the + session. `offline` does a test to make sure that the two bundles are the same. Do not add + to a bundle an item that is different in each clone. The upload then sends the bundle + again at each run. +- **Each owner has one read-only token, and `list` rejects a listing with a different + owner.** A fine-grained token has one resource owner. A listing with a repository of a + different owner is the result of interchanged tokens. A listing of less than `LIST_FLOOR` + repositories is the result of an incorrect token or an API problem, not an empty account. + In the two conditions, the run stops before `prune` can move a bundle to the trash. The + `vars` input of `sync.yml` can set `LIST_FLOOR` and `OWNERS` for `list` and `prune`, and + `LIST_FLOOR=0` there removes this guard. +- **This mirror has a session that no other mirror uses.** + [lib, The session](https://github.com/katoptra/lib#the-session) gives the rules for a + session. +- **Each person can read the run logs.** The stderr of git goes to `.run/git.err`, and the + stderr of the CLI goes to `.run/pd.err`. But the stderr of `list-folder`, which `prune` + uses, goes to the log. An error message gives the position of the repository in the + listing, not its name. The token goes to git in its environment, as a header, not on a + command line. +- **If a repository gets a new name, its bundle with the previous name goes to the trash.** + The mirror uploads a bundle with the new name. The trash in Proton Drive keeps the + previous bundle. Only a person runs `empty-trash`, and it shows a prompt first. +- **A run failure is the only alert.** The healthcheck in healthchecks.io has the cron + `42 17 * * *` UTC and a grace time of 3 hours. ## Verifying a change -- `task check` renders every command of the pipeline inside the image and diffs it - against `render.txt`; `task render-update` accepts a change. The `check` workflow does - the same on every pull request, then `task run -- task offline`. -- `task run -- task offline`: a throwaway repository bundled twice inside the image; the - two must match. -- `task plan`: the real listing, clones and bundles, nothing uploaded. Needs the vault - and the session. -- The engine's own checks run in lib: `cd ../lib/examples/proton && task run -- task offline`. +- `task check` makes a render of each command of the pipeline in the image. Then it compares + the render with `render.txt`. `task render-update` accepts a change. +- The `check` workflow uses the check workflow of lib, with `offline: true`. Thus, on each + pull request, it runs `task check` and then `task run -- task offline`, with no secrets. +- `task run -- task offline` makes two bundles of a test repository in the image, with no + credentials. The two bundles must be the same. +- `task plan` gets the listing from GitHub, and makes the clones and the bundles, but it + uploads no file. For `task plan`, the vault and the session are necessary. +- The checks of the engine are in lib: `cd ../lib/examples/proton && task run -- task offline`. diff --git a/README.md b/README.md index 46a42c5..766d790 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@

github

-

A nightly mirror of GitHub repositories into Proton Drive, one bundle each.

+

A daily mirror of GitHub repositories into Proton Drive, one bundle for each repository.

sync @@ -17,20 +17,24 @@ mirror

-A nightly mirror of every repository under the GitHub owners it names into one Proton -Drive folder. Each repository becomes one git bundle, `/.bundle`: a mirror -clone's every ref, branches, tags and pull-request heads, in one file that `git clone` -restores. A repository that did not change uploads nothing, because its bundle's bytes -are the same and Proton's CLI skips a file whose content it already holds; one that -changed becomes a new revision, and Proton's version history is the history of the -mirror. Nothing is kept between runs but the Proton CLI session, encrypted in a bucket. +This mirror copies each repository of the GitHub owners in `OWNERS` into one Proton Drive +folder, daily. Each repository becomes one git bundle, `/.bundle`. A bundle is +one file with all the refs of a mirror clone (the branches, the tags and the pull-request +heads). `git clone` makes the repository again from that file. -The repository is public and holds no account: every credential and every account -identifier lives in one vault and reaches a run by name. +If a repository did not change, its bundle has the same bytes as the bundle of the last run. +The Proton CLI does not upload a file that is the same as the file in Proton Drive. Thus, the +run uploads no file for that repository. If a repository changed, its bundle becomes a new +revision. Thus, the version history in Proton Drive is the history of the mirror. + +Between runs, the mirror keeps only the encrypted session of the Proton CLI in a bucket. + +This is a public repository. It contains no credential and no account identifier. These are +in one vault. A run uses the name of each value to get it. ## How to use -A bundle is a repository. To get one back: +A bundle is a repository. Use these commands to get a repository from its bundle: ```sh git clone owner/name.bundle name # every branch, tag and pull-request head @@ -38,16 +42,19 @@ git -C name remote set-url origin https://github.com/owner/name.git git bundle list-heads owner/name.bundle # what it carries, without cloning ``` -Last night's copy is the current revision in Proton Drive; earlier nights are its -version history, as many as the plan keeps. A repository that left GitHub is in Proton's -trash under its old name, and a renamed one is trashed under the old name and uploaded -under the new. +The copy from the last run is the newest revision in Proton Drive. The copies from the +previous runs are its version history. The Proton plan has a limit on the number of +revisions. + +If the listing from GitHub does not contain a repository, the mirror moves its bundle to the +trash in Proton Drive. If a repository gets a new name, the mirror uploads a bundle with the +new name. Then it moves the bundle with the previous name to the trash. ## How it works -Once a night a GitHub Actions job runs this pipeline inside the toolbox image from -[katoptra/lib](https://github.com/katoptra/lib). Every solid box is a verb of lib's -proton engine; the dashed ones are this mirror's own. +A GitHub Actions job operates this pipeline daily, in the toolbox image of +[katoptra/lib](https://github.com/katoptra/lib). In the diagram, each dashed box is a verb of +this mirror. Each other box is a verb of the toolbox or of the proton engine of lib. ```mermaid flowchart LR @@ -58,83 +65,120 @@ flowchart LR class stage,list,prune,rm own ``` -What this mirror owns, in [`Taskfile.yml`](Taskfile.yml): - -- **Its identity**, in root vars: `OWNERS`, the GitHub users or organizations to mirror, - and `LIST_FLOOR`, the count under which a listing is refused, so a broken token can - never turn an empty listing into a trash list. -- **`stage`**, the engine's hook for filling the staging tree. It runs `list`, which asks - the API for every repository under each owner with that owner's token and refuses a - listing that names another owner's repositories, then clones each as a mirror, bundles - it with `git bundle create --all`, and deletes the clone. -- **`prune`**, the engine's hook for what upstream dropped: list each owner's folder in - Proton and trash the bundles whose repository is no longer listed. -- **Its rows of the run summary**, and `offline`, which bundles a throwaway repository - twice inside the image and compares: the skip rests on bundles being byte-for-byte - reproducible for an unchanged repository. - -The session, the destination check, the one-call upload and its confirmation are the -engine's and are documented once in -[lib's README](https://github.com/katoptra/lib#the-proton-engine). +[`Taskfile.yml`](Taskfile.yml) contains the parts of this mirror: + +- **Root vars.** Root vars hold only the values of this mirror. Do not put an engine default + in a root var, because then the command line cannot set it + ([lib README, Rules a mirror keeps](https://github.com/katoptra/lib#rules-a-mirror-keeps)). + `API`, `CURL` and `ACCEPT` are only for `list`. This mirror has two more root vars: + - `OWNERS` gives the GitHub users or organizations that the mirror copies. + - `LIST_FLOOR` is the minimum number of repositories in a listing. `list` rejects a + listing that has less than `LIST_FLOOR` repositories. Thus, the empty listing of an + incorrect token cannot cause `prune` to move bundles to the trash. But the `vars` input + of `sync.yml` can set `LIST_FLOOR` and `OWNERS` for `list` and `prune`, and + `LIST_FLOOR=0` there removes this guard. +- **`stage`** is the hook of the engine that fills the staging tree. First, it runs `list`. + `list` gets each repository of each owner from the API, with the token of that owner. It + rejects a listing that contains a repository of a different owner. Then `stage` does + these steps for each repository: + 1. It clones the repository as a mirror. + 2. It makes a bundle of the clone with `git bundle create --all`. + 3. It deletes the clone. +- **`prune`** is the hook of the engine for the repositories that upstream does not have. + It gets the listing of the folder of each owner in Proton Drive. Then it moves a bundle to + the trash if the listing from GitHub does not contain its repository. +- **`report-mirror`** adds the rows of this mirror to the run summary. +- **`offline`** makes two bundles of a test repository in the image, and compares them. A + repository that did not change must give the same bundle, byte for byte. If the bundle + changes, the upload sends it again at each run. + +The engine has these verbs: + +- The session +- The destination check +- The upload in one CLI call +- The check of the upload. + +Only [lib's README](https://github.com/katoptra/lib#the-proton-engine) tells how they operate. ## Want your own? ### 1. Fork it -Fork [katoptra/github](https://github.com/katoptra/github). One line of `Taskfile.yml` is -yours to change, `OWNERS`; each owner needs a token line in `op.env` named -`MIRROR_GITHUB_TOKEN_`. Set `LIST_FLOOR` to about half the repositories you have. +Fork [katoptra/github](https://github.com/katoptra/github). Then set two root vars in +`Taskfile.yml`: + +- Set `OWNERS` to your GitHub users and organizations. +- Set `LIST_FLOOR` to approximately half the number of your repositories. + +For each owner, add a token line to `op.env`, with the name `MIRROR_GITHUB_TOKEN_`. ### 2. Storage -The bucket holds one object, the encrypted Proton CLI session, under `.state/`. Proton -holds the mirror. +The bucket contains one object: the encrypted session of the Proton CLI, in `.state/`. +Proton Drive contains the mirror. -| What | Why | +| Item | Function | |---|---| -| An R2 bucket, or any S3-compatible bucket | The session | -| An API token with Object Read & Write, scoped to that bucket | The `r2` values in step 3 | +| An R2 bucket, or a bucket of a different S3-compatible service | It contains the session | +| An API token with Object Read & Write, for that bucket only | It gives the `r2` values in step 3 | -What the engine keeps in a bucket and why the session gets no history: -[lib, Storage](https://github.com/katoptra/lib#storage). +[lib, Storage](https://github.com/katoptra/lib#storage) gives the keys that the engine keeps +in a bucket. [lib, The session](https://github.com/katoptra/lib#the-session) gives the cause +of a session without history. ### 3. Secrets -Ten values, in one vault item named `github`, one section per service: +Put ten values (for two owners) in one vault item with the name `github`. Use one section +for each service: -| Section | Field | What it is | Reaches the run as | +| Section | Field | Value | Name in the run | |---|---|---|---| -| `github` | `token_`, one per owner | A fine-grained token, step 4 | `MIRROR_GITHUB_TOKEN_` | +| `github` | `token_`, one for each owner | A fine-grained token, step 4 | `MIRROR_GITHUB_TOKEN_` | | `proton` | `destination` | The CLI path of the folder, `/my-files/GitHub` | `MIRROR_PROTON_DESTINATION` | -| `proton` | `destination_uid` | That folder's UID | `MIRROR_PROTON_DESTINATION_UID` | +| `proton` | `destination_uid` | The UID of that folder | `MIRROR_PROTON_DESTINATION_UID` | | `r2` | `access_key_id`, `secret_access_key` | The token from step 2 | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | | `r2` | `endpoint` | `https://.r2.cloudflarestorage.com` | `AWS_ENDPOINT_URL_S3` | -| `r2` | `bucket` | The bucket's name | `MIRROR_R2_BUCKET` | +| `r2` | `bucket` | The name of the bucket | `MIRROR_R2_BUCKET` | | `age` | `identity` | An `AGE-SECRET-KEY-...` line from `age-keygen` | `MIRROR_AGE_IDENTITY` | | `healthcheck` | `url` | Optional: a healthchecks.io ping URL | `HEALTHCHECK_URL` | -Put your vault's UUID into the references in [`op.env`](op.env), make a service account -that can read that vault, and store its token as the `OP_SERVICE_ACCOUNT_TOKEN` secret, -on the organization or on the repository. Finding a vault's UUID and why a UUID and not -a name: [lib, Secrets](https://github.com/katoptra/lib#secrets). +Put the UUID of your vault in the references in [`op.env`](op.env). Make a service account +that can read that vault. Put its token in the `OP_SERVICE_ACCOUNT_TOKEN` secret, on the +organization or on the repository. + +Each reference uses the UUID of the vault, not its name. +[lib, Secrets](https://github.com/katoptra/lib#secrets) tells how to find the UUID, and gives +the cause of this rule. ### 4. GitHub and Proton -**Tokens.** One fine-grained personal access token per owner, since a fine-grained token -has exactly one resource owner: for all repositories, with `Contents: read` and -`Metadata: read` and nothing else. The mirror can never write to GitHub. Store each as -the `token_` field of section `github`. +**Tokens.** Make one fine-grained personal access token for each owner. A fine-grained token +has only one resource owner. Give each token access to all repositories, with +`Contents: read`, `Metadata: read` and no other permission. Thus, the mirror cannot write to +GitHub. Put each token in the field `token_` of the section `github`. + +**The session.** Only a sign-in in a browser can make a session for the Proton CLI. Make the +session one time, on a laptop. Then each run gets the encrypted session from the bucket. +[lib, The session](https://github.com/katoptra/lib#the-session) gives: -**The session.** The Proton CLI can only be seeded by a browser sign-in, so the session -is made once on a laptop and carried to every run encrypted; the commands, the folder and -its UID, and `task session-seal -- .run/pd` are in -[lib, The session](https://github.com/katoptra/lib#the-session). Store the folder's CLI -path as `destination` and its `uid` from the listing as `destination_uid`. This mirror's -session is its own: two mirrors sharing one race its rotating refresh token. +- The commands +- How to make the folder and find its UID +- `task session-seal -- .run/pd`. -### 5. Prove it, run it, schedule it +Put the CLI path of the folder in the field `destination`. Put the `uid` of the folder from +the listing in the field `destination_uid`. This mirror has a session that no other mirror +uses. -On a laptop with go-task, the 1Password CLI and Docker or Apple `container`: +### 5. Do the checks, run it, schedule it + +Use a laptop with these tools: + +- go-task +- The 1Password CLI +- Docker or Apple `container`. + +Run these commands: ```sh task check # every pipeline command rendered inside the image, diffed against render.txt @@ -142,50 +186,84 @@ task run -- task offline # the bundle reproducibility check; no network task plan # the real thing, read-only: session, destination, list, clone and bundle; nothing uploaded ``` -Then Actions, sync, Run workflow. The first run uploads every bundle; a quiet night -afterwards uploads none. Nothing in this repository schedules a run: add a `schedule:` -trigger to `.github/workflows/sync.yml`, or dispatch it from outside as this mirror is. +Pause the healthcheck before the first run. Then, in Actions, open the sync workflow. Click +**Run workflow**. The first run uploads each bundle. After that, a run uploads no bundle if no +repository changed. + +No part of this repository starts a run. To schedule runs, add a `schedule:` trigger to +`.github/workflows/sync.yml`. Or dispatch the workflow from an external scheduler. This mirror +uses an external scheduler. ## Operating it -`task` alone prints the menu. Everything runs inside the toolbox. +`task` without a task name prints the menu. Each task operates in the toolbox. ```sh task sync # one run, the same thing Actions runs task plan # read-only: list, clone and bundle; prints what an upload would carry task session-seal -- .run/pd # encrypt a laptop Proton CLI session into the bucket task empty-trash # permanently delete Proton's trash; asks first; never scheduled +gh workflow run sync.yml # one run in Actions ``` -Never run `task sync` or `task plan` from a laptop while an Actions run may be in -progress: both hold the one Proton session, and the loser of a race needs a fresh login. - -Every run appends one table to its job page: when it started, the files and MB staged, -what Proton reported (uploaded, skipped as identical, failed), whether the session was -restored, repositories listed and bundled, bundles pruned. A quiet night reads "0 -uploaded, N skipped as identical", N being the bundles plus their owner folders. A failed -run is the only alert: healthchecks.io emails when a night passes without a ping. - -On a public repository the run logs are public. They carry counts and never a repository -name, a path in Proton, a token or an account identifier: git's stderr goes to a file, a -failure names a repository by its position in the listing, and the CLI's stderr goes to -`.run/pd.err`. - -- **The run fails at `session`.** The session is gone or its token was rotated out from - under it. Sign in again on a laptop and `task session-seal -- .run/pd`. -- **`destination` refuses the run.** The folder is not a direct child of its parent, or - its UID differs from the vault's. List the parent with `filesystem list -j` and fix the - field or the folder. -- **`list` refuses a listing.** Under `LIST_FLOOR`, or the token listed another owner's - repositories: a wrong or expired token. Nothing was trashed. -- **`confirm` fails.** Proton's summary did not account for every staged file. The next - night retries everything, which costs nothing for what already landed. +Do not run `task sync`, `task plan` or `task empty-trash` on a laptop while an Actions run can +be in progress. The two runs use one session, and then a new login can be necessary +([lib, The session](https://github.com/katoptra/lib#the-session)). + +Each run adds one table to its job page, with these rows: + +- The start of the run +- The image +- The next run +- The files and the MB in the staging tree +- The result from Proton: "uploaded", "skipped as identical" and "failed" +- The session: "restored" or "not restored" +- The repositories in the listing, and the bundles that the run made +- The bundles that the run moved to the trash. + +After a run with no change, the Proton row shows "0 uploaded" and "N skipped as identical". +N is the number of bundles plus the number of owner folders. + +A run failure is the only alert: if a day has no ping, healthchecks.io sends an e-mail. + +Each person can read the run logs: +[lib, Monitoring](https://github.com/katoptra/lib#monitoring) gives the data that a run writes +to them. + +If a run has a problem, find it in this list: + +- **The run stops at `session`.** The bucket has no session, or a different process changed + the token of the session. Make a new session on a laptop. Then run + `task session-seal -- .run/pd`. +- **`destination` stops the run.** The folder is not directly in its parent folder, or its + UID is different from the UID in the vault. Get the listing of the parent folder with + `filesystem list -j`. Then correct the field or the folder. +- **`list` rejects a listing.** The listing has less than `LIST_FLOOR` repositories, or it + contains a repository of a different owner. The cause is an incorrect token or an expired + token. The run moved no bundle to the trash. +- **`confirm` stops the run.** The summary from Proton does not include each staged file. + The next run tries all the bundles again. The CLI does not send a bundle again if Proton + Drive has the same file. +- **The run did not start.** No part of this repository starts a run. Examine the scheduler + ([katoptra/dispatch](https://github.com/katoptra/dispatch#when-something-goes-wrong)). + Then run `gh workflow list --all` to find if a person disabled the workflow. A disabled + workflow shows `disabled_manually`, not `active`. Until you correct the cause, start runs + with `gh workflow run sync.yml`. ## Reference -Issues, pull-request discussion, release notes and attachments are not in git and -GitHub's migration export API is closed to ordinary accounts, so they are not here. A -wiki would be one more clone each; no repository under either owner has one today. +These items are not in git: + +- Issues +- The comments on pull requests +- Release notes +- Attached files. + +GitHub does not let usual accounts use its migration export API. Thus, this mirror does not +contain these items. + +The mirror does not copy wikis. Each wiki is one more clone for its repository. No +repository of the two owners has a wiki. Pull requests are welcome.