Skip to content

chore(release): 0.5.0 - #7

Merged
fransallen merged 152 commits into
mainfrom
dev
Nov 21, 2026
Merged

fransallen merged 152 commits into
mainfrom
dev

Conversation

@fransallen

Copy link
Copy Markdown
Contributor

Pyo 0.5.0: everything on dev since 0.4.5, preview environments for pull and merge requests, folder deploys, request logs kept on the clusters, Tailscale in each cluster's agent, pyo scale, image scans in the release, security fixes, and the version bump.

A command can carry a signature from a separate signer: claims with
the cluster, a growing sequence number, expiry, the command's kind
and the SHA-256 of its exact JSON. Agents given the signer's public
key check them. The signer never sees the command itself.
Extensions gains a CommandSigner: when set, every command an agent
polls is signed first, with the signer choosing the sequence number
and expiry. The signer sees the command's kind and hash, never its
secrets. When signing fails the commands go back to the queue.
With PYO_AGENT_CA the agent pins that CA, makes its own key, gets a
day-long client certificate with the connect command's token, and
renews it at half its lifetime. With PYO_COMMAND_KEY and
PYO_CLUSTER_ID it refuses commands that aren't signed for its
cluster, are replayed, expired or changed, and reports why. The key,
certificate and last command number live in pyo-agent-identity.
With PYO_AGENT_MTLS_ADDR, Pyo also serves the agent routes on a TLS
1.3 listener with its own CA. An agent enrolls once with its connect
token and a CSR, gets a day-long certificate naming its team and
cluster, and renews with that certificate. The agent extractor takes
the cluster from the certificate, and PYO_AGENT_REQUIRE_CERT refuses
bearer tokens alone. Rotating a cluster's token revokes its
certificates. In this mode the install manifest points the agent at
PYO_AGENT_ENDPOINT and pins the CA, the cluster id and the signer's
keys, and connect commands use PYO_PUBLIC_API_URL.
With PYO_CLUSTER_REGISTRY=in-cluster, connect commands add
--local-registry: connect.sh sets up a registry in pyo-system and a
loopback forward on every server (127.0.0.1:5005), and builders on
that cluster push there instead of to Pyo's registry door. Images
stay on the customer's servers.
With Extensions::verify_domains, a team claims a new hostname only
when the TXT record _pyo-verify.<host> holds the token Pyo gives that
team for it. A team that proves a hostname takes it over from a team
that can't anymore. Free domains need no record but must point at one
of the team's own servers. PYO_DEV_TXT_FILE reads records from a file,
for tests only.
When the API says test payments are on (a development build of Pyo
Cloud), Billing shows a test payment option. No money moves.
On Pyo Cloud the machine Pyo runs on isn't a team's server, so a team
can't connect it as its cluster.
The signer answers each command with a signature or a refusal. A
refused command fails with the signer's reason instead of holding
back the cluster's queue.
TLS 1.3 resumption carried the enrollment's session, which had no
client certificate, into every later connection, so the API saw an
agent without one. Resumption is off on both ends of the agent
endpoint: each connection presents the current certificate.
With in-cluster registries every cluster builds for itself, and no
cluster is Pyo's own, so a team's first cluster to connect gets the
team's first builder. Git deploys work from the start.
An agent's identity can travel in each request instead of a client
certificate, so it survives a proxy that ends TLS. The signature
covers the key id, method, path with query, time, a nonce and the
body's hash, with an Ed25519 key made in the cluster.
PYO_AGENT_AUTH picks how agents prove who they are: token (the
default), mtls, or signed. In signed mode an agent enrolls an Ed25519
public key once with its connect token, gets a key id for a day,
renews with the key it has, and signs every request. Pyo checks the
signature, refuses times more than a minute off and nonces it has
seen, and takes the cluster from the key, so it works behind a proxy
that ends TLS. Rotating the token revokes the keys. Bearer tokens
alone stay refused outside token mode.
Behind Cloudflare the visitor's address comes from CF-Connecting-IP,
counted only when the connection comes from PYO_TRUSTED_PROXIES
(Cloudflare's ranges, or cloudflared on this machine).
With PYO_AGENT_AUTH=signed the agent makes an Ed25519 key in the
cluster, enrolls it with the connect token, renews it at half its
day, and signs heartbeats, long-polls, reports and the terminal's
WebSocket handshake. A new connect token makes it enroll again.
pyo-agent sign-request prints signed headers, for tests.
Proxies close idle connections (Cloudflare after 100 seconds), so a
quiet terminal session kept dropping.
Tailscale is set up in the API today, with the installation's own
credential. A hosted Pyo serves many teams and joins none of their
tailnets, so setting it up with a token or saving an OAuth client is
refused there until each cluster's agent does it.
A hosted Pyo has no Tailscale setup yet, and Hetzner servers join over
the tailnet. Their cards, the "set up Tailscale first" notes and the
public internet switch (its agents always connect that way) don't show.
A hosted Pyo runs on none of a team's machines, and its API refuses
"This machine". Add cluster, onboarding and the install command dialog
no longer offer it there, so the connect command is the default and
only option. Hetzner, which joins over the tailnet, is left out of Add
cluster and Add server too. Self-hosted is unchanged.
A hosted Pyo has no Tailscale setup. Its own backup store doesn't reach
a team's servers either, so there backups count as on only once they go
to the team's bucket.
The Pyo team updates a hosted Pyo, so the sidebar, menu and settings
tab don't point its users at an update.
Add sandbox server shows only the command, since Pyo's Hetzner servers
join over the tailnet. The registry doesn't offer pushing over the
tailnet, and Security says Pyo's own backups are managed for you.
A build job went to any ready builder of the team. With in-cluster
registries (Pyo Cloud) the image stays in the cluster that built it, so
a build now goes only to a builder of the deploy's own cluster. When
every cluster pulls from Pyo's registry, a builder of another cluster
still helps out while the deploy's cluster has none ready.

With in-cluster registries each cluster without a builder gets its
first one when it connects, and a Git deploy needs a builder in the
service's cluster.
The agent endpoints handed out any certificate or registry login of the
cluster's team by id. A certificate now goes only to a cluster where a
service's domain uses it, and a registry login only to a cluster with a
deployment that pulls with it. The cluster's team has to own both.
The installation's own sandbox servers and templates (no team) are
every team's. On a hosted Pyo, teams don't trust each other: none of
them can be added there, and lookups ignore any that exist. Pyo's
built-in templates stay for every team. Self-hosted is unchanged.
Defence in depth behind the API's own team checks. Pyo connects as its
tables' owner, which row security doesn't hold back, so migrations,
agents, builders, webhooks and background jobs see every row as before.
An operation run for a person or an API key lowers its transaction to
the pyo_team role with the teams it may see (SET LOCAL role and
pyo.team_ids). There policies on services, deployments and their logs,
secrets and credentials, certificates, registry logins, Git and alert
connections, DNS records, databases, agent keys and commands show and
accept only those teams' rows.

Hostname checks across teams and a hosted build's entitlement checks
lift the scope for their part. The installation's own settings run
unscoped for its admin. A test lists the team tables not covered yet.
Answers the agent acts on (the heartbeat's reply, a certificate, a
registry login) get a signature of their own, under the pyoreply1
prefix so it never passes for a command's. The claims name the
cluster, the nonce the agent sent with its request, a one-minute
expiry and the SHA-256 of the exact reply JSON.
With a command signer (Pyo Cloud), every answer the agent acts on goes
out with the signer's signature beside its plain fields, bound to the
cluster and the request's nonce. The signer still sees hashes only.
Without a signer (self-hosted) answers are as before.
The service logs view opens the log follow socket instead of reading
the last 300 lines every 3 seconds, and keeps the last 300.
Pyo Cloud's Terms say an ended team's agents stop receiving commands.
One check, Entitlements::can_command, refused by default whenever
deploys are: a person's or API key's request for a change gets a 402
with a plain message, and the agent's poll fails anything else held
back, unsigned. Heartbeats, reads the dashboard shows and removals
still go through. A deploy held back fails its deployment with why.
Self-hosted Pyo has no entitlements and is never affected.
An agent's terminal or log follow socket that lands on another web
replica than the reader's was forwarded with only its Authorization
header. With signed requests or mTLS the other replica refused it: a
signed request's nonce is spent on the first replica, and a client
certificate ends there. Now the replica that checked the agent leaves
a one-time ticket, good for a minute, and forwards the socket to the
reader's replica at /api/agent/v1/relay/{id} with it. Nothing the
agent sent is passed on. Works in every agent auth mode.
Syncs Pyo queued while a team's plan was paused or ended (routes,
firewall, cron jobs and similar) failed at the agent's poll and were
never sent again after payment. A held back command now marks the
team. The worker's sweep asks each marked team's entitlement, and once
commands are allowed it clears the mark and queues each cluster's
syncs once more: firewall and routes with the WAF's bundle, Pyo Static,
cron jobs, registry tokens and the tailnet's names. A held back bundle
install or removal now ends failed, so its next sync sends it again.
Deploys and changes a person asked for aren't repeated.
A follow_database_logs command opens a follow like a service's: the
agent dials back to Pyo and sends the database's pods with their parts,
its recent lines and events as fetch_database_logs reads them, then each
pod's new lines as Kubernetes streams them. New pods (a failover, a new
replica) are followed from their start, and the pods and events are
sent again when they change. The log frames gain pods and events.
GET /api/v1/databases/{id}/logs/follow is a WebSocket. It queues one
signed follow_database_logs command and relays the agent's lines as the
database's logs show them, each with its part, and all events again
whenever they change. Replaces a signed fetch_database_logs every few
seconds while the logs are live.
The database logs view opens the log follow socket while live instead
of reading the last 500 lines every 4 seconds, and keeps the last 500.
Its events follow too. Time ranges are still read once.
Each heartbeat reads whether the gateway's proxies carry a cluster
token, from one unpaged list of every proxy pod, whole. A server short
of disk evicted the proxy over and over and left about 4,000 Failed
proxy pods in pyo-gateway: that list was 69 MB of JSON, so the agent
was OOMKilled at its 256Mi limit seconds after starting, on every
restart. The 0.4.5 fix paged the heartbeat's own pod lists but not
this one.

The check now lists running proxies only, a page at a time, keeping a
flag per pod. The same goes for other pod lists that ended pods could
swell: a service's logs, log follows and terminal (running pods, and
the ended ones of its Jobs), a database's members (paged, only members
kept) and the waits for a claim to be let go (running pods).
The release builds each image into the runner's Docker first and scans
it before anything is pushed or signed. A CRITICAL CVE with a fix stops
the release, HIGH ones go to the job's summary and code scanning.

CI scans Cargo.lock, bun.lock (dev dependencies too) and the WAF's
go.mod on every pull request and push to main, with the same rule.

Trivy comes from a local action that pins its version and caches the
vulnerability database per day. .trivyignore holds CVEs found not to
apply.
Trivy scans the latest release's images (as its release.json pins
them) and the third-party database images Pyo pins: the databases
system's images, the CloudNativePG bases of postgres.Dockerfile and
plain Valkey 8.1. Each image with fixable HIGH or CRITICAL CVEs gets
one issue labelled security, listing the CVEs and fixed versions. The
next scans update it, comment on new CVEs and close it when nothing is
left. Only the built-in GITHUB_TOKEN is used.
Builds the Postgres 16, 17 and 18 images from the latest release's
Dockerfile on today's base digest and packages, without pushing, and
scans them next to the published ones. When the fresh build has fewer
fixable HIGH or CRITICAL CVEs, an issue says a patch release would pick
up the fixes and whether the base digest needs a bump first.
Covers Cargo, the dashboard's bun workspace, GitHub Actions (pinned by
commit with the version in a comment) and container images: Dockerfile
FROM and ARG images pinned by digest, deploy/k8s manifests, and regex
managers for the images pinned in the system bundles and the Rust code
that lists them (tag and digest together), the MySQL operator's
upstream tag and commit, the toolchains in the workflows and Trivy.

Minor and patch updates come weekly in groups, security fixes at once
as fix(security) commits. Database major versions and vendored operator
updates wait for approval on the Dependency Dashboard. No automerge.
deny.toml checks RustSec advisories (with the ignores of
.cargo/audit.toml), licenses against a permissive allow list,
banned and duplicate crates, and that crates come from crates.io only.
CI fails on advisories and sources. Licenses and bans only warn for
now, through continue-on-error.
scripts/patch-release.sh branches release/vX.Y.Z from the last release
tag, cherry-picks the given commits and bumps the version as a release
commit does. It's a dry run unless given --apply, and leaves pushing,
the pull request and the tag to the maintainer.
How a fix gets from dev to a patch release tag, how it reaches
self-hosted installs and Pyo Cloud, and how databases move to new
images. Also says that the release scans each image before pushing it.
How to report a vulnerability privately (GitHub's advisory form or
support@pyo.dev) and which versions get fixes.
Fixes CVE-2026-78661 (HIGH). x/net 0.62.0 also needs newer x/sync,
x/sys and x/text.
MySQL Shell 8.4.8 bundles certifi 2025.11.12, which still trusts the
emSign roots (GHSA-6q48-6c7w-9gw2, CRITICAL), plus cryptography 45.0.7
and pyOpenSSL 25.1.0 with HIGH CVEs. No MySQL Shell 8.4 release bundles
fixed versions yet, and 8.4.9 and 8.4.10 crash in loadDump (MySQL bug
121027), so the image keeps 8.4.8.

A new build stage downloads MySQL Shell and installs certifi 2026.10.23,
cryptography 50.0.2 and pyOpenSSL 26.4.0 over the bundled versions with
its own pip, from wheels pinned by hash in mysqlsh-python.txt. The
backup image copies the result, without pip's downloads or curl.
pyo scale <n> sets how many replicas the service runs through
scaleService, without a redeploy, and prints the API's warning when
they won't fit on the servers. A very large change asks first on a
terminal, and needs --yes without one. Without a number, it shows how
many replicas run and on which servers.

A linked pyo.jsonc that sets deploy.replicas is updated too, keeping
its comments, so the next pyo deploy doesn't scale back.
@fransallen
fransallen merged commit 25393ce into main Nov 21, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant