Repository navigation
chore(release): 0.5.0 - #7
Merged
Merged
Conversation
A command can carry a signature from a separate signer: claims with the cluster, a growing sequence number, expiry, the command's kind and the SHA-256 of its exact JSON. Agents given the signer's public key check them. The signer never sees the command itself.
Extensions gains a CommandSigner: when set, every command an agent polls is signed first, with the signer choosing the sequence number and expiry. The signer sees the command's kind and hash, never its secrets. When signing fails the commands go back to the queue.
With PYO_AGENT_CA the agent pins that CA, makes its own key, gets a day-long client certificate with the connect command's token, and renews it at half its lifetime. With PYO_COMMAND_KEY and PYO_CLUSTER_ID it refuses commands that aren't signed for its cluster, are replayed, expired or changed, and reports why. The key, certificate and last command number live in pyo-agent-identity.
With PYO_AGENT_MTLS_ADDR, Pyo also serves the agent routes on a TLS 1.3 listener with its own CA. An agent enrolls once with its connect token and a CSR, gets a day-long certificate naming its team and cluster, and renews with that certificate. The agent extractor takes the cluster from the certificate, and PYO_AGENT_REQUIRE_CERT refuses bearer tokens alone. Rotating a cluster's token revokes its certificates. In this mode the install manifest points the agent at PYO_AGENT_ENDPOINT and pins the CA, the cluster id and the signer's keys, and connect commands use PYO_PUBLIC_API_URL.
With PYO_CLUSTER_REGISTRY=in-cluster, connect commands add --local-registry: connect.sh sets up a registry in pyo-system and a loopback forward on every server (127.0.0.1:5005), and builders on that cluster push there instead of to Pyo's registry door. Images stay on the customer's servers.
With Extensions::verify_domains, a team claims a new hostname only when the TXT record _pyo-verify.<host> holds the token Pyo gives that team for it. A team that proves a hostname takes it over from a team that can't anymore. Free domains need no record but must point at one of the team's own servers. PYO_DEV_TXT_FILE reads records from a file, for tests only.
When the API says test payments are on (a development build of Pyo Cloud), Billing shows a test payment option. No money moves.
On Pyo Cloud the machine Pyo runs on isn't a team's server, so a team can't connect it as its cluster.
The signer answers each command with a signature or a refusal. A refused command fails with the signer's reason instead of holding back the cluster's queue.
TLS 1.3 resumption carried the enrollment's session, which had no client certificate, into every later connection, so the API saw an agent without one. Resumption is off on both ends of the agent endpoint: each connection presents the current certificate.
With in-cluster registries every cluster builds for itself, and no cluster is Pyo's own, so a team's first cluster to connect gets the team's first builder. Git deploys work from the start.
An agent's identity can travel in each request instead of a client certificate, so it survives a proxy that ends TLS. The signature covers the key id, method, path with query, time, a nonce and the body's hash, with an Ed25519 key made in the cluster.
PYO_AGENT_AUTH picks how agents prove who they are: token (the default), mtls, or signed. In signed mode an agent enrolls an Ed25519 public key once with its connect token, gets a key id for a day, renews with the key it has, and signs every request. Pyo checks the signature, refuses times more than a minute off and nonces it has seen, and takes the cluster from the key, so it works behind a proxy that ends TLS. Rotating the token revokes the keys. Bearer tokens alone stay refused outside token mode.
Behind Cloudflare the visitor's address comes from CF-Connecting-IP, counted only when the connection comes from PYO_TRUSTED_PROXIES (Cloudflare's ranges, or cloudflared on this machine).
With PYO_AGENT_AUTH=signed the agent makes an Ed25519 key in the cluster, enrolls it with the connect token, renews it at half its day, and signs heartbeats, long-polls, reports and the terminal's WebSocket handshake. A new connect token makes it enroll again. pyo-agent sign-request prints signed headers, for tests.
Proxies close idle connections (Cloudflare after 100 seconds), so a quiet terminal session kept dropping.
Tailscale is set up in the API today, with the installation's own credential. A hosted Pyo serves many teams and joins none of their tailnets, so setting it up with a token or saving an OAuth client is refused there until each cluster's agent does it.
A hosted Pyo has no Tailscale setup yet, and Hetzner servers join over the tailnet. Their cards, the "set up Tailscale first" notes and the public internet switch (its agents always connect that way) don't show.
A hosted Pyo runs on none of a team's machines, and its API refuses "This machine". Add cluster, onboarding and the install command dialog no longer offer it there, so the connect command is the default and only option. Hetzner, which joins over the tailnet, is left out of Add cluster and Add server too. Self-hosted is unchanged.
A hosted Pyo has no Tailscale setup. Its own backup store doesn't reach a team's servers either, so there backups count as on only once they go to the team's bucket.
The Pyo team updates a hosted Pyo, so the sidebar, menu and settings tab don't point its users at an update.
Add sandbox server shows only the command, since Pyo's Hetzner servers join over the tailnet. The registry doesn't offer pushing over the tailnet, and Security says Pyo's own backups are managed for you.
A build job went to any ready builder of the team. With in-cluster registries (Pyo Cloud) the image stays in the cluster that built it, so a build now goes only to a builder of the deploy's own cluster. When every cluster pulls from Pyo's registry, a builder of another cluster still helps out while the deploy's cluster has none ready. With in-cluster registries each cluster without a builder gets its first one when it connects, and a Git deploy needs a builder in the service's cluster.
The agent endpoints handed out any certificate or registry login of the cluster's team by id. A certificate now goes only to a cluster where a service's domain uses it, and a registry login only to a cluster with a deployment that pulls with it. The cluster's team has to own both.
The installation's own sandbox servers and templates (no team) are every team's. On a hosted Pyo, teams don't trust each other: none of them can be added there, and lookups ignore any that exist. Pyo's built-in templates stay for every team. Self-hosted is unchanged.
Defence in depth behind the API's own team checks. Pyo connects as its tables' owner, which row security doesn't hold back, so migrations, agents, builders, webhooks and background jobs see every row as before. An operation run for a person or an API key lowers its transaction to the pyo_team role with the teams it may see (SET LOCAL role and pyo.team_ids). There policies on services, deployments and their logs, secrets and credentials, certificates, registry logins, Git and alert connections, DNS records, databases, agent keys and commands show and accept only those teams' rows. Hostname checks across teams and a hosted build's entitlement checks lift the scope for their part. The installation's own settings run unscoped for its admin. A test lists the team tables not covered yet.
Answers the agent acts on (the heartbeat's reply, a certificate, a registry login) get a signature of their own, under the pyoreply1 prefix so it never passes for a command's. The claims name the cluster, the nonce the agent sent with its request, a one-minute expiry and the SHA-256 of the exact reply JSON.
With a command signer (Pyo Cloud), every answer the agent acts on goes out with the signer's signature beside its plain fields, bound to the cluster and the request's nonce. The signer still sees hashes only. Without a signer (self-hosted) answers are as before.
The service logs view opens the log follow socket instead of reading the last 300 lines every 3 seconds, and keeps the last 300.
Pyo Cloud's Terms say an ended team's agents stop receiving commands. One check, Entitlements::can_command, refused by default whenever deploys are: a person's or API key's request for a change gets a 402 with a plain message, and the agent's poll fails anything else held back, unsigned. Heartbeats, reads the dashboard shows and removals still go through. A deploy held back fails its deployment with why. Self-hosted Pyo has no entitlements and is never affected.
An agent's terminal or log follow socket that lands on another web
replica than the reader's was forwarded with only its Authorization
header. With signed requests or mTLS the other replica refused it: a
signed request's nonce is spent on the first replica, and a client
certificate ends there. Now the replica that checked the agent leaves
a one-time ticket, good for a minute, and forwards the socket to the
reader's replica at /api/agent/v1/relay/{id} with it. Nothing the
agent sent is passed on. Works in every agent auth mode.
Syncs Pyo queued while a team's plan was paused or ended (routes, firewall, cron jobs and similar) failed at the agent's poll and were never sent again after payment. A held back command now marks the team. The worker's sweep asks each marked team's entitlement, and once commands are allowed it clears the mark and queues each cluster's syncs once more: firewall and routes with the WAF's bundle, Pyo Static, cron jobs, registry tokens and the tailnet's names. A held back bundle install or removal now ends failed, so its next sync sends it again. Deploys and changes a person asked for aren't repeated.
A follow_database_logs command opens a follow like a service's: the agent dials back to Pyo and sends the database's pods with their parts, its recent lines and events as fetch_database_logs reads them, then each pod's new lines as Kubernetes streams them. New pods (a failover, a new replica) are followed from their start, and the pods and events are sent again when they change. The log frames gain pods and events.
GET /api/v1/databases/{id}/logs/follow is a WebSocket. It queues one
signed follow_database_logs command and relays the agent's lines as the
database's logs show them, each with its part, and all events again
whenever they change. Replaces a signed fetch_database_logs every few
seconds while the logs are live.
The database logs view opens the log follow socket while live instead of reading the last 500 lines every 4 seconds, and keeps the last 500. Its events follow too. Time ranges are still read once.
Each heartbeat reads whether the gateway's proxies carry a cluster token, from one unpaged list of every proxy pod, whole. A server short of disk evicted the proxy over and over and left about 4,000 Failed proxy pods in pyo-gateway: that list was 69 MB of JSON, so the agent was OOMKilled at its 256Mi limit seconds after starting, on every restart. The 0.4.5 fix paged the heartbeat's own pod lists but not this one. The check now lists running proxies only, a page at a time, keeping a flag per pod. The same goes for other pod lists that ended pods could swell: a service's logs, log follows and terminal (running pods, and the ended ones of its Jobs), a database's members (paged, only members kept) and the waits for a claim to be let go (running pods).
The release builds each image into the runner's Docker first and scans it before anything is pushed or signed. A CRITICAL CVE with a fix stops the release, HIGH ones go to the job's summary and code scanning. CI scans Cargo.lock, bun.lock (dev dependencies too) and the WAF's go.mod on every pull request and push to main, with the same rule. Trivy comes from a local action that pins its version and caches the vulnerability database per day. .trivyignore holds CVEs found not to apply.
Trivy scans the latest release's images (as its release.json pins them) and the third-party database images Pyo pins: the databases system's images, the CloudNativePG bases of postgres.Dockerfile and plain Valkey 8.1. Each image with fixable HIGH or CRITICAL CVEs gets one issue labelled security, listing the CVEs and fixed versions. The next scans update it, comment on new CVEs and close it when nothing is left. Only the built-in GITHUB_TOKEN is used.
Builds the Postgres 16, 17 and 18 images from the latest release's Dockerfile on today's base digest and packages, without pushing, and scans them next to the published ones. When the fresh build has fewer fixable HIGH or CRITICAL CVEs, an issue says a patch release would pick up the fixes and whether the base digest needs a bump first.
Covers Cargo, the dashboard's bun workspace, GitHub Actions (pinned by commit with the version in a comment) and container images: Dockerfile FROM and ARG images pinned by digest, deploy/k8s manifests, and regex managers for the images pinned in the system bundles and the Rust code that lists them (tag and digest together), the MySQL operator's upstream tag and commit, the toolchains in the workflows and Trivy. Minor and patch updates come weekly in groups, security fixes at once as fix(security) commits. Database major versions and vendored operator updates wait for approval on the Dependency Dashboard. No automerge.
deny.toml checks RustSec advisories (with the ignores of .cargo/audit.toml), licenses against a permissive allow list, banned and duplicate crates, and that crates come from crates.io only. CI fails on advisories and sources. Licenses and bans only warn for now, through continue-on-error.
scripts/patch-release.sh branches release/vX.Y.Z from the last release tag, cherry-picks the given commits and bumps the version as a release commit does. It's a dry run unless given --apply, and leaves pushing, the pull request and the tag to the maintainer.
How a fix gets from dev to a patch release tag, how it reaches self-hosted installs and Pyo Cloud, and how databases move to new images. Also says that the release scans each image before pushing it.
How to report a vulnerability privately (GitHub's advisory form or support@pyo.dev) and which versions get fixes.
Fixes CVE-2026-78661 (HIGH). x/net 0.62.0 also needs newer x/sync, x/sys and x/text.
MySQL Shell 8.4.8 bundles certifi 2025.11.12, which still trusts the emSign roots (GHSA-6q48-6c7w-9gw2, CRITICAL), plus cryptography 45.0.7 and pyOpenSSL 25.1.0 with HIGH CVEs. No MySQL Shell 8.4 release bundles fixed versions yet, and 8.4.9 and 8.4.10 crash in loadDump (MySQL bug 121027), so the image keeps 8.4.8. A new build stage downloads MySQL Shell and installs certifi 2026.10.23, cryptography 50.0.2 and pyOpenSSL 26.4.0 over the bundled versions with its own pip, from wheels pinned by hash in mysqlsh-python.txt. The backup image copies the result, without pip's downloads or curl.
pyo scale <n> sets how many replicas the service runs through scaleService, without a redeploy, and prints the API's warning when they won't fit on the servers. A very large change asks first on a terminal, and needs --yes without one. Without a number, it shows how many replicas run and on which servers. A linked pyo.jsonc that sets deploy.replicas is updated too, keeping its comments, so the next pyo deploy doesn't scale back.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pyo 0.5.0: everything on dev since 0.4.5, preview environments for pull and merge requests, folder deploys, request logs kept on the clusters, Tailscale in each cluster's agent, pyo scale, image scans in the release, security fixes, and the version bump.